Regime adjusted exp weighted covariance (a)
Types
PortfolioOptimisers.RegimeAdjustedTarget — Type
abstract type RegimeAdjustedTarget <: AbstractAlgorithmNames the regime statistic that RegimeAdjustedExpWeightedCovariance reads from one observation.
The statistic compares the realised risk of the observation with the risk that the covariance before it predicts, so the target names which part of the covariance the regime multiplier calibrates. The cross-sectional bias statistic of [79] reads the volatilities alone, and it is DiagonalTarget. The Short-Term Volatility Updating of [80] reads the whole standardised vector, and it is MahalanobisTarget.
Interfaces
To implement a new regime-adjustment target, subtype RegimeAdjustedTarget and implement regime_statistic. Implement min_active_assets where the statistic needs more than one scored asset. Implement regime_denom to use the target with FirstMomentRegimeAdjusted, and regime_kappa to use it with LogRegimeAdjusted. A new target takes no bias correction, under every debias rule.
regime_statistic
regime_statistic(target::MyTarget, X::VecNum, C::MatNum, idx::AbstractVector{<:Integer}, min_val::Number) -> Option{<:VecNum}: One statistic for each direction that the target reads, from the deviationsXof the scored assets, their bias-corrected covariance blockCbefore the observation, and their indexidxin the universe. Under correct calibration the statistic has the meanlength(X).nothingskips the observation.
min_active_assets
min_active_assets(target::MyTarget) -> Int: The smallest count of scored assets at which the target gives a statistic. It is one by default.
regime_denom
regime_denom(method::FirstMomentRegimeAdjusted, target::MyTarget, n::Integer) -> Number: The expectation of the root of the statistic ofnscored assets under correct calibration.
regime_kappa
regime_kappa(method::LogRegimeAdjusted, target::MyTarget, n::Integer) -> Number: The expectation of the log of the statistic ofnscored assets under correct calibration.
Examples
julia> struct MyTarget <: PortfolioOptimisers.RegimeAdjustedTarget endjulia> function PortfolioOptimisers.regime_statistic(::MyTarget, X::AbstractVector, C::AbstractMatrix, ::AbstractVector{<:Integer}, min_val::Number) return [sum(abs2, X ./ sqrt.(max.(LinearAlgebra.diag(C), min_val)))] end;julia> PortfolioOptimisers.min_active_assets(MyTarget())1julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012; 0.008 -0.02; -0.02 0.03];julia> kw = (; decay = 0.9, min_obs = 2, regime_min_obs = 1, debias = nothing, regime_method = RootMeanSquaredAdjusted());julia> cov(RegimeAdjustedExpWeightedCovariance(; regime_target = MyTarget(), kw...), X) ≈ cov(RegimeAdjustedExpWeightedCovariance(; regime_target = DiagonalTarget(), kw...), X)trueRelated
MahalanobisTargetDiagonalTargetPortfolioTargetRegimeAdjustedExpWeightedCovarianceRegimeAdjustedMethod
References
PortfolioOptimisers.MahalanobisTarget — Type
struct MahalanobisTarget{__T_recursion} <: RegimeAdjustedTargetScores an observation by its squared Mahalanobis distance against the covariance that stands before it.
The distance reads every eigen-direction of the block of the scored assets, so the target calibrates the whole covariance structure. A direction with a small estimated variance takes as much weight as the market direction. The target needs two scored assets.
The squared distance is the $\lVert \boldsymbol{u}_{t} \rVert^{2}$ of the Short-Term Volatility Updating of [80], with $\boldsymbol{u}_{t} = \mathbf{C}_{t}^{-1/2} \mathbf{V}_{t}^{-1} \boldsymbol{f}_{t}$. There, $\mathbf{C}_{t}$ is the correlation, $\mathbf{V}_{t}$ the diagonal of the volatilities and $\boldsymbol{f}_{t}$ the returns of the model. Under Model 6.8 it is $e^{x_{t}}$ times a $\chi^{2}_{n}$ variate. The book defines $\kappa$ as the expectation of the log of that variate and prints no closed form. At the default law, regime_kappa is that expectation, $\psi(n/2) + \ln 2$. The book reads the covariance of the model, and this target reads the estimate before the observation.
The statistic reads the inverse of an estimated block, and the inverse of an estimate is too large on average (Jensen's inequality). At 12 assets and a half-life of 10, the squared distance of a correctly calibrated return is 1.6 times its dimension. Before the block has more than n + 1 observations the mean is not finite, and before it has more than n + 3 the variance is not finite, so one observation can hold the smoothed regime state. Where the debias field of RegimeAdjustedExpWeightedCovariance is ExactDebias or LawDebias, the target skips those observations. It divides the other statistics by the bias that its regime method reads, so the statistic has the calibration that the method assumes.
Mathematical definition
\[\begin{align} d^{2} &= \frac{u^{\top} \hat{C}^{-1} u}{\beta}\,, & u^{\top} \hat{C}^{-1} u &= \chi^{2}_{n}\, R\,, & R &= \frac{1}{S}\,, & w_{j} &= \frac{(1 - \lambda)\, \lambda^{j}}{1 - \lambda^{K}}\,. \end{align}\]
Where:
- $u$: Returns of the $n$ contributing assets at the observation.
- $\hat{C}$: Bias-corrected covariance block of those assets, from the observations before it.
- $K$: Smallest count of observations among the contributing assets. The target skips the update while $K \le n + 3$. With a HAC adjustment, it skips the update while the effective count of the banded weight matrix is $n + 1$ or less.
- $\lambda$:
cor_decaywhere the separate correlation path runs, elsedecay. - $S$: Schur complement of one direction in $W = \sum_{j} w_{j} z_{j} z_{j}^{\top}$, $z_{j} \sim N(0, I_{n})$. $R$ is independent of the $\chi^{2}_{n}$ factor.
- $\beta$: Bias factor of the regime method without a HAC adjustment, from
mahalanobis_regime_bias!, times the variance factor ofvariance_noise_bias!on the separate correlation path: $\mathbb{E}[R]$ forRootMeanSquaredAdjusted, $\mathbb{E}[\sqrt{R}]^{2}$ forFirstMomentRegimeAdjusted, and $\exp(\mathbb{E}[\ln R])$ forLogRegimeAdjusted. With equal weights $R = K / \chi^{2}_{K - n + 1}$ exactly. On exponential weights the factor comes from a recursion over the $n - 1$ other directions. The recursion is exact at one asset and at equal weights, and it is within 0.15 % of a Monte Carlo at half-lives of 10 and more.
With debias = nothing, $\beta = 1$, and the target scores every observation above min_obs. A block that is not positive definite then takes the ridge of safe_regime_cholesky. Its statistic can be about $10^{12}$, and that one value holds the regime state for many half-lives.
The factor assumes one shared history of returns, and the weights of the recursion. On iid Normal returns at 12 assets and a half-life of 10, the squared multiplier of each of the three methods is then 1.000 within 0.002.
A HAC estimate reads the same recursion on the spectrum of its banded weight matrix, and it skips a block with too few effective observations. At two lags, 12 assets and a half-life of 10, the three methods read 1.0035, 1.0031 and 1.0023 within 0.004 over 8 seeds. With the fixed point of the mean they read 0.984, 0.952 and 0.920.
On the separate correlation path the factor reads cor_decay. variance_noise_bias! adds the noise of the variance at decay, to the mean and to the spread of each method. At a correlation half-life of 20, the three methods read 0.998, 0.997 and 0.996 over 8 seeds, and 1.032, 1.030 and 1.027 at cor_decay alone. At two HAC lags, each correlation row divides by the volatility before its update, and the tables at cor_decay read the kernel of hac_row_kernel. The three methods then read 1.001, 1.001 and 1.000 over 8 seeds. RootMeanSquaredAdjusted reads 1.090 at cor_decay alone, and 1.026 with the rule after the update.
Fields
recursion: The rule that computes the bias factor of the Mahalanobis statistic from the level recursion over its directions.MeanLevelRecursion, the default, takes the mean log-determinant of each level inside the exponential, which reads 0.2 % to 1.3 % low near the gate.GaussianLevelRecursioncarries the covariance of the log-determinant of each level too, which reads within the noise of a Monte Carlo of the pre-centred law. Its nodes cost 2 s to 5.4 s for each count of assets that a fit meets, where the default costs 0.3 s to 0.9 s, and it may be removed if its gain on the forecasts of the estimator does not pay that cost.
Constructors
MahalanobisTarget(; recursion::AbstractLevelRecursion = MeanLevelRecursion()) -> MahalanobisTargetKeywords correspond to the struct's fields.
Examples
julia> MahalanobisTarget()MahalanobisTarget recursion ┴ MeanLevelRecursion()Related
RegimeAdjustedTargetDiagonalTargetPortfolioTargetRegimeAdjustedExpWeightedCovariancemahalanobis_regime_bias!variance_noise_bias!AbstractLevelRecursion
References
- [80] G. A. Paleologo. The Elements of Quantitative Investing. Wiley (2025). Section 6.3.3, pages 122 and 123, Model 6.8 and the estimate of the state.
PortfolioOptimisers.DiagonalTarget — Type
struct DiagonalTarget <: RegimeAdjustedTargetTargets the diagonal of a covariance matrix, in a regime adjustment and in a geodesic shrinkage.
In a regime adjustment, the statistic divides each return by its own volatility and sums the squares, so it calibrates the volatilities and reads no correlation. This is the cross-sectional bias statistic of Equation 4.3 of [79], times the count of the assets. In a GeodesicShrinkageCovariance, the target matrix is the diagonal of the matrix that the estimator shrinks. It keeps the variances and removes every correlation.
Each term of the regime statistic reads one estimated variance, and the inverse of an estimate is too large on average. Where the estimator has debias = ExactDebias(), regime_target_statistic divides each term by the mean of that inverse at the count of its asset, so the statistic has the mean $n$ at every correlation. The root and the log of the statistic read its law, which the correlation of the assets and the noise of each estimate set, so for the first-moment and the log methods it then divides the sum by diagonal_law_factor.
Mathematical definition
\[\begin{align} d_{t}^{2} &= \frac{1}{\gamma} \sum_{i \in \mathcal{A}_{t}} \frac{u_{i,t}^{2}}{f_{i}\, \hat{C}_{ii}}\,. \end{align}\]
Where:
- $d_{t}^{2}$: Regime statistic of observation $t$.
- $\mathcal{A}_{t}$, $n_{t}$: Scored assets of observation $t$, the assets whose returns the regime statistic reads, and their count.
- $u_{i,t}$: Deviation of the return of asset $i$ at observation $t$.
- $\hat{C}_{ii}$: Bias-corrected variance of asset $i$ before observation $t$.
- $f_{i}$: Mean of the inverse of that estimated variance, from
regime_bias_table, or one underdebias = nothing. - $\gamma$: Factor of
diagonal_law_factorfor the first-moment and the log methods, and one for every other method and underdebias = nothing.
With $f_{i} = \gamma = 1$, the root-mean-squared method reads $d_{t}^{2} / n_{t}$, the square of the bias statistic of [79].
Related
RegimeAdjustedTargetMahalanobisTargetPortfolioTargetRegimeAdjustedExpWeightedCovarianceGeodesicShrinkageCovarianceAbstractCovarianceShrinkageTarget: the other target rules of a geodesic shrinkage.
References
- [79] J. Menchero, D. J. Orr and J. Wang. The Barra US Equity Model (USE4): Methodology Notes. Model Insight, MSCI (2011). Section 4.3, Equation 4.3.
PortfolioOptimisers.PortfolioTarget — Type
struct PortfolioTarget{__T_w, __T_indefinite} <: RegimeAdjustedTargetScores an observation by its squared return along each portfolio direction over the variance of that direction.
The target calibrates the covariance along the directions that a portfolio holds, and reads no other direction. Each direction gives one statistic, and the regime method averages the transformed statistics.
The Short-Term Volatility Updating of [80] reads the whole standardised vector. Under its Model 6.8, the return along a fixed direction $\boldsymbol{p}$ is $e^{x_{t}/2} \sqrt{\boldsymbol{p}^{\intercal} \boldsymbol{\Omega} \boldsymbol{p}}\, z_{t}$, with the covariance $\boldsymbol{\Omega}$ of the model and a standard normal $z_{t}$. So each direction reads the same latent $x_{t}$, as $e^{x_{t}}$ times a $\chi^{2}_{1}$ variate. This target is that case of one dimension, times $n_{t}$ so that its mean is $n_{t}$. The inverse-volatility direction of w = nothing is a construction of this library.
Each direction reads one estimated variance, and the inverse of an estimate is too large on average. Where the estimator has debias = ExactDebias(), regime_target_statistic divides each direction by the bias that the regime method reads. The factor is exact for fixed weights. The inverse-volatility direction of w = nothing is built from the same estimate, so a smaller bias remains, which that function states.
A direction reads the block as a covariance. A HAC block in the warm-up, and a block of assets with different histories, need not be positive definite. A direction with a small positive variance on such a block scores a large statistic. At 8 assets, a half-life of 10 and two lags, one row of a block with the smallest eigenvalue $-0.31$ had $r^{2} = 0.51$ against a direction variance of 0.0057, and it scored 51. The indefinite field names what the statistic does with such a block.
Mathematical definition
\[\begin{align} d_{k,t}^{2} &= \frac{n_{t}\, (\boldsymbol{p}_{k}^{\intercal} \boldsymbol{u}_{t})^{2}} {f_{k}\, \boldsymbol{p}_{k}^{\intercal} \hat{\mathbf{C}} \boldsymbol{p}_{k}}\,. \end{align}\]
Where:
- $d_{k,t}^{2}$: Regime statistic of direction $k$ at observation $t$.
- $\mathcal{A}_{t}$, $n_{t}$: Scored assets of observation $t$, the assets whose returns the regime statistic reads, and their count.
- $\boldsymbol{u}_{t}$: Deviations of the returns of the assets of $\mathcal{A}_{t}$.
- $\boldsymbol{p}_{k}$: Direction $k$. Each row of
wwith a positive sum over $\mathcal{A}_{t}$ gives one direction, its entries on $\mathcal{A}_{t}$ divided by that sum. Wherewisnothing, the one direction is $p_{i} \propto 1 / \sqrt{\hat{C}_{ii}}$, with a unit sum. - $\hat{\mathbf{C}}$: Bias-corrected covariance block of $\mathcal{A}_{t}$ before observation $t$.
- $f_{k}$: Bias factor of direction $k$, which
regime_target_statisticstates, or one underdebias = nothing.
The target does not score an observation whose direction has a variance at or below min_val. It does not score an observation whose block the rule of indefinite refuses either, under every debias. The default rule refuses a young block with a negative eigenvalue.
Fields
w: Optional portfolio weights for the portfolio target, as one vector over the assets or as a matrix whose rows are portfolios. Ifnothing, the target uses inverse-volatility weights, and it makes them again at each observation from the bias-corrected covariance block before it.
indefinite: The rule for a covariance block that is not positive definite. A HAC block in the warm-up, and a block of assets with different histories, need not be one, and a direction with a small positive variance on it scores a large statistic.RefuseIndefiniteWarmUp, the default, refuses a block with a negative eigenvalue while the block is young, and scores it once the gate that the Mahalanobis target reads underExactDebias()opens for the same assets, under everydebias.ClipIndefiniteBlockclips the eigenvalues of its correlation at zero and keeps the variances.ScoreIndefiniteBlockscores the block as it is.
Constructors
PortfolioTarget(; w::Option{<:Union{VecNum, MatNum}} = nothing, indefinite::AbstractIndefiniteBlockRule = RefuseIndefiniteWarmUp()) -> PortfolioTargetKeywords correspond to the struct's fields. The bound is the pair of shapes the fit can honour, because Statistics.cov(ce::RegimeAdjustedExpWeightedCovariance, X::MatNum; dims::Int = 1, estimation_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...) reads a bare matrix and carries no asset names. The count of assets, the sign of each weight and the sum of each row are checked at the fit, where the universe is known.
Validation
- If
wis notnothing,!isempty(w).
Examples
julia> PortfolioTarget()PortfolioTarget w ┼ nothing indefinite ┴ RefuseIndefiniteWarmUp()Related
References
- [80] G. A. Paleologo. The Elements of Quantitative Investing. Wiley (2025). Section 6.3.3, page 123, Model 6.8.
PortfolioOptimisers.RegimeAdjustedExpWeightedCovariance — Type
struct RegimeAdjustedExpWeightedCovariance{__T_decay, __T_cor_decay, __T_min_obs, __T_hac_lags, __T_hac_floor, __T_hac_vol_before, __T_cor_vol, __T_regime_method, __T_regime_decay, __T_regime_min_obs, __T_regime_target, __T_regime_lohi_mult, __T_min_val, __T_centring, __T_debias, __T_cache} <: AbstractCovarianceEstimatorOnline exponentially weighted covariance estimator with regime-state adjustment.
At each observation it updates a running exponentially weighted covariance. It also makes a one-step-ahead statistic, which compares the realised risk of that observation with the risk that the state predicted before it. regime_decay smooths the statistic into a scalar regime state, and the estimator multiplies the covariance by the square of the multiplier of that state.
The estimator is the covariance twin of RegimeAdjustedExpWeightedVariance. It shares the RegimeAdjustedMethod family of that type, and it adds a RegimeAdjustedTarget. The target states what the statistic measures: one portfolio direction, the volatilities alone, or the whole covariance structure.
The regime adjustment is the Volatility Regime Adjustment of factor risk of [79]. It multiplies the whole matrix by the square of one multiplier, so it does not change a correlation. The log and the first-moment methods, and the Mahalanobis and the portfolio targets, are the Short-Term Volatility Updating of [80]. The separate correlation path is the exponential smoother of [81]. The HAC adjustment is the estimator of [18], applied to each term of the recursion.
The estimator reads the masks of the data. A prior fitted with it keeps a young asset investable, and scenario_fill puts a zero return in the rows where the asset had none. Every consumer of a Prior Result reads its returns matrix, so a measure of the scenarios then reads a zero return over those rows, and it gives the asset less risk than it has. The covariance stays the estimate that this recursion made from the rows that it saw. The fill is silent at or below the fill_limit field of the prior, a share of the observations of the asset. Above it the fill warns, and under strict the prior refuses every fill. fill_limit is nothing by default, and this family has no CoveragePolicy to give a limit, so the prior names every fill.
A regime_method of nothing turns the adjustment off. Then no regime state advances, the multiplier stays at one, and the estimator is the plain exponentially weighted recursion.
The statistic divides a realised square by an estimated variance, and the inverse of an estimate is too large on average (Jensen's inequality). With debias = ExactDebias(), the default, the statistic skips an estimate too young for a finite variance. It divides the other statistics by the known size of the bias, as each target states. debias = nothing scores the raw statistic.
With centring = EstimatedCentring(), the default, each deviation is taken from the normalised location of the returns of its asset before it. Each product is divided by its exact factor, centring_pair_factor, plus centring_lag_factor under HAC. The covariance is then unbiased from the second valid return of each asset, whatever the mean. The regime statistic reads each deviation divided by the root of its own factor. This is exact on the diagonal, and where the assets share one history. centring = PreCentred() takes the returns as deviations from zero.
Mathematical definition
Write $\lambda$ for decay and $\lambda_c$ for cor_decay. Where cor_decay is nothing, or equal to decay (see has_separate_cor_decay), one recursion carries the whole matrix:
\[\begin{align} S_{ij,t} &= \lambda S_{ij,t-1} + (1-\lambda) u_{i,t} u_{j,t}\,, \\ W_{ij,t} &= \lambda W_{ij,t-1} + (1-\lambda)\,, \end{align}\]
on the pairs of which both assets are valid at $t$. Every other entry holds.
Where:
- $S_{ij,t}$: Raw exponentially weighted covariance state at time $t$, seeded at zero.
- $W_{ij,t}$: Weight that the pair holds in $S$, seeded at zero. It is $1 - \lambda^{n_{ij}}$, with $n_{ij}$ the count of the common valid observations of the pair.
- $u_{i,t} u_{j,t}$: The product of the deviations of observation $t$, HAC-adjusted where
hac_lagsis notnothing, and divided by its exact factor. UnderEstimatedCentringa pair steps only where both assets have a valid return before $t$.
Where cor_decay is not nothing and differs from decay, the variance and the correlation run at their own decays, and the report combines them again:
\[\begin{align} v_{i,t} &= \lambda v_{i,t-1} + (1-\lambda) u_{i,t}^{2}\,, \\ Q_{ij,t} &= \lambda_c Q_{ij,t-1} + (1-\lambda_c) \frac{u_{i,t} u_{j,t}}{\sqrt{v_{i,t} v_{j,t}}}\,, \\ W_{ij,t} &= \lambda_c W_{ij,t-1} + (1-\lambda_c)\,, \\ \rho_{ij,t} &= \frac{Q_{ij,t} / W_{ij,t}}{\sqrt{(Q_{ii,t} / W_{ii,t})(Q_{jj,t} / W_{jj,t})}}\,. \end{align}\]
Where:
- $v_{i,t}$: Raw exponentially weighted variance of asset $i$.
- $Q_{ij,t}$: Raw exponentially weighted correlation state. Like $S$, it steps only on the pairs of which both assets are valid, so a holiday of asset $i$ holds every entry of $i$. Where
hac_lagsis notnothingandhac_vol_beforeisVolatilityBeforeUpdate(), the product divides by $\sqrt{v_{i,t-1} v_{j,t-1}}$, the variances before the step, because the diagonal of a HAC product can be negative and a variance that holds it amplifies it. Undercor_vol = BiasCorrectedVolatility(), the default, each variance is first divided by its weight $1 - \lambda^{K_i}$, so the first products weigh no more than the later ones. - $W_{ij,t}$: Weight that the pair holds in $Q$, with the step of $Q$ on a unit product: $1 - \lambda_c^{n_{ij}}$.
- $\rho_{ij,t}$: Correlation, the weighted mean of the standardised product over the common observations of the pair, over the root of the two weighted means of the squares.
Each observation scores the assets of $\mathcal{A}_{t}$ against the covariance that stands before it. The target gives one statistic $d_{k,t}^{2}$ for each of its $D_{t}$ directions, the method transforms each one, and the regime state smooths their mean:
\[\begin{align} g_{t} &= \frac{1}{D_{t}} \sum_{k=1}^{D_{t}} \phi(d_{k,t}^{2})\,, \\ s_{t} &= \lambda_{r} s_{t-1} + (1 - \lambda_{r})\, g_{t}\,, \quad s_{t_{0}} = g_{t_{0}}\,. \end{align}\]
Where:
- $\mathcal{A}_{t}$, $n_{t}$: Scored assets of observation $t$, the assets whose returns the regime statistic reads, and their count. An asset is scored where its return is valid, where it is in the estimation mask, and where its count of observations is at least
min_obs. UnderExactDebiasandLawDebiasan estimate too young for a finite variance of its statistic is not scored, as each target states. - $d_{k,t}^{2}$, $D_{t}$: The statistic of direction $k$, and the count of the directions, which the
RegimeAdjustedTargetstates. - $\phi$: Transform of the
RegimeAdjustedMethod: $d^{2} / n_{t}$ forRootMeanSquaredAdjusted, $\sqrt{d^{2}} / c_{n}$ forFirstMomentRegimeAdjusted, and $\ln \max(d^{2}, \varepsilon) - \kappa_{n}$ forLogRegimeAdjusted. $c_{n}$ and $\kappa_{n}$ are the expectations of the root and of the log of the statistic of the target under correct calibration, fromregime_denomandregime_kappa, and $\varepsilon$ ismin_val. - $g_{t}$: Regime statistic of observation $t$, which the regime state smooths. An observation that the target does not score leaves the state as it is.
- $s_{t}$: Smoothed regime state after observation $t$. $t_{0}$ is the first observation that the target scores.
- $\lambda_{r}$: Decay of the regime state, the
regime_decayfield.
A zero seed damps the state, so bias_corrected_covariance divides each pair by its weight before it reports:
\[\begin{align} \hat{\Sigma}_{ij} &= \mathrm{mult}(s_T)^{2}\, \frac{S_{ij,T}}{W_{ij,T}}\,. \end{align}\]
Where:
- $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
- $\mathrm{mult}(s_T)^{2}\, S_{ij,T} / W_{ij,T}$: The entry where
cor_decayisnothing. On the separate path it is $\mathrm{mult}(s_T)^{2}\, \rho_{ij,T} \sqrt{\hat{v}_{i} \hat{v}_{j}}$, with $\hat{v}_{i} = v_{i,T} / (1 - \lambda^{K_i})$ and $K_i$ the count of the terms of asset $i$. - $\mathrm{mult}(s_T)$: Regime multiplier of the smoothed regime state $s_T$, clamped to
regime_lohi_multwhere that field is notnothing. It is one while fewer thanregime_min_obsobservations have been scored, and whereregime_methodisnothing. - $\mathrm{mult}(s)$: Volatility multiplier of a regime state $s$. The estimator scales its variance or its covariance by the square of the multiplier.
Each entry reads its own common observations, so where the assets do not share one history, a holiday or a late listing, the matrix need not be positive semidefinite. Where the smallest eigenvalue of its correlation is below $-n\,\varepsilon$ times its largest, the report clips the negative eigenvalues to zero, restores the unit diagonal and keeps the variances. Where every pair shares one history, the division is a congruence of a sum of outer products, so the matrix is positive semidefinite and the repair changes nothing. On different histories a congruence shrinks each correlation of the pair towards zero, so the report divides each pair by its own weight instead.
Fields
decay: Exponential decay factor for the exponentially weighted estimator.
cor_decay: Exponential decay factor for the correlation smoother.
min_obs: Minimum number of observations required before the estimator produces a valid result.
hac_lags: Optional number of lags for Heteroskedasticity and Autocorrelation Consistent (HAC) kernel correction of squared returns. Ifnothing, no HAC correction is applied.
hac_floor: Whether each HAC-adjusted square is floored at zero before it enters the variance recursion. On returns with no autocorrelation the floor makes the variance too large, by 16 % at two lags, sonothing, the default, keeps the recursion an unbiased quadratic form and floors only the variance that the estimator returns.PerTermHacFloorfloors each square.
hac_vol_before: Whether, on the separate correlation path under HAC, each row of the correlation recursion divides by the volatility before the variance step. The diagonal of a HAC row can be negative, and a divisor that holds the row damps a positive row and amplifies a negative one, so the diagonal of the correlation state is skewed down and can fall to zero.VolatilityBeforeUpdate, the default, divides by the volatility before the step: at two lags the rule after the step has 9 % to 19 % more mean squared error in the correlation on most returns, and at eight lags up to 3 % of its rows fall on a guard, where the rule before the step puts none.VolatilityAfterUpdatedivides by the volatility after the step, as the plain path does. It has no effect wherehac_lagsorcor_decayisnothing.
cor_vol: The variance that standardises each product of the separate correlation path. The variance state is seeded at zero, so in the warm-up the raw state gives the first products of the correlation more weight than the bias tables of the regime statistic assume.BiasCorrectedVolatility, the default, divides the state by its weight first.nothingreads the raw state, the rule of the oracle. The steady state is the same under both. It has no effect wherecor_decayisnothing.
regime_method: Regime adjustment method that computes the regime statistic of each observation, ornothingfor no regime adjustment.
regime_decay: Exponential decay factor of the regime state. A value above2^(-1/138) ≈ 0.99499, which is a regime half-life above 138 observations, makes the regime state slow to follow the variance that it corrects. After the variance changes level, the multiplier keeps correcting a mismatch that the variance has already absorbed, so it can overshoot and act erratically. The estimator gives no warning, and it computes the same recursion at every value in(0, 1).
regime_min_obs: Count of regime observations that the estimator needs before it applies the regime multiplier. Before that count the multiplier is one.
regime_target: Regime-adjustment target, which names the regime statistic of one observation and so the part of the covariance that the multiplier calibrates.
regime_lohi_mult: Optional(lo, hi)bounds of the regime multiplier. Ifnothing, no clamp runs.
min_val: Minimum threshold to prevent division by zero or degenerate estimates.
centring: The location each deviation is taken from.EstimatedCentring, the default, estimates the location of each asset and divides each term by the bias that the estimate puts in it, so the moment is unbiased from the second valid return whatever the mean.PreCentredtakes the returns as deviations from a mean of zero.ZeroStartCentringtakes the location of a recursion from zero that is not divided by its weight.
debias: Whether the regime statistic corrects the bias of the estimate it reads. The inverse of an estimated variance is too large on average (Jensen's inequality). UnderExactDebias, the default, the statistic skips an estimate ofn + 3terms or fewer, forndirections. Wherehac_lagsis set, it also skips an estimate whose effective count of terms isn + 1or fewer. It divides the rest by the bias factor that itsregime_methodreads. The diagonal target also divides its sum by the factor of its law, which reads the correlation of the assets and the noise of each estimate, for the first-moment and the log methods.LawDebiasdivides by the law of the estimate alone, and takes the deviation as independent of the estimate.nothingscores the raw statistic.
cache: Running state of an incremental fit, ornothingbefore the first call topartial_fit!. It is the one Result this estimator holds, and its type bound is the enforcement of that exception.Statistics.cov(ce::RegimeAdjustedExpWeightedCovariance)reads it, and a fit over a matrix ignores it.
Constructors
RegimeAdjustedExpWeightedCovariance(; decay::Number = exp2(-inv(40.0)), cor_decay::Option{<:Number} = nothing, min_obs::Integer = round(Int, max(1, decay_half_life(decay), isnothing(cor_decay) ? 1 : decay_half_life(cor_decay, :cor_decay))), hac_lags::Option{<:Integer} = nothing, hac_floor::Option{<:AbstractHacFloor} = nothing, hac_vol_before::AbstractHacVolatilityTiming = VolatilityBeforeUpdate(), cor_vol::Option{<:AbstractCorrelationVolatility} = BiasCorrectedVolatility(), regime_method::Option{<:RegimeAdjustedMethod} = FirstMomentRegimeAdjusted(), regime_decay::Number = exp2(-2 / decay_half_life(decay)), regime_min_obs::Integer = round(Int, max(1, decay_half_life(decay) / 2)), regime_target::RegimeAdjustedTarget = PortfolioTarget(), regime_lohi_mult::Option{<:Tuple{Number, Number}} = (0.7, 1.6), min_val::Number = 1e-12, centring::AbstractCentring = EstimatedCentring(), debias::Option{<:AbstractRegimeDebias} = ExactDebias(), cache::Option{<:AbstractPartialFitState} = nothing) -> RegimeAdjustedExpWeightedCovarianceKeywords correspond to the struct's fields. Where cor_decay is not nothing, the default min_obs reads the slower of the two decays.
Validation
0 < decay < 1.- If
cor_decayis notnothing,0 < cor_decay < 1. min_obs > 0.- If
hac_lagsis notnothing,hac_lags > 0. 0 < regime_decay < 1.regime_min_obs > 0.- If
regime_lohi_multis notnothing,0 < regime_lohi_mult[1] < regime_lohi_mult[2].
Examples
julia> ce = RegimeAdjustedExpWeightedCovariance();julia> ce.decay ≈ exp2(-inv(40.0))truejulia> isnothing(ce.cor_decay)trueRelated
RegimeAdjustedTargetRegimeAdjustedMethodAbstractCovarianceEstimatorRegimeAdjustedExpWeightedVarianceRegimeAdjustedCovarianceStatepartial_fit!scenario_fillEmpiricalPrior
References
- [79] J. Menchero, D. J. Orr and J. Wang. The Barra US Equity Model (USE4): Methodology Notes. Model Insight, MSCI (2011). Section 4.3, Equations 4.3 to 4.5.
- [80] G. A. Paleologo. The Elements of Quantitative Investing. Wiley (2025). Section 6.3.3, Model 6.8 and Equations 6.9 to 6.11.
- [18] W. K. Newey and K. D. West. A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55, 703–708 (1987). Equation 5.
- [81] R. Engle. Dynamic conditional correlation: A simple class of multivariate generalized autoregressive conditional heteroskedasticity models. Journal of Business & Economic Statistics 20, 339–350 (2002). Equation 17.
Functions
PortfolioOptimisers.min_active_assets — Function
min_active_assets(_::RegimeAdjustedTarget) -> Int64
Returns the smallest count of scored assets at which a regime-adjustment target gives a statistic.
Arguments
::RegimeAdjustedTarget: Regime-adjustment target (unused by this default method).
Returns
1::Int: The default is one scored asset.
Related
min_active_assets(_::MahalanobisTarget) -> Int64
Returns the smallest count of scored assets at which the Mahalanobis target gives a statistic.
Arguments
::MahalanobisTarget: Mahalanobis regime-adjustment target (unused).
Returns
2::Int: The target needs two scored assets.
Related
PortfolioOptimisers.regime_statistic — Function
regime_statistic(
_::MahalanobisTarget,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
_::AbstractVector{<:Integer},
min_val::Number
) -> Union{Nothing, Vector}
Computes the squared Mahalanobis distance of one observation against the covariance block.
The target reads every eigen-direction of the block, so it calibrates the whole covariance structure. It needs two active assets, which min_active_assets states.
Arguments
::MahalanobisTarget: Mahalanobis regime-adjustment target.X::VecNum: Centred returns of the assets that contribute to the statistic.C::MatNum: Bias-corrected covariance block of those assets.::AbstractVector{<:Integer}: Ignored index of those assets.min_val::Number: Scale of the ridge thatsafe_regime_choleskyapplies.
Returns
stats::Option{<:VecNum}: One statistic, ornothingwhere the block does not factorise.
Related
regime_statistic(
_::DiagonalTarget,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
_::AbstractVector{<:Integer},
min_val::Number
) -> Vector
Computes the squared standardised Euclidean distance of one observation against the marginal volatilities of the covariance block.
The target divides each return by its own volatility and sums the squares, so it calibrates the diagonal risk scale and reads no correlation.
Arguments
::DiagonalTarget: Diagonal regime-adjustment target.X::VecNum: Centred returns of the assets that contribute to the statistic.C::MatNum: Bias-corrected covariance block of those assets.::AbstractVector{<:Integer}: Ignored index of those assets.min_val::Number: Floor applied to each variance before its root is taken.
Returns
stats::VecNum: One statistic.
Related
regime_statistic(
target::PortfolioTarget,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer},
min_val::Number
) -> Any
Computes the squared standardised return of one observation along each portfolio direction.
Each direction of portfolio_directions gives its own statistic, and the caller averages the transformed values.
Algorithm
- Build the directions over the contributing assets with
portfolio_directions. Returnnothingwhere no row keeps a positive weight. - Return
nothingwhere the variance of a direction is not abovemin_val. A HAC block in the warm-up need not be positive definite, and a floor atmin_valscores the row at about $r^{2} / 10^{-12}$. - Return the count of contributing assets times the squared portfolio return, divided by the portfolio variance, for each row.
Arguments
target::PortfolioTarget: Portfolio regime-adjustment target.X::VecNum: Centred returns of the assets that contribute to the statistic.C::MatNum: Bias-corrected covariance block of those assets.idx::AbstractVector{<:Integer}: Index of those assets in the universe.min_val::Number: Floor applied to each variance, and the smallest portfolio variance that the statistic reads.
Returns
stats::Option{<:VecNum}: One statistic per direction, ornothingwhere no row keeps a positive weight or where the variance of a direction is not abovemin_val.
Related
PortfolioOptimisers.regime_kappa — Function
regime_kappa(
method::LogRegimeAdjusted,
_::Union{DiagonalTarget, MahalanobisTarget},
n::Integer
) -> Any
Computes the stationary expectation of the log regime statistic for a target that reads the whole active block.
MahalanobisTarget and DiagonalTarget both sum n standardised squares, so the statistic is a $\chi^2(n)$ variate under correct calibration and its log has expectation $\psi(x n) + \ln y$. The scalar case of RegimeAdjustedExpWeightedVariance is this expression at n = 1. The squares of the diagonal target are correlated, so its sum is not a $\chi^2(n)$ variate. So the target first divides the debiased statistic by diagonal_law_factor. Then this constant holds at the correlation of the assets and at the noise of each estimate, to the second order in that noise.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.::Union{MahalanobisTarget, DiagonalTarget}: Regime-adjustment target.n::Integer: Count of assets that contribute to the statistic.
Returns
kappa::Number:digamma(method.x * n) + log(method.y).
Related
regime_kappa(
method::LogRegimeAdjusted,
_::PortfolioTarget,
n::Integer
) -> Any
Computes the stationary expectation of the log regime statistic for the portfolio target.
PortfolioTarget reads one direction and multiplies by n, so its statistic is n times a $\chi^2(1)$ variate and the log has expectation $\ln n + \psi(x) + \ln y$.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.::PortfolioTarget: Portfolio regime-adjustment target.n::Integer: Count of assets that contribute to the statistic.
Returns
kappa::Number:log(n) + digamma(method.x) + log(method.y).
Related
PortfolioOptimisers.regime_denom — Function
regime_denom(
method::FirstMomentRegimeAdjusted,
_::PortfolioTarget,
n::Integer
) -> Any
Computes the first-moment normalisation of the regime statistic for the portfolio target.
The statistic is n times a squared standard normal, so the root has expectation $\sqrt{n}\,x$ with $x = \sqrt{2/\pi}$.
Arguments
method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.::PortfolioTarget: Portfolio regime-adjustment target.n::Integer: Count of assets that contribute to the statistic.
Returns
denom::Number:sqrt(n) * method.x.
Related
regime_denom(
method::FirstMomentRegimeAdjusted,
_::Union{DiagonalTarget, MahalanobisTarget},
n::Integer
) -> Any
Computes the first-moment normalisation of the regime statistic for a target that reads the whole active block.
MahalanobisTarget and DiagonalTarget both sum n standardised squares. The Mahalanobis statistic is a $\chi^2(n)$ variate, so the root has expectation $\sqrt{2}\,\Gamma((n+1)/2)/\Gamma(n/2)$. The expression is evaluated through the log gamma function, which stays finite for a wide universe. At n = 1 it is $\sqrt{2/\pi}$, which is the constant the scalar case of RegimeAdjustedExpWeightedVariance carries.
The squares of the diagonal target are correlated, so its sum is a $\chi^2(n)$ variate only at independent returns. There the constant is exact. The root of n, which is the root of the mean of the statistic, is too large at every correlation. The raw statistic of debias = nothing divides by this constant. The target first divides the debiased statistic by diagonal_law_factor. Then this constant holds at the correlation of the assets and at the noise of each estimate, to the second order in that noise. That factor reads the constant, so the debiased statistic does not depend on it.
Arguments
method::FirstMomentRegimeAdjusted: First-moment regime adjustment method. The type of its fieldxis the type of the constant.::Union{MahalanobisTarget, DiagonalTarget}: Regime-adjustment target.n::Integer: Count of assets that contribute to the statistic.
Returns
denom::Number:sqrt(2) * exp(loggamma((n + 1) / 2) - loggamma(n / 2)).
Related
PortfolioOptimisers.get_regime_state — Method
get_regime_state(
_::RootMeanSquaredAdjusted,
_::RegimeAdjustedTarget,
stats::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
n::Integer,
_
) -> Any
Transforms the root-mean-squared regime statistics into the values the regime state smooths.
Arguments
::RootMeanSquaredAdjusted: Root-mean-squared regime adjustment method (unused).::RegimeAdjustedTarget: Regime-adjustment target (unused).stats::VecNum: One statistic per calibration direction.n::Integer: Count of assets that contribute to the statistic.::Any: Ignored minimum value argument.
Returns
val::VecNum:stats ./ n.
Related
PortfolioOptimisers.get_regime_state — Method
get_regime_state(
method::FirstMomentRegimeAdjusted,
target::RegimeAdjustedTarget,
stats::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
n::Integer,
_
) -> Any
Transforms the first-moment regime statistics into the values the regime state smooths.
Arguments
method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.target::RegimeAdjustedTarget: Regime-adjustment target, which names the normalisation.stats::VecNum: One statistic per calibration direction.n::Integer: Count of assets that contribute to the statistic.::Any: Ignored minimum value argument.
Returns
val::VecNum:sqrt.(max.(stats, 0)) ./ regime_denom(method, target, n).
Related
PortfolioOptimisers.get_regime_state — Method
get_regime_state(
method::LogRegimeAdjusted,
target::RegimeAdjustedTarget,
stats::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
n::Integer,
min_val::Number
) -> Any
Transforms the log regime statistics into the values the regime state smooths.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.target::RegimeAdjustedTarget: Regime-adjustment target, which names the expectation.stats::VecNum: One statistic per calibration direction.n::Integer: Count of assets that contribute to the statistic.min_val::Number: Floor applied before the logarithm.
Returns
val::VecNum:log.(max.(stats, min_val)) .- regime_kappa(method, target, n).
Related
References
- [18]
- W. K. Newey and K. D. West. A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55, 703–708 (1987).
- [79]
- J. Menchero, D. J. Orr and J. Wang. The Barra US Equity Model (USE4): Methodology Notes (MSCI, 2011).
- [80]
- G. A. Paleologo. The Elements of Quantitative Investing (Wiley, 2025).
- [81]
- R. Engle. Dynamic conditional correlation: A simple class of multivariate generalized autoregressive conditional heteroskedasticity models. Journal of Business & Economic Statistics 20, 339–350 (2002).