Regime adjusted exp weighted covariance (a): private API
Types
PortfolioOptimisers.RegimeAdjustedCovarianceState — Type
struct RegimeAdjustedCovarianceState{__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, __T_n_regime_obs, __T_bias, __T_overlap, __T_lag_records, __T_rows} <: AbstractPartialFitStateHolds the running state of an incremental fit of a RegimeAdjustedExpWeightedCovariance.
partial_fit! builds it and keeps it in the cache field of the estimator, and cov(ce, state) reads it. A caller can hold a state outside the estimator, but only the forward pass of the estimator writes it. The pass changes the arrays in place. It builds a new state with the two new scalar fields, regime_state and n_regime_obs, through the positional constructor.
The three fields of the separate correlation recursion, variance, cor_state and cor_weight, are nothing where one decay carries the whole matrix. The field weight is nothing on the separate path. On that path, covariance is not a running state: update_var_cor! writes into it, at each observation, the correlation times the raw volatilities of the built assets, and only the regime statistic reads it. The report makes the covariance again from the variance and the correlation.
Fields
ret_buffer: Optional circular buffer of recent centred returns for HAC kernel correction.
covariance: Running exponentially weighted covariance matrix, seeded at zero.
weight: Running total of the weights that each pair of assets holds incovariance. It takes the step ofcovarianceon a unit outer product, so a pair gains weight only at an observation where both of its assets are valid. It isnothingwherecor_decaycarries the correlation incor_state.
variance: Running per-asset variance vector.
cor_state: Running exponentially weighted correlation state, ornothingwhere one decay governs both the variance and the correlation.
cor_weight: Running total of the weights that each pair of assets holds incor_state. It takes the step ofcor_stateon a unit outer product, so a pair gains weight only at an observation where both of its assets are valid. It isnothingwhere one decay governs both the variance and the correlation.
XXt: Working matrix for the current (possibly HAC-adjusted) outer product of the returns.
Xi: Working array for the current centred returns.
X_old_i: Working array for lagged centred returns.
location: Exponentially smoothed location (mean) vector.
obs_count: Per-asset count of observations processed.
active: Boolean mask of the assets that are active now.
regime_state: Current smoothed regime state, ornothingbefore the first scored observation.
n_regime_obs: Count of the observations that advanced the regime state.
bias: Store of the bias factors of the regime statistic. A portfolio target keeps a table with one factor per count of observations, and the diagonal target a table with the moments of one term per count inmoments, and each table grows as the counts grow. The Mahalanobis target keeps the interpolation nodes of its factor over the count of observations, one set for each count of assets that it meets, innodes, and inkappaandspreadthe tables of the factor of the variance noise on the separate correlation path, one entry per count. The diagonal and the Mahalanobis targets keep inkernelthe HAC kernel of the correlation, which only the separate path under HAC fills. It isnothingwhere the estimator hasdebias = nothingor has no regime method, and a target that reads no store leaves it empty.
overlap: Overlap of the locations of each pair of assets, the sum over their common valid returns of the products of the two unnormalised weights, which the factor of a pair reads. It isnothingunderPreCentred, which takes no location.
lag_records: Circular buffer of the records of the recent observations, the counts, the valid assets and, for a covariance, the overlap of the locations before each one, which the exact factor of a HAC product reads. It isnothingunderPreCentredor without HAC.
rows: Number of the observations folded so far, the rows that hold aNaNincluded, in aRefthat the fold writes in place. Unlikeobs_count, an asset that leaves the active mask does not reset it, so it is thesize(X, 1)of a batch fit over the same rows, whichobservation_countreads.
Related
Functions
PortfolioOptimisers.pair_weighted_correlation — Function
pair_weighted_correlation(
cache::RegimeAdjustedCovarianceState,
idx::AbstractVector{<:Integer},
min_val::Number
) -> Any
Returns the correlation of the separate path on a block of assets, each pair normalised by the weight it holds.
The correlation state $Q$ and the weight state $W$ step only on the pairs of which both assets are valid, $W$ on a unit product, so $Q_{ij} / W_{ij}$ is the weighted mean of the standardised product over the common observations of the pair. The correlation divides it by the root of the two weighted means of the squares, each over the observations of its own asset. An asset that lists late, or that has a holiday, holds fewer observations than an asset that did not, so its pairs hold less weight than the product of the two diagonal weights. A normalisation of $Q$ alone divides by that product, and it shrinks the correlation towards zero. Where every pair shares one history, $W$ is one scalar on every entry, and the two normalisations agree.
A matrix of pairs, each normalised by its own weight, need not be positive semidefinite. The function makes no repair: regime_adjusted_covariance restores the report with restore_psd!, and the regime statistic reads the block as it is.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache.idx::AbstractVector{<:Integer}: Index of the assets of the block.min_val::Number: Floor applied to each diagonal mean before its root is taken.
Returns
rho::MatNum: The correlation block, with a unit diagonal.
Related
PortfolioOptimisers.update_var_cor! — Function
update_var_cor!(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
valid::AbstractVector{<:Bool},
pair_valid::AbstractMatrix{<:Bool}
)
Advances the separate variance and correlation recursions, and writes from them the block that the regime statistic reads.
This is the path that cor_decay opens. The variance runs at decay and the correlation state at cor_decay. So a volatility that reverts quickly to its mean can sit beside a correlation that needs more data. The function combines the two into cache.covariance, which only the regime statistic reads.
Algorithm
- Advance the variance with the diagonal of the outer product. Each term is floored at zero only under
hac_floor = PerTermHacFloor(). - Standardise the outer product by the running volatilities, after the step of 1. Where
hac_lagsis notnothingandhac_vol_beforeisVolatilityBeforeUpdate(), use the volatilities before the step. The diagonal of a HAC row can be negative, and a volatility that holds the row makes a negative row larger, which moves the diagonal of the correlation state down. Undercor_vol = BiasCorrectedVolatility(), first divide the variance by its weight $1 - \lambda^{K}$ withcorrelation_variance. An asset whose variance is not abovemin_valcontributes zero. So under the rule before the step, an asset contributes nothing on its first row. - On the pairs of which both assets are valid, advance the correlation state by the step $Q_{ij} \leftarrow \lambda_c Q_{ij} + (1 - \lambda_c) \Delta_{ij}$, where $\Delta$ is the standardised outer product, and the weight state by the same step with $\Delta_{ij}$ one. Every other entry holds, so a holiday holds every entry of its asset.
- Find the built assets: the active assets with a positive variance whose diagonal of the correlation state is above
min_val. Set to zero the row and the column ofcache.covarianceof every other active asset with a positive variance, so thatregime_block_builtmarks it as not built. - Normalise each pair of the built block by its weight with
pair_weighted_correlation, and multiply the correlation by the running volatilities intocache.covariance.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache (mutated).ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.valid::AbstractVector{<:Bool}: Assets with a finite return that are active.pair_valid::AbstractMatrix{<:Bool}: Pairs whose two assets are both valid.
Returns
nothing: The cache is mutated in place.
Related
PortfolioOptimisers.bias_corrected_covariance — Function
bias_corrected_covariance(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance
) -> Any
Removes the damping a zero seed leaves in the running state, and returns the covariance that regime_adjusted_covariance reports.
The recursion starts at zero, so it damps each pair by the weight $W_{ij}$ that the pair holds. A holiday or a late listing leaves a pair less weight than a congruence assumes. The division by the weight of each pair is then exact, where a congruence shrinks the correlation. The division can break positive semidefiniteness. regime_adjusted_covariance restores it on the block of the ready assets, so an asset in its warm-up moves nothing.
Algorithm
- Where one decay runs, divide each entry of the covariance state of the active assets by its weight with
pair_weighted_block, and return it withNaNon every other entry. - Else, on the active assets with at least one term, divide each variance by $1 - \lambda^{K_{i}}$, with $K_{i}$ the count of its terms. Divide each pair of the correlation state by its weight, and normalise it to a unit diagonal, with
pair_weighted_correlation. - Return $\rho_{ij} \sqrt{\hat{v}_{i} \hat{v}_{j}}$ on those assets, with
NaNon every other entry.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
Returns
sigma::MatNum: The bias-corrected covariance matrix. An inactive asset isNaNin its own row and column.
Related
PortfolioOptimisers.update_regime! — Function
update_regime!(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
valid::AbstractVector{<:Bool},
estimation_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> RegimeAdjustedCovarianceState
Advances the smoothed regime state with one observation, one step ahead of the covariance update.
The statistic scores the observation against the state that stands before it, so it compares the realised risk with the predicted risk, and not with itself. An estimate that is too young gives a statistic of large noise, so only an asset that is ready contributes. On the separate path an asset contributes only when regime_block_built holds, so no row is scored against a block at its zero seed. A regime_method of nothing advances nothing.
Algorithm
- Find the scored assets: the assets with a deviation that are active, that are ready under
min_obs, thatregime_bias_openaccepts at their count of terms, that are built, and that are inestimation_maskwhere it is notnothing. - Return the cache unchanged where the count $n$ of the scored assets is below
min_active_assetsof the target. - Compute the statistics $d_{k}^{2}$ of the target with
regime_target_statistic. Return the cache unchanged where the result isnothing. - Transform each statistic with
get_regime_state, and take the mean $g_{t}$. - Blend the state, $s_{t} = \lambda_{r} s_{t-1} + (1 - \lambda_{r}) g_{t}$, or $s_{t} = g_{t}$ at the first scored observation, and return a new state with the count of the scored observations plus one.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Current centred returns vector, zeroed where the asset is not valid.valid::AbstractVector{<:Bool}: Assets with a finite return that are active.estimation_mask::Option{<:AbstractVector{<:Bool}}: Optional mask restricting which assets contribute to the statistic.
Returns
cache::RegimeAdjustedCovarianceState: The cache to read on and to pass to the next observation.regime_stateandn_regime_obsare immutable fields that a new state holds, so the caller must rebind the cache to this return value.
Related
PortfolioOptimisers.process_observation! — Method
process_observation!(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
estimation_mask::Union{Nothing, AbstractVector{<:Bool}},
active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> RegimeAdjustedCovarianceState
Processes one observation, a row or a column, and advances the online covariance state.
The regime state takes its step before the covariance, so the statistic of the observation reads the covariance that stands before it. A holiday holds every entry of its asset, as in ExpWeightedCovariance. An asset that leaves the universe at this observation loses its state, so a later listing starts from zero.
Algorithm
- Add one to
cache.rows, because every row counts. Find the valid assets: a finite return, and active whereactive_maskis notnothing. Find the assets with a deviation: underEstimatedCentring, a valid asset with a valid return before; underPreCentred, every valid asset. - Advance the location, and store the deviations $u_{t}$ in
cache.Xi, with zero for an asset with no deviation. - Divide each deviation by the root of its centring factor, and advance the regime state with
update_regime!. - Make the outer product $\Delta$ of the deviations with
hac_outer_product!, with the HAC terms wherehac_lagsis notnothing, each product divided by its exact factor. - On the separate path, advance the variance and the correlation with
update_var_cor!. Else, on the pairs of which both assets have a deviation, take the step $S_{ij} \leftarrow \lambda S_{ij} + (1 - \lambda) \Delta_{ij}$ on the covariance state and the same step with $\Delta_{ij} = 1$ on the weight state. Every other entry holds. - Where
hac_lagsis notnothing, push the deviations into the return buffer, withNaNfor an asset with no deviation, and push the record of the lag. - Advance the overlap of the locations, add one to the count of each valid asset, and apply the active mask with
apply_active_mask!.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache (mutated).ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Returns vector for the current observation.estimation_mask::Option{<:AbstractVector{<:Bool}}: Optional mask restricting which assets contribute to the regime state update.active_mask::Option{<:AbstractVector{<:Bool}}: Optional mask of currently active assets.
Returns
cache::RegimeAdjustedCovarianceState: The cache to read and to pass to the next observation. The function changes the arrays in place. The caller must bind the cache to this return value, becauseregime_stateandn_regime_obsare immutable fields.
Related
PortfolioOptimisers.apply_active_mask! — Function
apply_active_mask!(cache, ce, active_mask::Nothing)
apply_active_mask!(cache, ce, active_mask::AbstractVector{<:Bool})Applies the active mask of one observation to the online state of a RegimeAdjustedExpWeightedCovariance, as the last step of process_observation!.
The type of active_mask selects the method. Without a mask every asset is active. With a mask, an asset that leaves the universe at this observation loses its state, so it starts again from zero when it comes back.
Algorithm
active_maskisnothing: mark every asset active, and returncache.- Otherwise find the assets that were active and are now inactive. For each one, set its rows and columns of the covariance to zero, and of the weights and the correlation state when the state holds them. Set its count to zero and its location to
NaN, and reset its overlap, its return buffer and its lag records. - Store
active_maskas the active assets, and returncache.
Arguments
cache: The online covariance state.ce: The covariance estimator.active_mask: The assets in the universe at this observation, ornothing.
Returns
cache::RegimeAdjustedCovarianceState: The updated state.
Related
PortfolioOptimisers.assert_regime_target — Function
assert_regime_target(_::RegimeAdjustedTarget, _::Integer)
Accepts every regime-adjustment target that names no weights.
Arguments
::RegimeAdjustedTarget: Regime-adjustment target (unused).::Integer: Ignored count of assets.
Returns
nothing.
Related
assert_regime_target(target::PortfolioTarget, N::Integer)
Checks the portfolio weights of the regime-adjustment target against the universe of the fit.
The target holds the weights, and the fit holds the universe, so the two meet here and nowhere earlier. Weights of nothing name the inverse-volatility direction, which needs no check.
Arguments
target::PortfolioTarget: Portfolio regime-adjustment target.N::Integer: Count of assets in the fit.
Validation
target.wnamesNassets.- Every weight is non-negative, and every row sums to a positive number.
Returns
nothing.
Related
PortfolioOptimisers.regime_adjusted_covariance_pass! — Function
regime_adjusted_covariance_pass!(
f,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
dims::Int64,
estimation_mask::Union{Nothing, AbstractMatrix{<:Bool}},
active_mask::Union{Nothing, AbstractMatrix{<:Bool}}
) -> RegimeAdjustedCovarianceState{__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, Int64, __T_bias, __T_overlap, __T_lag_records, Base.RefValue{Int64}} where {__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, __T_bias, __T_overlap, __T_lag_records}
regime_adjusted_covariance_pass!(
f,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
dims::Int64,
estimation_mask::Union{Nothing, AbstractMatrix{<:Bool}},
active_mask::Union{Nothing, AbstractMatrix{<:Bool}},
state::Union{Nothing, RegimeAdjustedCovarianceState}
) -> RegimeAdjustedCovarianceState
Runs one forward pass of the online covariance update over the observations of X, and calls f after each observation.
The pass owns the argument validation, the orientation and the cache, so every verb that reads it states the recursion once.
Algorithm
- Check
dims, the sizes of the two masks, and the weights ofce.regime_targetagainst the count $N$ of the assets. - Where
stateisnothing, make a cold state in the floating point type ofX: every state at zero, the location atNaN, every asset active, and no regime state. The separate path takes a variance and a correlation state, and the path with one decay a weight state. Else check thatstateholds $N$ assets, and continue it. - For each observation, advance the state with
process_observation!, with the rows of the two masks, and callf(i, state). - Return the last state.
Arguments
f: Called asf(i, cache)after observationi. A caller that wants the final cache alone passesReturns(nothing), ascovandpartial_fit!do.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::MatNum: Observation matrix.dims::Int: Dimension along which the observations lie.estimation_mask::Option{<:AbstractMatrix{<:Bool}}: Optional mask restricting which assets contribute to the regime state update.active_mask::Option{<:AbstractMatrix{<:Bool}}: Optional mask of active assets.state::Option{<:RegimeAdjustedCovarianceState}: Optional state to continue. Anothingstarts a cold cache, which is what a fit over a whole sample needs.partial_fit!passes the estimator's own state.
Validation
dims in (1, 2).- If
estimation_maskis notnothing,size(X) == size(estimation_mask). - If
active_maskis notnothing,size(X) == size(active_mask). - The portfolio weights of
ce.regime_target, where it names any. - If
stateis notnothing, it holds as many assets asX.
Returns
cache::RegimeAdjustedCovarianceState: The cache after the last observation.
Related
PortfolioOptimisers.regime_adjusted_covariance — Function
regime_adjusted_covariance(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance;
repair
) -> Any
Reads the regime-adjusted covariance out of a cache, as it stands.
The function reads the cache and never writes it, so the same cache answers this call after every observation of a forward pass.
Where ce.regime_method is nothing, process_observation! advances no regime state. Then cache.n_regime_obs stays at zero, which is below every valid regime_min_obs, so the multiplier is one and the covariance is the plain recursion.
The clamp bounds an estimate. The multiplier of one before regime_min_obs is no estimate, so it holds even where lo > 1 or hi < 1.
Algorithm
- Remove the damping of the zero seed with
bias_corrected_covariance. - Set to
NaNthe row and the column of every asset that is not ready undermin_obs. - Make the block of the assets with a finite variance symmetric. Where
repairistrue, restore a positive semidefinite block withrestore_psd!, which keeps the diagonal. - Find the multiplier: one while
cache.n_regime_obs < ce.regime_min_obs, elseregime_multiplierof the regime state, clamped toce.regime_lohi_multwhere it is notnothing. - Return the matrix times the square of the multiplier.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.repair::Bool: Whether to restore a positive semidefinite matrix. A caller that reads the diagonal alone passesfalse.
Returns
sigma::MatNum: Regime-adjusted covariance matrix. An asset with fewer thance.min_obsobservations, or one that is not active, isNaNin its own row and column.
Related
PortfolioOptimisers.regime_adjusted_correlation — Function
regime_adjusted_correlation(
sigma::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Rescales a regime-adjusted covariance matrix to a unit diagonal.
An asset that the covariance blanks has a NaN variance, so its row, its column and its diagonal entry stay NaN. An asset with a zero variance, a constant return, gets the same NaN row: its correlation with every asset, itself included, is 0 / 0. Statistics.cor writes one on that diagonal entry, and the library writes NaN, see cov2cor_nan!. A finite entry outside [-1, 1] is round-off, and the function clamps it.
Arguments
sigma::MatNum: Regime-adjusted covariance matrix.
Returns
rho::MatNum: The correlation matrix ofsigma.
Related