Regime adjusted exp weighted covariance (a): private API

Types

PortfolioOptimisers.RegimeAdjustedCovarianceState — Type
struct RegimeAdjustedCovarianceState{__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, __T_n_regime_obs, __T_bias, __T_overlap, __T_lag_records, __T_rows} <: AbstractPartialFitState

Holds the running state of an incremental fit of a RegimeAdjustedExpWeightedCovariance.

partial_fit! builds it and keeps it in the cache field of the estimator, and cov(ce, state) reads it. A caller can hold a state outside the estimator, but only the forward pass of the estimator writes it. The pass changes the arrays in place. It builds a new state with the two new scalar fields, regime_state and n_regime_obs, through the positional constructor.

The three fields of the separate correlation recursion, variance, cor_state and cor_weight, are nothing where one decay carries the whole matrix. The field weight is nothing on the separate path. On that path, covariance is not a running state: update_var_cor! writes into it, at each observation, the correlation times the raw volatilities of the built assets, and only the regime statistic reads it. The report makes the covariance again from the variance and the correlation.

Fields

  • ret_buffer: Optional circular buffer of recent centred returns for HAC kernel correction.
  • covariance: Running exponentially weighted covariance matrix, seeded at zero.
  • weight: Running total of the weights that each pair of assets holds in covariance. It takes the step of covariance on a unit outer product, so a pair gains weight only at an observation where both of its assets are valid. It is nothing where cor_decay carries the correlation in cor_state.
  • variance: Running per-asset variance vector.
  • cor_state: Running exponentially weighted correlation state, or nothing where one decay governs both the variance and the correlation.
  • cor_weight: Running total of the weights that each pair of assets holds in cor_state. It takes the step of cor_state on a unit outer product, so a pair gains weight only at an observation where both of its assets are valid. It is nothing where one decay governs both the variance and the correlation.
  • XXt: Working matrix for the current (possibly HAC-adjusted) outer product of the returns.
  • Xi: Working array for the current centred returns.
  • X_old_i: Working array for lagged centred returns.
  • location: Exponentially smoothed location (mean) vector.
  • obs_count: Per-asset count of observations processed.
  • active: Boolean mask of the assets that are active now.
  • regime_state: Current smoothed regime state, or nothing before the first scored observation.
  • n_regime_obs: Count of the observations that advanced the regime state.
  • bias: Store of the bias factors of the regime statistic. A portfolio target keeps a table with one factor per count of observations, and the diagonal target a table with the moments of one term per count in moments, and each table grows as the counts grow. The Mahalanobis target keeps the interpolation nodes of its factor over the count of observations, one set for each count of assets that it meets, in nodes, and in kappa and spread the tables of the factor of the variance noise on the separate correlation path, one entry per count. The diagonal and the Mahalanobis targets keep in kernel the HAC kernel of the correlation, which only the separate path under HAC fills. It is nothing where the estimator has debias = nothing or has no regime method, and a target that reads no store leaves it empty.
  • overlap: Overlap of the locations of each pair of assets, the sum over their common valid returns of the products of the two unnormalised weights, which the factor of a pair reads. It is nothing under PreCentred, which takes no location.
  • lag_records: Circular buffer of the records of the recent observations, the counts, the valid assets and, for a covariance, the overlap of the locations before each one, which the exact factor of a HAC product reads. It is nothing under PreCentred or without HAC.
  • rows: Number of the observations folded so far, the rows that hold a NaN included, in a Ref that the fold writes in place. Unlike obs_count, an asset that leaves the active mask does not reset it, so it is the size(X, 1) of a batch fit over the same rows, which observation_count reads.

Related

source

Functions

PortfolioOptimisers.pair_weighted_correlation — Function
pair_weighted_correlation(
    cache::RegimeAdjustedCovarianceState,
    idx::AbstractVector{<:Integer},
    min_val::Number
) -> Any

Returns the correlation of the separate path on a block of assets, each pair normalised by the weight it holds.

The correlation state $Q$ and the weight state $W$ step only on the pairs of which both assets are valid, $W$ on a unit product, so $Q_{ij} / W_{ij}$ is the weighted mean of the standardised product over the common observations of the pair. The correlation divides it by the root of the two weighted means of the squares, each over the observations of its own asset. An asset that lists late, or that has a holiday, holds fewer observations than an asset that did not, so its pairs hold less weight than the product of the two diagonal weights. A normalisation of $Q$ alone divides by that product, and it shrinks the correlation towards zero. Where every pair shares one history, $W$ is one scalar on every entry, and the two normalisations agree.

A matrix of pairs, each normalised by its own weight, need not be positive semidefinite. The function makes no repair: regime_adjusted_covariance restores the report with restore_psd!, and the regime statistic reads the block as it is.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • idx::AbstractVector{<:Integer}: Index of the assets of the block.
  • min_val::Number: Floor applied to each diagonal mean before its root is taken.

Returns

  • rho::MatNum: The correlation block, with a unit diagonal.

Related

source
PortfolioOptimisers.update_var_cor! — Function
update_var_cor!(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    valid::AbstractVector{<:Bool},
    pair_valid::AbstractMatrix{<:Bool}
)

Advances the separate variance and correlation recursions, and writes from them the block that the regime statistic reads.

This is the path that cor_decay opens. The variance runs at decay and the correlation state at cor_decay. So a volatility that reverts quickly to its mean can sit beside a correlation that needs more data. The function combines the two into cache.covariance, which only the regime statistic reads.

Algorithm

  1. Advance the variance with the diagonal of the outer product. Each term is floored at zero only under hac_floor = PerTermHacFloor().
  2. Standardise the outer product by the running volatilities, after the step of 1. Where hac_lags is not nothing and hac_vol_before is VolatilityBeforeUpdate(), use the volatilities before the step. The diagonal of a HAC row can be negative, and a volatility that holds the row makes a negative row larger, which moves the diagonal of the correlation state down. Under cor_vol = BiasCorrectedVolatility(), first divide the variance by its weight $1 - \lambda^{K}$ with correlation_variance. An asset whose variance is not above min_val contributes zero. So under the rule before the step, an asset contributes nothing on its first row.
  3. On the pairs of which both assets are valid, advance the correlation state by the step $Q_{ij} \leftarrow \lambda_c Q_{ij} + (1 - \lambda_c) \Delta_{ij}$, where $\Delta$ is the standardised outer product, and the weight state by the same step with $\Delta_{ij}$ one. Every other entry holds, so a holiday holds every entry of its asset.
  4. Find the built assets: the active assets with a positive variance whose diagonal of the correlation state is above min_val. Set to zero the row and the column of cache.covariance of every other active asset with a positive variance, so that regime_block_built marks it as not built.
  5. Normalise each pair of the built block by its weight with pair_weighted_correlation, and multiply the correlation by the running volatilities into cache.covariance.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache (mutated).
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • valid::AbstractVector{<:Bool}: Assets with a finite return that are active.
  • pair_valid::AbstractMatrix{<:Bool}: Pairs whose two assets are both valid.

Returns

  • nothing: The cache is mutated in place.

Related

source
PortfolioOptimisers.bias_corrected_covariance — Function
bias_corrected_covariance(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance
) -> Any

Removes the damping a zero seed leaves in the running state, and returns the covariance that regime_adjusted_covariance reports.

The recursion starts at zero, so it damps each pair by the weight $W_{ij}$ that the pair holds. A holiday or a late listing leaves a pair less weight than a congruence assumes. The division by the weight of each pair is then exact, where a congruence shrinks the correlation. The division can break positive semidefiniteness. regime_adjusted_covariance restores it on the block of the ready assets, so an asset in its warm-up moves nothing.

Algorithm

  1. Where one decay runs, divide each entry of the covariance state of the active assets by its weight with pair_weighted_block, and return it with NaN on every other entry.
  2. Else, on the active assets with at least one term, divide each variance by $1 - \lambda^{K_{i}}$, with $K_{i}$ the count of its terms. Divide each pair of the correlation state by its weight, and normalise it to a unit diagonal, with pair_weighted_correlation.
  3. Return $\rho_{ij} \sqrt{\hat{v}_{i} \hat{v}_{j}}$ on those assets, with NaN on every other entry.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.

Returns

  • sigma::MatNum: The bias-corrected covariance matrix. An inactive asset is NaN in its own row and column.

Related

source
PortfolioOptimisers.update_regime! — Function
update_regime!(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    valid::AbstractVector{<:Bool},
    estimation_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> RegimeAdjustedCovarianceState

Advances the smoothed regime state with one observation, one step ahead of the covariance update.

The statistic scores the observation against the state that stands before it, so it compares the realised risk with the predicted risk, and not with itself. An estimate that is too young gives a statistic of large noise, so only an asset that is ready contributes. On the separate path an asset contributes only when regime_block_built holds, so no row is scored against a block at its zero seed. A regime_method of nothing advances nothing.

Algorithm

  1. Find the scored assets: the assets with a deviation that are active, that are ready under min_obs, that regime_bias_open accepts at their count of terms, that are built, and that are in estimation_mask where it is not nothing.
  2. Return the cache unchanged where the count $n$ of the scored assets is below min_active_assets of the target.
  3. Compute the statistics $d_{k}^{2}$ of the target with regime_target_statistic. Return the cache unchanged where the result is nothing.
  4. Transform each statistic with get_regime_state, and take the mean $g_{t}$.
  5. Blend the state, $s_{t} = \lambda_{r} s_{t-1} + (1 - \lambda_{r}) g_{t}$, or $s_{t} = g_{t}$ at the first scored observation, and return a new state with the count of the scored observations plus one.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Current centred returns vector, zeroed where the asset is not valid.
  • valid::AbstractVector{<:Bool}: Assets with a finite return that are active.
  • estimation_mask::Option{<:AbstractVector{<:Bool}}: Optional mask restricting which assets contribute to the statistic.

Returns

  • cache::RegimeAdjustedCovarianceState: The cache to read on and to pass to the next observation. regime_state and n_regime_obs are immutable fields that a new state holds, so the caller must rebind the cache to this return value.

Related

source
PortfolioOptimisers.process_observation! — Method
process_observation!(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    estimation_mask::Union{Nothing, AbstractVector{<:Bool}},
    active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> RegimeAdjustedCovarianceState

Processes one observation, a row or a column, and advances the online covariance state.

The regime state takes its step before the covariance, so the statistic of the observation reads the covariance that stands before it. A holiday holds every entry of its asset, as in ExpWeightedCovariance. An asset that leaves the universe at this observation loses its state, so a later listing starts from zero.

Algorithm

  1. Add one to cache.rows, because every row counts. Find the valid assets: a finite return, and active where active_mask is not nothing. Find the assets with a deviation: under EstimatedCentring, a valid asset with a valid return before; under PreCentred, every valid asset.
  2. Advance the location, and store the deviations $u_{t}$ in cache.Xi, with zero for an asset with no deviation.
  3. Divide each deviation by the root of its centring factor, and advance the regime state with update_regime!.
  4. Make the outer product $\Delta$ of the deviations with hac_outer_product!, with the HAC terms where hac_lags is not nothing, each product divided by its exact factor.
  5. On the separate path, advance the variance and the correlation with update_var_cor!. Else, on the pairs of which both assets have a deviation, take the step $S_{ij} \leftarrow \lambda S_{ij} + (1 - \lambda) \Delta_{ij}$ on the covariance state and the same step with $\Delta_{ij} = 1$ on the weight state. Every other entry holds.
  6. Where hac_lags is not nothing, push the deviations into the return buffer, with NaN for an asset with no deviation, and push the record of the lag.
  7. Advance the overlap of the locations, add one to the count of each valid asset, and apply the active mask with apply_active_mask!.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache (mutated).
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Returns vector for the current observation.
  • estimation_mask::Option{<:AbstractVector{<:Bool}}: Optional mask restricting which assets contribute to the regime state update.
  • active_mask::Option{<:AbstractVector{<:Bool}}: Optional mask of currently active assets.

Returns

  • cache::RegimeAdjustedCovarianceState: The cache to read and to pass to the next observation. The function changes the arrays in place. The caller must bind the cache to this return value, because regime_state and n_regime_obs are immutable fields.

Related

source
PortfolioOptimisers.apply_active_mask! — Function
apply_active_mask!(cache, ce, active_mask::Nothing)
apply_active_mask!(cache, ce, active_mask::AbstractVector{<:Bool})

Applies the active mask of one observation to the online state of a RegimeAdjustedExpWeightedCovariance, as the last step of process_observation!.

The type of active_mask selects the method. Without a mask every asset is active. With a mask, an asset that leaves the universe at this observation loses its state, so it starts again from zero when it comes back.

Algorithm

  1. active_mask is nothing: mark every asset active, and return cache.
  2. Otherwise find the assets that were active and are now inactive. For each one, set its rows and columns of the covariance to zero, and of the weights and the correlation state when the state holds them. Set its count to zero and its location to NaN, and reset its overlap, its return buffer and its lag records.
  3. Store active_mask as the active assets, and return cache.

Arguments

  • cache: The online covariance state.
  • ce: The covariance estimator.
  • active_mask: The assets in the universe at this observation, or nothing.

Returns

  • cache::RegimeAdjustedCovarianceState: The updated state.

Related

source
PortfolioOptimisers.assert_regime_target — Function
assert_regime_target(_::RegimeAdjustedTarget, _::Integer)

Accepts every regime-adjustment target that names no weights.

Arguments

  • ::RegimeAdjustedTarget: Regime-adjustment target (unused).
  • ::Integer: Ignored count of assets.

Returns

  • nothing.

Related

source
assert_regime_target(target::PortfolioTarget, N::Integer)

Checks the portfolio weights of the regime-adjustment target against the universe of the fit.

The target holds the weights, and the fit holds the universe, so the two meet here and nowhere earlier. Weights of nothing name the inverse-volatility direction, which needs no check.

Arguments

  • target::PortfolioTarget: Portfolio regime-adjustment target.
  • N::Integer: Count of assets in the fit.

Validation

  • target.w names N assets.
  • Every weight is non-negative, and every row sums to a positive number.

Returns

  • nothing.

Related

source
PortfolioOptimisers.regime_adjusted_covariance_pass! — Function
regime_adjusted_covariance_pass!(
    f,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    dims::Int64,
    estimation_mask::Union{Nothing, AbstractMatrix{<:Bool}},
    active_mask::Union{Nothing, AbstractMatrix{<:Bool}}
) -> RegimeAdjustedCovarianceState{__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, Int64, __T_bias, __T_overlap, __T_lag_records, Base.RefValue{Int64}} where {__T_ret_buffer, __T_covariance, __T_weight, __T_variance, __T_cor_state, __T_cor_weight, __T_XXt, __T_Xi, __T_X_old_i, __T_location, __T_obs_count, __T_active, __T_regime_state, __T_bias, __T_overlap, __T_lag_records}
regime_adjusted_covariance_pass!(
    f,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    dims::Int64,
    estimation_mask::Union{Nothing, AbstractMatrix{<:Bool}},
    active_mask::Union{Nothing, AbstractMatrix{<:Bool}},
    state::Union{Nothing, RegimeAdjustedCovarianceState}
) -> RegimeAdjustedCovarianceState

Runs one forward pass of the online covariance update over the observations of X, and calls f after each observation.

The pass owns the argument validation, the orientation and the cache, so every verb that reads it states the recursion once.

Algorithm

  1. Check dims, the sizes of the two masks, and the weights of ce.regime_target against the count $N$ of the assets.
  2. Where state is nothing, make a cold state in the floating point type of X: every state at zero, the location at NaN, every asset active, and no regime state. The separate path takes a variance and a correlation state, and the path with one decay a weight state. Else check that state holds $N$ assets, and continue it.
  3. For each observation, advance the state with process_observation!, with the rows of the two masks, and call f(i, state).
  4. Return the last state.

Arguments

  • f: Called as f(i, cache) after observation i. A caller that wants the final cache alone passes Returns(nothing), as cov and partial_fit! do.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::MatNum: Observation matrix.
  • dims::Int: Dimension along which the observations lie.
  • estimation_mask::Option{<:AbstractMatrix{<:Bool}}: Optional mask restricting which assets contribute to the regime state update.
  • active_mask::Option{<:AbstractMatrix{<:Bool}}: Optional mask of active assets.
  • state::Option{<:RegimeAdjustedCovarianceState}: Optional state to continue. A nothing starts a cold cache, which is what a fit over a whole sample needs. partial_fit! passes the estimator's own state.

Validation

  • dims in (1, 2).
  • If estimation_mask is not nothing, size(X) == size(estimation_mask).
  • If active_mask is not nothing, size(X) == size(active_mask).
  • The portfolio weights of ce.regime_target, where it names any.
  • If state is not nothing, it holds as many assets as X.

Returns

  • cache::RegimeAdjustedCovarianceState: The cache after the last observation.

Related

source
PortfolioOptimisers.regime_adjusted_covariance — Function
regime_adjusted_covariance(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance;
    repair
) -> Any

Reads the regime-adjusted covariance out of a cache, as it stands.

The function reads the cache and never writes it, so the same cache answers this call after every observation of a forward pass.

Where ce.regime_method is nothing, process_observation! advances no regime state. Then cache.n_regime_obs stays at zero, which is below every valid regime_min_obs, so the multiplier is one and the covariance is the plain recursion.

The clamp bounds an estimate. The multiplier of one before regime_min_obs is no estimate, so it holds even where lo > 1 or hi < 1.

Algorithm

  1. Remove the damping of the zero seed with bias_corrected_covariance.
  2. Set to NaN the row and the column of every asset that is not ready under min_obs.
  3. Make the block of the assets with a finite variance symmetric. Where repair is true, restore a positive semidefinite block with restore_psd!, which keeps the diagonal.
  4. Find the multiplier: one while cache.n_regime_obs < ce.regime_min_obs, else regime_multiplier of the regime state, clamped to ce.regime_lohi_mult where it is not nothing.
  5. Return the matrix times the square of the multiplier.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • repair::Bool: Whether to restore a positive semidefinite matrix. A caller that reads the diagonal alone passes false.

Returns

  • sigma::MatNum: Regime-adjusted covariance matrix. An asset with fewer than ce.min_obs observations, or one that is not active, is NaN in its own row and column.

Related

source
PortfolioOptimisers.regime_adjusted_correlation — Function
regime_adjusted_correlation(
    sigma::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Rescales a regime-adjusted covariance matrix to a unit diagonal.

An asset that the covariance blanks has a NaN variance, so its row, its column and its diagonal entry stay NaN. An asset with a zero variance, a constant return, gets the same NaN row: its correlation with every asset, itself included, is 0 / 0. Statistics.cor writes one on that diagonal entry, and the library writes NaN, see cov2cor_nan!. A finite entry outside [-1, 1] is round-off, and the function clamps it.

Arguments

  • sigma::MatNum: Regime-adjusted covariance matrix.

Returns

  • rho::MatNum: The correlation matrix of sigma.

Related

source