Exponentially Weighted Covariance

struct ExpWeightedCovariance{__T_decay, __T_min_obs, __T_centred, __T_cache} <: AbstractCovarianceEstimator

Estimates a covariance matrix by an exponentially weighted recursion that freezes on a holiday and resets on an inactive period.

Each observation updates the sub-block of the assets that are valid at it, so a gap never reaches an entry it did not touch. The recursion is seeded at zero, and the output divides out the damping that the cold start costs through a congruence transform, which keeps the state positive semidefinite and leaves every correlation unchanged.

Keeping a young asset investable has a cost the prior pays for it. A prior fitted with this estimator zero-fills the rows the asset was missing through scenario_fill, because every consumer of a Prior Result reads its returns matrix; a scenario-based measure then reads a zero return where the asset had none and understates that asset's risk over those rows, while the covariance stays the estimate this recursion made from the rows it saw. The fill is silent at or below the fitting prior's own fill_limit field, a share of that asset's own observations, warns above it, and refuses any fill under strict; fill_limit defaults to nothing, and this family carries no CoveragePolicy to derive a limit from, so every fill is named.

Fields

• decay: Exponential decay factor for the exponentially weighted estimator.

• min_obs: Minimum number of observations required before the estimator produces a valid result.

• centred: Whether to treat the returns as pre-centred (mean zero). If false, the location is estimated online.

• cache: Running state of an incremental fit, or nothing before the first call to partial_fit!. It is the one Result this estimator holds, and its type bound is the enforcement of that exception. A fit over a matrix ignores it.

Constructors

ExpWeightedCovariance(;    decay::Number = exp2(-inv(40.0)),    min_obs::Integer = round(Int, max(1, inv(log2(inv(decay))))),    centred::Bool = false,    cache::Option{<:AbstractPartialFitState} = nothing) -> ExpWeightedCovariance

Keywords correspond to the struct's fields.

Validation

• decay > 0.
• min_obs > 0.

Mathematical definition

The internal state after the valid observations of each asset is

\[S = (1 - \lambda) \sum_{k} \lambda^{k} e_{k} e_{k}^{\top},\]

taken on the sub-block of the assets that are valid at each observation, and the reported covariance is the congruence transform

\[\hat{\Sigma} = D S D, \qquad D = \operatorname{diag}\left(\frac{1}{\sqrt{1 - \lambda^{n_i}}}\right).\]

Where:

• $\lambda$: decay.
• $e_{k}$: the returns of the valid assets at the observation $k$, less the running location where centred is false.
• $n_i$: the count of valid observations of asset $i$.

A congruence transform preserves positive semidefiniteness, so $\hat{\Sigma}$ inherits the property from $S$, and it cancels in a correlation, so the correction moves no correlation. The correction is the square root of $1 - \lambda^{n_i}$, where a first-moment estimator uses the first power.

Examples

julia> ce = ExpWeightedCovariance();julia> ce.decay ≈ exp2(-inv(40.0))truejulia> ce.min_obs40

Related

• AbstractCovarianceEstimator
• ExpWeightedCovarianceState
• ExpWeightedExpectedReturns
• ExpWeightedVariance
• RegimeAdjustedExpWeightedCovariance
• partial_fit!
• scenario_fill
• EmpiricalPrior

Statistics.cov(
    ce::ExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> MatNum

Compute the exponentially weighted covariance matrix.

Iterates over the observation dimension of X, updating an online covariance cache at each step, then applies the congruence correction and blanks every asset that is not ready.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• active_mask: Optional boolean matrix with the same size as X. An asset whose entry is false is inactive at that observation: its row and column are reset and its answer is NaN while it stays inactive. With nothing every asset is active, so a non-finite return reads as a holiday and every entry that touches the asset freezes.
• kwargs: Additional keyword arguments (ignored).

Validation

• dims in (1, 2).
• If active_mask is not nothing, size(X) == size(active_mask).

Returns

• sigma::MatNum: Covariance matrix of size assets × assets. The row and the column of an asset with fewer than ce.min_obs valid observations are NaN.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = ExpWeightedCovariance(; decay = 0.9, min_obs = 2);julia> size(cov(ce, X))(2, 2)

Related

• ExpWeightedCovariance
• ExpWeightedCovarianceState
• Statistics.cor(ce::ExpWeightedCovariance, X::MatNum; dims::Int = 1, active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)

Statistics.cor(
    ce::ExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> MatNum

Compute the exponentially weighted correlation matrix.

This is the covariance of the same call, rescaled to a unit diagonal. The congruence correction cancels in the rescale, so the correlation is the correlation of the raw state.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• active_mask: Optional boolean matrix with the same size as X.
• kwargs: Additional keyword arguments (ignored).

Returns

• rho::MatNum: Correlation matrix of size assets × assets. The row and the column of an asset that is not ready are NaN, and so is its diagonal entry.

Related

• ExpWeightedCovariance
• Statistics.cov(ce::ExpWeightedCovariance, X::MatNum; dims::Int = 1, active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)
• regime_adjusted_correlation: the shared rescale, which keeps a blanked asset NaN on the diagonal.

Statistics.cov(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the exponentially weighted covariance from a window of an Asset Panel.

This estimator is mask-aware, so it overrides the reduce-and-expand root of the verb and reads the panel's active mask itself. The answer therefore lives on the whole universe rather than on the Coverage Universe: a young asset that lists inside the window is answered from the observations it has, and it is NaN only while it stays below ce.min_obs.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• pnl: Optional AssetPanel, whose active mask the Coverage Universe of the fit is derived from. nothing makes the rule finiteness alone.
• dims: Dimension along which to perform the computation.
• kwargs: Additional keyword arguments (ignored).

Returns

• sigma::MatNum: Covariance matrix of size assets × assets.

Related

• ExpWeightedCovariance
• panel_moment_masks
• Statistics.cov(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)

Statistics.cor(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the exponentially weighted correlation from a window of an Asset Panel.

This is the covariance of the same call, rescaled to a unit diagonal, and it reads the panel's active mask through the same override.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• pnl: Optional AssetPanel, whose active mask the Coverage Universe of the fit is derived from. nothing makes the rule finiteness alone.
• dims: Dimension along which to perform the computation.
• kwargs: Additional keyword arguments (ignored).

Returns

• rho::MatNum: Correlation matrix of size assets × assets.

Related

• ExpWeightedCovariance
• Statistics.cov(ce::ExpWeightedCovariance, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)

Statistics.var(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the marginal exponentially weighted variance from a window of an Asset Panel.

This is the diagonal of the covariance of the same call, and it reads the panel's active mask through the same override.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• pnl: Optional AssetPanel, whose active mask the Coverage Universe of the fit is derived from. nothing makes the rule finiteness alone.
• dims: Dimension along which to perform the computation.
• kwargs: Additional keyword arguments (ignored).

Returns

• var::MatNum: Marginal variance, as a row where dims is 1 and as a column otherwise.

Related

• ExpWeightedCovariance
• Statistics.cov(ce::ExpWeightedCovariance, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)

Statistics.std(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the marginal exponentially weighted volatility from a window of an Asset Panel.

This is the square root of the diagonal of the covariance of the same call, and it reads the panel's active mask through the same override.

Arguments

• ce: Exponentially weighted covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• pnl: Optional AssetPanel, whose active mask the Coverage Universe of the fit is derived from. nothing makes the rule finiteness alone.
• dims: Dimension along which to perform the computation.
• kwargs: Additional keyword arguments (ignored).

Returns

• std::MatNum: Marginal volatility, as a row where dims is 1 and as a column otherwise.

Related

• ExpWeightedCovariance
• Statistics.var(ce::ExpWeightedCovariance, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

Statistics.cov(ce::ExpWeightedCovariance, state::ExpWeightedCovarianceState; kwargs...) -> MatNum

Read the exponentially weighted covariance out of a state the caller holds.

Arguments

• ce: Exponentially weighted covariance estimator.
• state::ExpWeightedCovarianceState: The state to read.
• kwargs: Additional keyword arguments (ignored).

Returns

• sigma::MatNum: Covariance matrix.

Related

• ExpWeightedCovarianceState
• exp_weighted_moment

Statistics.cov(ce::ExpWeightedCovariance; kwargs...) -> MatNum

Read the exponentially weighted covariance out of the estimator's own state.

The one-argument form is what an incremental fit answers: partial_fit! leaves the state in the cache field, and this verb turns it into the ordinary answer. An estimator that has been given no observation carries no state, so the call is refused rather than answered with a zero.

Arguments

• ce: Exponentially weighted covariance estimator carrying a state.
• kwargs: Additional keyword arguments (ignored).

Validation

• ce.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

• sigma::MatNum: Covariance matrix of size assets × assets.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(ExpWeightedCovariance(; decay = 0.9, min_obs = 2), X);julia> size(cov(ce))(2, 2)julia> cov(ExpWeightedCovariance())ERROR: ArgumentError: `ce` holds no partial-fit state, so there is nothing to read. Call `partial_fit!(ce, X)` first, or `cov(ce, X)` for a fit over a whole sample.[...]

Related

• partial_fit!(ce::ExpWeightedCovariance, X::MatNum; dims::Int = 1, active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)
• Statistics.cov(ce::ExpWeightedCovariance, state::ExpWeightedCovarianceState; kwargs...)

Statistics.cor(ce::ExpWeightedCovariance, state::ExpWeightedCovarianceState; kwargs...) -> MatNum

Read the exponentially weighted correlation out of a state the caller holds.

Arguments

• ce: Exponentially weighted covariance estimator.
• state::ExpWeightedCovarianceState: The state to read.
• kwargs: Additional keyword arguments (ignored).

Returns

• rho::MatNum: Correlation matrix.

Related

• ExpWeightedCovarianceState
• Statistics.cov(ce::ExpWeightedCovariance, state::ExpWeightedCovarianceState; kwargs...)

Statistics.cor(ce::ExpWeightedCovariance; kwargs...) -> MatNum

Read the exponentially weighted correlation out of the estimator's own state.

Arguments

• ce: Exponentially weighted covariance estimator carrying a state.
• kwargs: Additional keyword arguments (ignored).

Validation

• ce.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

• rho::MatNum: Correlation matrix of size assets × assets.

Related

• Statistics.cov(ce::ExpWeightedCovariance; kwargs...)
• ExpWeightedCovarianceState

merge_states(
    a::ExpWeightedCovarianceState,
    b::ExpWeightedCovarianceState
)

Refuses to merge two ExpWeightedCovarianceState.

An exponentially weighted state folds forward exactly, S = λ^{n_b} S_a + S_b, but only while no asset resets inside the second block. The state records the count that a reset zeroed and not the reset itself, so the two cases are indistinguishable after the fact and a merge would silently keep a history the reset discarded. Fold the second block into the first with partial_fit! instead.

Arguments

• a: State of the first block.
• b: State of the second block.

Validation

• The pair is refused with an ArgumentError.

Related

• ExpWeightedCovarianceState
• merge_states
• assert_mergeable_states