Simple covariance

struct GeneralCovariance{__T_ce, __T_w, __T_cache} <: AbstractCovarianceEstimator

Adapts any StatsBase.CovarianceEstimator to the library's calling convention, carrying its observation weights alongside it.

The estimator and the weights travel together in one object, so a caller passes one value where the StatsBase API takes a separate estimator and weight vector at every call site. ce accepts any subtype of StatsBase.CovarianceEstimator, so an estimator from a package such as CovarianceEstimation.jl reaches the library unchanged.

Fields

• ce: Covariance estimator.

• w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.

• cache: Optional partial-fit state. It is nothing until partial_fit! writes one, and the estimator's read-out verb reads it when the caller gives no data matrix. Each propagation channel does one thing with it: factory carries it unchanged, because a factory call resolves configuration rather than the sample; port_opt_view slices it to the selected assets by index copy, so the viewed estimator answers over those assets alone; and obs_weights_view drops it, because no slice of a state exists on the observation axis. A family whose state has no exact asset slice drops it on both axes and names the reason.

Constructors

GeneralCovariance(;    ce::StatsBase.CovarianceEstimator = StatsBase.SimpleCovariance(;        corrected = true),    w::Option{<:ObsWeights} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing) -> GeneralCovariance

Keywords correspond to the struct's fields.

Validation

• If w is not nothing, !isempty(w).

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• ce: Recursively updated via factory.
• w: Replaced with the incoming ObsWeights.
• cache: Carried unchanged via factory.

View parameters

When port_opt_view is called on this type, the following @vprop-tagged fields are automatically subset to the selected indices:

• ce: Recursively viewed via port_opt_view.
• cache: Sliced to the selected assets via port_opt_view.

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

• ce: Recursively indexed via obs_weights_view.
• w: Indexed to the selected observations via obs_weights_view.
• cache: Dropped via obs_weights_view, because no slice of a state exists on the observation axis.

Examples

julia> GeneralCovariance()GeneralCovariance  ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)   w ┴ nothingjulia> GeneralCovariance(; w = StatsBase.Weights([0.1, 0.2, 0.7]))GeneralCovariance  ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)   w ┴ StatsBase.Weights{Float64, Float64, Vector{Float64}}: [0.1, 0.2, 0.7]

Related

• AbstractCovarianceEstimator
• Option
• StatsBase.CovarianceEstimator
• StatsBase.AbstractWeights
• cov(ce::GeneralCovariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)
• CovarianceState
• partial_fit!
• factory
• port_opt_view
• obs_weights_view

Statistics.cov(
    ce::GeneralCovariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the covariance matrix using a GeneralCovariance estimator.

This method dispatches to the appropriate robust_cov depending on ce.w, which computes the covariance matrix using ce.ce.

Algorithm

1. Resolve the observation weights from ce.w against X, giving w.
2. When w is nothing, call robust_cov with ce.ce and X alone.
3. Otherwise call robust_cov with ce.ce, X and w.

Arguments

• ce: Covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• mean: Optional mean value to use for centering.
• kwargs...: Additional keyword arguments passed to robust_cov.

Returns

• sigma::MatNum: Covariance matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cov(GeneralCovariance(), X)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001

Related

• MatNum
• GeneralCovariance
• robust_cov
• cor(ce::GeneralCovariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)

Statistics.cor(
    ce::GeneralCovariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the correlation matrix using a GeneralCovariance estimator.

This method dispatches to the appropriate robust_cor depending on ce.w, which computes the correlation matrix using ce.ce.

Algorithm

1. Resolve the observation weights from ce.w against X, giving w.
2. When w is nothing, call robust_cor with ce.ce and X alone.
3. Otherwise call robust_cor with ce.ce, X and w.

Arguments

• ce: Covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• mean: Optional mean value to use for centering.
• kwargs...: Additional keyword arguments passed to robust_cor.

Returns

• rho::MatNum: Correlation matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cor(GeneralCovariance(), X)2×2 Matrix{Float64}: 1.0  1.0 1.0  1.0

Related

• MatNum
• GeneralCovariance
• robust_cor
• cov(ce::GeneralCovariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)

struct Covariance{__T_me, __T_ce, __T_alg, __T_w, __T_cvg, __T_cache} <: AbstractCovarianceEstimator

Estimates the covariance matrix of asset returns from a centring estimator, a covariance estimator, and a moment algorithm.

Covariance encapsulates all components required for estimating the covariance matrix of asset returns, including the expected returns estimator for centering the data, the covariance estimator, and the moment algorithm.

w weights the whole estimate, so it reaches the centre as well as the deviations. The four methods send me and ce through factory, which replaces the weights of each with w, so w wins over the weights that me and ce carry. Pass mean for a centre that w does not describe.

ce admits any StatsBase.CovarianceEstimator, and no verb of this library reads the weights of one that the library does not own. A ce from a package such as CovarianceEstimation.jl therefore keeps its own configuration, and w is the field that weights a Covariance.

Fields

• me: Expected returns estimator.

• ce: Covariance estimator.

• alg: Moment algorithm.

• w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.

• cvg: Optional CoveragePolicy. nothing is the reduce-and-expand path of the Coverage Universe, in which an asset that is non-finite or inactive at any observation of the window is NaN throughout the answer. A policy replaces it by available-case estimation: every cell is fitted on the observations at which the assets of that cell are all finite and active, each cell carries its own denominator, and an asset reaches the answer where admits says so.

• cache: Optional partial-fit state. It is nothing until partial_fit! writes one, and the estimator's read-out verb reads it when the caller gives no data matrix. Each propagation channel does one thing with it: factory carries it unchanged, because a factory call resolves configuration rather than the sample; port_opt_view slices it to the selected assets by index copy, so the viewed estimator answers over those assets alone; and obs_weights_view drops it, because no slice of a state exists on the observation axis. A family whose state has no exact asset slice drops it on both axes and names the reason.

Constructors

Covariance(;    me::AbstractExpectedReturnsEstimator = SimpleExpectedReturns(),    ce::StatsBase.CovarianceEstimator = GeneralCovariance(),    alg::AbstractMomentAlgorithm = FullMoment(),    w::Option{<:ObsWeights} = nothing,    cvg::Option{<:CoveragePolicy} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing) -> Covariance

Keywords correspond to the struct's fields.

Validation

• If w is not nothing, !isempty(w).

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• me: Recursively updated via factory.
• ce: Recursively updated via factory.
• w: Replaced with the incoming ObsWeights.
• cache: Carried unchanged via factory.

View parameters

When port_opt_view is called on this type, the following @vprop-tagged fields are automatically subset to the selected indices:

• me: Recursively viewed via port_opt_view.
• ce: Recursively viewed via port_opt_view.
• cache: Sliced to the selected assets via port_opt_view.

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

• me: Recursively indexed via obs_weights_view.
• ce: Recursively indexed via obs_weights_view.
• w: Indexed to the selected observations via obs_weights_view.
• cache: Dropped via obs_weights_view, because no slice of a state exists on the observation axis.

Examples

julia> Covariance()Covariance   me ┼ SimpleExpectedReturns      │   w ┴ nothing   ce ┼ GeneralCovariance      │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │    w ┴ nothing  alg ┼ FullMoment()    w ┴ nothingjulia> Covariance(; w = StatsBase.AnalyticWeights([0.2, 0.3, 0.5]))Covariance   me ┼ SimpleExpectedReturns      │   w ┴ nothing   ce ┼ GeneralCovariance      │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │    w ┴ nothing  alg ┼ FullMoment()    w ┴ StatsBase.AnalyticWeights{Float64, Float64, Vector{Float64}}: [0.2, 0.3, 0.5]

Related

• AbstractCovarianceEstimator
• GeneralCovariance
• SimpleExpectedReturns
• FullMoment
• SemiMoment
• Option
• StatsBase.AbstractWeights
• covariance_centre_and_estimator
• CovarianceState
• partial_fit!
• factory
• port_opt_view
• obs_weights_view

Statistics.cov(
    ce::Covariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the covariance matrix using a Covariance estimator.

Mathematical definition

FullMoment covariance:

\[\begin{align} \hat{\mathbf{\Sigma}}_{ij} &= \frac{1}{T-1} \sum_{t=1}^{T} (r_{ti} - \hat{\mu}_i)(r_{tj} - \hat{\mu}_j)\,. \end{align}\]

SemiMoment (downside) covariance, from the de-meaned returns clipped at zero:

\[\begin{align} \tilde{r}_{tj} &= \min(r_{tj} - \hat{\mu}_j,\, 0)\,,\\ \hat{\mathbf{\Sigma}}^{\text{semi}}_{ij} &= \frac{1}{T-1} \sum_{t=1}^{T} \tilde{r}_{ti} \, \tilde{r}_{tj}\,. \end{align}\]

Where:

• $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
• $\hat{\mathbf{\Sigma}}^{\text{semi}}_{ij}$: Estimated semi-covariance between assets $i$ and $j$.
• $r_{tj}$: Return of asset $j$ at time $t$.
• $r_{ti}$: Return of asset $i$ at time $t$.
• $\hat{\mu}_j$: Estimated mean of asset $j$.
• $\hat{\mu}_i$: Estimated mean of asset $i$.
• $\tilde{r}_{ti}$, $\tilde{r}_{tj}$: De-meaned returns of assets $i$ and $j$, clipped at zero.
• $T$: Number of observations.

The semi-covariance keeps the $T-1$ divisor of the full moment, so it is not the covariance of the clipped returns about their own mean.

Algorithm

1. Resolve the centring vector mu and the inner estimator cel with covariance_centre_and_estimator. When mean is nothing, mu comes from ce.me; otherwise it comes from mean. When ce.w is not nothing, ce.w reaches ce.me and ce.ce through factory first.
2. Delegate to Statistics.cov(cel, X; dims = dims, mean = mu, kwargs...).

Arguments

• ce: Covariance estimator.
  • ce::Covariance{<:Any, <:Any, <:FullMoment}: Covariance estimator with FullMoment moment algorithm.
  • ce::Covariance{<:Any, <:Any, <:SemiMoment}: Covariance estimator with SemiMoment moment algorithm.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• mean: Optional mean value to use for centering. If not provided, computed using ce.me.
• kwargs...: Additional keyword arguments passed to the underlying covariance estimator.

Returns

• sigma::MatNum: Covariance matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cov(Covariance(), X)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001julia> cov(Covariance(; alg = SemiMoment()), X)2×2 Matrix{Float64}: 5.0e-5  5.0e-5 5.0e-5  5.0e-5

Related

• Covariance
• AbstractCovarianceEstimator
• GeneralCovariance
• FullMoment
• SemiMoment
• cor(ce::Covariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)

cov(
    ce::Covariance{<:Any, <:Any, <:SemiMoment},
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

SemiMoment variant of cov(ce::Covariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...). Clips de-meaned returns to zero before computing the covariance matrix, capturing only downside co-movements.

Algorithm

1. Resolve the centring vector mu and the inner estimator cel with covariance_centre_and_estimator.
2. Replace X with min.(X .- mu, 0), the de-meaned returns clipped at zero.
3. Delegate to Statistics.cov(cel, X; dims = dims, mean = 0, kwargs...). The zero mean is what stops the clipped returns being centred a second time.

Statistics.cor(
    ce::Covariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the correlation matrix using a Covariance estimator.

Mathematical definition

\[\begin{align} \hat{\mathbf{P}}_{ij} &= \frac{\hat{\mathbf{\Sigma}}_{ij}}{\hat{\sigma}_i \hat{\sigma}_j}\,. \end{align}\]

Where:

• $\hat{\mathbf{P}}_{ij}$: Estimated correlation between assets $i$ and $j$.
• $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
• $\hat{\sigma}_i$: Estimated standard deviation of asset $i$.
• $\hat{\sigma}_j$: Estimated standard deviation of asset $j$.

The alg field of ce reaches $\hat{\mathbf{\Sigma}}$: SemiMoment standardises the semi-covariance, so the diagonal is one and an off-diagonal entry is a downside correlation.

Algorithm

1. Resolve the centring vector mu and the inner estimator cel with covariance_centre_and_estimator. When mean is nothing, mu comes from ce.me; otherwise it comes from mean. When ce.w is not nothing, ce.w reaches ce.me and ce.ce through factory first.
2. Delegate to Statistics.cor(cel, X; dims = dims, mean = mu, kwargs...).

Arguments

• ce: Covariance estimator.

  • ce::Covariance{<:Any, <:Any, <:FullMoment}: Covariance estimator with FullMoment moment algorithm.
  • ce::Covariance{<:Any, <:Any, <:SemiMoment}: Covariance estimator with SemiMoment moment algorithm.

• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.

• dims: Dimension along which to perform the computation.

• mean: Optional mean value to use for centering. If not provided, computed using ce.me.

• kwargs...: Additional keyword arguments passed to the underlying correlation estimator.

Returns

• rho::MatNum: Correlation matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cor(Covariance(), X)2×2 Matrix{Float64}: 1.0  1.0 1.0  1.0

Related

• Covariance
• AbstractCovarianceEstimator
• GeneralCovariance
• FullMoment
• SemiMoment
• cov(ce::Covariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)

cor(
    ce::Covariance{<:Any, <:Any, <:SemiMoment},
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

SemiMoment variant of cor(ce::Covariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...). Clips de-meaned returns to zero before computing the correlation matrix, capturing only downside co-movements.

Algorithm

1. Resolve the centring vector mu and the inner estimator cel with covariance_centre_and_estimator.
2. Replace X with min.(X .- mu, 0), the de-meaned returns clipped at zero.
3. Delegate to Statistics.cor(cel, X; dims = dims, mean = 0, kwargs...). The zero mean is what stops the clipped returns being centred a second time.

partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Any

CovarianceState method of partial_fit!. Folds one observation into the running count, mean and co-moment accumulator.

Mathematical definition

\[\begin{align} n &\leftarrow n + 1\\ \boldsymbol{d} &= \boldsymbol{x} - \boldsymbol{\mu}\\ \boldsymbol{\mu} &\leftarrow \boldsymbol{\mu} + \frac{\boldsymbol{d}}{n}\\ \boldsymbol{M} &\leftarrow \boldsymbol{M} + \boldsymbol{d} (\boldsymbol{x} - \boldsymbol{\mu})^{\intercal}\, . \end{align}\]

Where:

• $n$: observation count.
• $\boldsymbol{x}$: the observation.
• $\boldsymbol{\mu}$: the running mean.
• $\boldsymbol{d}$: deviation of the observation from the mean before the fold.
• $\boldsymbol{M}$: the running co-moment accumulator.

The last line reads $\boldsymbol{\mu}$ after the third line moved it, where $\boldsymbol{d}$ read it before. That asymmetry is Welford's, and it is what keeps the accumulator positive semi-definite.

Algorithm

1. Refuse an observation whose length is not the number of assets the state describes.
2. Add one to the count.
3. Take the deviation of the observation from the mean before the fold, giving d.
4. Move mu in place along d, by the reciprocal of the new count.
5. Add the outer product of d and the deviation from the mean after the fold to M, in place.
6. Rebind the count with Accessors.@reset, and return the state.

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

Statistics.cov(
    ce::Union{<:GeneralCovariance, <:Covariance{<:Any, <:Any, <:FullMoment}},
    state::CovarianceState
) -> MatNum
Statistics.cov(ce::Union{<:GeneralCovariance, <:Covariance}) -> MatNum

Read the covariance matrix of an incremental fit out of a CovarianceState.

The two-argument method reads a state the caller holds, and the one-argument method reads the state the cache field of ce carries. The bias correction comes from the innermost StatsBase.SimpleCovariance, which partial_fit_corrected resolves.

The one-argument method is bound to every moment algorithm, not to FullMoment alone, because an estimator wrapped in Online carries a SampleBufferState rather than a CovarianceState, and reading it runs the batch verb over the buffer's rows — which every algorithm answers. It dispatches on what cache holds, so a SemiMoment estimator answers when it was wrapped and meets the named refusal of partial_fit_cache when it carries nothing.

Mathematical definition

\[\begin{align} \hat{\mathbf{\Sigma}} &= \frac{M}{n - c}\,. \end{align}\]

Where:

• $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
• $M$: Running co-moment accumulator.
• $n$: Observation count.
• $c$: One when the innermost StatsBase.SimpleCovariance is corrected, and zero otherwise.

Algorithm

1. Resolve the bias correction with partial_fit_corrected, which refuses every estimator an incremental fit does not reproduce.
2. Take the divisor n - c, and return a matrix of NaN when it is below one, in the way min_obs reads an asset with too few observations.
3. Otherwise divide the accumulator by the divisor.

Arguments

• ce: Covariance estimator.
• state: The state to read.

Validation

• ce passes partial_fit_corrected. An ArgumentError is thrown otherwise.
• ce.cache is not nothing, for the one-argument method. An ArgumentError is thrown otherwise.

Returns

• sigma::MatNum: Covariance matrix assets x assets. NaN where the state holds too few observations.

Examples

julia> ce = foldl(partial_fit!, eachrow([0.01 0.02; 0.03 0.04; 0.02 0.03]); init = Covariance());julia> cov(ce)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001

Related

• GeneralCovariance
• Covariance
• CovarianceState
• partial_fit!
• partial_fit_corrected
• cov(ce::Covariance, X::MatNum; dims::Int = 1, mean = nothing, kwargs...)

Statistics.cor(ce::Union{<:GeneralCovariance,
                         <:Covariance{<:Any, <:Any, <:FullMoment}},
               state::CovarianceState)
Statistics.cor(ce::Union{<:GeneralCovariance, <:Covariance})

Reads a correlation matrix out of a folded covariance estimator.

The correlation twin of the state read-out of Statistics.cov, and the same two steps the batch method takes: the folded covariance first, then coverage_correlation, which is the one place the conversion lives. So a folded estimator answers cor for the same configurations it answers cov, and the composite PortfolioOptimisersCovariance can offer both.

Arguments

• ce: Covariance estimator.
• state: The state the estimator carries.

Validation

• ce.cache is not nothing, for the one-argument form. An ArgumentError is thrown otherwise.

Returns

• rho::MatNum: Correlation matrix of the observations the state was fitted on.

Related

• CovarianceState
• coverage_correlation
• partial_fit!

port_opt_view(
    x::CovarianceState,
    i,
    args...
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}

Slices a CovarianceState to the selected assets.

The Welford accumulator of one pair of assets reads those two assets' observations alone, and reads no third asset. So the slice of the state is the state of the sliced universe, entry for entry, and the count is shared by every asset and passes through. The slice copies by index and does not view: a later partial_fit! on the viewed estimator would otherwise write through into the arrays of the estimator the view was taken from.

Arguments

• x: The state to slice.
• i: Index or indices of the assets to keep.
• args...: Additional positional arguments (ignored).

Returns

• state::CovarianceState: The state of the same sample over the selected assets.

Related

• CovarianceState
• port_opt_view
• partial_fit!

merge_states(
    a::CovarianceState,
    b::CovarianceState
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}

Folds two CovarianceState fitted on disjoint blocks into the state of the concatenated block.

Algorithm

1. Refuse the pair with assert_mergeable_states.
2. Fold the counts, the means and the accumulators with chan_merge, whose outer-product method reads a co-moment accumulator.

Arguments

• a: The state of the first block of observations.
• b: The state of the second block of observations.

Validation

• a and b pass assert_mergeable_states.

Returns

• state::CovarianceState: The state the two blocks give when they are fitted as one block.

Related

• CovarianceState
• merge_states
• chan_merge

partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    _::Nothing,
    _::Union{Nothing, AbstractVector{<:Bool}}
) -> Any

Nothing method of the coverage arm of partial_fit! for a CovarianceState. An estimator that carries no CoveragePolicy folds through partial_fit!(state::CovarianceState, x::VecNum), and the active mask is ignored.

Related

• partial_fit!
• CoveragePolicy
• CovarianceState

partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    cvg::CoveragePolicy,
    active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> Any

CoveragePolicy method of the coverage arm of partial_fit! for a CovarianceState. Folds one observation into the running per-pair count, centre and co-moment accumulator, reading each pair's own observations alone.

Mathematical definition

\[\begin{align} \nu_{jk} &\leftarrow \nu_{jk} + 1\\ d_{jk} &= r_{tj} - c_{jk}\\ c_{jk} &\leftarrow c_{jk} + \frac{d_{jk}}{\nu_{jk}}\\ d_{kj} &= r_{tk} - c_{kj}\\ c_{kj} &\leftarrow c_{kj} + \frac{d_{kj}}{\nu_{jk}}\\ M_{jk} &\leftarrow M_{jk} + d_{jk} (r_{tk} - c_{kj})\, , \end{align}\]

for every pair $(j, k)$ both of whose assets are finite and active at observation $t$, and no line at all for a pair that is not. Where:

• $\nu_{jk}$: the number of observations at which both assets of the pair were finite and active.
• $r_{tj}$: Return of asset $j$ at time $t$.
• $c_{jk}$: the running mean of asset $j$ over the observations of the pair $(j, k)$, so that $c_{kj}$ is the running mean of asset $k$ over those same observations.
• $M_{jk}$: the running co-moment accumulator of the pair.

This is Welford's recursion per pair, so a covariance folded observation by observation is the covariance of the same rows fitted as a block, and the diagonal is each asset's ordinary available-case variance. The pair is the unit of the centre as well as of the count, which is what makes the recursion exact: an entry centred on the whole window's mean would need every past term corrected as the window grows.

Algorithm

1. Refuse an observation whose length is not the number of assets the state describes.
2. Read the valid assets and the newly inactive ones with coverage_valid.
3. Apply the algorithm's fold-time rule with fold_inactive!.
4. Fold the observation into every pair of valid assets, taking the upper triangle and mirroring it, so that the accumulator stays exactly symmetric.
5. Copy the diagonal of the centre onto mu, which is each asset's own available-case mean.
6. Move the per-asset bookkeeping on with coverage_step!, add one to the observation count, and return the state.

Arguments

• state: The state to fold into, mutated in place.
• x: One observation, one entry per asset.
• cvg: The policy the estimator carries.
• active_mask: The active mask of the Asset Panel at this observation, or nothing.

Validation

• length(x) is the number of assets the state describes. A DimensionMismatch is thrown otherwise.

Returns

• state::CovarianceState: The state after the observation.

Related

• partial_fit!
• CoveragePolicy
• coverage_valid
• fold_inactive!
• coverage_step!

fold_inactive!(
    _::ResetCoverage,
    state::CovarianceState,
    ni::AbstractVector{<:Bool}
) -> CovarianceState

CovarianceState method of fold_inactive! under ResetCoverage. Zeroes the count, the centre and the accumulator of every row and column that touches an asset that has just gone inactive, so that a relisting starts every pair the asset belongs to cold.

Related

• fold_inactive!
• ResetCoverage
• CovarianceState