Simple covariance: private API

show_fields(_::GeneralCovariance) -> Tuple{Symbol, Symbol}

Renders every field of a GeneralCovariance except cache.

The state a cache holds is the running detail of an incremental fit, not the configuration a reader looks the type up for, and it prints under the estimator at every site that renders one. Set set_show_nothing_fields!(:GeneralCovariance, true) to render it.

Arguments

• ::GeneralCovariance: Covariance estimator, read for its type alone.

Returns

• fields::Tuple: The field names to render, which is (:ce, :w).

Related

• GeneralCovariance
• show_fields
• set_show_nothing_fields!

show_fields(
    ce::Covariance
) -> Union{NTuple{4, Symbol}, NTuple{5, Symbol}}

Renders every field of a Covariance except cache.

The state a cache holds is the running detail of an incremental fit, not the configuration a reader looks the type up for, and it prints under the estimator at every site that renders one. Set set_show_nothing_fields!(:Covariance, true) to render it.

Arguments

• ::Covariance: Covariance estimator, read for its type alone.

Returns

• fields::Tuple: The field names to render, which is (:me, :ce, :alg, :w) with no policy and (:me, :ce, :alg, :w, :cvg) with one.

Related

• Covariance
• CoveragePolicy
• show_fields
• set_show_nothing_fields!

covariance_centre_and_estimator(
    ce::Covariance,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    kwargs...
) -> Tuple{Any, Any}

Resolve the centring vector and the inner covariance estimator that a Covariance method computes with.

The four methods of Statistics.cov and Statistics.cor that take a Covariance reach one centre and one inner estimator by this verb, so ce.w reaches the centre and the deviations by one rule.

Algorithm

1. Resolve the centre mu from ce.me and ce.w with weighted_centre, which reads mean when the caller gave one. ce.w reaches ce.me through factory, so the centre carries the weights of the deviations.
2. ce.w is nothing: return ce.ce unchanged.
3. ce.w is not nothing: send ce.ce through factory_child with ce.w. An estimator of the library takes the weights; a StatsBase.CovarianceEstimator that is not one of them passes through unchanged, because no verb of this library reads its weights.

Step 2 is a performance guard and not a second contract. ce.w is a field, so its type decides the branch, and the guard keeps a windowed loop from rebuilding the estimator tree of ce once per window. A ce.ce that holds weights of its own therefore keeps them when ce.w is nothing, and loses them to ce.w when it is not. That is what factory does on every other path.

Arguments

• ce: Covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• mean: Optional mean value to use for centering.
• kwargs...: Additional keyword arguments passed to the mean estimator.

Returns

• mu::Union{<:Number, <:ArrNum}: Centring vector.
• cel::StatsBase.CovarianceEstimator: Inner covariance estimator, weighted by ce.w when it is not nothing.

Related

• Covariance
• weighted_centre
• factory
• factory_child

struct CovarianceState{__T_n, __T_mu, __T_M, __T_cvg} <: AbstractPartialFitState

Carries the running observation count, mean and co-moment accumulator of an incremental covariance fit.

The state of GeneralCovariance and of Covariance{<:Any, <:Any, <:FullMoment} under partial_fit!. One struct serves both, because the two estimators run the same recursion over the same three quantities. M is the accumulator $\sum_t (\boldsymbol{r}_t - \hat{\boldsymbol{\mu}}) (\boldsymbol{r}_t - \hat{\boldsymbol{\mu}})^{\intercal}$ and not the covariance, so cov(ce::GeneralCovariance, state::CovarianceState) divides it by the count, or by the count less one when the inner StatsBase.SimpleCovariance is corrected.

Fields

• n: Number of observations folded into the state.

• mu: Running mean of the observations folded into the state, assets × 1.

• M: Running second-moment accumulator of the observations folded into the state, about mu.

• cvg: Optional CoverageCounts, the per-cell denominators and per-asset bookkeeping of an available-case fold. It is nothing when the estimator carries no CoveragePolicy, so the plain state costs nothing.

Constructors

CovarianceState(;    n::Integer = 0,    mu::VecNum,    M::MatNum = zeros(eltype(mu), length(mu), length(mu)),    cvg::Option{<:CoverageCounts} = nothing) -> CovarianceState

Keywords correspond to the struct's fields. A state seeded for N assets is CovarianceState(; mu = zeros(N)), which partial_fit! builds when the cache field of the estimator holds nothing.

Validation

• n >= 0. A DomainError is thrown otherwise.
• !isempty(mu). An IsEmptyError is thrown otherwise.
• Every entry of mu and of M is finite. An IsNonFiniteError is thrown otherwise.
• size(M) == (length(mu), length(mu)). A DimensionMismatch is thrown otherwise.

View parameters

When port_opt_view is called on this type, its fields are subset to the selected assets:

• mu: Sliced to the selected indices via port_opt_view.
• M: Sliced to the selected indices on both axes via port_opt_view.

Examples

julia> PortfolioOptimisers.CovarianceState(; mu = [0.0, 0.0])PortfolioOptimisers.CovarianceState    n ┼ Int64: 0   mu ┼ Vector{Float64}: [0.0, 0.0]    M ┼ 2×2 Matrix{Float64}  cvg ┴ nothing

Related

• AbstractPartialFitState
• GeneralCovariance
• Covariance
• partial_fit!
• merge_states

copy(x::CovarianceState) -> CovarianceState

Copies a CovarianceState, so the copy shares no array with the original.

The copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. The count is a scalar and passes through, and the running mean and the co-moment accumulator are copied.

Arguments

• x: The state to copy.

Returns

• state::CovarianceState: A fresh state, equal to x, whose mu and M are fresh arrays.

Related

• CovarianceState
• partial_fit
• AbstractPartialFitState

covariance_state_seed(
    cache::Union{Nothing, CovarianceState},
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Any
covariance_state_seed(
    cache::Union{Nothing, CovarianceState},
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    cvg::Union{Nothing, CoveragePolicy}
) -> Any

Returns the CovarianceState an incremental covariance fit folds into, seeding one of zeros when the estimator carries none.

Both covariance estimators of the seam seed the same state from the same observation, so the seed is written once here rather than at each partial_fit! method.

Arguments

• cache: The state the estimator carries, or nothing.
• x: One observation, assets × 1, read for its length and its element type.

Returns

• state::CovarianceState: The state cache holds, or a state of zeros over length(x) assets.

Related

• CovarianceState
• partial_fit!

partial_fit_corrected(ce::SimpleCovariance) -> Bool

Resolves the bias correction of the covariance estimator an incremental fit reproduces, and refuses every other estimator.

A CovarianceState accumulates the full-moment sum of outer products about the running mean, which is what StatsBase.SimpleCovariance divides by the count or by the count less one. Every other covariance estimator reads the sample in a way the accumulator cannot answer, so it is refused rather than answered wrongly. The verb walks the wrappers, so it reads the flag through a GeneralCovariance and through a Covariance whose algorithm is FullMoment.

Arguments

• ce: Covariance estimator.

Validation

• ce carries no observation weights, at every level it is walked through. An ArgumentError is thrown otherwise.
• The innermost estimator is a StatsBase.SimpleCovariance. An ArgumentError is thrown otherwise.

Returns

• corrected::Bool: Bias correction of the innermost StatsBase.SimpleCovariance.

Related

• CovarianceState
• GeneralCovariance
• Covariance
• partial_fit!

partial_fit_corrected(ce::GeneralCovariance) -> Any

GeneralCovariance method of partial_fit_corrected. Refuses the observation weights of ce with assert_partial_fittable, then reads the flag off ce.ce.

partial_fit_corrected(
    ce::Covariance{<:Any, <:Any, <:FullMoment}
) -> Any

Covariance{<:Any, <:Any, <:FullMoment} method of partial_fit_corrected. Refuses the observation weights and the centring estimator of ce with assert_partial_fittable, then reads the flag off ce.ce.

partial_fit_corrected(ce::CovarianceEstimator) -> Bool

Fallback method of partial_fit_corrected. Refuses every covariance estimator an incremental fit does not reproduce, naming the type that was handed over.

coverage_covariance(f, ce, cvg, X; dims::Int = 1, mean = nothing,
                    active_mask = nothing, kwargs...) -> MatNum

Routes a covariance or correlation fit to the Coverage Universe arm or to the available-case arm.

The cvg field of the estimator is passed as the third argument, so the arm is chosen by dispatch on the policy rather than by a branch on its value, exactly as coverage_mean does for a sample mean. f is Statistics.cov or Statistics.cor, and the moment algorithm of ce chooses between the FullMoment arms, which fold, and the SemiMoment arms, which have no recursion and run a two-pass over the block.

Arguments

• f: Statistics.cov or Statistics.cor.
• ce: Covariance estimator.
• cvg: The policy the estimator carries, which selects the arm.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• mean: A precomputed centre, or nothing.
• active_mask: The active mask of the Asset Panel, observations × assets, or nothing. The Coverage Universe arms ignore it.
• kwargs...: Additional keyword arguments passed to the centring estimator and the inner covariance estimator.

Returns

• sigma::MatNum: The covariance or correlation matrix.

Related

• CoveragePolicy
• Covariance
• coverage_mean
• coverage_valid_block

coverage_covariance(
    f,
    ce::Covariance{<:Any, <:Any, <:FullMoment},
    ::Nothing,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

Nothing method of the FullMoment arm of coverage_covariance. The Coverage Universe arm, which resolves the centre and the inner estimator with covariance_centre_and_estimator and delegates, and which is the body the verb has always had.

Related

• coverage_covariance
• covariance_centre_and_estimator

coverage_covariance(
    f,
    ce::Covariance{<:Any, <:Any, <:FullMoment},
    cvg::CoveragePolicy,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

CoveragePolicy method of the FullMoment arm of coverage_covariance. The available-case arm: each pair is fitted on the observations at which both of its assets are finite and active, and the batch answer is the incremental one, because the arm folds the block through partial_fit! and reads the state out.

The centre of a pair is that pair's own mean, which is what a pairwise Welford recursion carries, so the diagonal is each asset's ordinary available-case variance and an off-diagonal entry is centred on the observations the pair shares. ce.ce, ce.me and ce.w are refused where an incremental fit does not reproduce them, by partial_fit_corrected.

Algorithm

1. Read the Bessel correction, and refuse a configuration no incremental fit reproduces, with partial_fit_corrected.
2. Fold every row of X into a fresh state with partial_fit!, carrying the active mask.
3. Read the state out with cov(ce::Covariance, state::CovarianceState), and rescale to a unit diagonal when the verb is Statistics.cor.

Related

• coverage_covariance
• CoveragePolicy
• coverage_correlation
• partial_fit!

coverage_covariance(
    f,
    ce::Covariance{<:Any, <:Any, <:SemiMoment},
    ::Nothing,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

Nothing method of the SemiMoment arm of coverage_covariance. The Coverage Universe arm, which clips the de-meaned returns at zero and delegates with a zero centre, and which is the body the verb has always had.

Related

• coverage_covariance
• SemiMoment

coverage_covariance(
    f,
    ce::Covariance{<:Any, <:Any, <:SemiMoment},
    cvg::CoveragePolicy,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

CoveragePolicy method of the SemiMoment arm of coverage_covariance. The available-case arm, run as a two-pass over the block rather than as a fold, because a semi-covariance has no incremental recursion: the clip is taken about a centre the whole window fixes, so a new observation moves every past term.

The centre is each asset's available-case mean, its own finite and active observations alone, and not the pair's own mean as the FullMoment arm's is. That is the same asymmetry the plain path already has, where the clip is taken about a centre and the co-moment about zero.

Algorithm

1. Read the valid entries of the block with coverage_valid_block.
2. Centre each asset on its own available-case mean, clip the deviations at zero and zero the invalid entries.
3. Take the numerator as the cross product of the clipped block, and the denominator as the cross product of the valid mask, which counts the observations of each pair.
4. Divide and frame with coverage_divide, and rescale to a unit diagonal when the verb is Statistics.cor.

Related

• coverage_covariance
• coverage_valid_block
• coverage_divide

coverage_covariance(
    ce::Union{Covariance{<:Any, <:Any, <:FullMoment}, GeneralCovariance},
    _::Nothing,
    state::CovarianceState
) -> Any

Nothing method of the read-out arm of coverage_covariance. Every pair shares one count, so the whole answer is NaN until the count passes the Bessel correction.

GeneralCovariance carries no cvg field and reads the same state, so it always reaches this arm.

Related

• coverage_covariance
• CovarianceState

coverage_covariance(
    ce::Covariance{<:Any, <:Any, <:FullMoment},
    cvg::CoveragePolicy,
    state::CovarianceState
) -> Any

CoveragePolicy method of the read-out arm of coverage_covariance. Each pair's accumulator is divided by that pair's own count less the Bessel correction, and an asset the policy refuses is NaN across its whole row and column.

A pair whose two assets are each admitted but which share no observation is NaN on its own, because a covariance of no observations is not a number. That is available-case estimation's own cost, and it is what a consumer of the matrix must be ready for.

Related

• coverage_covariance
• coverage_admission
• coverage_divide

coverage_correlation(f::typeof(Statistics.cov), sigma::MatNum) -> MatNum
coverage_correlation(f::typeof(Statistics.cor), sigma::MatNum) -> MatNum

Maps an available-case covariance onto the answer the caller's verb asks for.

An available-case fit produces a covariance whichever verb called it, because the state carries a co-moment accumulator and nothing else, so the correlation is that covariance rescaled by the square roots of its own diagonal. Dispatching on the verb rather than comparing it keeps the choice at compile time.

Arguments

• f: Statistics.cov or Statistics.cor.
• sigma: The available-case covariance.

Returns

• val::MatNum: sigma itself, or sigma rescaled to a unit diagonal.

Related

• coverage_covariance
• Covariance

coverage_policy(ce::GeneralCovariance) -> Nothing
coverage_policy(ce::Covariance) -> Option{<:CoveragePolicy}

Reads the CoveragePolicy a covariance estimator carries, out of an estimator that may have no such field.

GeneralCovariance and Covariance share one state and one read-out, and only the second has a cvg field. The read-out therefore asks for the policy through this verb rather than for the field, so that the arm is still chosen by dispatch and the estimator without the field answers nothing.

Arguments

• ce: Covariance estimator.

Returns

• cvg::Option{<:CoveragePolicy}: The policy the estimator carries, or nothing.

Related

• CoveragePolicy
• GeneralCovariance
• Covariance
• coverage_covariance