Covariance

The covariance matrix measures how the returns of the assets vary together. The mean-variance portfolio of Markowitz [32] takes the portfolio variance as its risk, which is w' Σ w for the weights w and the covariance matrix Σ. This page has the sample covariance and correlation estimators.

General covariance

PortfolioOptimisers.GeneralCovariance — Type
struct GeneralCovariance{__T_ce, __T_w, __T_cache} <: AbstractCovarianceEstimator

Adapts a StatsBase.CovarianceEstimator to the calling convention of the library, and holds its observation weights with it.

The estimator and the weights are in one object, so a caller passes one value. The StatsBase API takes an estimator and a separate weight vector at each call site. ce accepts all subtypes of StatsBase.CovarianceEstimator, so an estimator from another package, such as CovarianceEstimation.jl, gets to the library without change.

Fields

  • ce: Covariance estimator.
  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.

Constructors

GeneralCovariance(;    ce::StatsBase.CovarianceEstimator = StatsBase.SimpleCovariance(;        corrected = true),    w::Option{<:ObsWeights} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing) -> GeneralCovariance

Keywords correspond to the struct's fields.

Validation

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • ce: Recursively updated through factory.
  • w: Replaced with the incoming ObsWeights.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Examples

julia> GeneralCovariance()GeneralCovariance  ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)   w ┴ nothingjulia> GeneralCovariance(; w = StatsBase.Weights([0.1, 0.2, 0.7]))GeneralCovariance  ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)   w ┴ StatsBase.Weights{Float64, Float64, Vector{Float64}}: [0.1, 0.2, 0.7]

Related

source
Statistics.cov — Method
Statistics.cov(
    ce::GeneralCovariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the covariance matrix with a GeneralCovariance estimator.

The method calls robust_cov with or without observation weights, as ce.w sets. robust_cov computes the covariance matrix with ce.ce.

Algorithm

  1. Refuse a gapped sample with assert_finite_sample.
  2. Get the observation weights w from ce.w and X.
  3. When w is nothing, call robust_cov with ce.ce and X only.
  4. Otherwise call robust_cov with ce.ce, X and w. Write NaN where the sample gives the covariance no value, with weighted_comoment_guard. A corrected StatsBase.SimpleCovariance on one observation of positive weight is then NaN, as the unweighted fit is.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • mean: Optional precomputed centre of the data.
  • kwargs...: More keyword arguments for robust_cov and for the observation weights.

Returns

  • sigma::MatNum: Covariance matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cov(GeneralCovariance(), X)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001

Related

source
Statistics.cor — Method
Statistics.cor(
    ce::GeneralCovariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the correlation matrix with a GeneralCovariance estimator.

The method calls robust_cor with or without observation weights, as ce.w sets. robust_cor computes the correlation matrix with ce.ce.

Algorithm

  1. Refuse a gapped sample with assert_finite_sample.
  2. Get the observation weights w from ce.w and X.
  3. When w is nothing, call robust_cor with ce.ce and X only.
  4. Otherwise call robust_cor with the estimator of weighted_correlation_estimator, X and w. Write NaN where the sample gives the correlation no value, with weighted_comoment_guard. A StatsBase.SimpleCovariance on one observation of positive weight is then NaN at every entry, as the unweighted fit is.
  5. Give each constant return the NaN row and column of an undefined correlation with mark_constant_correlation!. Under the Pearson estimator StatsBase.SimpleCovariance, it tests the data, because the variance of a constant can be about 1e-38 and not zero. Another estimator keeps its values, and only a zero variance in a matrix that holds a NaN gets a NaN diagonal entry.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • mean: Optional precomputed centre of the data.
  • kwargs...: More keyword arguments for robust_cor and for the observation weights.

Returns

  • rho::MatNum: Correlation matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cor(GeneralCovariance(), X)2×2 Matrix{Float64}: 1.0  1.0 1.0  1.0

Related

source

Covariance

PortfolioOptimisers.Covariance — Type
struct Covariance{__T_me, __T_ce, __T_alg, __T_w, __T_cvg, __T_cache} <: AbstractCovarianceEstimator

Estimates the covariance matrix of asset returns from a centring estimator, a covariance estimator, and a moment algorithm.

me centres the data, ce computes the covariance matrix, and alg selects the full moment or the semi-moment.

w weights the whole estimate, so it weights the centre and the deviations. The four methods send me and ce through factory, which replaces the weights that each of them holds with w. Pass mean for a centre that w does not describe.

ce accepts all StatsBase.CovarianceEstimator types. No verb of the library reads the weights of an estimator that the library does not own. A ce from another package, such as CovarianceEstimation.jl, keeps its own configuration, and w is the field that weights a Covariance.

Fields

  • me: Expected returns estimator.
  • ce: Covariance estimator.
  • alg: Moment algorithm.
  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • cvg: Optional CoveragePolicy. nothing selects the reduce-and-expand path of the Coverage Universe. On that path, an asset that is not finite or not active at an observation of the window is NaN in all of the result. A policy replaces that path with available-case estimation. The fit of each cell reads the observations at which all the assets of that cell are finite and active, and each cell has its own denominator. An asset is in the result where admits admits it.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.

Constructors

Covariance(;    me::AbstractExpectedReturnsEstimator = SimpleExpectedReturns(),    ce::StatsBase.CovarianceEstimator = GeneralCovariance(),    alg::AbstractMomentAlgorithm = FullMoment(),    w::Option{<:ObsWeights} = nothing,    cvg::Option{<:CoveragePolicy} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing) -> Covariance

Keywords correspond to the struct's fields.

Validation

  • ce is not an AbstractVarianceEstimator. A variance estimator has no cov or cor, so assert_covariance_estimator raises an ArgumentError that names ce.
  • If w is not nothing, !isempty(w).
  • If cvg is a CoveragePolicy, me is nothing or a SimpleExpectedReturns, through assert_coverage_fittable. The available-case fit centres each asset on the mean of its own observations, and on no other centre. With no w, the fit folds, so me holds no weights. With w, the fit centres on the weighted mean, and w replaces the weights of me. An ArgumentError is thrown otherwise.
  • If cvg is a CoveragePolicy, ce reads the sample as a StatsBase.SimpleCovariance, through assert_coverage_fittable. With no w, the available-case fit folds a Welford accumulator, so ce holds no weights of its own. With w, the weights of w replace the weights of ce. An ArgumentError is thrown otherwise.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • me: Recursively updated through factory.
  • ce: Recursively updated through factory.
  • w: Replaced with the incoming ObsWeights.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Examples

julia> Covariance()Covariance   me ┼ SimpleExpectedReturns      │   w ┴ nothing   ce ┼ GeneralCovariance      │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │    w ┴ nothing  alg ┼ FullMoment()    w ┴ nothingjulia> Covariance(; w = StatsBase.AnalyticWeights([0.2, 0.3, 0.5]))Covariance   me ┼ SimpleExpectedReturns      │   w ┴ nothing   ce ┼ GeneralCovariance      │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │    w ┴ nothing  alg ┼ FullMoment()    w ┴ StatsBase.AnalyticWeights{Float64, Float64, Vector{Float64}}: [0.2, 0.3, 0.5]

Related

source
Statistics.cov — Method
Statistics.cov(
    ce::Covariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the covariance matrix with a Covariance estimator.

Mathematical definition

FullMoment covariance:

\[\begin{align} \hat{\mathbf{\Sigma}}_{ij} &= \frac{1}{T-1} \sum_{t=1}^{T} (r_{ti} - \hat{\mu}_i)(r_{tj} - \hat{\mu}_j)\,. \end{align}\]

SemiMoment covariance, the downside covariance of the de-meaned returns clipped at zero:

\[\begin{align} \tilde{r}_{tj} &= \min(r_{tj} - \hat{\mu}_j,\, 0)\,,\\ \hat{\mathbf{\Sigma}}^{\text{semi}}_{ij} &= \frac{1}{T-1} \sum_{t=1}^{T} \tilde{r}_{ti} \, \tilde{r}_{tj}\,. \end{align}\]

Where:

  • $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
  • $\hat{\mathbf{\Sigma}}^{\text{semi}}_{ij}$: The estimated semi-covariance between assets $i$ and $j$.
  • $r_{tj}$: Return of asset $j$ at time $t$.
  • $r_{ti}$: The return of asset $i$ at time $t$.
  • $\hat{\mu}_j$: Estimated mean of asset $j$.
  • $\hat{\mu}_i$: The estimated mean of asset $i$.
  • $\tilde{r}_{ti}$, $\tilde{r}_{tj}$: The de-meaned returns of assets $i$ and $j$, clipped at zero.
  • $T$: Number of observations.

The semi-covariance keeps the $T-1$ divisor of the full moment, so it is not the covariance of the clipped returns about their own mean.

Algorithm

  1. Get the centring vector mu and the inner estimator cel with covariance_centre_and_estimator. When mean is nothing, mu comes from ce.me. Otherwise it comes from mean. When ce.w is not nothing, ce.w first gets to ce.me and ce.ce through factory.
  2. Call Statistics.cov(cel, X; dims = dims, mean = mu, kwargs...). Under SemiMoment, first replace X with min.(X .- mu, 0), and call the inner estimator with a zero mean.

coverage_covariance does these steps. It dispatches on the alg and cvg fields of ce, so one method serves every moment algorithm.

Arguments

  • ce: Covariance estimator.
    • ce.alg::FullMoment: The FullMoment moment algorithm gives the covariance matrix.
    • ce.alg::SemiMoment: The SemiMoment moment algorithm gives the semi-covariance matrix.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • mean: Optional precomputed centre of the data. If it is nothing, the method computes it with ce.me.
  • kwargs...: More keyword arguments for the mean estimator and for the inner covariance estimator.

Returns

  • sigma::MatNum: Covariance matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cov(Covariance(), X)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001julia> cov(Covariance(; alg = SemiMoment()), X)2×2 Matrix{Float64}: 5.0e-5  5.0e-5 5.0e-5  5.0e-5

Related

source
Statistics.cor — Method
Statistics.cor(
    ce::Covariance,
    X::MatNum;
    dims::Int = 1,
    mean = nothing,
    kwargs...
) -> MatNum

Compute the correlation matrix with a Covariance estimator.

Mathematical definition

\[\begin{align} \hat{\mathbf{P}}_{ij} &= \frac{\hat{\mathbf{\Sigma}}_{ij}}{\hat{\sigma}_i \hat{\sigma}_j}\,. \end{align}\]

Where:

  • $\hat{\mathbf{P}}_{ij}$: The estimated correlation between assets $i$ and $j$.
  • $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
  • $\hat{\sigma}_i$: Estimated standard deviation of asset $i$.
  • $\hat{\sigma}_j$: The estimated standard deviation of asset $j$.

The alg field of ce changes $\hat{\mathbf{\Sigma}}$. SemiMoment standardises the semi-covariance, so the diagonal is one and an off-diagonal entry is a downside correlation.

Algorithm

  1. Get the centring vector mu and the inner estimator cel with covariance_centre_and_estimator. When mean is nothing, mu comes from ce.me. Otherwise it comes from mean. When ce.w is not nothing, ce.w first gets to ce.me and ce.ce through factory.
  2. Call Statistics.cor(cel, X; dims = dims, mean = mu, kwargs...). Under SemiMoment, first replace X with min.(X .- mu, 0), and call the inner estimator with a zero mean.

coverage_covariance does these steps. It dispatches on the alg and cvg fields of ce, so one method serves every moment algorithm.

Arguments

  • ce: Covariance estimator.

    • ce.alg::FullMoment: The FullMoment moment algorithm gives the correlation matrix.
    • ce.alg::SemiMoment: The SemiMoment moment algorithm gives the downside correlation matrix.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.

  • dims: Dimension along which the computation runs.

  • mean: Optional precomputed centre of the data. If it is nothing, the method computes it with ce.me.

  • kwargs...: More keyword arguments for the mean estimator and for the inner correlation estimator.

Returns

  • rho::MatNum: Correlation matrix assets x assets.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> cor(Covariance(), X)2×2 Matrix{Float64}: 1.0  1.0 1.0  1.0

Related

source

Incremental fit

Like the sample mean, the sample covariance with FullMoment updates with each new block of observations. GeneralCovariance and Covariance use the same state, CovarianceState, and update it in the same way. partial_fit! stores the state in the cache field of the estimator it returns, and cov(ce) computes the estimate from that state, with no data.

PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{Number, AbstractJuMPScalar}}
) -> CovarianceState

The CovarianceState method of partial_fit! folds one observation into the count, the running mean and the co-moment accumulator.

Mathematical definition

\[\begin{align} n &\leftarrow n + 1\\ \boldsymbol{d} &= \boldsymbol{x} - \boldsymbol{\mu}\\ \boldsymbol{\mu} &\leftarrow \boldsymbol{\mu} + \frac{\boldsymbol{d}}{n}\\ \boldsymbol{M} &\leftarrow \boldsymbol{M} + \boldsymbol{d} (\boldsymbol{x} - \boldsymbol{\mu})^{\intercal}\, . \end{align}\]

Where:

  • $n$: The observation count.
  • $\boldsymbol{x}$: The observation.
  • $\boldsymbol{\mu}$: The running mean.
  • $\boldsymbol{d}$: The deviation of the observation from the mean before the fold.
  • $\boldsymbol{M}$: The running co-moment accumulator.

The last line reads $\boldsymbol{\mu}$ after the third line changed it, but $\boldsymbol{d}$ reads it before the change. This asymmetry is the one of Welford, and it keeps the accumulator positive semi-definite.

Algorithm

  1. Refuse an observation whose length is not the number of assets that the state describes.
  2. Add one to the count.
  3. Take the deviation d of the observation from the mean before the fold.
  4. Add d, divided by the new count, to mu in place.
  5. Add the outer product of d and the deviation from the mean after the fold to M, in place.
  6. Build the state with the new count through its positional constructor, and return it.
source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Union{ExpWeightedVariance{<:Any, <:Any, <:Any, <:Union{Nothing, var"#s2433"} where var"#s2433"<:ExpWeightedVarianceState}, ExpWeightedVariance{<:Number, <:Integer, <:AbstractCentring, <:AbstractPartialFitState}}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Union{ExpWeightedVariance{<:Any, <:Any, <:Any, <:Union{Nothing, var"#s2433"} where var"#s2433"<:ExpWeightedVarianceState}, ExpWeightedVariance{<:Number, <:Integer, <:AbstractCentring, <:AbstractPartialFitState}}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Union{ExpWeightedVariance{<:Any, <:Any, <:Any, <:Union{Nothing, var"#s2433"} where var"#s2433"<:ExpWeightedVarianceState}, ExpWeightedVariance{<:Number, <:Integer, <:AbstractCentring, <:AbstractPartialFitState}}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Union{ExpWeightedVariance{<:Any, <:Any, <:Any, <:Union{Nothing, var"#s2433"} where var"#s2433"<:ExpWeightedVarianceState}, ExpWeightedVariance{<:Number, <:Integer, <:AbstractCentring, <:AbstractPartialFitState}}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Union{ExpWeightedVariance{<:Any, <:Any, <:Any, <:Union{Nothing, var"#s2433"} where var"#s2433"<:ExpWeightedVarianceState}, ExpWeightedVariance{<:Number, <:Integer, <:AbstractCentring, <:AbstractPartialFitState}}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
Statistics.cov — Method
Statistics.cov(
    ce::Union{GeneralCovariance, Covariance{<:Any, <:Any, <:FullMoment}},
    state::CovarianceState;
    strict::Bool = false
) -> MatNum
Statistics.cov(ce::Union{GeneralCovariance, Covariance}; strict::Bool = false) -> MatNum

Read the covariance matrix of an incremental fit from a CovarianceState.

The two-argument method reads a state that the caller holds. The one-argument method reads the state in the cache field of ce. The bias correction comes from the innermost StatsBase.SimpleCovariance, and partial_fit_corrected reads it.

The one-argument method accepts all moment algorithms, not only FullMoment. An estimator wrapped in Online holds a SampleBufferState and not a CovarianceState. To read that state, the method runs the batch verb over the rows of the buffer, and all algorithms accept that. The method dispatches on what cache holds, so a SemiMoment estimator returns a result when a wrapper gave it a buffer. When it holds no state, partial_fit_cache refuses it with an error that names the cause.

Mathematical definition

\[\begin{align} \hat{\mathbf{\Sigma}} &= \frac{M}{n - c}\,. \end{align}\]

Where:

  • $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
  • $M$: The running co-moment accumulator.
  • $n$: The observation count.
  • $c$: One when the innermost StatsBase.SimpleCovariance is corrected, and zero otherwise.

Algorithm

  1. Get the bias correction with partial_fit_corrected, which refuses each estimator that an incremental fit cannot reproduce.
  2. Take the divisor n - c. Return a matrix of NaN when the divisor is less than one. min_obs reads an asset with too few observations in the same way.
  3. Otherwise divide the accumulator by the divisor.

Arguments

  • ce: Covariance estimator.
  • state: The state to read.
  • strict: If true, a peel of the CoveragePolicy of ce throws an error. If false, the peel gives a warning. See coverage_peel.

Validation

  • ce passes partial_fit_corrected. An ArgumentError is thrown otherwise.
  • ce.cache is not nothing, for the one-argument method. An ArgumentError is thrown otherwise.

Returns

  • sigma::MatNum: Covariance matrix assets x assets. NaN where the state holds too few observations.

Examples

julia> ce = foldl(partial_fit!, eachrow([0.01 0.02; 0.03 0.04; 0.02 0.03]); init = Covariance());julia> cov(ce)2×2 Matrix{Float64}: 0.0001  0.0001 0.0001  0.0001

Related

source
Statistics.cor — Method
Statistics.cor(ce::Union{GeneralCovariance,
                         Covariance{<:Any, <:Any, <:FullMoment}},
               state::CovarianceState)
Statistics.cor(ce::Union{GeneralCovariance, Covariance})

Reads a correlation matrix from a folded covariance estimator.

This method is the correlation form of cov(ce) with no data. It takes the same two steps as the batch method. First it computes the folded covariance. Then it applies coverage_correlation, which is the one place that holds the conversion. A folded estimator returns cor for the same configurations as cov, and the composite PortfolioOptimisersCovariance can give both.

Arguments

  • ce: Covariance estimator.
  • state: The state that the estimator holds.
  • strict: If true, a peel of the CoveragePolicy of ce throws an error. If false, the peel gives a warning. See coverage_peel.

Validation

  • ce.cache is not nothing, for the one-argument form. An ArgumentError is thrown otherwise.

Returns

  • rho::MatNum: The correlation matrix of the observations that the state holds.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
    x::CovarianceState,
    i,
    args...
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}

Slices a CovarianceState to the selected assets.

The Welford accumulator of one pair of assets reads only the observations of those two assets, and it reads no third asset. The slice of the state is the state of the sliced universe, entry for entry. All the assets share the count, so the method uses it without change. The slice copies by index and does not use a view. With a view, a later partial_fit! on the viewed estimator writes into the arrays of the original estimator.

Arguments

  • x: The state to slice.
  • i: The index or the indices of the assets to keep.
  • args...: More positional arguments, which the method ignores.

Returns

  • state::CovarianceState: The state of the same sample over the selected assets.

Related

source
PortfolioOptimisers.merge_states — Method
merge_states(
    a::CovarianceState,
    b::CovarianceState
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}

Folds two CovarianceState fitted on disjoint blocks into the state of the concatenated block.

Algorithm

  1. Refuse the pair with assert_mergeable_states.
  2. If one of the two states holds no coverage counts, fold the counts, the means and the accumulators with chan_merge. Its outer-product method reads a co-moment accumulator.
  3. Otherwise, fold the centre and the accumulator of each pair with the pairwise counts nu of the two states. A pair with no shared observation in the two blocks keeps a centre and an accumulator of zero. Read the mean of each asset from the diagonal of the centre, and merge the coverage counts too.

Arguments

  • a: The state of the first block of observations.
  • b: The state of the second block of observations.

Validation

Returns

  • state::CovarianceState: The state of the two blocks, as if one fit read them as one block.

Related

source

Available-case fit

With a CoveragePolicy in its cvg field, the estimator fits each pair of assets on the observations where both assets are finite and active. Without a policy, it fits on the coverage universe, the assets that are finite and active at every observation. PortfolioOptimisers.coverage_covariance chooses between the two fits.

PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    _::Nothing,
    active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> CovarianceState

This is the Nothing method of the coverage arm of partial_fit! for a CovarianceState. An estimator that holds no CoveragePolicy folds through partial_fit!(state::CovarianceState, x::VecNum). A plain state holds one count for all the assets, so it cannot skip an inactive cell. inactive_as_nonfinite writes NaN in that cell, and the row and the column of the asset are then NaN. The Asset Panel method gives the same NaN for an asset outside the Coverage Universe of the same rows.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    state::CovarianceState,
    x::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    cvg::CoveragePolicy,
    active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> CovarianceState

This is the CoveragePolicy method of the coverage arm of partial_fit! for a CovarianceState. It folds one observation into the count, the centre and the co-moment accumulator of each pair. Each pair reads only its own observations.

Mathematical definition

\[\begin{align} \nu_{jk} &\leftarrow \nu_{jk} + 1\\ d_{jk} &= r_{tj} - c_{jk}\\ c_{jk} &\leftarrow c_{jk} + \frac{d_{jk}}{\nu_{jk}}\\ d_{kj} &= r_{tk} - c_{kj}\\ c_{kj} &\leftarrow c_{kj} + \frac{d_{kj}}{\nu_{jk}}\\ M_{jk} &\leftarrow M_{jk} + d_{jk} (r_{tk} - c_{kj})\, , \end{align}\]

for each pair $(j, k)$ whose two assets are finite and active at observation $t$. The method applies no line to a pair that is not. Where:

  • $\nu_{jk}$: The number of observations at which the two assets of the pair are finite and active.
  • $r_{tj}$: Return of asset $j$ at time $t$.
  • $c_{jk}$: The running mean of asset $j$ over the observations of the pair $(j, k)$, and $c_{kj}$ is the running mean of asset $k$ over the same observations.
  • $M_{jk}$: The running co-moment accumulator of the pair.

This is the Welford recursion for each pair. A covariance that folds one observation at a time is the covariance of the same rows in one block fit. The diagonal is the ordinary available-case variance of each asset. The pair is the unit of the centre and of the count, and this makes the recursion exact. An entry centred on the mean of the whole window needs a correction of all past terms when the window grows.

Algorithm

  1. Refuse an observation whose length is not the number of assets that the state describes.
  2. Read the valid assets, and the assets that became inactive at this observation, with coverage_valid.
  3. Apply the fold-time rule of the coverage algorithm with fold_inactive!.
  4. Fold the observation into each pair of valid assets. Compute the upper triangle and copy it to the lower triangle, so that the accumulator stays exactly symmetric.
  5. Copy the diagonal of the centre to mu. The diagonal is the available-case mean of each asset.
  6. Update the per-asset records with coverage_step!. Add one to the observation count, and return the state.

Arguments

  • state: The state to fold into. The method changes it in place.
  • x: One observation, with one entry for each asset.
  • cvg: The policy that the estimator holds.
  • active_mask: The active mask of the Asset Panel at this observation, or nothing.

Validation

  • length(x) is the number of assets that the state describes. A DimensionMismatch is thrown otherwise.

Returns

  • state::CovarianceState: The state after the observation.

Related

source

References

[32]
H. Markowitz. Portfolio Selection. The Journal of Finance 7, 77–91 (1952).