Geodesic shrinkage covariance

abstract type AbstractCovarianceShrinkageTarget <: AbstractMomentAlgorithm

Abstract supertype for the rules that build a covariance shrinkage target from the matrix being shrunk.

Each rule builds a target from the table of Schäfer and Strimmer. The target is positive definite when the matrix being shrunk is, so a geodesic reaches it. The target of the table with perfect positive correlation has rank one, so no rule builds it. DiagonalTarget builds the diagonal target of the same table. It is a RegimeAdjustedTarget, so it joins the rules through GeodesicShrinkageTarget and not through this type.

Interfaces

To implement a new target rule, subtype AbstractCovarianceShrinkageTarget and implement the following method.

shrinkage_target

• shrinkage_target(tgt::MyShrinkageTarget, sigma::MatNum) -> MatNum: Builds the target matrix from the matrix being shrunk.

Arguments

• tgt: The target rule.
• sigma: Covariance-like or correlation-like matrix assets × assets.

Returns

• target::MatNum: A positive definite matrix of the size of sigma.

Examples

julia> struct MyShrinkageTarget <: PortfolioOptimisers.AbstractCovarianceShrinkageTarget endjulia> function PortfolioOptimisers.shrinkage_target(::MyShrinkageTarget,                                                     sigma::PortfolioOptimisers.MatNum)           return 2 * PortfolioOptimisers.shrinkage_target(IdentityTarget(), sigma)       endjulia> ce = GeodesicShrinkageCovariance(; tgt = MyShrinkageTarget(), alpha = 1);julia> cov(ce, [1.0 2.0; 2.0 1.0; 0.0 0.5])2×2 Matrix{Float64}: 2.0  0.0 0.0  2.0

Related

• IdentityTarget
• ScaledIdentityTarget
• CommonCovarianceTarget
• ConstantCorrelationTarget
• GeodesicShrinkageTarget
• shrinkage_target

References

• [65] J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology 4, Article 32 (2005).

shrinkage_target(
    _::IdentityTarget,
    sigma::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Any

Builds the target matrix of a covariance shrinkage from the matrix being shrunk.

Each rule states its target under # Mathematical definition. The function returns a fixed matrix as it is, after it checks that the matrix has the size of sigma.

Arguments

• tgt: The target.
  • ::IdentityTarget: the identity matrix, as a Diagonal.
  • ::ScaledIdentityTarget: the average variance of sigma times the identity matrix, as a Diagonal.
  • ::CommonCovarianceTarget: the average variance on the diagonal and the average covariance elsewhere.
  • ::ConstantCorrelationTarget: the variances of sigma and the average correlation of its pairs.
  • ::DiagonalTarget: the diagonal of sigma, as a Diagonal.
  • ::MatNum: the matrix itself.
• sigma: Covariance-like or correlation-like matrix assets × assets.

Validation

• A matrix tgt has the size of sigma. The function throws a DimensionMismatch otherwise.

Returns

• target::MatNum: The target matrix.

Related

• AbstractCovarianceShrinkageTarget
• GeodesicShrinkageTarget

struct IdentityTarget <: AbstractCovarianceShrinkageTarget

Targets the identity matrix, which ignores the scale of the matrix being shrunk.

Mathematical definition

\[\begin{align} \mathbf{T} &= \mathbf{I}\,. \end{align}\]

Where:

• $\mathbf{T}$: Covariance shrinkage target, a positive definite matrix.
• $\mathbf{I}$: Identity matrix.

Examples

julia> IdentityTarget()IdentityTarget()

Related

• AbstractCovarianceShrinkageTarget
• ScaledIdentityTarget: the same shape at the scale of the average variance.
• GeodesicShrinkageCovariance

References

• [65] J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology 4, Article 32 (2005). Table 2, target A.

struct ScaledIdentityTarget <: AbstractCovarianceShrinkageTarget

Targets the identity matrix scaled by the average variance, the target with condition number one.

Mathematical definition

\[\begin{align} \mathbf{T} &= \bar{v} \, \mathbf{I}\,, \\ \bar{v} &= \frac{1}{N} \sum_{i = 1}^{N} \hat{\mathbf{\Sigma}}_{ii}\,. \end{align}\]

Where:

• $\mathbf{T}$: Covariance shrinkage target, a positive definite matrix.
• $\bar{v}$: Average variance, the mean of the diagonal of $\hat{\mathbf{\Sigma}}$.
• $\mathbf{I}$: Identity matrix.
• $\hat{\mathbf{\Sigma}}_{ii}$: $i$-th diagonal entry of $\hat{\mathbf{\Sigma}}$.
• $N$: Number of assets.

The target has the trace of $\hat{\mathbf{\Sigma}}$. It commutes with every matrix, so a geodesic towards it keeps the eigenvectors of $\hat{\mathbf{\Sigma}}$ and changes only the eigenvalues.

Examples

julia> ScaledIdentityTarget()ScaledIdentityTarget()

Related

• AbstractCovarianceShrinkageTarget
• IdentityTarget
• GeodesicShrinkageCovariance

References

• [65] J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology 4, Article 32 (2005). Table 2, target B.

struct CommonCovarianceTarget <: AbstractCovarianceShrinkageTarget

Targets a matrix with the average variance on the diagonal and the average covariance everywhere else.

Mathematical definition

\[\begin{align} T_{ij} &= \begin{cases} \bar{v} & i = j\,, \\ \bar{c} & i \neq j\,, \end{cases} \\ \bar{v} &= \frac{1}{N} \sum_{i = 1}^{N} \hat{\mathbf{\Sigma}}_{ii}\,, \\ \bar{c} &= \frac{1}{N (N - 1)} \sum_{i \neq j} \hat{\mathbf{\Sigma}}_{ij}\,. \end{align}\]

Where:

• $T_{ij}$: Entry of $\mathbf{T}$ in row $i$ and column $j$.
• $\bar{v}$: Average variance, the mean of the diagonal of $\hat{\mathbf{\Sigma}}$.
• $\bar{c}$: Average covariance of the distinct pairs.
• $\hat{\mathbf{\Sigma}}_{ii}$: $i$-th diagonal entry of $\hat{\mathbf{\Sigma}}$.
• $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
• $N$: Number of assets.

The target has the eigenvalue $\bar{v} - \bar{c}$ with multiplicity $N - 1$ and the eigenvalue $\bar{v} + (N - 1) \bar{c}$ once. So it is positive definite when $-\bar{v} / (N - 1) < \bar{c} < \bar{v}$. A positive definite $\hat{\mathbf{\Sigma}}$ meets both bounds. A singular $\hat{\mathbf{\Sigma}}$ fails the lower bound when the portfolio that holds every asset equally has zero variance. It fails the upper bound when every variance is equal and every pair has correlation one.

Examples

julia> CommonCovarianceTarget()CommonCovarianceTarget()

Related

• AbstractCovarianceShrinkageTarget
• ConstantCorrelationTarget: keeps the variances and averages the correlations.
• GeodesicShrinkageCovariance

References

• [65] J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology 4, Article 32 (2005). Table 2, target C.

struct ConstantCorrelationTarget <: AbstractCovarianceShrinkageTarget

Targets a matrix that keeps the variances and gives every pair of assets the average correlation.

Mathematical definition

\[\begin{align} T_{ij} &= \begin{cases} \hat{\mathbf{\Sigma}}_{ii} & i = j\,, \\ \bar{r} \sqrt{\hat{\mathbf{\Sigma}}_{ii} \, \hat{\mathbf{\Sigma}}_{jj}} & i \neq j\,, \end{cases} \\ \bar{r} &= \frac{1}{N (N - 1)} \sum_{i \neq j} \frac{\hat{\mathbf{\Sigma}}_{ij}}{\sqrt{\hat{\mathbf{\Sigma}}_{ii} \, \hat{\mathbf{\Sigma}}_{jj}}}\,. \end{align}\]

Where:

• $T_{ij}$: Entry of $\mathbf{T}$ in row $i$ and column $j$.
• $\bar{r}$: Average correlation of the distinct pairs.
• $\hat{\mathbf{\Sigma}}_{ii}$: $i$-th diagonal entry of $\hat{\mathbf{\Sigma}}$.
• $\hat{\mathbf{\Sigma}}_{ij}$: Estimated covariance between assets $i$ and $j$.
• $N$: Number of assets.

The correlation matrix of the target has the eigenvalue $1 - \bar{r}$ with multiplicity $N - 1$ and the eigenvalue $1 + (N - 1) \bar{r}$ once. So the target is positive definite when every variance is positive and $-1 / (N - 1) < \bar{r} < 1$. A positive definite $\hat{\mathbf{\Sigma}}$ meets both conditions.

Examples

julia> ConstantCorrelationTarget()ConstantCorrelationTarget()

Related

• AbstractCovarianceShrinkageTarget
• CommonCovarianceTarget
• GeodesicShrinkageCovariance

References

• [65] J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology 4, Article 32 (2005). Table 2, target F.

struct GeodesicShrinkageCovariance{__T_ce, __T_pdm, __T_tgt, __T_alpha} <: AbstractCovarianceEstimator

Shrinks a covariance matrix towards a target along the geodesic of the positive definite matrices.

Linear shrinkage averages the matrix and its target entry by entry. This estimator follows the shortest path between them under the affine-invariant metric instead. So the estimate is positive definite at every intensity when both ends are positive definite.

A zero eigenvalue of the start stays zero at every intensity below one, so the shrinkage does not repair a singular matrix. The pdm field repairs the matrix that ce computes, before the shrinkage runs. cor returns the correlation matrix of the shrunk covariance.

Mathematical definition

\[\begin{align} \hat{\mathbf{\Sigma}}_{\alpha} &= \hat{\mathbf{\Sigma}}^{1/2} \left(\hat{\mathbf{\Sigma}}^{-1/2} \, \mathbf{T} \, \hat{\mathbf{\Sigma}}^{-1/2}\right)^{\alpha} \hat{\mathbf{\Sigma}}^{1/2} \\ &= \mathbf{T}^{1/2} \left(\mathbf{T}^{-1/2} \, \hat{\mathbf{\Sigma}} \, \mathbf{T}^{-1/2}\right)^{1 - \alpha} \mathbf{T}^{1/2}\,. \end{align}\]

The second form needs only $\mathbf{T}$ to be positive definite. It is the continuous extension of the first to a positive semidefinite $\hat{\mathbf{\Sigma}}$. Two targets give closed forms:

\[\begin{align} \mathbf{T} = \bar{v} \mathbf{I} &\implies \hat{\mathbf{\Sigma}}_{\alpha} = \mathbf{V} \operatorname{Diag}\left(\lambda_i^{1 - \alpha} \bar{v}^{\alpha}\right) \mathbf{V}^\intercal\,, \\ \mathbf{T} = \mathbf{D} &\implies \hat{\mathbf{\Sigma}}_{\alpha} = \mathbf{D}^{1/2} \, \mathbf{R}^{1 - \alpha} \, \mathbf{D}^{1/2}\,. \end{align}\]

Where:

• $\hat{\mathbf{\Sigma}}_{\alpha}$: Shrunk covariance matrix.
• $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
• $\mathbf{T}$: Covariance shrinkage target, a positive definite matrix.
• $\alpha \in [0, 1]$: Shrinkage intensity, the fraction of the geodesic distance from $\hat{\mathbf{\Sigma}}$ to $\mathbf{T}$ that the estimate travels.
• $\bar{v}$: Average variance, the mean of the diagonal of $\hat{\mathbf{\Sigma}}$.
• $\mathbf{I}$: Identity matrix.
• $\mathbf{V}$, $\lambda_i$: Eigenvectors and eigenvalues of $\hat{\mathbf{\Sigma}}$.
• $\mathbf{D} = \operatorname{Diag}(\hat{\mathbf{\Sigma}})$: Diagonal matrix of the variances.
• $\mathbf{R} = \mathbf{D}^{-1/2} \hat{\mathbf{\Sigma}} \mathbf{D}^{-1/2}$: Correlation matrix of $\hat{\mathbf{\Sigma}}$.

The geodesic distance from $\hat{\mathbf{\Sigma}}$ to $\hat{\mathbf{\Sigma}}_{\alpha}$ is $\alpha$ times the distance from $\hat{\mathbf{\Sigma}}$ to $\mathbf{T}$.

Towards $\bar{v} \mathbf{I}$, each eigenvalue moves to the weighted geometric mean $\lambda_i^{1 - \alpha} \bar{v}^{\alpha}$. Linear shrinkage moves it to the weighted arithmetic mean $(1 - \alpha) \lambda_i + \alpha \bar{v}$, which is never smaller. So at an intermediate intensity the trace of $\hat{\mathbf{\Sigma}}_{\alpha}$ is below the trace of $\hat{\mathbf{\Sigma}}$, unless every eigenvalue equals $\bar{v}$. The condition number of $\hat{\mathbf{\Sigma}}_{\alpha}$ is the condition number of $\hat{\mathbf{\Sigma}}$ to the power $1 - \alpha$.

Fields

• ce: Covariance estimator. It computes the matrix that the estimator shrinks.

• pdm: Optional positive definite matrix estimator. It repairs the matrix ce computes before the shrinkage runs.

• tgt: Shrinkage target. A rule of GeodesicShrinkageTarget, or a fixed positive definite matrix.

• alpha: Shrinkage intensity, the fraction of the geodesic from the matrix to the target that the estimate travels. 0 returns the matrix unchanged and 1 returns the target.

Constructors

GeodesicShrinkageCovariance(;    ce::StatsBase.CovarianceEstimator = Covariance(),    pdm::Option{<:AbstractPosdefEstimator} = Posdef(),    tgt::GeodesicShrinkageTarget = ScaledIdentityTarget(),    alpha::Number = 0.1) -> GeodesicShrinkageCovariance

Keywords correspond to the struct's fields.

Validation

• 0 <= alpha <= 1.
• A matrix tgt is square, finite, symmetric and positive definite, as assert_shrinkage_target states.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• ce: Recursively updated via factory.

View parameters

GeodesicShrinkageCovariance defines its own port_opt_view method rather than deriving one from field tags.

• ce recurses through port_opt_view with every argument.
• The view slices a matrix tgt on both axes, because it is an asset-by-asset matrix. It keeps a target rule unchanged, because shrinkage_target builds the target of a rule from the viewed matrix.

Examples

julia> GeodesicShrinkageCovariance()GeodesicShrinkageCovariance     ce ┼ Covariance        │    me ┼ SimpleExpectedReturns        │       │   w ┴ nothing        │    ce ┼ GeneralCovariance        │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)        │       │    w ┴ nothing        │   alg ┼ FullMoment()        │     w ┴ nothing    pdm ┼ Posdef        │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │   kwargs ┴ @NamedTuple{}: NamedTuple()    tgt ┼ ScaledIdentityTarget()  alpha ┴ Float64: 0.1julia> X = [0.01 0.02 0.0; -0.02 0.01 0.01; 0.03 -0.01 0.02; 0.0 0.01 -0.01];julia> cond(cov(GeodesicShrinkageCovariance(; alpha = 0.5), X)) ≈ sqrt(cond(cov(Covariance(), X)))true

Related

• AbstractCovarianceShrinkageTarget
• GeodesicShrinkageTarget
• Covariance
• Posdef
• factory
• port_opt_view

References

• [66] R. Bhatia. Positive Definite Matrices (Princeton University Press, 2007). Chapter 6.
• [67] A. Musolas, S. T. Smith and Y. Marzouk. Geodesically parameterized covariance estimation. SIAM Journal on Matrix Analysis and Applications 42, 528–556 (2021). Section 2.

Statistics.cov(ce::GeodesicShrinkageCovariance, X::MatNum; dims::Int = 1, kwargs...)
Statistics.cor(ce::GeodesicShrinkageCovariance, X::MatNum; dims::Int = 1, kwargs...)

Computes the covariance matrix with ce.ce, and shrinks it towards ce.tgt along the geodesic of the positive definite matrices.

GeodesicShrinkageCovariance states the mathematics. The estimator forwards the sample and its keywords to ce.ce unchanged. So an active mask reaches an inner estimator that reads one, and the inner estimator keeps or refuses a gap.

Algorithm

1. Orient X to observations × assets with dims_oriented.
2. Compute sigma with Statistics.cov(library_covariance_estimator(ce.ce), X; kwargs...).
3. Copy sigma into a new Matrix, because step 4 writes in place and the inner estimator can return an immutable matrix.
4. Repair and shrink the finite block of sigma with geodesic_shrinkage!.
5. cov returns sigma. cor rescales it to a correlation matrix with frame_cov2cor! and returns it, so the correlation is that of the shrunk covariance.

Arguments

• ce: The geodesic shrinkage covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• dims: Dimension along which to perform the computation.
• kwargs...: Additional keyword arguments passed to ce.ce.

Validation

• dims in (1, 2).
• Every check of geodesic_shrinkage! applies.

Returns

• sigma::MatNum: The shrunk covariance matrix, or rho, its correlation matrix.

Related

• GeodesicShrinkageCovariance
• geodesic_shrinkage!

Statistics.cov(ce::GeodesicShrinkageCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNum
Statistics.cor(ce::GeodesicShrinkageCovariance, X::MatNum,
               pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...) -> MatNum

Forwards the Asset Panel to ce.ce, then repairs and shrinks the finite block of the frame it gets back.

This method overrides the reduce-and-expand root. The inner estimator does the reduction, because its own method decides whether to read the gaps of the sample. This method does the shrinkage. An asset outside the Coverage Universe keeps its NaN row and column.

Algorithm

1. Orient X to observations × assets with dims_oriented.
2. Compute sigma with the inner estimator, forwarding pnl as its third positional argument.
3. Copy sigma into a new Matrix, because step 4 writes in place and the inner estimator can return an immutable matrix.
4. Repair and shrink the finite block of sigma with geodesic_shrinkage!.
5. cov returns sigma. cor rescales it to a correlation matrix with frame_cov2cor! and returns it.

Arguments

• ce: The geodesic shrinkage covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• pnl: Optional AssetPanel, whose active mask the Coverage Universe of the fit is derived from. nothing makes the rule finiteness alone.
• dims: Dimension along which to perform the computation.
• kwargs...: Additional keyword arguments passed to ce.ce.

Validation

• dims in (1, 2).
• Every check of geodesic_shrinkage! applies.

Returns

• sigma::MatNum: The shrunk covariance matrix, or rho, its correlation matrix, on the full asset universe.

Related

• GeodesicShrinkageCovariance
• geodesic_shrinkage!
• coverage_reduction

partial_fit!(ce::GeodesicShrinkageCovariance, X::MatNum; dims::Int = 1, kwargs...)
partial_fit!(ce::GeodesicShrinkageCovariance, x::VecNum; kwargs...)

Folds observations into a GeodesicShrinkageCovariance by forwarding them to ce.ce.

The shrinkage reads the matrix that ce.ce returns and no observation. So the estimator keeps no state of its own, and it folds when ce.ce folds. The read-out applies the shrinkage to the folded matrix.

Algorithm

1. Build a copy of ce whose ce.ce is the estimator that partial_fit! returns, with Accessors.@reset.

Arguments

• ce: The geodesic shrinkage covariance estimator.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• x: One observation, whose entries are the assets.
• dims: Dimension along which to perform the computation.
• kwargs...: Additional keyword arguments, forwarded to ce.ce.

Returns

• ce::GeodesicShrinkageCovariance: A new estimator, equal to ce except that its ce.ce holds the state after the last observation.

Related

• GeodesicShrinkageCovariance
• partial_fit!

Statistics.cov(ce::GeodesicShrinkageCovariance; kwargs...)
Statistics.cor(ce::GeodesicShrinkageCovariance; kwargs...)

Reads the shrunk covariance, or its correlation, of a folded GeodesicShrinkageCovariance.

ce.ce returns its folded matrix, and the shrinkage runs on it as it runs on a batch fit. So this method returns what a batch fit over the same observations returns.

Algorithm

1. Read the folded matrix sigma of ce.ce with the one-argument Statistics.cov.
2. Copy sigma into a new Matrix, because step 3 writes in place and the inner estimator can return an immutable matrix.
3. Repair and shrink the finite block of sigma with geodesic_shrinkage!.
4. cov returns sigma. cor rescales it to a correlation matrix with frame_cov2cor! and returns it.

Arguments

• ce: The geodesic shrinkage covariance estimator.
• kwargs...: Additional keyword arguments, forwarded to ce.ce.

Validation

• ce.ce carries a partial-fit state. The read-out of ce.ce throws an ArgumentError otherwise.

Returns

• sigma::MatNum: The shrunk covariance matrix, or rho, its correlation matrix.

Related

• GeodesicShrinkageCovariance
• partial_fit!

port_opt_view(ce::GeodesicShrinkageCovariance, i, args...) -> GeodesicShrinkageCovariance

Restricts a GeodesicShrinkageCovariance to the assets i.

A fixed target is an asset-by-asset matrix, so this method slices it on both axes, as it slices a covariance matrix. shrinkage_target builds the target of a rule from the matrix it shrinks, so a rule needs no slice.

Algorithm

1. View ce.ce with port_opt_view and every argument.
2. Slice a matrix ce.tgt with nothing_scalar_array_view, and keep a target rule as it is.
3. Rebuild the estimator with the viewed fields, ce.pdm and ce.alpha.

Arguments

• ce: The geodesic shrinkage covariance estimator.
• i: Indices of the selected assets.
• args...: Additional positional arguments, forwarded to the view of ce.ce.

Returns

• ce::GeodesicShrinkageCovariance: The estimator restricted to the assets i.

Related

• GeodesicShrinkageCovariance
• port_opt_view
• nothing_scalar_array_view