Exp weighted covariance: private API

Types

PortfolioOptimisers.ExpWeightedCovarianceState — Type
struct ExpWeightedCovarianceState{__T_covariance, __T_weight, __T_location, __T_overlap, __T_obs_count, __T_active, __T_rows} <: AbstractPartialFitState

Holds the running state of an incremental exponentially weighted covariance fit.

The struct is immutable, and each of its fields is an array that the recursion writes in place, or nothing. partial_fit! stores the state in the cache field of ExpWeightedCovariance, and cov(ce, state) reads the estimate out of it.

Fields

  • covariance: Running exponentially weighted covariance matrix, seeded at zero.
  • weight: Running total of the weights that each pair of assets holds in covariance. It takes the step of covariance on a unit outer product, so a pair gains weight only at an observation where both of its assets are valid.
  • location: Exponentially smoothed location (mean) vector.
  • overlap: Overlap of the locations of each pair of assets, the sum over their common valid returns of the products of the two unnormalised weights, which the factor of a pair reads. It is nothing under PreCentred, which takes no location.
  • obs_count: Per-asset count of observations processed.
  • active: Boolean mask of the assets that are active now.
  • rows: Number of the observations folded so far, the rows that hold a NaN included, in a Ref that the fold writes in place. Unlike obs_count, an asset that leaves the active mask does not reset it, so it is the size(X, 1) of a batch fit over the same rows, which observation_count reads.

Related

source

Functions

PortfolioOptimisers.process_observation! — Method
process_observation!(
    cache::ExpWeightedCovarianceState,
    ce::ExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> ExpWeightedCovarianceState

Folds one observation into the state of an exponentially weighted covariance fit.

An asset is valid when its return is finite and the active mask admits it. An active asset with a non-finite return is on a holiday. Its variance and each of its covariances keep their values, and so do their weights. An asset that leaves the active mask returns to the zero state, so its weights start again if it lists again.

Algorithm

  1. Add one to cache.rows, because every row counts. Mark as valid each asset whose return in X is finite and that active_mask admits.
  2. When active_mask is not nothing, take newly_inactive as the assets that the mask excludes and that cache.active admits. Set their rows and columns of cache.covariance, cache.weight and cache.overlap, and their counts in cache.obs_count, to zero, and their entries of cache.location to NaN.
  3. Copy the mask into cache.active, or set every entry to true when the mask is nothing.
  4. Return cache when no asset is valid.
  5. Take the factor of each pair from the overlap and the counts before the observation with centring_pair_factor. Take the deviations and the mask dvalid of the assets that give a deviation with centring_location!, which also moves the location of each valid asset.
  6. Take e as the deviations of the assets of dvalid. On their block, move cache.covariance to ce.decay * block + (1 - ce.decay) * (e * e') ./ F, with F the block of the factors, and cache.weight to ce.decay * block + (1 - ce.decay). Every other entry holds.
  7. Fold the valid assets into the overlap with centring_overlap!, and add one to the count of each valid asset in cache.obs_count.

Arguments

  • cache::ExpWeightedCovarianceState: State of the fit. The method writes it in place.
  • ce::ExpWeightedCovariance: Exponentially weighted covariance estimator.
  • X::VecNum: Returns of one observation, one entry per asset.
  • active_mask::Option{<:AbstractVector{<:Bool}}: Mask of the active assets at this observation. With nothing every asset is active, so a non-finite return is a holiday.

Returns

  • cache::ExpWeightedCovarianceState: The state after the observation. It is the same object as the argument.

Related

source
PortfolioOptimisers.exp_weighted_pass! — Function
exp_weighted_pass!(
    f,
    est::ExpWeightedCovariance,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    dims::Int64,
    active_mask::Union{Nothing, AbstractMatrix{<:Bool}}
) -> Union{ExpWeightedCovarianceState{_A, _B, _C, Nothing, _D, _E, Base.RefValue{Int64}} where {_A, _B, _C, _D, _E}, ExpWeightedCovarianceState{_A, _B, _C, __T_overlap, _D, _E, Base.RefValue{Int64}} where {_A, _B, _C, __T_overlap<:(Matrix{T} where T<:Union{Number, AbstractJuMPScalar}), _D, _E}}
exp_weighted_pass!(
    f,
    est::ExpWeightedCovariance,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    dims::Int64,
    active_mask::Union{Nothing, AbstractMatrix{<:Bool}},
    state::Union{Nothing, ExpWeightedCovarianceState}
) -> ExpWeightedCovarianceState

Covariance method of exp_weighted_pass!. Runs one forward pass of the covariance recursion over the observations of X, and calls f after each observation. A verb that needs only the last state passes Returns(nothing) as f.

Algorithm

  1. Check dims, and check that active_mask has the size of X when it is not nothing.
  2. Take N as the number of assets.
  3. When state is nothing, make a zero state of N assets in the type T of X. Its covariance, weight and obs_count are zero, its active entries are true, its location is NaN, its rows is a Ref of zero, and its overlap is centring_overlap of est.centring. When state is given, check that it holds N assets and take it as it is.
  4. For each observation i, fold the observation and its row of active_mask into the state with process_observation!, then call f(i, cache).
  5. Return the state.

Arguments

  • f: Function called as f(i, cache) after the observation i. Its return value is discarded.
  • est::ExpWeightedCovariance: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask::Option{<:AbstractMatrix{<:Bool}}: Mask of the active assets, with the size of X. With nothing every asset is active.
  • state::Option{<:ExpWeightedCovarianceState}: State to continue. With nothing the pass starts from the zero state.

Validation

  • dims in (1, 2).
  • If active_mask is not nothing, size(X) == size(active_mask). A DimensionMismatch is thrown otherwise.
  • If state is not nothing, it holds as many assets as X. A DimensionMismatch is thrown otherwise.

Returns

  • cache::ExpWeightedCovarianceState: The state after the last observation. A given state is the same object, written in place.

Related

source
PortfolioOptimisers.exp_weighted_moment — Method
exp_weighted_moment(
    cache::ExpWeightedCovarianceState,
    est::ExpWeightedCovariance;
    repair
) -> Any

Covariance method of exp_weighted_moment. Reads the exponentially weighted covariance out of a state.

The method reads the state and does not write it, so a forward pass can call it after every observation. The struct docstring of ExpWeightedCovariance states the estimate that it computes.

Algorithm

  1. Fill sigma with NaN, and take ready as the assets that cache.active admits and that centring_ready finds ready at est.min_obs.
  2. On the block of the ready assets, divide each entry of cache.covariance by its entry of cache.weight with pair_weighted_block, and replace the block with the mean of the block and its transpose, which removes the round-off asymmetry.
  3. Where repair is true, restore a positive semidefinite block with restore_psd!. The repair keeps the diagonal.
  4. Write the block into sigma.

Arguments

  • cache::ExpWeightedCovarianceState: State to read.
  • est::ExpWeightedCovariance: Exponentially weighted covariance estimator.
  • repair::Bool: Whether to restore a positive semidefinite matrix. A caller that reads the diagonal alone passes false.

Returns

  • sigma::MatNum: Covariance matrix. The row and the column of an asset that is inactive, or that has fewer than est.min_obs valid observations, are NaN.

Related

source
PortfolioOptimisers.pair_weighted_block — Function
pair_weighted_block(
    S::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    W::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Divides each entry of a block of an exponentially weighted state by the weight that its pair holds.

The weight state takes the step of S on a unit outer product, so S[i, j] / W[i, j] is the exponentially weighted mean of the product over the common observations of the pair. A pair that holds no weight has no common observation, and its entry is zero.

Algorithm

  1. Take the block S[idx, idx] and the block W[idx, idx].
  2. Divide each entry of the first by the entry of the second where that entry is above zero, and take zero elsewhere.

Arguments

  • S::MatNum: Exponentially weighted state, seeded at zero.
  • W::MatNum: Weight that each pair holds in S.
  • idx::AbstractVector{<:Integer}: Index of the assets of the block.

Returns

  • block::MatNum: The block of the weighted means.

Related

source
PortfolioOptimisers.restore_psd! — Function
restore_psd!(
    sigma::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}

Restores a positive semidefinite matrix from a block whose pairs are each divided by their own weight, and keeps its diagonal.

Where every asset has the same valid observations, the division by the weight is a congruence of a sum of outer products, and the block is positive semidefinite. After a holiday it need not be, whatever the weights are: a weight that saturates at one does not show which observations its pair read. So the method tests the block itself. A correlation with a Cholesky factor is positive definite to working precision, and the block is returned unchanged with no eigen decomposition. Otherwise, where the correlation has an eigenvalue below -n eps times the largest, the method clips the negative eigenvalues to zero, which is the nearest positive semidefinite matrix in the Frobenius norm, and restores the unit diagonal and the variances. A smaller negative eigenvalue is round-off, and the block is returned unchanged.

The Cholesky test runs first because it costs a fraction of the decomposition. It can accept a correlation whose smallest eigenvalue is slightly below -n eps times the largest, because the backward error of the factor is about n^2 eps. Such an eigenvalue is round-off, and the block is kept as it is.

Algorithm

  1. Take s as the root of the diagonal of sigma, floored at zero, and pos as the assets whose s is above zero. Return sigma when pos is empty.
  2. Take the correlation R = sigma[pos, pos] ./ (s * s'). Return sigma when R has a Cholesky factor.
  3. Take the eigen decomposition of R. Return sigma when the smallest eigenvalue is at least -length(pos) * eps times the largest magnitude.
  4. Rebuild R with its eigenvalues floored at zero, divide it by the root of its diagonal on both sides, clamp it to [-1, 1], symmetrise it and set its diagonal to one.
  5. Write R .* (s * s') into sigma[pos, pos], and return sigma.

Arguments

  • sigma::MatNum: Block to repair, a covariance or a correlation. It is written in place.

Returns

  • sigma::MatNum: The repaired block. It is the same object as the argument.

Related

source
PortfolioOptimisers.gap_fill_value — Method
gap_fill_value(ce::ExpWeightedCovariance) -> Float64

Answer NaN, so a gapped sample reaches the recursion with its gaps intact.

The recursion moves only the entries of the assets that are valid at an observation. A finite fill value would make a gapped asset valid, and the recursion would then move entries that a gap keeps. So the consumer passes the sample as it is, together with the active mask that explains each gap.

Arguments

  • ce: Covariance estimator.

Returns

  • fv::Float64: NaN.

Related

source
PortfolioOptimisers.variance_series — Method
variance_series(
    ce::ExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> Matrix{<:Number}

Compute the point-in-time exponentially weighted variance series.

Row t holds the diagonal of what cov returns for the first t observations of X, so no row reads an observation after its own. The recursion folds one observation at a time, so this method replaces the expanding-window fallback with one forward pass that reads the state after each observation.

The fallback cannot serve this estimator. It slices X once per row and passes every keyword as it is, so a mask of the whole window meets a window of t observations, and the size check refuses the call.

Algorithm

  1. Check dims, and allocate val with one row per observation and one column per asset.
  2. Run one forward pass over X with exp_weighted_pass!. After the observation i, write the diagonal of exp_weighted_moment of the state into row i of val, with repair = false, because the repair keeps the diagonal.
  3. Return val, transposed when dims == 2.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: Optional boolean matrix with the same size as X.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If active_mask is not nothing, size(X) == size(active_mask). A DimensionMismatch is thrown otherwise.

Returns

  • val::Matrix{<:Number}: Variance series, shaped as (T, N) if dims == 1 or (N, T) if dims == 2. An asset is NaN at row t when it has fewer than ce.min_obs valid observations up to t, or when active_mask excludes it at t.

Related

source
PortfolioOptimisers.variance_series — Method
variance_series(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> Matrix{<:Number}

Compute the point-in-time exponentially weighted variance series from a window of an Asset Panel.

This is the diagonal of the covariance of the same call, read after each observation, and it reads the active mask of the panel in the same way. A Descriptor that holds this estimator thus keeps a number for a young asset. The reduce-and-expand root would reduce every window and answer NaN for it.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If pnl is not nothing, its active mask has the size of X after the orientation that dims sets. A DimensionMismatch is thrown otherwise.

Returns

  • val::Matrix{<:Number}: Variance series on the full asset universe, shaped as (T, N) if dims == 1 or (N, T) if dims == 2.

Related

source
Base.copy — Method
copy(
    x::ExpWeightedCovarianceState
) -> ExpWeightedCovarianceState{_A, _B, _C, _D, _E, _F, __T_rows} where {_A, _B, _C, _D, _E, _F, __T_rows<:Base.RefValue}

Copies an ExpWeightedCovarianceState, so the copy shares no array with the original.

It is the copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. It copies each array, and keeps an overlap of nothing. It puts the row count in a new Ref, so a fold of the copy does not count in the original.

Arguments

  • x: The state to copy.

Returns

  • state::ExpWeightedCovarianceState: A new state equal to x, with new arrays.

Related

source
PortfolioOptimisers.observation_count — Method
observation_count(state::ExpWeightedCovarianceState) -> Any

ExpWeightedCovarianceState method of observation_count.

The state counts the observations of each asset in obs_count, and a reset sets such a count to zero. So the method reads rows, the number of folded rows, which is the size(X, 1) of a batch fit over the same rows.

Arguments

  • state: The state that holds the count.

Returns

  • n::Integer: The number of observations folded so far.

Related

source
PortfolioOptimisers.exp_weighted_covariance_count — Function
exp_weighted_covariance_count(
    decay::Number,
    n::Integer,
    centring::AbstractCentring
) -> Any

Return the effective number of observations of an exponentially weighted covariance over n rows.

It is the count of variance_count(decay::Number, X::MatNum, centring::AbstractCentring, amsk::Option{<:AbstractMatrix{<:Bool}}) for one column of n finite rows. The covariance reads every row with the weight of its age, so the count reads all n rows, the rows with a NaN too, as size(X, 1) counts them. The count tends to $(1 + \lambda) / (1 - \lambda)$ as n grows. Under equal weights it is n.

Mathematical definition

\[\begin{align} T_{e} &= \kappa\left(K\right) + \ell\,, \\ K &= \max\left(n - \ell, 0\right)\,. \end{align}\]

Where:

  • $T_{e}$: Effective sample size, which is Kish's when the observation weights are stated.
  • $\kappa(K)$: Kish count of $K$ exponential weights, from exp_weighted_kish_count.
  • $n$: Number of rows.
  • $\ell$: The lag of the centring, from centring_lag.

Arguments

  • decay: Decay of the recursion.
  • n: Number of rows.
  • centring: The centring of the estimator.

Returns

  • T_e::Number: The effective number of observations.

Related

source