Exp weighted covariance: private API
Types
PortfolioOptimisers.ExpWeightedCovarianceState — Type
struct ExpWeightedCovarianceState{__T_covariance, __T_weight, __T_location, __T_overlap, __T_obs_count, __T_active, __T_rows} <: AbstractPartialFitStateHolds the running state of an incremental exponentially weighted covariance fit.
The struct is immutable, and each of its fields is an array that the recursion writes in place, or nothing. partial_fit! stores the state in the cache field of ExpWeightedCovariance, and cov(ce, state) reads the estimate out of it.
Fields
covariance: Running exponentially weighted covariance matrix, seeded at zero.
weight: Running total of the weights that each pair of assets holds incovariance. It takes the step ofcovarianceon a unit outer product, so a pair gains weight only at an observation where both of its assets are valid.
location: Exponentially smoothed location (mean) vector.
overlap: Overlap of the locations of each pair of assets, the sum over their common valid returns of the products of the two unnormalised weights, which the factor of a pair reads. It isnothingunderPreCentred, which takes no location.
obs_count: Per-asset count of observations processed.
active: Boolean mask of the assets that are active now.
rows: Number of the observations folded so far, the rows that hold aNaNincluded, in aRefthat the fold writes in place. Unlikeobs_count, an asset that leaves the active mask does not reset it, so it is thesize(X, 1)of a batch fit over the same rows, whichobservation_countreads.
Related
Functions
PortfolioOptimisers.process_observation! — Method
process_observation!(
cache::ExpWeightedCovarianceState,
ce::ExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
active_mask::Union{Nothing, AbstractVector{<:Bool}}
) -> ExpWeightedCovarianceState
Folds one observation into the state of an exponentially weighted covariance fit.
An asset is valid when its return is finite and the active mask admits it. An active asset with a non-finite return is on a holiday. Its variance and each of its covariances keep their values, and so do their weights. An asset that leaves the active mask returns to the zero state, so its weights start again if it lists again.
Algorithm
- Add one to
cache.rows, because every row counts. Mark asvalideach asset whose return inXis finite and thatactive_maskadmits. - When
active_maskis notnothing, takenewly_inactiveas the assets that the mask excludes and thatcache.activeadmits. Set their rows and columns ofcache.covariance,cache.weightandcache.overlap, and their counts incache.obs_count, to zero, and their entries ofcache.locationtoNaN. - Copy the mask into
cache.active, or set every entry totruewhen the mask isnothing. - Return
cachewhen no asset is valid. - Take the factor of each pair from the overlap and the counts before the observation with
centring_pair_factor. Take the deviations and the maskdvalidof the assets that give a deviation withcentring_location!, which also moves the location of each valid asset. - Take
eas the deviations of the assets ofdvalid. On their block, movecache.covariancetoce.decay * block + (1 - ce.decay) * (e * e') ./ F, withFthe block of the factors, andcache.weighttoce.decay * block + (1 - ce.decay). Every other entry holds. - Fold the valid assets into the overlap with
centring_overlap!, and add one to the count of each valid asset incache.obs_count.
Arguments
cache::ExpWeightedCovarianceState: State of the fit. The method writes it in place.ce::ExpWeightedCovariance: Exponentially weighted covariance estimator.X::VecNum: Returns of one observation, one entry per asset.active_mask::Option{<:AbstractVector{<:Bool}}: Mask of the active assets at this observation. Withnothingevery asset is active, so a non-finite return is a holiday.
Returns
cache::ExpWeightedCovarianceState: The state after the observation. It is the same object as the argument.
Related
PortfolioOptimisers.exp_weighted_pass! — Function
exp_weighted_pass!(
f,
est::ExpWeightedCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
dims::Int64,
active_mask::Union{Nothing, AbstractMatrix{<:Bool}}
) -> Union{ExpWeightedCovarianceState{_A, _B, _C, Nothing, _D, _E, Base.RefValue{Int64}} where {_A, _B, _C, _D, _E}, ExpWeightedCovarianceState{_A, _B, _C, __T_overlap, _D, _E, Base.RefValue{Int64}} where {_A, _B, _C, __T_overlap<:(Matrix{T} where T<:Union{Number, AbstractJuMPScalar}), _D, _E}}
exp_weighted_pass!(
f,
est::ExpWeightedCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
dims::Int64,
active_mask::Union{Nothing, AbstractMatrix{<:Bool}},
state::Union{Nothing, ExpWeightedCovarianceState}
) -> ExpWeightedCovarianceState
Covariance method of exp_weighted_pass!. Runs one forward pass of the covariance recursion over the observations of X, and calls f after each observation. A verb that needs only the last state passes Returns(nothing) as f.
Algorithm
- Check
dims, and check thatactive_maskhas the size ofXwhen it is notnothing. - Take
Nas the number of assets. - When
stateisnothing, make a zero state ofNassets in the typeTofX. Itscovariance,weightandobs_countare zero, itsactiveentries aretrue, itslocationisNaN, itsrowsis aRefof zero, and itsoverlapiscentring_overlapofest.centring. Whenstateis given, check that it holdsNassets and take it as it is. - For each observation
i, fold the observation and its row ofactive_maskinto the state withprocess_observation!, then callf(i, cache). - Return the state.
Arguments
f: Function called asf(i, cache)after the observationi. Its return value is discarded.est::ExpWeightedCovariance: Exponentially weighted covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask::Option{<:AbstractMatrix{<:Bool}}: Mask of the active assets, with the size ofX. Withnothingevery asset is active.state::Option{<:ExpWeightedCovarianceState}: State to continue. Withnothingthe pass starts from the zero state.
Validation
dims in (1, 2).- If
active_maskis notnothing,size(X) == size(active_mask). ADimensionMismatchis thrown otherwise. - If
stateis notnothing, it holds as many assets asX. ADimensionMismatchis thrown otherwise.
Returns
cache::ExpWeightedCovarianceState: The state after the last observation. A givenstateis the same object, written in place.
Related
PortfolioOptimisers.exp_weighted_moment — Method
exp_weighted_moment(
cache::ExpWeightedCovarianceState,
est::ExpWeightedCovariance;
repair
) -> Any
Covariance method of exp_weighted_moment. Reads the exponentially weighted covariance out of a state.
The method reads the state and does not write it, so a forward pass can call it after every observation. The struct docstring of ExpWeightedCovariance states the estimate that it computes.
Algorithm
- Fill
sigmawithNaN, and takereadyas the assets thatcache.activeadmits and thatcentring_readyfinds ready atest.min_obs. - On the block of the ready assets, divide each entry of
cache.covarianceby its entry ofcache.weightwithpair_weighted_block, and replace the block with the mean of the block and its transpose, which removes the round-off asymmetry. - Where
repairistrue, restore a positive semidefinite block withrestore_psd!. The repair keeps the diagonal. - Write the block into
sigma.
Arguments
cache::ExpWeightedCovarianceState: State to read.est::ExpWeightedCovariance: Exponentially weighted covariance estimator.repair::Bool: Whether to restore a positive semidefinite matrix. A caller that reads the diagonal alone passesfalse.
Returns
sigma::MatNum: Covariance matrix. The row and the column of an asset that is inactive, or that has fewer thanest.min_obsvalid observations, areNaN.
Related
PortfolioOptimisers.pair_weighted_block — Function
pair_weighted_block(
S::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
W::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Divides each entry of a block of an exponentially weighted state by the weight that its pair holds.
The weight state takes the step of S on a unit outer product, so S[i, j] / W[i, j] is the exponentially weighted mean of the product over the common observations of the pair. A pair that holds no weight has no common observation, and its entry is zero.
Algorithm
- Take the block
S[idx, idx]and the blockW[idx, idx]. - Divide each entry of the first by the entry of the second where that entry is above zero, and take zero elsewhere.
Arguments
S::MatNum: Exponentially weighted state, seeded at zero.W::MatNum: Weight that each pair holds inS.idx::AbstractVector{<:Integer}: Index of the assets of the block.
Returns
block::MatNum: The block of the weighted means.
Related
PortfolioOptimisers.restore_psd! — Function
restore_psd!(
sigma::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
Restores a positive semidefinite matrix from a block whose pairs are each divided by their own weight, and keeps its diagonal.
Where every asset has the same valid observations, the division by the weight is a congruence of a sum of outer products, and the block is positive semidefinite. After a holiday it need not be, whatever the weights are: a weight that saturates at one does not show which observations its pair read. So the method tests the block itself. A correlation with a Cholesky factor is positive definite to working precision, and the block is returned unchanged with no eigen decomposition. Otherwise, where the correlation has an eigenvalue below -n eps times the largest, the method clips the negative eigenvalues to zero, which is the nearest positive semidefinite matrix in the Frobenius norm, and restores the unit diagonal and the variances. A smaller negative eigenvalue is round-off, and the block is returned unchanged.
The Cholesky test runs first because it costs a fraction of the decomposition. It can accept a correlation whose smallest eigenvalue is slightly below -n eps times the largest, because the backward error of the factor is about n^2 eps. Such an eigenvalue is round-off, and the block is kept as it is.
Algorithm
- Take
sas the root of the diagonal ofsigma, floored at zero, andposas the assets whosesis above zero. Returnsigmawhenposis empty. - Take the correlation
R = sigma[pos, pos] ./ (s * s'). ReturnsigmawhenRhas a Cholesky factor. - Take the eigen decomposition of
R. Returnsigmawhen the smallest eigenvalue is at least-length(pos) * epstimes the largest magnitude. - Rebuild
Rwith its eigenvalues floored at zero, divide it by the root of its diagonal on both sides, clamp it to[-1, 1], symmetrise it and set its diagonal to one. - Write
R .* (s * s')intosigma[pos, pos], and returnsigma.
Arguments
sigma::MatNum: Block to repair, a covariance or a correlation. It is written in place.
Returns
sigma::MatNum: The repaired block. It is the same object as the argument.
Related
PortfolioOptimisers.gap_fill_value — Method
gap_fill_value(ce::ExpWeightedCovariance) -> Float64Answer NaN, so a gapped sample reaches the recursion with its gaps intact.
The recursion moves only the entries of the assets that are valid at an observation. A finite fill value would make a gapped asset valid, and the recursion would then move entries that a gap keeps. So the consumer passes the sample as it is, together with the active mask that explains each gap.
Arguments
ce: Covariance estimator.
Returns
fv::Float64:NaN.
Related
PortfolioOptimisers.variance_series — Method
variance_series(
ce::ExpWeightedCovariance,
X::MatNum;
dims::Int = 1,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
kwargs...
) -> Matrix{<:Number}Compute the point-in-time exponentially weighted variance series.
Row t holds the diagonal of what cov returns for the first t observations of X, so no row reads an observation after its own. The recursion folds one observation at a time, so this method replaces the expanding-window fallback with one forward pass that reads the state after each observation.
The fallback cannot serve this estimator. It slices X once per row and passes every keyword as it is, so a mask of the whole window meets a window of t observations, and the size check refuses the call.
Algorithm
- Check
dims, and allocatevalwith one row per observation and one column per asset. - Run one forward pass over
Xwithexp_weighted_pass!. After the observationi, write the diagonal ofexp_weighted_momentof the state into rowiofval, withrepair = false, because the repair keeps the diagonal. - Return
val, transposed whendims == 2.
Arguments
ce: Exponentially weighted covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask: Optional boolean matrix with the same size asX.kwargs: More keyword arguments, which the method ignores.
Validation
dims in (1, 2).- If
active_maskis notnothing,size(X) == size(active_mask). ADimensionMismatchis thrown otherwise.
Returns
val::Matrix{<:Number}: Variance series, shaped as(T, N)ifdims == 1or(N, T)ifdims == 2. An asset isNaNat rowtwhen it has fewer thance.min_obsvalid observations up tot, or whenactive_maskexcludes it att.
Related
PortfolioOptimisers.variance_series — Method
variance_series(
ce::ExpWeightedCovariance,
X::MatNum,
pnl::Option{<:AssetPanel};
dims::Int = 1,
kwargs...
) -> Matrix{<:Number}Compute the point-in-time exponentially weighted variance series from a window of an Asset Panel.
This is the diagonal of the covariance of the same call, read after each observation, and it reads the active mask of the panel in the same way. A Descriptor that holds this estimator thus keeps a number for a young asset. The reduce-and-expand root would reduce every window and answer NaN for it.
Arguments
ce: Exponentially weighted covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs: More keyword arguments, which the method ignores.
Validation
dims in (1, 2).- If
pnlis notnothing, its active mask has the size ofXafter the orientation thatdimssets. ADimensionMismatchis thrown otherwise.
Returns
val::Matrix{<:Number}: Variance series on the full asset universe, shaped as(T, N)ifdims == 1or(N, T)ifdims == 2.
Related
Base.copy — Method
copy(
x::ExpWeightedCovarianceState
) -> ExpWeightedCovarianceState{_A, _B, _C, _D, _E, _F, __T_rows} where {_A, _B, _C, _D, _E, _F, __T_rows<:Base.RefValue}
Copies an ExpWeightedCovarianceState, so the copy shares no array with the original.
It is the copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. It copies each array, and keeps an overlap of nothing. It puts the row count in a new Ref, so a fold of the copy does not count in the original.
Arguments
x: The state to copy.
Returns
state::ExpWeightedCovarianceState: A new state equal tox, with new arrays.
Related
PortfolioOptimisers.observation_count — Method
observation_count(state::ExpWeightedCovarianceState) -> Any
ExpWeightedCovarianceState method of observation_count.
The state counts the observations of each asset in obs_count, and a reset sets such a count to zero. So the method reads rows, the number of folded rows, which is the size(X, 1) of a batch fit over the same rows.
Arguments
state: The state that holds the count.
Returns
n::Integer: The number of observations folded so far.
Related
PortfolioOptimisers.exp_weighted_covariance_count — Function
exp_weighted_covariance_count(
decay::Number,
n::Integer,
centring::AbstractCentring
) -> Any
Return the effective number of observations of an exponentially weighted covariance over n rows.
It is the count of variance_count(decay::Number, X::MatNum, centring::AbstractCentring, amsk::Option{<:AbstractMatrix{<:Bool}}) for one column of n finite rows. The covariance reads every row with the weight of its age, so the count reads all n rows, the rows with a NaN too, as size(X, 1) counts them. The count tends to $(1 + \lambda) / (1 - \lambda)$ as n grows. Under equal weights it is n.
Mathematical definition
\[\begin{align} T_{e} &= \kappa\left(K\right) + \ell\,, \\ K &= \max\left(n - \ell, 0\right)\,. \end{align}\]
Where:
- $T_{e}$: Effective sample size, which is Kish's when the observation weights are stated.
- $\kappa(K)$: Kish count of $K$ exponential weights, from
exp_weighted_kish_count. - $n$: Number of rows.
- $\ell$: The lag of the centring, from
centring_lag.
Arguments
decay: Decay of the recursion.n: Number of rows.centring: The centring of the estimator.
Returns
T_e::Number: The effective number of observations.
Related