Exp weighted covariance

Types

PortfolioOptimisers.ExpWeightedCovariance — Type
struct ExpWeightedCovariance{__T_decay, __T_min_obs, __T_centring, __T_cache} <: AbstractCovarianceEstimator

Estimates a covariance matrix by an exponentially weighted recursion in which each pair of assets ages on its own common observations, and which resets on an inactive period.

An entry for a pair of assets steps only at an observation where both assets are valid. A holiday of one asset thus holds its variance and each of its covariances, because it carries no information about them. The state starts at zero, and a second state holds the weight that each pair has gained. The estimate divides each entry by that weight, so each entry is the exponentially weighted mean of the product over the common observations of its pair.

A matrix whose entries each read their own observations need not be positive semidefinite. Where the assets do not share one history, for example after a holiday or a late listing, and the matrix is not positive semidefinite, the estimate clips the negative eigenvalues of its correlation to zero and keeps its variances. The congruence that held the correlation on a holiday is positive semidefinite by construction, but it shrinks every correlation of a pair with different histories towards zero, and a simulation measured the cost. Where every asset has the same valid observations, the two rules agree.

A young asset that stays investable has a cost for the prior. A prior fitted with this estimator fills the rows that the asset lacks with zeros through scenario_fill, because every consumer of a Prior Result reads its returns matrix. A scenario-based measure then reads a zero return where the asset had none, and it understates the risk of that asset over those rows. The covariance stays the estimate that this recursion made from the rows it saw. The fill is silent at or below the fill_limit field of the fitting prior, which is a share of the observations of that asset. Above the limit it warns, and under strict it refuses every fill. fill_limit defaults to nothing, and this family has no CoveragePolicy to derive a limit from, so the prior names every fill.

The estimator has no window of its own. WindowedCovariance is its window: the wrapper keeps the last window observations of the returns and of the active mask, and the recursion starts cold at the first of them. The example below fits the covariance of an EmpiricalPrior on the last 500 observations.

Fields

  • decay: Exponential decay factor for the exponentially weighted estimator.
  • min_obs: Minimum number of observations required before the estimator produces a valid result.
  • centring: The location each deviation is taken from. EstimatedCentring, the default, estimates the location of each asset and divides each term by the bias that the estimate puts in it, so the moment is unbiased from the second valid return whatever the mean. PreCentred takes the returns as deviations from a mean of zero. ZeroStartCentring takes the location of a recursion from zero that is not divided by its weight.
  • cache: Running state of an incremental fit, or nothing before the first call to partial_fit!. It is the one Result this estimator holds, and its type bound is the enforcement of that exception. A fit over a matrix ignores it.

Constructors

ExpWeightedCovariance(;    decay::Number = exp2(-inv(40.0)),    min_obs::Integer = round(Int, max(1, decay_half_life(decay))),    centring::AbstractCentring = EstimatedCentring(),    cache::Option{<:AbstractPartialFitState} = nothing) -> ExpWeightedCovariance

Keywords correspond to the struct's fields. The default min_obs is the half-life of decay, the count of observations after which a weight halves.

Validation

  • 0 < decay < 1.
  • min_obs > 0.

Mathematical definition

\[\begin{align} m_{t,\,i} &= \frac{\sum_{u \in \mathcal{V}_i,\, u < t} \lambda^{c_i(u,\, t)} x_{u,\,i}}{\sum_{u \in \mathcal{V}_i,\, u < t} \lambda^{c_i(u,\, t)}}\,, \\ e_{t,\,i} &= x_{t,\,i} - m_{t,\,i}\,, \\ S_{ij} &= (1 - \lambda) \sum_{t \in \mathcal{D}_i \cap \mathcal{D}_j} \lambda^{c_{ij}(t)} \frac{e_{t,\,i} e_{t,\,j}}{1 + q_{t,\,ij}}\,, \\ W_{ij} &= (1 - \lambda) \sum_{t \in \mathcal{D}_i \cap \mathcal{D}_j} \lambda^{c_{ij}(t)} = 1 - \lambda^{n_{ij}}\,, \\ \hat{\Sigma}_{ij} &= \frac{S_{ij}}{W_{ij}}\,. \end{align}\]

Where:

  • $\lambda$: decay, the decay factor of the exponentially weighted recursion.
  • $x_{t,\,i}$: Return of asset $i$ at observation $t$.
  • $\mathcal{V}_i$: Valid observations of asset $i$. They are the observations after the last one at which the active mask excludes the asset, where the return of the asset is finite.
  • $\mathcal{D}_i$: The observations of $\mathcal{V}_i$ that give a deviation. Under EstimatedCentring, the default, they are every one but the first, because the first has no location before it. Under PreCentred they are every one.
  • $n_i$: Valid-history length of asset $i$, the count of its valid observations since its last reset.
  • $c_i(u,\, t)$: Count of the observations of $\mathcal{V}_i$ that lie strictly between $u$ and $t$.
  • $c_{ij}(t)$: Count of the observations of $\mathcal{D}_i \cap \mathcal{D}_j$ that lie after $t$.
  • $n_{ij}$: Count of the observations of $\mathcal{D}_i \cap \mathcal{D}_j$.
  • $m_{t,\,i}$: Location of asset $i$ before the observation $t$, the normalised exponentially weighted mean of its valid returns. It is zero under PreCentred.
  • $e_{t,\,i}$: Deviation of asset $i$ at the observation $t$.
  • $q_{t,\,ij}$: The overlap of the two locations, the sum over their common returns of the products of their normalised weights, so $1 + q_{t,\,ij}$ is the factor of centring_pair_factor. It is zero under PreCentred.
  • $S_{ij}$: Entry of the internal state.
  • $W_{ij}$: Weight that the pair holds in $S_{ij}$, the entry of the second state.
  • $\hat{\Sigma}_{ij}$: Entry of the estimate, before the repair below. It is NaN where asset $i$ or asset $j$ has fewer than min_obs valid observations, or no deviation, or where the active mask excludes it at the last observation.

For returns that are independent in time, $\mathbb{E}[e_{t,\,i} e_{t,\,j}] = \sigma_{ij} (1 + q_{t,\,ij})$, so each term of $S_{ij}$ has the mean $\sigma_{ij}$, and the estimate is unbiased from the second return of each asset, whatever the mean. Under PreCentred it is too large by the product of the two means. ZeroStartCentring keeps the uncorrected rule: $\mathcal{D}_i = \mathcal{V}_i$, $q_{t,\,ij} = 0$, and the location is the recursion $m \leftarrow \lambda m + (1 - \lambda) x$ from zero, not divided by its weight.

The weights of $S_{ij}$ sum to $W_{ij}$, so $\hat{\Sigma}_{ij}$ is a weighted mean of products whose weights sum to one, over the common observations of the pair. At each observation the recursion is $S_{\mathcal{V}\mathcal{V}} \leftarrow \lambda S_{\mathcal{V}\mathcal{V}} + (1 - \lambda) e_{\mathcal{V}} e_{\mathcal{V}}^\intercal$ on the block of the valid assets, and $W$ takes the same step with every product one. Every other entry holds.

Where every asset has the same valid observations, $W_{ij} = \sqrt{W_{ii} W_{jj}}$, so the estimate is the congruence $D S D$ with $D = \operatorname{diag}(1 / \sqrt{1 - \lambda^{n_i}})$. $S$ is then a sum of positive semidefinite outer products, and so is the estimate. Where the histories differ, $n_{ij} \le \min(n_i, n_j)$, so $W_{ij} \le \sqrt{W_{ii} W_{jj}}$, and the congruence would divide $S_{ij}$ by more than its weight and shrink the correlation of the pair towards zero. The division by $W_{ij}$ does not, but its matrix can fail to be positive semidefinite: a variance that falls while the covariances of its asset hold can imply a correlation above one. Where the smallest eigenvalue of the correlation is below $-n\,\varepsilon$ times its largest, the estimate clips the negative eigenvalues to zero, restores the unit diagonal, and keeps the variances.

Examples

julia> ce = ExpWeightedCovariance();julia> ce.decay ≈ exp2(-inv(40.0))truejulia> ce.min_obs40julia> pe = EmpiricalPrior(; ce = WindowedCovariance(; ce = ce, window = 500));julia> pe.ce.window500

Related

source

Functions

Statistics.cov — Method
Statistics.cov(
    ce::ExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> MatNum

Compute the exponentially weighted covariance matrix.

The struct docstring of ExpWeightedCovariance states the estimate.

Algorithm

  1. Run one forward pass over the observations of X from the zero state with exp_weighted_pass!, giving the state cache.
  2. Read sigma out of cache with exp_weighted_moment.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: Optional boolean matrix with the size of X. An asset whose entry is false is inactive at that observation. Its row and column return to the zero state, and its answer is NaN while it stays inactive. With nothing every asset is active, so a non-finite return is a holiday, and every entry of that asset keeps its value.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If active_mask is not nothing, size(X) == size(active_mask). A DimensionMismatch is thrown otherwise.

Returns

  • sigma::MatNum: Covariance matrix of size assets × assets. The row and the column of an asset are NaN when the asset has fewer than ce.min_obs valid observations, or when active_mask excludes it at the last observation.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = ExpWeightedCovariance(; decay = 0.9, min_obs = 2);julia> size(cov(ce, X))(2, 2)

Related

source
Statistics.cor — Method
Statistics.cor(
    ce::ExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> MatNum

Compute the exponentially weighted correlation matrix.

This is the covariance of the same call, rescaled to a unit diagonal. The correction of the covariance cancels in the rescale, so the correlation is the correlation of the internal state.

Mathematical definition

\[\begin{align} \rho_{ij} &= \max\left(-1,\, \min\left(1,\, \frac{S_{ij}}{\sqrt{S_{ii} S_{jj}}}\right)\right)\,, \quad i \neq j\,, \\ \rho_{ii} &= 1\,. \end{align}\]

Where:

  • $\rho_{ij}$: Entry of the correlation matrix.
  • $S_{ij}$: Entry of the internal state, as the struct docstring of ExpWeightedCovariance defines it.

The state is positive semidefinite, so every ratio lies in $[-1, 1]$ and the clamp removes only round-off. A blanked asset has NaN in its row, in its column and on the diagonal.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: Optional boolean matrix with the same size as X.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • The checks of the covariance of the same call.

Returns

  • rho::MatNum: Correlation matrix of size assets × assets. The row and the column of an asset that is not ready are NaN, and so is its diagonal entry.

Related

source
Statistics.cov — Method
Statistics.cov(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the exponentially weighted covariance from a window of an Asset Panel.

This estimator reads an active mask, so it replaces the reduce-and-expand root of the verb and reads the active mask of the panel itself. The answer covers the whole universe, not only the Coverage Universe. A young asset that lists inside the window gets an estimate from the observations it has, and it is NaN only while its count stays below ce.min_obs.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If pnl is not nothing, its active mask has the size of X after the orientation that dims sets. A DimensionMismatch is thrown otherwise.

Returns

  • sigma::MatNum: Covariance matrix of size assets × assets.

Related

source
Statistics.cor — Method
Statistics.cor(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the exponentially weighted correlation from a window of an Asset Panel.

This is the covariance of the same call, rescaled to a unit diagonal, and it reads the active mask of the panel in the same way.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If pnl is not nothing, its active mask has the size of X after the orientation that dims sets. A DimensionMismatch is thrown otherwise.

Returns

  • rho::MatNum: Correlation matrix of size assets × assets.

Related

source
Statistics.var — Method
Statistics.var(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the marginal exponentially weighted variance from a window of an Asset Panel.

This is the diagonal of the covariance of the same call, and it reads the active mask of the panel in the same way.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If pnl is not nothing, its active mask has the size of X after the orientation that dims sets. A DimensionMismatch is thrown otherwise.

Returns

  • var::MatNum: Marginal variance, as a row where dims is 1 and as a column otherwise.

Related

source
Statistics.std — Method
Statistics.std(
    ce::ExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> MatNum

Compute the marginal exponentially weighted volatility from a window of an Asset Panel.

This is the square root of the diagonal of the covariance of the same call, and it reads the active mask of the panel in the same way.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If pnl is not nothing, its active mask has the size of X after the orientation that dims sets. A DimensionMismatch is thrown otherwise.

Returns

  • std::MatNum: Marginal volatility, as a row where dims is 1 and as a column otherwise.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
Statistics.cov — Method
Statistics.cov(ce::ExpWeightedCovariance; kwargs...) -> MatNum

Read the exponentially weighted covariance out of the state that the estimator holds.

The one-argument form reads the result of an incremental fit. partial_fit! leaves the state in the cache field, and this verb turns the state into the estimate. An estimator that has seen no observation has no state, so the call throws and does not answer zero.

Arguments

  • ce: Exponentially weighted covariance estimator that holds a state.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • ce.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

  • sigma::MatNum: Covariance matrix of size assets × assets.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(ExpWeightedCovariance(; decay = 0.9, min_obs = 2), X);julia> size(cov(ce))(2, 2)julia> cov(ExpWeightedCovariance())ERROR: ArgumentError: `ce` holds no partial-fit state, so there is nothing to read. Call `partial_fit!(ce, X)` first, or `cov(ce, X)` for a fit over a whole sample.[...]

Related

source
Statistics.cor — Method
Statistics.cor(ce::ExpWeightedCovariance, state::ExpWeightedCovarianceState; kwargs...) -> MatNum

Read the exponentially weighted correlation out of a state that the caller holds.

Arguments

  • ce: Exponentially weighted covariance estimator.
  • state::ExpWeightedCovarianceState: The state to read.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • rho::MatNum: Correlation matrix.

Related

source
Statistics.cor — Method
Statistics.cor(ce::ExpWeightedCovariance; kwargs...) -> MatNum

Read the exponentially weighted correlation out of the state that the estimator holds.

Arguments

  • ce: Exponentially weighted covariance estimator that holds a state.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • ce.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

  • rho::MatNum: Correlation matrix of size assets × assets.

Related

source
PortfolioOptimisers.merge_states — Method
merge_states(
    a::ExpWeightedCovarianceState,
    b::ExpWeightedCovarianceState
)

Refuses to merge two ExpWeightedCovarianceState.

The fold $S = \lambda^{n_b} S_a + S_b$ is exact only under PreCentred over a second block in which every asset is valid at every observation. Under EstimatedCentring a block reads the location and the overlap that the first block carries. A holiday makes the decay of an entry depend on the common valid observations of its two assets, and a reset discards the first block for its asset. The state records neither the holidays nor the resets, so a merge cannot tell the cases apart. Fold the second block into the first with partial_fit!.

Arguments

  • a: State of the first block.
  • b: State of the second block.

Validation

  • The method throws an ArgumentError for every pair.

Related

source