Abstract covariance variance: private API

Types

PortfolioOptimisers.VarianceCount — Type
struct VarianceCount{__T_n, __T_m}

Holds the effective count and the divisor of a variance, for one column or for each column of the data.

This type is an implementation detail and is not intended for direct use. variance_count and simple_variance_count make it. residual_variance_counts and cross_sectional_variance_counts read it.

Fields

  • n: n: The effective count of the observations, a number for one column or a vector with one entry for each column.
  • m: m: The divisor of the variance for that count, of the shape of n.

Related

source

Functions

PortfolioOptimisers.variance_series — Method
variance_series(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)

Compute the point-in-time variance series, with one row for each observation.

Row t holds the variance of each asset, as estimated from observations 1 to t only, so row t - 1 holds a variance that does not contain observation t. A weight for observation t needs this variance. A variance that contains the squared residual of date t gives less weight to an asset for its own shock, and it correlates the weights with the residuals.

This method is the fallback that gives a series to each AbstractCovarianceEstimator, so a member does not need a method of its own. It fits the estimator again on an expanding window, which costs one fit for each observation. A member whose estimate is a recursion overrides this method with one forward pass.

Each fit reads only the observations 1 to t, so every input with one entry for each observation is cut to those rows with the returns. These inputs are the observation weights that ce holds, the implied volatility iv, the active mask and the estimation mask. Row t under observation weights is the weighted variance of the first t observations. The weights of these observations keep the values that the full sample gives them. A moment estimator of the library normalises its weights over the rows that it fits, so each row is a weighted variance with weights that sum to one over its window.

The expanding window is the windowed series with no window and no weights of its own. So this method calls the windowed method variance_series(est, w::Option{<:ObsWeights}, window::Option{<:Int_VecInt}, X::MatNum, pnl::Option{<:AssetPanel}), and the two series share one definition of row t.

Mathematical definition

\[\begin{align} \mathbf{V}_{tj} &= \hat{\sigma}^2_j\left(\mathbf{X}_{1:t};\, \boldsymbol{w}_{1:t}\right)\,. \end{align}\]

Where:

  • $\mathbf{V}_{tj}$: Point-in-time variance of asset $j$ at observation $t$. No observation after $t$ enters it.
  • $\hat{\sigma}^2_j$: Estimated variance of asset $j$. Here ce estimates it.
  • $\mathbf{X}_{1:t}$: The first $t$ observations of the data matrix.
  • $\boldsymbol{w}_{1:t}$: The observation weights of ce at the first $t$ observations, or no weights if ce holds none.
  • $T$: Number of observations.

Algorithm

  1. Call the windowed method with no window and no wrapper weights. For each observation t, it cuts X, the observation weights of ce, iv and the two universe masks to the rows 1 to t, and it writes the variance of that fit into row t.
  2. Return the series of the windowed method, which it transposes when dims == 2.

Arguments

  • ce: Covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • kwargs...: More keyword arguments for the variance estimator. The method cuts iv, active_mask and estimation_mask to the rows of each fit. It passes every other keyword unchanged, so an estimator that takes another keyword with one entry for each observation must override this method.

Validation

  • dims in (1, 2).

Returns

  • val::Matrix{<:Number}: The variance series. Its shape is (T, N) if dims == 1, or (N, T) if dims == 2.

Examples

julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> PortfolioOptimisers.variance_series(SimpleVariance(), X)3×2 Matrix{Float64}: NaN       NaN   0.0002    0.0002   0.0001    0.0001

Row 1 is a fit on one observation. An estimator that needs two observations returns NaN in that row.

Related

source
PortfolioOptimisers.variance_count — Function
variance_count(ve::AbstractCovarianceEstimator, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::SimpleVariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::ExpWeightedVariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::RegimeAdjustedExpWeightedVariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::WindowedVariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::GeneralCovariance{<:StatsBase.SimpleCovariance}, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::Covariance{<:Any, <:Any, <:FullMoment}, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::ExpWeightedCovariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::RegimeAdjustedExpWeightedCovariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::WindowedCovariance, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)
variance_count(ve::PortfolioOptimisersCovariance{<:Any,
                                                 <:MatrixProcessing{<:Any, <:Any, <:Any,
                                                                    Nothing}}, X::MatNum,
               amsk::Option{<:AbstractMatrix{<:Bool}} = nothing)

Return the effective count and the divisor of the variance that ve measures on each column of X.

A variance that reads an active mask restarts an asset where the mask turns false, so the count of that variance reads the same rows. Pass the mask that the variance read as amsk. An estimator whose fit reads no mask ignores it.

A consumer that prices the sampling error of a variance needs two facts that only the estimator knows. The first fact is the number of observations that the estimate is worth. The second fact is the divisor of the sum of squares. This function returns both. A prior that writes the idiosyncratic variances of a loadings block calls this function on the residuals. The prior writes the result next to the variances, so that ResidualInflation reads it from the block.

The method for AbstractCovarianceEstimator returns nothing, which means that the estimator states no count. nothing is not a default count. A consumer that needs a count refuses a block that has none, and its error names the fix. The fallback is on the covariance family and not on the variance family alone, because CrossSectionalFactorPrior takes any covariance estimator as its variance estimator, and it asks each one for its count. A covariance estimator whose diagonal weights the squares as a variance estimator with a count does states that count, so the two estimators give one count.

Mathematical definition

Let the estimate of asset $i$ be a weighted sum of squares, $\hat{d}_{i} = \sum_{t} a_{t} e_{t,i}^{2}$, of residuals with variance $d_{i}$. A scaled chi-squared law with the same mean and the same variance approximates the law of the estimate:

\[\begin{align} \dfrac{m_{i} \hat{d}_{i}}{d_{i}} &\sim \chi^{2}_{n_{i}}\,, \\ n_{i} &= \dfrac{\left(\sum_{t} a_{t}\right)^{2}}{\sum_{t} a_{t}^{2}}\,, \\ m_{i} &= \dfrac{n_{i}}{\sum_{t} a_{t}}\,. \end{align}\]

Where:

  • $n_{i}$: The effective count of the observations of asset $i$. It is the Kish count of the weights $a_{t}$.
  • $m_{i}$: The divisor of the variance of asset $i$ for that count. The estimate $\hat{d}_{i}$ equals the sum of squares of $n_{i}$ observations of equal weight, divided by $m_{i}$.
  • $a_{t}$: The weight of observation $t$ in the estimate.
  • $e_{t,i}$: The residual of asset $i$ at observation $t$.
  • $d_{i}$: The true variance of asset $i$.

On an unweighted sample of $T$ observations, $n_{i} = T$. Then $m_{i}$ is T - 1 if the estimator corrects the bias, and T if it does not. For Gaussian residuals the relation is then exact. For all other weights it is an approximation.

Algorithm

  1. On AbstractCovarianceEstimator, return nothing.
  2. On SimpleVariance, take the finite rows of each column with their weights, through observed_sample, and read the count from those weights with simple_variance_count. A dynamic weight resolves on the rows of the column.
  3. On ExpWeightedVariance and RegimeAdjustedExpWeightedVariance, take the Kish count of the exponential weights over the finite rows of each column after its last inactive row of amsk, with variance_count(decay::Number, X::MatNum, centring::AbstractCentring, amsk::Option{<:AbstractMatrix{<:Bool}}). The weights sum to one after the correction of the cold start, so the divisor equals the count. The count ignores the regime multiplier and the lags of a Heteroskedasticity and Autocorrelation Consistent (HAC) adjustment.
  4. On WindowedVariance and WindowedCovariance, apply the window and the weights of the wrapper with windowed_preamble, which cuts amsk to the window too. Then return the count of the inner estimator.
  5. On a GeneralCovariance over a StatsBase.SimpleCovariance, return the count of a SimpleVariance with the weights of the estimator and the corrected flag of StatsBase.SimpleCovariance. The diagonal of the covariance is that variance.
  6. On a Covariance with FullMoment, return the count of its inner estimator, with the weights of ve, from covariance_inner_estimator. An inner estimator with no count of its own, such as a shrinkage estimator, gives nothing. A SemiMoment variance is not a sum of squares of every residual, so it states no count.
  7. On ExpWeightedCovariance and RegimeAdjustedExpWeightedCovariance, take the count of step 3 at the decay and the centring of the estimator. The diagonal of the covariance weights the squares as the matching variance estimator does. The regime multiplier of the covariance can differ from the one of RegimeAdjustedExpWeightedVariance, and the count ignores it, as step 3 states.
  8. On a PortfolioOptimisersCovariance whose MatrixProcessing has no custom alg step, return the count of its ce. The repair, the denoising and the detoning act on the fitted matrix and drop no observation, so the variance rests on the observations that ce counts. A custom alg step reads the sample itself, and its matrix need not rest on those observations, so the composite then states no count.

Arguments

  • ve: Variance estimator.
  • X: The data matrix, of shape observations × assets.
  • amsk: The active mask that the variance read, of the shape of X, or nothing for a variance that read no mask.

Validation

Returns

  • cnt::Option{<:VarianceCount}: The count of two vectors, with one entry for each column of X. n is the effective count and m is the divisor. The value is nothing if the estimator states no count.

Related

source
variance_count(decay::Number, X::MatNum, centring::AbstractCentring,
               amsk::Option{<:AbstractMatrix{<:Bool}})

Effective count and divisor of an exponentially weighted variance, for each column of X.

ExpWeightedVariance, ExpWeightedCovariance, RegimeAdjustedExpWeightedVariance and RegimeAdjustedExpWeightedCovariance pass their decay and their centring to this method.

Each column enters the recursion on its finite rows alone, and the recursion divides out the weights that the cold start never accumulated. A column of $n_{i}$ finite rows gives $K_{i} = n_{i} - \ell$ terms, with $\ell$ the lag of the centring of centring_lag. The weights of the terms are proportional to $\lambda^{k}$, $k = 0, \ldots, K_{i} - 1$, and they sum to one. ExpWeightedVariance and RegimeAdjustedExpWeightedVariance share this count.

The recursion restarts an asset where its active mask turns false, so the count reads the same rows. With a mask, $n_{i}$ counts the finite rows after the last inactive row of column $i$. With no mask, the recursion does not restart, and $n_{i}$ counts every finite row.

The Kish count of the weights of the terms is the count of the terms that the estimate is worth. Under PreCentred each term reads one observation, so it is both the effective count and the divisor. Under EstimatedCentring the location spends one observation, as the mean of a sample variance does, so the effective count is one more than the divisor. With equal weights the pair is $(n_{i}, n_{i} - 1)$, the count and the divisor of a sample variance.

Mathematical definition

\[\begin{align} \kappa_{i} &= \dfrac{\left(\sum_{k=0}^{K_{i}-1} \lambda^{k}\right)^{2}}{\sum_{k=0}^{K_{i}-1} \lambda^{2k}} = \dfrac{\left(1 - \lambda^{K_{i}}\right)\left(1 + \lambda\right)}{\left(1 - \lambda\right)\left(1 + \lambda^{K_{i}}\right)}\,, \\ n_{i}^{\mathrm{eff}} &= \kappa_{i} + \ell\,, \\ m_{i} &= \kappa_{i}\,. \end{align}\]

Where:

  • $\kappa_{i}$: Kish count of the weights of the terms of asset $i$.
  • $n_{i}^{\mathrm{eff}}$: Effective count of the observations of asset $i$.
  • $m_{i}$: The divisor of its variance.
  • $n_{i}$: Number of finite rows of column $i$ after its last inactive row, or of the whole column with no mask.
  • $\ell$: The lag of the centring.
  • $\lambda$: decay, the decay factor of the exponentially weighted recursion.

The count tends to $(1 + \lambda) / (1 - \lambda)$ as $n_{i}$ grows, which is about $115$ at the default half-life of $40$ observations.

Arguments

  • decay: Decay of the recursion.
  • X: Data matrix observations × assets.
  • centring: The centring of the estimator.
  • amsk: The active mask that the variance read, observations × assets, or nothing for a variance that read no mask.

Validation

  • size(amsk) == size(X) when amsk is a mask. Raises a DimensionMismatch.

Returns

  • cnt::VarianceCount: The effective count n and the divisor m, one entry per column of X.

Related

source