Exp weighted variance

Types

PortfolioOptimisers.ExpWeightedVariance — Type
struct ExpWeightedVariance{__T_decay, __T_min_obs, __T_centring, __T_cache} <: AbstractVarianceEstimator

Estimates per-asset variance by an exponentially weighted recursion that freezes on a holiday and resets on an inactive period.

The recursion starts at zero, so a newly listed asset starts from a cold state, and the output divides out the weight that the cold start lacks. An asset with fewer than min_obs valid observations since its last reset is NaN, and so is an asset that the active mask leaves inactive at the last observation.

A young asset stays investable, and the returns matrix of the prior carries the cost. A prior fitted with this estimator fills the rows that the asset was missing with zero through scenario_fill, because every consumer of a Prior Result reads its returns matrix. A scenario-based measure then reads a zero return where the asset had none, so it understates the risk of that asset over those rows. The variance stays the estimate that this recursion made from the rows it saw. The fill is silent up to the fill_limit field of the fitting prior, a share of the asset's own observations. Above that share it warns, and strict refuses any fill. fill_limit defaults to nothing, and this family carries no CoveragePolicy to derive a limit from, so every fill warns.

The estimator has no window of its own. WindowedVariance is its window: the wrapper keeps the last window observations of the returns and of the active mask, and the recursion starts cold at the first of them. In a variance series, row t keeps the window that ends at t. The example below puts the windowed estimator in the variance slot of a CrossSectionalFactorPrior.

Fields

  • decay: Exponential decay factor for the exponentially weighted estimator.
  • min_obs: Minimum number of observations required before the estimator produces a valid result.
  • centring: The location each deviation is taken from. EstimatedCentring, the default, estimates the location of each asset and divides each term by the bias that the estimate puts in it, so the moment is unbiased from the second valid return whatever the mean. PreCentred takes the returns as deviations from a mean of zero. ZeroStartCentring takes the location of a recursion from zero that is not divided by its weight.
  • cache: Running state of an incremental fit, or nothing before the first call to partial_fit!. It is the one Result this estimator holds, and its type bound is the enforcement of that exception. A fit over a matrix ignores it.

Constructors

ExpWeightedVariance(;    decay::Number = exp2(-inv(40.0)),    min_obs::Integer = round(Int, max(1, decay_half_life(decay))),    centring::AbstractCentring = EstimatedCentring(),    cache::Option{<:AbstractPartialFitState} = nothing) -> ExpWeightedVariance

Keywords correspond to the struct's fields.

Validation

  • 0 < decay < 1.
  • min_obs > 0.

Mathematical definition

Let $x_{i, k}$ be the $k$-th valid return of asset $i$ since its last reset, $k = 1, \ldots, n_i$. Under EstimatedCentring, the default, the location is the exponentially weighted mean of the returns, divided by the sum of its weights. The deviation of a return is taken from the location of the returns before it, and its square is divided by its exact factor, because the location is itself an estimate,

\[\begin{align} m_{i, k} &= \frac{\sum_{j = 1}^{k} \lambda^{k - j} x_{i, j}}{\sum_{j = 1}^{k} \lambda^{k - j}}\,, \\ e_{i, k}^{2} &= \frac{\left(x_{i, k} - m_{i, k - 1}\right)^{2}}{1 + 1 / n^{\mathrm{eff}}_{k - 1}}\,, \quad k = 2, \ldots, n_i\,. \end{align}\]

The first return gives no term, so the lag of the centring is $\ell = 1$. Under PreCentred the deviation is the return, $e_{i, k} = x_{i, k}$ for $k = 1, \ldots, n_i$, and $\ell = 0$. The internal state of asset $i$ is the exponentially weighted sum of its $K_i = n_i - \ell$ terms, and the reported variance divides out the weights that the cold start never accumulated,

\[\begin{align} S_i &= (1 - \lambda) \sum_{k = \ell + 1}^{n_i} \lambda^{n_i - k} e_{i, k}^{2}\,, \\ \hat{\sigma}^{2}_i &= \frac{S_i}{1 - \lambda^{K_i}}\,. \end{align}\]

For returns that are independent in time, each term has the mean $\sigma^{2}_i$, so under EstimatedCentring the estimate is unbiased from the second return, whatever the mean. Under PreCentred it is too large by the square of the mean. ZeroStartCentring keeps the uncorrected rule: $\ell = 0$, the location takes the step $m_{i, k} = \lambda m_{i, k - 1} + (1 - \lambda) x_{i, k}$ from $m_{i, 0} = 0$, and $e_{i, k} = x_{i, k} - m_{i, k - 1}$ with no factor.

Where:

  • $\lambda$: decay, the decay factor of the exponentially weighted recursion.
  • $n_i$: Valid-history length of asset $i$, the count of its valid observations since its last reset.
  • $x_{i, k}$: The $k$-th valid return of asset $i$ since its last reset.
  • $m_{i, k}$: The location of asset $i$ after $k$ valid returns.
  • $n^{\mathrm{eff}}_{k}$: The Kish count of the weights of a location of $k$ returns, so $1 + 1 / n^{\mathrm{eff}}_{k}$ is the factor of centring_factor.
  • $e_{i, k}^{2}$: The term of the $k$-th valid return of asset $i$.
  • $\ell$: The lag of the centring, the count of the leading returns that give no term.
  • $K_i$: The count of the terms of asset $i$.
  • $S_i$: The internal state of asset $i$, the variance field of ExpWeightedVarianceState.
  • $\hat{\sigma}^{2}_i$: The reported variance of asset $i$.

Examples

julia> ce = ExpWeightedVariance();julia> ce.decay ≈ exp2(-inv(40.0))truejulia> ce.min_obs40julia> ve = WindowedVariance(; ve = ce, window = 500);julia> pe = CrossSectionalFactorPrior(; factors = ["market" => ConstantExposure()], ve = ve);julia> pe.ve.window500

Related

source

Functions

Statistics.var — Method
Statistics.var(
    ce::ExpWeightedVariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> Vector{<:Number}

Compute the exponentially weighted variance of each asset.

Runs the recursion over the observations of X in one forward pass. Then it applies the cold-start bias correction and sets every asset that is not ready to NaN.

Arguments

  • ce: Exponentially weighted variance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: Optional boolean matrix with the same size as X. An asset whose entry is false is inactive at that observation. The estimator resets its state, and answers NaN for it while it stays inactive. With nothing every asset is active, so a non-finite return reads as a holiday and the variance freezes.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If active_mask is not nothing, size(X) == size(active_mask).

Returns

  • var::Vector{<:Number}: Per-asset variance vector of length assets. An asset with fewer than ce.min_obs valid observations since its last reset is NaN, and so is an asset that is inactive at the last observation.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = ExpWeightedVariance(; decay = 0.9, min_obs = 2);julia> length(var(ce, X))2

Related

source
Statistics.std — Method
Statistics.std(
    ce::ExpWeightedVariance,
    X::MatNum;
    dims::Int = 1,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> Vector{<:Number}

Compute the exponentially weighted volatility of each asset.

This is the square root of the variance of the same call.

Arguments

  • ce: Exponentially weighted variance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • active_mask: Optional boolean matrix with the same size as X.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • std::Vector{<:Number}: Per-asset volatility vector of length assets.

Related

source
Statistics.var — Method
Statistics.var(
    ce::ExpWeightedVariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> Vector{<:Number}

Compute the exponentially weighted variance from a window of an Asset Panel.

This estimator reads the active mask of the panel itself, so it overrides the reduce-and-expand root of the verb. Its answer covers the whole universe, not only the Coverage Universe. A young asset that lists inside the window gets an answer from the observations it has, and it is NaN only while it has fewer than ce.min_obs.

Arguments

  • ce: Exponentially weighted variance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • var::Vector{<:Number}: Per-asset variance vector of length assets.

Related

source
Statistics.std — Method
Statistics.std(
    ce::ExpWeightedVariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> Vector{<:Number}

Compute the exponentially weighted volatility from a window of an Asset Panel.

This is the square root of the variance of the same call, and it reads the panel's active mask through the same override.

Arguments

  • ce: Exponentially weighted variance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • std::Vector{<:Number}: Per-asset volatility vector of length assets.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

  1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
  2. Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
  3. Rebuild the estimator with the new buffer in cache through rebuild_struct, and return it.

Arguments

  • est: Estimator whose buffer is folded forward.
  • X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
  • estimation_mask: The estimation mask, on the same terms as active_mask.
  • iv: The implied volatilities, on the same terms as active_mask.
  • ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

  • est carries a SampleBufferState. An ArgumentError is thrown otherwise.
  • The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
  • A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
  • ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
  • dims in (1, 2).

Returns

  • est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source
Statistics.var — Method
Statistics.var(ce::ExpWeightedVariance; kwargs...) -> Vector{<:Number}

Read the exponentially weighted variance out of the estimator's own state.

The one-argument form is what an incremental fit answers: partial_fit! leaves the state in the cache field, and this verb turns it into the ordinary answer. An estimator that has seen no observation carries no state, so this method refuses the call instead of answering zero.

Arguments

  • ce: Exponentially weighted variance estimator carrying a state.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • ce.cache is not nothing. Otherwise the method throws an ArgumentError.

Returns

  • var::Vector{<:Number}: Per-asset variance vector of length assets.

Examples

julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(ExpWeightedVariance(; decay = 0.9, min_obs = 2), X);julia> length(var(ce))2julia> var(ExpWeightedVariance())ERROR: ArgumentError: `ce` holds no partial-fit state, so there is nothing to read. Call `partial_fit!(ce, X)` first, or `var(ce, X)` for a fit over a whole sample.[...]

Related

source
Statistics.std — Method
Statistics.std(ce::ExpWeightedVariance, state::ExpWeightedVarianceState; kwargs...) -> Vector{<:Number}

Read the exponentially weighted volatility out of a state the caller holds.

Arguments

  • ce: Exponentially weighted variance estimator.
  • state::ExpWeightedVarianceState: The state to read.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • std::Vector{<:Number}: Per-asset volatility vector.

Related

source
Statistics.std — Method
Statistics.std(ce::ExpWeightedVariance; kwargs...) -> Vector{<:Number}

Read the exponentially weighted volatility out of the estimator's own state.

Arguments

  • ce: Exponentially weighted variance estimator carrying a state.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • ce.cache is not nothing. Otherwise the method throws an ArgumentError.

Returns

  • std::Vector{<:Number}: Per-asset volatility vector of length assets.

Related

source
PortfolioOptimisers.merge_states — Method
merge_states(
    a::ExpWeightedVarianceState,
    b::ExpWeightedVarianceState
)

Refuses to merge two ExpWeightedVarianceState.

A state under PreCentred folds forward exactly, S = λ^{n_b} S_a + S_b with n_b the valid observations of the second block, but only while no asset resets inside that block. The state records the count that a reset zeroed and not the reset itself, so a merge cannot tell the two cases apart, and it would keep a history that the reset discarded. A state under EstimatedCentring does not fold at all, because each deviation of the second block reads a location that carries the first block. Fold the second block into the first with partial_fit! instead.

Arguments

  • a: State of the first block.
  • b: State of the second block.

Validation

  • The method throws an ArgumentError for every pair.

Related

source