Exp weighted variance
Types
PortfolioOptimisers.ExpWeightedVariance — Type
struct ExpWeightedVariance{__T_decay, __T_min_obs, __T_centring, __T_cache} <: AbstractVarianceEstimatorEstimates per-asset variance by an exponentially weighted recursion that freezes on a holiday and resets on an inactive period.
The recursion starts at zero, so a newly listed asset starts from a cold state, and the output divides out the weight that the cold start lacks. An asset with fewer than min_obs valid observations since its last reset is NaN, and so is an asset that the active mask leaves inactive at the last observation.
A young asset stays investable, and the returns matrix of the prior carries the cost. A prior fitted with this estimator fills the rows that the asset was missing with zero through scenario_fill, because every consumer of a Prior Result reads its returns matrix. A scenario-based measure then reads a zero return where the asset had none, so it understates the risk of that asset over those rows. The variance stays the estimate that this recursion made from the rows it saw. The fill is silent up to the fill_limit field of the fitting prior, a share of the asset's own observations. Above that share it warns, and strict refuses any fill. fill_limit defaults to nothing, and this family carries no CoveragePolicy to derive a limit from, so every fill warns.
The estimator has no window of its own. WindowedVariance is its window: the wrapper keeps the last window observations of the returns and of the active mask, and the recursion starts cold at the first of them. In a variance series, row t keeps the window that ends at t. The example below puts the windowed estimator in the variance slot of a CrossSectionalFactorPrior.
Fields
decay: Exponential decay factor for the exponentially weighted estimator.
min_obs: Minimum number of observations required before the estimator produces a valid result.
centring: The location each deviation is taken from.EstimatedCentring, the default, estimates the location of each asset and divides each term by the bias that the estimate puts in it, so the moment is unbiased from the second valid return whatever the mean.PreCentredtakes the returns as deviations from a mean of zero.ZeroStartCentringtakes the location of a recursion from zero that is not divided by its weight.
cache: Running state of an incremental fit, ornothingbefore the first call topartial_fit!. It is the one Result this estimator holds, and its type bound is the enforcement of that exception. A fit over a matrix ignores it.
Constructors
ExpWeightedVariance(; decay::Number = exp2(-inv(40.0)), min_obs::Integer = round(Int, max(1, decay_half_life(decay))), centring::AbstractCentring = EstimatedCentring(), cache::Option{<:AbstractPartialFitState} = nothing) -> ExpWeightedVarianceKeywords correspond to the struct's fields.
Validation
0 < decay < 1.min_obs > 0.
Mathematical definition
Let $x_{i, k}$ be the $k$-th valid return of asset $i$ since its last reset, $k = 1, \ldots, n_i$. Under EstimatedCentring, the default, the location is the exponentially weighted mean of the returns, divided by the sum of its weights. The deviation of a return is taken from the location of the returns before it, and its square is divided by its exact factor, because the location is itself an estimate,
\[\begin{align} m_{i, k} &= \frac{\sum_{j = 1}^{k} \lambda^{k - j} x_{i, j}}{\sum_{j = 1}^{k} \lambda^{k - j}}\,, \\ e_{i, k}^{2} &= \frac{\left(x_{i, k} - m_{i, k - 1}\right)^{2}}{1 + 1 / n^{\mathrm{eff}}_{k - 1}}\,, \quad k = 2, \ldots, n_i\,. \end{align}\]
The first return gives no term, so the lag of the centring is $\ell = 1$. Under PreCentred the deviation is the return, $e_{i, k} = x_{i, k}$ for $k = 1, \ldots, n_i$, and $\ell = 0$. The internal state of asset $i$ is the exponentially weighted sum of its $K_i = n_i - \ell$ terms, and the reported variance divides out the weights that the cold start never accumulated,
\[\begin{align} S_i &= (1 - \lambda) \sum_{k = \ell + 1}^{n_i} \lambda^{n_i - k} e_{i, k}^{2}\,, \\ \hat{\sigma}^{2}_i &= \frac{S_i}{1 - \lambda^{K_i}}\,. \end{align}\]
For returns that are independent in time, each term has the mean $\sigma^{2}_i$, so under EstimatedCentring the estimate is unbiased from the second return, whatever the mean. Under PreCentred it is too large by the square of the mean. ZeroStartCentring keeps the uncorrected rule: $\ell = 0$, the location takes the step $m_{i, k} = \lambda m_{i, k - 1} + (1 - \lambda) x_{i, k}$ from $m_{i, 0} = 0$, and $e_{i, k} = x_{i, k} - m_{i, k - 1}$ with no factor.
Where:
- $\lambda$:
decay, the decay factor of the exponentially weighted recursion. - $n_i$: Valid-history length of asset $i$, the count of its valid observations since its last reset.
- $x_{i, k}$: The $k$-th valid return of asset $i$ since its last reset.
- $m_{i, k}$: The location of asset $i$ after $k$ valid returns.
- $n^{\mathrm{eff}}_{k}$: The Kish count of the weights of a location of $k$ returns, so $1 + 1 / n^{\mathrm{eff}}_{k}$ is the factor of
centring_factor. - $e_{i, k}^{2}$: The term of the $k$-th valid return of asset $i$.
- $\ell$: The lag of the centring, the count of the leading returns that give no term.
- $K_i$: The count of the terms of asset $i$.
- $S_i$: The internal state of asset $i$, the
variancefield ofExpWeightedVarianceState. - $\hat{\sigma}^{2}_i$: The reported variance of asset $i$.
Examples
julia> ce = ExpWeightedVariance();julia> ce.decay ≈ exp2(-inv(40.0))truejulia> ce.min_obs40julia> ve = WindowedVariance(; ve = ce, window = 500);julia> pe = CrossSectionalFactorPrior(; factors = ["market" => ConstantExposure()], ve = ve);julia> pe.ve.window500Related
Functions
Statistics.var — Method
Statistics.var(
ce::ExpWeightedVariance,
X::MatNum;
dims::Int = 1,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
kwargs...
) -> Vector{<:Number}Compute the exponentially weighted variance of each asset.
Runs the recursion over the observations of X in one forward pass. Then it applies the cold-start bias correction and sets every asset that is not ready to NaN.
Arguments
ce: Exponentially weighted variance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask: Optional boolean matrix with the same size asX. An asset whose entry isfalseis inactive at that observation. The estimator resets its state, and answersNaNfor it while it stays inactive. Withnothingevery asset is active, so a non-finite return reads as a holiday and the variance freezes.kwargs: More keyword arguments, which the method ignores.
Validation
dims in (1, 2).- If
active_maskis notnothing,size(X) == size(active_mask).
Returns
var::Vector{<:Number}: Per-asset variance vector of lengthassets. An asset with fewer thance.min_obsvalid observations since its last reset isNaN, and so is an asset that is inactive at the last observation.
Examples
julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = ExpWeightedVariance(; decay = 0.9, min_obs = 2);julia> length(var(ce, X))2Related
Statistics.std — Method
Statistics.std(
ce::ExpWeightedVariance,
X::MatNum;
dims::Int = 1,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
kwargs...
) -> Vector{<:Number}Compute the exponentially weighted volatility of each asset.
This is the square root of the variance of the same call.
Arguments
ce: Exponentially weighted variance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.active_mask: Optional boolean matrix with the same size asX.kwargs: More keyword arguments, which the method ignores.
Returns
std::Vector{<:Number}: Per-asset volatility vector of lengthassets.
Related
Statistics.var — Method
Statistics.var(
ce::ExpWeightedVariance,
X::MatNum,
pnl::Option{<:AssetPanel};
dims::Int = 1,
kwargs...
) -> Vector{<:Number}Compute the exponentially weighted variance from a window of an Asset Panel.
This estimator reads the active mask of the panel itself, so it overrides the reduce-and-expand root of the verb. Its answer covers the whole universe, not only the Coverage Universe. A young asset that lists inside the window gets an answer from the observations it has, and it is NaN only while it has fewer than ce.min_obs.
Arguments
ce: Exponentially weighted variance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs: More keyword arguments, which the method ignores.
Returns
var::Vector{<:Number}: Per-asset variance vector of lengthassets.
Related
Statistics.std — Method
Statistics.std(
ce::ExpWeightedVariance,
X::MatNum,
pnl::Option{<:AssetPanel};
dims::Int = 1,
kwargs...
) -> Vector{<:Number}Compute the exponentially weighted volatility from a window of an Asset Panel.
This is the square root of the variance of the same call, and it reads the panel's active mask through the same override.
Arguments
ce: Exponentially weighted variance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs: More keyword arguments, which the method ignores.
Returns
std::Vector{<:Number}: Per-asset volatility vector of lengthassets.
Related
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
est::Union{AbstractEstimator, CovarianceEstimator},
X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
dims,
active_mask,
estimation_mask,
iv,
ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}
Folds observations into the sample buffer an estimator carries.
This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.
The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.
A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.
Algorithm
- Read the buffer out of the
cachefield withassert_sample_buffer, which refuses an estimator that was never wrapped inOnline. - Fold a matrix, its masks and its implied volatilities through the block arm of
partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm. - Rebuild the estimator with the new buffer in
cachethroughrebuild_struct, and return it.
Arguments
est: Estimator whose buffer is folded forward.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets, anddimsis ignored.dims: Dimension along which the computation runs.active_mask: The active mask of the block, of the shape ofX, or of one entry per asset whenXis one observation, ornothing.estimation_mask: The estimation mask, on the same terms asactive_mask.iv: The implied volatilities, on the same terms asactive_mask.ivpa: The implied volatility risk premium adjustment, ornothing.
Validation
estcarries aSampleBufferState. AnArgumentErroris thrown otherwise.- The masks and
iv, when they are notnothing, have the shape ofX. ADimensionMismatchis thrown otherwise. - A buffer that holds observations receives the masks that it records, and it receives
ivif and only if it records implied volatilities. AnArgumentErroris thrown otherwise. ivpaequals the adjustment that the first append pinned. AnArgumentErroris thrown otherwise.dims in (1, 2).
Returns
est: The estimator, with itscachefield rebound to the buffer after the last observation.
Related
PortfolioOptimisers.partial_fit! — Method
partial_fit!(
est::Union{AbstractEstimator, CovarianceEstimator},
X::Union{AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:Union{Number, AbstractJuMPScalar}}};
dims,
active_mask,
estimation_mask,
iv,
ivpa
) -> Covariance{<:AbstractExpectedReturnsEstimator, <:CovarianceEstimator, <:AbstractMomentAlgorithm}
Folds observations into the sample buffer an estimator carries.
This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.
The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.
A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.
Algorithm
- Read the buffer out of the
cachefield withassert_sample_buffer, which refuses an estimator that was never wrapped inOnline. - Fold a matrix, its masks and its implied volatilities through the block arm of
partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm. - Rebuild the estimator with the new buffer in
cachethroughrebuild_struct, and return it.
Arguments
est: Estimator whose buffer is folded forward.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets, anddimsis ignored.dims: Dimension along which the computation runs.active_mask: The active mask of the block, of the shape ofX, or of one entry per asset whenXis one observation, ornothing.estimation_mask: The estimation mask, on the same terms asactive_mask.iv: The implied volatilities, on the same terms asactive_mask.ivpa: The implied volatility risk premium adjustment, ornothing.
Validation
estcarries aSampleBufferState. AnArgumentErroris thrown otherwise.- The masks and
iv, when they are notnothing, have the shape ofX. ADimensionMismatchis thrown otherwise. - A buffer that holds observations receives the masks that it records, and it receives
ivif and only if it records implied volatilities. AnArgumentErroris thrown otherwise. ivpaequals the adjustment that the first append pinned. AnArgumentErroris thrown otherwise.dims in (1, 2).
Returns
est: The estimator, with itscachefield rebound to the buffer after the last observation.
Related
Statistics.var — Method
Statistics.var(ce::ExpWeightedVariance, state::ExpWeightedVarianceState; kwargs...) -> Vector{<:Number}Read the exponentially weighted variance out of a state the caller holds.
Arguments
ce: Exponentially weighted variance estimator.state::ExpWeightedVarianceState: The state to read.kwargs: More keyword arguments, which the method ignores.
Returns
var::Vector{<:Number}: Per-asset variance vector.
Related
Statistics.var — Method
Statistics.var(ce::ExpWeightedVariance; kwargs...) -> Vector{<:Number}Read the exponentially weighted variance out of the estimator's own state.
The one-argument form is what an incremental fit answers: partial_fit! leaves the state in the cache field, and this verb turns it into the ordinary answer. An estimator that has seen no observation carries no state, so this method refuses the call instead of answering zero.
Arguments
ce: Exponentially weighted variance estimator carrying a state.kwargs: More keyword arguments, which the method ignores.
Validation
ce.cacheis notnothing. Otherwise the method throws anArgumentError.
Returns
var::Vector{<:Number}: Per-asset variance vector of lengthassets.
Examples
julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(ExpWeightedVariance(; decay = 0.9, min_obs = 2), X);julia> length(var(ce))2julia> var(ExpWeightedVariance())ERROR: ArgumentError: `ce` holds no partial-fit state, so there is nothing to read. Call `partial_fit!(ce, X)` first, or `var(ce, X)` for a fit over a whole sample.[...]Related
Statistics.std — Method
Statistics.std(ce::ExpWeightedVariance, state::ExpWeightedVarianceState; kwargs...) -> Vector{<:Number}Read the exponentially weighted volatility out of a state the caller holds.
Arguments
ce: Exponentially weighted variance estimator.state::ExpWeightedVarianceState: The state to read.kwargs: More keyword arguments, which the method ignores.
Returns
std::Vector{<:Number}: Per-asset volatility vector.
Related
Statistics.std — Method
Statistics.std(ce::ExpWeightedVariance; kwargs...) -> Vector{<:Number}Read the exponentially weighted volatility out of the estimator's own state.
Arguments
ce: Exponentially weighted variance estimator carrying a state.kwargs: More keyword arguments, which the method ignores.
Validation
ce.cacheis notnothing. Otherwise the method throws anArgumentError.
Returns
std::Vector{<:Number}: Per-asset volatility vector of lengthassets.
Related
PortfolioOptimisers.merge_states — Method
merge_states(
a::ExpWeightedVarianceState,
b::ExpWeightedVarianceState
)
Refuses to merge two ExpWeightedVarianceState.
A state under PreCentred folds forward exactly, S = λ^{n_b} S_a + S_b with n_b the valid observations of the second block, but only while no asset resets inside that block. The state records the count that a reset zeroed and not the reset itself, so a merge cannot tell the two cases apart, and it would keep a history that the reset discarded. A state under EstimatedCentring does not fold at all, because each deviation of the second block reads a location that carries the first block. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The method throws an
ArgumentErrorfor every pair.
Related