Abstract covariance variance
Statistics.var — Method
Statistics.var(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)Compute the variance vector from the diagonal of the covariance matrix.
This method is the fallback that gives a marginal variance to each AbstractCovarianceEstimator. A caller does not need a separate variance estimator to read the diagonal of the covariance matrix.
Mathematical definition
\[\begin{align} \hat{\sigma}_i^2 &= \hat{\mathbf{\Sigma}}_{ii}\,. \end{align}\]
Where:
- $\hat{\sigma}_i^2$: The variance of asset $i$.
- $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
- $\hat{\mathbf{\Sigma}}_{ii}$: $i$-th diagonal entry of $\hat{\mathbf{\Sigma}}$.
Algorithm
- Compute the covariance matrix with
Statistics.cov(ce, X; dims = dims, kwargs...). - Read its diagonal into
val, the variance of each asset. - Reshape
valto a1 × Nrow vector whendims == 1, and to anN × 1column vector otherwise.
Arguments
ce: Covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the covariance estimator.
Returns
var::Matrix{<:Number}: The variance vector. Its shape is(1, N)ifdims == 1, or(N, 1)ifdims == 2.
Examples
julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> var(Covariance(), X)1×2 Matrix{Float64}: 0.0001 0.0001Related
Statistics.std — Method
Statistics.std(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)Compute the standard deviation vector from the diagonal of the covariance matrix.
This method is the fallback that gives a marginal standard deviation to each AbstractCovarianceEstimator. A caller does not need a separate variance estimator to read the diagonal of the covariance matrix.
Mathematical definition
\[\begin{align} \hat{\sigma}_i &= \sqrt{\hat{\mathbf{\Sigma}}_{ii}}\,. \end{align}\]
Where:
- $\hat{\sigma}_i$: Estimated standard deviation of asset $i$.
- $\hat{\mathbf{\Sigma}}$: Estimated covariance matrix.
- $\hat{\mathbf{\Sigma}}_{ii}$: $i$-th diagonal entry of $\hat{\mathbf{\Sigma}}$.
Algorithm
- Compute the covariance matrix with
Statistics.cov(ce, X; dims = dims, kwargs...). - Read its diagonal, take the element-wise square root, and store the result in
val. - Reshape
valto a1 × Nrow vector whendims == 1, and to anN × 1column vector otherwise.
Arguments
ce: Covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the covariance estimator.
Returns
sd::Matrix{<:Number}: The standard deviation vector. Its shape is(1, N)ifdims == 1, or(N, 1)ifdims == 2.
Examples
julia> X = [0.01 0.02; 0.03 0.04; 0.02 0.03];julia> std(Covariance(), X)1×2 Matrix{Float64}: 0.01 0.01Related
Statistics.cov — Method
Statistics.cov(ve::AbstractVarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)Always throw a MethodError. An AbstractVarianceEstimator estimates one marginal variance for each asset. It has no structure across assets, so it has no covariance matrix to return.
This method stops a recursion. Without it, cov falls through to Statistics.cov(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...), which calls cor. A variance estimator has no cor method of its own, so a generic fallback of cor calls cov again. The two calls then recurse without end. This method throws before that fallback runs.
Use a covariance estimator, for example Covariance, for a covariance matrix.
Examples
julia> try Statistics.cov(SimpleVariance(), [0.01 0.02; 0.03 0.04]) catch e e isa MethodError endtrueRelated
Statistics.cor — Method
Statistics.cor(ve::AbstractVarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)Always throw a MethodError. An AbstractVarianceEstimator estimates one marginal variance for each asset. It has no structure across assets, so it has no correlation matrix to return.
Without this method, cor falls through to a generic fallback that calls Statistics.cov(ve::AbstractVarianceEstimator, X::MatNum; dims::Int = 1, kwargs...), and that method throws a MethodError. This method throws the same kind of error one call before. The error names cor, which is the verb that the caller used.
Use a covariance estimator, for example Covariance, for a correlation matrix.
Examples
julia> try Statistics.cor(SimpleVariance(), [0.01 0.02; 0.03 0.04]) catch e e isa MethodError endtrueRelated
Statistics.std — Method
Statistics.std(ve::AbstractVarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)Compute the standard deviation vector as the element-wise square root of the variance vector.
This method is the fallback for an AbstractVarianceEstimator that defines only Statistics.var. It computes std from var, so it does not call Statistics.cov, which throws for an AbstractVarianceEstimator. A member that computes its own standard deviation overrides this method. SimpleVariance is an example.
Mathematical definition
\[\begin{align} \hat{\sigma}_i &= \sqrt{\hat{\sigma}_i^2}\,. \end{align}\]
Where:
- $\hat{\sigma}_i$: Estimated standard deviation of asset $i$.
Algorithm
- Compute the variance vector with
Statistics.var(ve, X; dims = dims, kwargs...). - Take the element-wise square root.
Arguments
ve: Variance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.kwargs...: More keyword arguments for the variance estimator.
Returns
res::ArrNum: Variance or standard deviation vector ofX, with the shape that matches the dimension of the computation.
Related
Statistics.cov — Method
Statistics.cov(ce::AbstractCovarianceEstimator; kwargs...)
Statistics.cor(ce::AbstractCovarianceEstimator; kwargs...)
Statistics.var(ve::AbstractVarianceEstimator)
Statistics.std(ve::AbstractVarianceEstimator)Reads a moment from the partial-fit state that a covariance or variance estimator holds.
This method is the call with no data for an estimator that has no such call of its own. It reads the state from the cache field with partial_fit_cache. Then it gives the state to the method of the same verb that takes a state. For a SampleBufferState, which Online seeds, that method runs the batch verb over the rows of the buffer. A family that folds exactly has its own call with no data, so this method serves the estimators that fold only through a buffer, for example GerberCovariance and WindowedCovariance.
Arguments
ce: Covariance estimator.ve: Variance estimator.kwargs...: The keyword arguments for the method that reads the state, for examplestrict.
Validation
- The estimator holds a state. An
ArgumentErroris thrown otherwise.
Returns
sigma::MatNum: The covariance matrix forcov, or the correlation matrixrhoforcor.sigma::ArrNum: The variance forvar, or the standard deviation forstd, with one entry for each asset.
Related
PortfolioOptimisers.effective_observation_count — Function
effective_observation_count(ce::StatsBase.CovarianceEstimator, X::MatNum; kwargs...) -> Real
effective_observation_count(me::AbstractExpectedReturnsEstimator, X::MatNum; kwargs...) -> Real
effective_observation_count(ce, state::AbstractPartialFitState = partial_fit_cache(ce);
kwargs...) -> Real
effective_observation_count(ce, state::SampleBufferState; kwargs...) -> RealReturn the number of observations that the moment of ce or of me is worth, as one number for the whole moment. The moment is a covariance, a mean, a coskewness or a cokurtosis.
The denoising step of MatrixProcessing reads the ratio T / N of the sample. An unweighted moment reads every row of X with the same weight, so T is size(X, 1). A weighted moment is worth fewer observations than X has rows. A denoiser that reads size(X, 1) then gets a ratio that is too large, and it removes too little noise. So PortfolioOptimisersCovariance reads T from this function.
EmpiricalPrior reads the count of its mean estimator and of its covariance estimator, and states them in the field eoc of its result. A normal uncertainty set, a calibration rule and the denoiser of a matrix that a prior assembles read that field.
The form with X is the batch fit. Its default is size(X, 1), so an unweighted estimator pays nothing. It takes the keywords of the fit, such as the mean of a covariance, so a count that reads the data reads the same values as the fit. A method passes them where its fit passes them: to its inner estimator, to get_observation_weights, or to the centring. The form with no data is the fit from a fold. It reads the state of ce with partial_fit_cache, and the form with a state reads the count from it. The default of a state is the folded row count of observation_count. A SampleBufferState holds the observations, so it reads the batch form over the rows of the buffer, with the masks and the implied volatilities of the buffer, as the read of the moment does. The form with no data takes the keywords of the read of the moment too.
The count covers every row of X, the rows with a NaN too, as size(X, 1) does. It does not count the rows of each asset.
Interfaces
An estimator that weights its observations adds a method of the batch form. That method takes kwargs..., because a caller passes the keywords of the fit to it. A method of the state form takes kwargs... for the same reason. A method with no keywords raises a MethodError when the caller passes a keyword, so a method that a user wrote for an earlier version of the library must add kwargs.... A method of the state form is necessary when the estimator folds into a state that holds no observations.
Mathematical definition
\[\begin{align} T_{e} &= \dfrac{\left(\sum_{t} w_{t}\right)^{2}}{\sum_{t} w_{t}^{2}}\,. \end{align}\]
Where:
- $T_{e}$: Effective sample size, which is Kish's when the observation weights are stated.
- $w_{t}$: Observation weight of observation $t$.
The count is $T$ under equal weights. Frequency weights count replicated observations, so their count is $\sum_{t} w_{t}$.
Algorithm
The method that Julia selects is the algorithm.
- On a covariance, mean, coskewness or cokurtosis estimator with no method of its own, return
size(X, 1). - On
Covariance,GeneralCovarianceandDistanceCovariance, return the count of the observation weights withobservation_weight_count. ACovariancewith no weights reads the count of its inner estimator, because the inner estimator can hold weights of its own.SimpleExpectedReturns,MedianExpectedReturns,CoskewnessandCokurtosisdo the same with their weights.ExpWeightedExpectedReturnsreturns the Kish count of its weights over the rows ofX, fromexp_weighted_kish_count. - On
GerberIQCovariance, return the count that the rulece.ocstates for the temporal weights of the decay, with one method for each form ofce.oc. The keywords go to the centring, so ameanof the caller centres the returns that a rule of the decay reads.ce.oc = nothinggivessize(X, 1). - On
ExpWeightedCovariance, return the count ofexp_weighted_covariance_countover the rows ofX. OnRegimeAdjustedExpWeightedCovariance, do the same with the decay and the centring of its correlation. The denoising step processes the correlation, so it reads the count of the correlation. - On
WindowedCovariance,WindowedExpectedReturns,WindowedCoskewnessandWindowedCokurtosis, apply the window and the weights of the wrapper withwindowed_preamble. Then return the count of the inner estimator over the rows that the window keeps. - On
PortfolioOptimisersCovariance,CorrelationCovariance,GeodesicShrinkageCovarianceandImpliedVolatility, return the count of the inner estimator, because these estimators passXto it without change.ExcessExpectedReturnsandShrunkExpectedReturnsreturn the count of their mean estimatorme.me, andEquilibriumExpectedReturns,StandardDeviationExpectedReturnsandVarianceExpectedReturnsreturn the count of their covariance estimatorme.ce.PriorExpectedReturnsfits its prior, and returns the count of the mean of the prior result frommoment_observation_count. The form with no data reads the folded prior. - With no data, read the state of
ce, and return the count of the state. A composite that keeps no state of its own reads its inner estimator, asobservation_countdoes. A windowed estimator under aSeedWindowreads its inner estimator too. The two exponentially weighted states return the count of step 4 over the folded rows.
Arguments
ce: Covariance estimator.me: The expected returns estimator.X: The data matrix, of shapeobservations × assets.kwargs...: The keywords of the batch fit, or of the read of the folded moment. A method that reads no data ignores them.state: The partial-fit state ofce.
Returns
T::Real: The effective number of observations. It is anIntegerfor an unweighted estimator.
Related
References
- [9] L. Kish. Survey Sampling (John Wiley & Sons, 1965). Whole book. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.reads_active_mask — Function
reads_active_mask(ce::AbstractCovarianceEstimator) -> Bool
reads_active_mask(ce::Union{Covariance{<:Any, <:Any, <:Any, <:Any, <:CoveragePolicy},
SimpleVariance{<:Any, <:Any, <:Any, <:CoveragePolicy}}) -> Bool
reads_active_mask(ce::Union{ExpWeightedVariance, ExpWeightedCovariance}) -> Bool
reads_active_mask(ce::Union{RegimeAdjustedExpWeightedVariance,
RegimeAdjustedExpWeightedCovariance}) -> Bool
reads_active_mask(ce::Union{WindowedVariance, WindowedCovariance}) -> Bool
reads_active_mask(ce::PortfolioOptimisersCovariance) -> BoolAnswer whether ce is a mask-aware estimator: whether the Asset Panel method of its variance reads the active mask itself, and not the Coverage Universe of the window.
A plain estimator takes the reduce-and-expand root of Statistics.cov(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...), so it answers false, which is the fallback. Each mask-aware family overrides the panel method and answers true with a method in its own file: a Covariance or a SimpleVariance with a CoveragePolicy, the exponentially weighted family and the regime-adjusted family. A windowed estimator and a PortfolioOptimisersCovariance give the panel to their inner estimator, so they answer for it.
panel_series_row, covers_an_asset and assert_mask_aware_variance read the answer. A variance estimator of your own whose panel method of var and whose variance_series over an Asset Panel read the active mask adds a method that returns true. Then an EWResidualVolatility admits it as its ce, and a windowed series and a PortfolioOptimisersCovariance give it the whole window.
Arguments
ce: Covariance estimator.
Returns
flag::Bool:truefor a mask-aware estimator.
Related
References
- [9]
- L. Kish. Survey Sampling (John Wiley & Sons, New York, 1965).