Simple variance: private API
The library uses the variance of each asset in several places, such as some estimators of the expected returns and of the covariance, performance analysis, and the constraints it generates. SimpleVariance computes the sample variance, and the standard deviation as its square root.
PortfolioOptimisers.show_fields — Method
show_fields(
ve::SimpleVariance
) -> Union{Tuple{Symbol, Symbol, Symbol}, NTuple{4, Symbol}}
Renders every field of a SimpleVariance except cache, and cvg only where a policy is set.
A cache holds the state of an incremental fit. That state is not part of the configuration, and it prints under the estimator at each site that renders the estimator. Set set_show_nothing_fields!(:SimpleVariance, true) to render it. The method reads cvg from the instance and not from the type, as the method for SimpleExpectedReturns does. That method states the reason.
Arguments
ve: The variance estimator. The method reads only itscvgfield.
Returns
fields::Tuple: The field names to render, which is(:me, :w, :corrected)with no policy and(:me, :w, :corrected, :cvg)with one.
Related
PortfolioOptimisers.simple_variance_count — Function
simple_variance_count(w::Nothing, corrected::Bool, f::AbstractVector{Bool})
simple_variance_count(w::StatsBase.FrequencyWeights, corrected::Bool,
f::AbstractVector{Bool})
simple_variance_count(w::StatsBase.AbstractWeights, corrected::Bool,
f::AbstractVector{Bool})Return the effective count and the divisor of one column of a SimpleVariance, over the rows that f marks.
The function dispatches on the weight type. Frequency weights count replicated observations, so their count is their sum. All other weight types count with the Kish formula. A column with no gap reads the weights without change, so the usual case copies nothing. The divisor is the divisor of StatsBase, stated in the count. It follows the bias correction of each weight type, and this function holds no copy of that rule.
Mathematical definition
\[\begin{align} n &= \begin{cases} \left\lvert \mathcal{F} \right\rvert & \textrm{with no weights}\\ \sum_{t \in \mathcal{F}} w_{t} & \textrm{with frequency weights}\\ \dfrac{\left(\sum_{t \in \mathcal{F}} w_{t}\right)^{2}}{\sum_{t \in \mathcal{F}} w_{t}^{2}} & \textrm{otherwise} \end{cases}\,, \\ m &= \dfrac{n \, D}{\sum_{t \in \mathcal{F}} w_{t}}\,. \end{align}\]
Where:
- $n$: The effective count of the observations.
- $m$: The divisor of the variance for that count. With no weights it is $n - 1$ when
correctedistrue, and $n$ otherwise. - $\mathcal{F}$: The rows that
fmarks. - $w_{t}$: Observation weight of observation $t$.
- $D$: The divisor of the weighted variance of
StatsBase. It is the inverse ofStatsBase.varcorrection(w, corrected).
Arguments
w: The observation weights, ornothing.corrected: Thecorrectedfield of the estimator.f: The rows of the column that the estimate reads.
Returns
cnt::VarianceCount: The effective count and the divisor, two numbers.
Related
Incremental fit
The sample variance can take the observations one at a time as well. partial_fit! returns a new estimator whose cache field holds the running state, and var(ve) computes the estimate from that field, with no data.
PortfolioOptimisers.SimpleVarianceState — Type
struct SimpleVarianceState{__T_n, __T_mu, __T_M, __T_cvg} <: AbstractPartialFitStateHolds the observation count, the running mean and the per-asset second-moment accumulator of an incremental variance fit.
It is the state of SimpleVariance under partial_fit!. M is the accumulator $\sum_t (r_{tj} - \hat{\mu}_j)^2$ and not the variance, so var(ve::SimpleVariance, state::SimpleVarianceState) divides it by the count, or by the count less one when ve.corrected is true.
Fields
n: Number of observations folded into the state.
mu: Running mean of the observations folded into the state,assets × 1.
M: Running second-moment accumulator of the observations folded into the state, aboutmu.
cvg: OptionalCoverageCounts, the per-cell denominators and the per-asset records of an available-case fold. It isnothingwhen the estimator holds noCoveragePolicy, so the plain state costs nothing.
Constructors
SimpleVarianceState(; n::Integer = 0, mu::VecNum, M::VecNum = zeros(eltype(mu), length(mu)), cvg::Option{<:CoverageCounts} = nothing) -> SimpleVarianceStateKeywords correspond to the struct's fields. A state seeded for N assets is SimpleVarianceState(; mu = zeros(N)), which partial_fit! builds when the cache field of the estimator holds nothing.
Validation
n >= 0. ADomainErroris thrown otherwise.!isempty(mu). AnIsEmptyErroris thrown otherwise.- Every entry of
muand ofMis finite. AnIsNonFiniteErroris thrown otherwise. length(M) == length(mu). ADimensionMismatchis thrown otherwise.
The positional constructor runs each check except the scan for finite values, see assert_partial_fit_state, so a fold and a lens write run them too. The keyword constructor alone runs the scan, see assert_partial_fit_values.
View parameters
port_opt_view takes the view of its fields at the selected assets:
mu: Sliced to the selected indices throughport_opt_view.M: Sliced to the selected indices throughport_opt_view.
Examples
julia> PortfolioOptimisers.SimpleVarianceState(; mu = [0.0, 0.0])PortfolioOptimisers.SimpleVarianceState n ┼ Int64: 0 mu ┼ Vector{Float64}: [0.0, 0.0] M ┼ Vector{Float64}: [0.0, 0.0] cvg ┴ nothingRelated
PortfolioOptimisers.variance_state_seed — Function
variance_state_seed(cache::Nothing, x::VecNum,
cvg::Option{<:CoveragePolicy} = nothing) -> SimpleVarianceState
variance_state_seed(cache::SimpleVarianceState, x::VecNum,
cvg::Option{<:CoveragePolicy} = nothing) -> SimpleVarianceStateReturns the SimpleVarianceState that an incremental variance fit folds into. If the estimator holds no state, the function seeds a state of zeros.
The seed is in this function and not in partial_fit!, so the fold is one line, and the dispatch on the type of the cache field is in one place.
Arguments
cache: The state that the estimator holds, ornothing.x: One observation,assets × 1. The function reads only its length and its element type.cvg: The policy that the estimator holds. It decides if the seed holds per-asset counts.
Returns
state::SimpleVarianceState: The state thatcacheholds, or a state of zeros overlength(x)assets.
Related
Base.copy — Method
copy(x::SimpleVarianceState) -> SimpleVarianceState
Copies a SimpleVarianceState, so the copy shares no array with the original.
This method is the copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. The count is a scalar, and the method uses it without change. The method copies the running mean, the per-asset accumulator and the coverage counts.
Arguments
x: The state to copy.
Returns
state::SimpleVarianceState: A new state, equal tox, whosemuandMare new vectors.
Related
Available-case fit
With a CoveragePolicy in its cvg field, the estimator computes the variance of each asset from the rows where that asset's return is finite and the asset is active. Without a policy, it computes the ordinary sample variance over the assets that have no gap in the window. PortfolioOptimisers.coverage_variance selects one of the two fits from the type of the cvg field.
PortfolioOptimisers.coverage_variance — Function
coverage_variance(f, ve, cvg, me, X; dims::Int = 1, mean = nothing,
active_mask = nothing, kwargs...) -> ArrNum
coverage_variance(f, ve, cvg, X::VecNum; mean = nothing) -> NumberRoutes a dispersion fit to the Coverage Universe arm or to the available-case arm.
The cvg field of the estimator is the third argument, so dispatch on the policy selects the arm, not a branch on its value. coverage_mean does the same for a sample mean. f is Statistics.var or Statistics.std. The available-case arm computes a variance, and coverage_moment_map changes it to the result that f asks for.
Arguments
f:Statistics.varorStatistics.std.ve: The variance estimator.cvg: The policy that the estimator holds. It selects the arm.me: The estimator that centres the fit.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.mean: A precomputed centre, ornothing.active_mask: The active mask of the Asset Panel,observations × assets, ornothing. The Coverage Universe arm refuses an inactive cell, as it refuses a return that is not finite.kwargs...: More keyword arguments for the centring estimator and for the observation weights.
Returns
val: The dispersion, with the shape thatfgives.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
f,
ve::SimpleVariance,
::Nothing,
me::AbstractExpectedReturnsEstimator,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
dims,
mean,
active_mask,
kwargs...
) -> Any
The Nothing method of coverage_variance is the Coverage Universe arm, which the SimpleVariance methods of Statistics.std and Statistics.var reach when ve.cvg is nothing. A cell is a gap when its return is not finite or when active_mask is false at that cell, so the arm refuses an inactive cell too.
Algorithm
The matrix method, this one:
- Check that
dimsis1or2. Refuse a gapped sample withassert_finite_sample. - Get the centring vector
mufrommeandve.wwithweighted_centre. It usesmeanif the caller gave one. - Get the observation weights
wfromve.wandXwithget_observation_weights. - When
wisnothing, callf(X; dims = dims, corrected = ve.corrected, mean = mu). - Otherwise call
Statistics.var(X, w, dims; corrected = ve.corrected, mean = mu). WriteNaNin each entry when the divisor ofwis not positive, withweighted_moment_guard. Then change the variance to the result offwithcoverage_moment_map.
The vector method, the Nothing method of the vector arm:
- Refuse a gapped series with
assert_finite_sample. Get the observation weightswfromve.wandX. - When
wisnothing, callf(X; corrected = ve.corrected, mean = mean). - Otherwise call
Statistics.var(X, w; corrected = ve.corrected, mean = mean). ReturnNaNwhen the divisor ofwis not positive, withweighted_moment_guard. Then change the variance to the result offwithcoverage_moment_map.
A corrected fit on one observation of positive weight has a divisor of zero. Its result is then NaN, as the unweighted fit on one observation is, and not the Inf that the rounding of the weighted mean gives. The guard reads the variance before the square root, because a divisor below zero gives a variance below zero. StatsBase takes the weighted standard deviation as the square root of the weighted variance, so a valid result does not change.
The two methods get one centre by two routes. The matrix method gets a centre before it calls f. weighted_centre takes that centre from me after factory writes ve.w into me. The vector method passes mean to f without change, so a mean of nothing lets f centre on the weighted mean of X.
One SimpleVariance gives the same number for a one-column matrix and for the matching vector. ve.w replaces the weights that me holds, as factory does on all other paths. Pass mean for any other centre.
weighted_centre calls factory only when ve.w is not nothing. That test is for performance and is not a second contract. ve.w is a field, so its type decides the branch. The test stops a windowed loop from a rebuild of the estimator tree of me in each window.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
f,
ve::SimpleVariance,
cvg::CoveragePolicy,
::AbstractExpectedReturnsEstimator,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
dims,
mean,
active_mask,
kwargs...
) -> Any
The CoveragePolicy method of coverage_variance is the available-case arm. It fits the dispersion of each asset on the finite and active observations of that asset. The batch result is the incremental result, because the arm folds the block through partial_fit! and then reads the state.
The centre is the running mean of the fold, so the arm refuses a mean from the caller, and it does not ignore that mean without a message. An available-case fit centres each asset on the observations of that asset, and a centre fitted over the whole window is different.
Algorithm
- Fold each row of
Xinto a new state withpartial_fit!, together with its row of the active mask. - Read the state with
var(ve::SimpleVariance, state::SimpleVarianceState). - Change the variance to the result that
fasks for withcoverage_moment_map. Orient the result as thedimsof the caller sets.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
f,
ve::SimpleVariance,
::Nothing,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}};
mean
) -> Any
The Nothing method of the vector arm of coverage_variance is the Coverage Universe arm over the series of one asset. It refuses a gapped series with assert_finite_sample. The matrix method of this arm states the algorithm of both.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
f,
ve::SimpleVariance,
cvg::CoveragePolicy,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}};
mean
) -> Any
This is the CoveragePolicy method of the vector arm of coverage_variance. One asset has no pair, so the available-case fit over its series is the ordinary fit over its finite entries. The method compares the coverage floor with the fraction of finite entries in the series. It returns NaN if too few finite entries remain, or if the policy does not admit the asset. It refuses a mean from the caller, as the matrix method does.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
ve::SimpleVariance,
_::Nothing,
state::SimpleVarianceState
) -> Any
This is the Nothing method of coverage_variance for a state. var(ve) with no data calls it. All the assets share one count, so the result is all NaN until the count is more than the Bessel correction.
Related
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
ve::SimpleVariance,
cvg::CoveragePolicy,
state::SimpleVarianceState
) -> Any
This is the CoveragePolicy method of coverage_variance for a state. var(ve) with no data calls it. The method divides the accumulator of each asset by the count of that asset less the Bessel correction. An asset that the policy refuses is NaN.
Related
PortfolioOptimisers.coverage_moment_map — Function
coverage_moment_map(f::typeof(Statistics.var), v::Num_ArrNum) -> Num_ArrNum
coverage_moment_map(f::typeof(Statistics.std), v::Num_ArrNum) -> Num_ArrNumChanges a variance to the result that the verb of the caller asks for.
coverage_variance fits a variance for both verbs. In the available-case arm, the state holds only a second-moment accumulator. In the weighted Coverage Universe arm, weighted_moment_guard reads the variance before the square root, because a divisor below zero gives a variance below zero, which has no square root. The standard deviation is the square root of that variance, as in StatsBase. The method dispatches on the verb and does not compare it, so the compiler makes the choice.
Arguments
f:Statistics.varorStatistics.std.v: The variance, one number or an array.
Returns
val::Num_ArrNum:vwithout change, or its element-wise square root.
Related