Simple variance: private API

The library uses the variance of each asset in several places, such as some estimators of the expected returns and of the covariance, performance analysis, and the constraints it generates. SimpleVariance computes the sample variance, and the standard deviation as its square root.

PortfolioOptimisers.show_fields — Method
show_fields(
    ve::SimpleVariance
) -> Union{Tuple{Symbol, Symbol, Symbol}, NTuple{4, Symbol}}

Renders every field of a SimpleVariance except cache, and cvg only where a policy is set.

A cache holds the state of an incremental fit. That state is not part of the configuration, and it prints under the estimator at each site that renders the estimator. Set set_show_nothing_fields!(:SimpleVariance, true) to render it. The method reads cvg from the instance and not from the type, as the method for SimpleExpectedReturns does. That method states the reason.

Arguments

  • ve: The variance estimator. The method reads only its cvg field.

Returns

  • fields::Tuple: The field names to render, which is (:me, :w, :corrected) with no policy and (:me, :w, :corrected, :cvg) with one.

Related

source
PortfolioOptimisers.simple_variance_count — Function
simple_variance_count(w::Nothing, corrected::Bool, f::AbstractVector{Bool})
simple_variance_count(w::StatsBase.FrequencyWeights, corrected::Bool,
                      f::AbstractVector{Bool})
simple_variance_count(w::StatsBase.AbstractWeights, corrected::Bool,
                      f::AbstractVector{Bool})

Return the effective count and the divisor of one column of a SimpleVariance, over the rows that f marks.

The function dispatches on the weight type. Frequency weights count replicated observations, so their count is their sum. All other weight types count with the Kish formula. A column with no gap reads the weights without change, so the usual case copies nothing. The divisor is the divisor of StatsBase, stated in the count. It follows the bias correction of each weight type, and this function holds no copy of that rule.

Mathematical definition

\[\begin{align} n &= \begin{cases} \left\lvert \mathcal{F} \right\rvert & \textrm{with no weights}\\ \sum_{t \in \mathcal{F}} w_{t} & \textrm{with frequency weights}\\ \dfrac{\left(\sum_{t \in \mathcal{F}} w_{t}\right)^{2}}{\sum_{t \in \mathcal{F}} w_{t}^{2}} & \textrm{otherwise} \end{cases}\,, \\ m &= \dfrac{n \, D}{\sum_{t \in \mathcal{F}} w_{t}}\,. \end{align}\]

Where:

  • $n$: The effective count of the observations.
  • $m$: The divisor of the variance for that count. With no weights it is $n - 1$ when corrected is true, and $n$ otherwise.
  • $\mathcal{F}$: The rows that f marks.
  • $w_{t}$: Observation weight of observation $t$.
  • $D$: The divisor of the weighted variance of StatsBase. It is the inverse of StatsBase.varcorrection(w, corrected).

Arguments

  • w: The observation weights, or nothing.
  • corrected: The corrected field of the estimator.
  • f: The rows of the column that the estimate reads.

Returns

  • cnt::VarianceCount: The effective count and the divisor, two numbers.

Related

source

Incremental fit

The sample variance can take the observations one at a time as well. partial_fit! returns a new estimator whose cache field holds the running state, and var(ve) computes the estimate from that field, with no data.

PortfolioOptimisers.SimpleVarianceState — Type
struct SimpleVarianceState{__T_n, __T_mu, __T_M, __T_cvg} <: AbstractPartialFitState

Holds the observation count, the running mean and the per-asset second-moment accumulator of an incremental variance fit.

It is the state of SimpleVariance under partial_fit!. M is the accumulator $\sum_t (r_{tj} - \hat{\mu}_j)^2$ and not the variance, so var(ve::SimpleVariance, state::SimpleVarianceState) divides it by the count, or by the count less one when ve.corrected is true.

Fields

  • n: Number of observations folded into the state.
  • mu: Running mean of the observations folded into the state, assets × 1.
  • M: Running second-moment accumulator of the observations folded into the state, about mu.
  • cvg: Optional CoverageCounts, the per-cell denominators and the per-asset records of an available-case fold. It is nothing when the estimator holds no CoveragePolicy, so the plain state costs nothing.

Constructors

SimpleVarianceState(;    n::Integer = 0,    mu::VecNum,    M::VecNum = zeros(eltype(mu), length(mu)),    cvg::Option{<:CoverageCounts} = nothing) -> SimpleVarianceState

Keywords correspond to the struct's fields. A state seeded for N assets is SimpleVarianceState(; mu = zeros(N)), which partial_fit! builds when the cache field of the estimator holds nothing.

Validation

  • n >= 0. A DomainError is thrown otherwise.
  • !isempty(mu). An IsEmptyError is thrown otherwise.
  • Every entry of mu and of M is finite. An IsNonFiniteError is thrown otherwise.
  • length(M) == length(mu). A DimensionMismatch is thrown otherwise.

The positional constructor runs each check except the scan for finite values, see assert_partial_fit_state, so a fold and a lens write run them too. The keyword constructor alone runs the scan, see assert_partial_fit_values.

View parameters

port_opt_view takes the view of its fields at the selected assets:

Examples

julia> PortfolioOptimisers.SimpleVarianceState(; mu = [0.0, 0.0])PortfolioOptimisers.SimpleVarianceState    n ┼ Int64: 0   mu ┼ Vector{Float64}: [0.0, 0.0]    M ┼ Vector{Float64}: [0.0, 0.0]  cvg ┴ nothing

Related

source
PortfolioOptimisers.variance_state_seed — Function
variance_state_seed(cache::Nothing, x::VecNum,
                    cvg::Option{<:CoveragePolicy} = nothing) -> SimpleVarianceState
variance_state_seed(cache::SimpleVarianceState, x::VecNum,
                    cvg::Option{<:CoveragePolicy} = nothing) -> SimpleVarianceState

Returns the SimpleVarianceState that an incremental variance fit folds into. If the estimator holds no state, the function seeds a state of zeros.

The seed is in this function and not in partial_fit!, so the fold is one line, and the dispatch on the type of the cache field is in one place.

Arguments

  • cache: The state that the estimator holds, or nothing.
  • x: One observation, assets × 1. The function reads only its length and its element type.
  • cvg: The policy that the estimator holds. It decides if the seed holds per-asset counts.

Returns

  • state::SimpleVarianceState: The state that cache holds, or a state of zeros over length(x) assets.

Related

source
Base.copy — Method
copy(x::SimpleVarianceState) -> SimpleVarianceState

Copies a SimpleVarianceState, so the copy shares no array with the original.

This method is the copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. The count is a scalar, and the method uses it without change. The method copies the running mean, the per-asset accumulator and the coverage counts.

Arguments

  • x: The state to copy.

Returns

  • state::SimpleVarianceState: A new state, equal to x, whose mu and M are new vectors.

Related

source

Available-case fit

With a CoveragePolicy in its cvg field, the estimator computes the variance of each asset from the rows where that asset's return is finite and the asset is active. Without a policy, it computes the ordinary sample variance over the assets that have no gap in the window. PortfolioOptimisers.coverage_variance selects one of the two fits from the type of the cvg field.

PortfolioOptimisers.coverage_variance — Function
coverage_variance(f, ve, cvg, me, X; dims::Int = 1, mean = nothing,
                  active_mask = nothing, kwargs...) -> ArrNum
coverage_variance(f, ve, cvg, X::VecNum; mean = nothing) -> Number

Routes a dispersion fit to the Coverage Universe arm or to the available-case arm.

The cvg field of the estimator is the third argument, so dispatch on the policy selects the arm, not a branch on its value. coverage_mean does the same for a sample mean. f is Statistics.var or Statistics.std. The available-case arm computes a variance, and coverage_moment_map changes it to the result that f asks for.

Arguments

  • f: Statistics.var or Statistics.std.
  • ve: The variance estimator.
  • cvg: The policy that the estimator holds. It selects the arm.
  • me: The estimator that centres the fit.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • mean: A precomputed centre, or nothing.
  • active_mask: The active mask of the Asset Panel, observations × assets, or nothing. The Coverage Universe arm refuses an inactive cell, as it refuses a return that is not finite.
  • kwargs...: More keyword arguments for the centring estimator and for the observation weights.

Returns

  • val: The dispersion, with the shape that f gives.

Related

source
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
    f,
    ve::SimpleVariance,
    ::Nothing,
    me::AbstractExpectedReturnsEstimator,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

The Nothing method of coverage_variance is the Coverage Universe arm, which the SimpleVariance methods of Statistics.std and Statistics.var reach when ve.cvg is nothing. A cell is a gap when its return is not finite or when active_mask is false at that cell, so the arm refuses an inactive cell too.

Algorithm

The matrix method, this one:

  1. Check that dims is 1 or 2. Refuse a gapped sample with assert_finite_sample.
  2. Get the centring vector mu from me and ve.w with weighted_centre. It uses mean if the caller gave one.
  3. Get the observation weights w from ve.w and X with get_observation_weights.
  4. When w is nothing, call f(X; dims = dims, corrected = ve.corrected, mean = mu).
  5. Otherwise call Statistics.var(X, w, dims; corrected = ve.corrected, mean = mu). Write NaN in each entry when the divisor of w is not positive, with weighted_moment_guard. Then change the variance to the result of f with coverage_moment_map.

The vector method, the Nothing method of the vector arm:

  1. Refuse a gapped series with assert_finite_sample. Get the observation weights w from ve.w and X.
  2. When w is nothing, call f(X; corrected = ve.corrected, mean = mean).
  3. Otherwise call Statistics.var(X, w; corrected = ve.corrected, mean = mean). Return NaN when the divisor of w is not positive, with weighted_moment_guard. Then change the variance to the result of f with coverage_moment_map.

A corrected fit on one observation of positive weight has a divisor of zero. Its result is then NaN, as the unweighted fit on one observation is, and not the Inf that the rounding of the weighted mean gives. The guard reads the variance before the square root, because a divisor below zero gives a variance below zero. StatsBase takes the weighted standard deviation as the square root of the weighted variance, so a valid result does not change.

The two methods get one centre by two routes. The matrix method gets a centre before it calls f. weighted_centre takes that centre from me after factory writes ve.w into me. The vector method passes mean to f without change, so a mean of nothing lets f centre on the weighted mean of X.

One SimpleVariance gives the same number for a one-column matrix and for the matching vector. ve.w replaces the weights that me holds, as factory does on all other paths. Pass mean for any other centre.

weighted_centre calls factory only when ve.w is not nothing. That test is for performance and is not a second contract. ve.w is a field, so its type decides the branch. The test stops a windowed loop from a rebuild of the estimator tree of me in each window.

Related

source
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
    f,
    ve::SimpleVariance,
    cvg::CoveragePolicy,
    ::AbstractExpectedReturnsEstimator,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
    dims,
    mean,
    active_mask,
    kwargs...
) -> Any

The CoveragePolicy method of coverage_variance is the available-case arm. It fits the dispersion of each asset on the finite and active observations of that asset. The batch result is the incremental result, because the arm folds the block through partial_fit! and then reads the state.

The centre is the running mean of the fold, so the arm refuses a mean from the caller, and it does not ignore that mean without a message. An available-case fit centres each asset on the observations of that asset, and a centre fitted over the whole window is different.

Algorithm

  1. Fold each row of X into a new state with partial_fit!, together with its row of the active mask.
  2. Read the state with var(ve::SimpleVariance, state::SimpleVarianceState).
  3. Change the variance to the result that f asks for with coverage_moment_map. Orient the result as the dims of the caller sets.

Related

source
PortfolioOptimisers.coverage_variance — Method
coverage_variance(
    f,
    ve::SimpleVariance,
    cvg::CoveragePolicy,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}};
    mean
) -> Any

This is the CoveragePolicy method of the vector arm of coverage_variance. One asset has no pair, so the available-case fit over its series is the ordinary fit over its finite entries. The method compares the coverage floor with the fraction of finite entries in the series. It returns NaN if too few finite entries remain, or if the policy does not admit the asset. It refuses a mean from the caller, as the matrix method does.

Related

source
PortfolioOptimisers.coverage_moment_map — Function
coverage_moment_map(f::typeof(Statistics.var), v::Num_ArrNum) -> Num_ArrNum
coverage_moment_map(f::typeof(Statistics.std), v::Num_ArrNum) -> Num_ArrNum

Changes a variance to the result that the verb of the caller asks for.

coverage_variance fits a variance for both verbs. In the available-case arm, the state holds only a second-moment accumulator. In the weighted Coverage Universe arm, weighted_moment_guard reads the variance before the square root, because a divisor below zero gives a variance below zero, which has no square root. The standard deviation is the square root of that variance, as in StatsBase. The method dispatches on the verb and does not compare it, so the compiler makes the choice.

Arguments

  • f: Statistics.var or Statistics.std.
  • v: The variance, one number or an array.

Returns

  • val::Num_ArrNum: v without change, or its element-wise square root.

Related

source