Observation weights: private API

PortfolioOptimisers.ScenarioProbabilities — Type
const ScenarioProbabilities = Union{StatsBase.AnalyticWeights,
                                    StatsBase.ProbabilityWeights}

Union type for the probabilities of the scenarios of a prior that reweights its scenarios.

The type of the weights fixes the bias correction of a weighted variance, and corrected = true is the default of the library. Probabilities of scenarios sum to one. Under that sum, StatsBase.AnalyticWeights divides by $1 - \sum_t p_t^2$, the reliability correction $n_{eff} / (n_{eff} - 1)$ of the Kish count $n_{eff} = 1 / \sum_t p_t^2$. StatsBase.ProbabilityWeights gives $T / (T - 1)$ whatever the weights are. The other kinds give no usable correction: StatsBase.FrequencyWeights divides by $\sum_t p_t - 1 = 0$, and StatsBase.Weights refuses corrected = true.

Related

source
PortfolioOptimisers.scenario_probabilities — Function
scenario_probabilities(w::VecNum)
scenario_probabilities(w, x::VecNum)

Wrap the probabilities x of the scenarios in the kind of the weights w. The one-argument method reads w as the probabilities, and returns it unchanged when it already has a kind of ScenarioProbabilities.

A prior that reweights its scenarios returns posterior probabilities of the same kind as its prior probabilities. A caller who gives StatsBase.ProbabilityWeights thus keeps that correction. With no such weights, the probabilities are StatsBase.AnalyticWeights, whose correction is the correction of reliability weights. ScenarioProbabilities states the two corrections.

Algorithm

The method that Julia selects is the algorithm.

  1. When w is a ScenarioProbabilities, return weights of the kind of w over x. With no x, return w.
  2. When w is a different value, which includes nothing and a plain vector, return StatsBase.aweights(x). With no x, return StatsBase.aweights(w).

Arguments

  • w: The weights whose kind the result takes, or nothing.
  • x: The probabilities of the scenarios. It is w when it is not given.

Returns

  • p::ScenarioProbabilities: The weights over x.

Examples

julia> PortfolioOptimisers.scenario_probabilities(nothing, [0.25, 0.75])2-element AnalyticWeights{Float64, Float64, Vector{Float64}}: 0.25 0.75julia> PortfolioOptimisers.scenario_probabilities(pweights([0.5, 0.5]), [0.25, 0.75])2-element ProbabilityWeights{Float64, Float64, Vector{Float64}}: 0.25 0.75

Related

source
PortfolioOptimisers.observation_weight_count — Function
observation_weight_count(w::Nothing, T::Integer) -> Integer
observation_weight_count(w::StatsBase.FrequencyWeights, T::Integer) -> Number
observation_weight_count(w::AbstractVector, T::Integer) -> Number

Return the number of observations that a sample of T rows under the weights w is worth.

The function dispatches on the weight type, as simple_variance_count and glm_weights do. Frequency weights count replicated observations, so their count is their sum. Every other weight vector states the relative weight of each observation and no count, so its count is the Kish count. With no weights the count is T.

Mathematical definition

\[\begin{align} T_{e} &= \dfrac{\left(\sum_{t} w_{t}\right)^{2}}{\sum_{t} w_{t}^{2}}\,. \end{align}\]

Where:

  • $T_{e}$: Effective sample size, which is Kish's when the observation weights are stated.
  • $w_{t}$: Observation weight of observation $t$.

Arguments

  • w: The resolved observation weights, or nothing. get_observation_weights resolves them.
  • T: The number of rows of the sample.

Returns

  • T_e::Number: The effective number of observations.

Related

References

  • [9] L. Kish. Survey Sampling (John Wiley & Sons, 1965). Whole book. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.weighted_moment_guard — Function
weighted_moment_guard(val::Number, w::StatsBase.AbstractWeights, corrected::Bool) -> Number
weighted_moment_guard(val::AbstractArray, w::StatsBase.AbstractWeights,
                      corrected::Bool) -> AbstractArray

Return a weighted variance or covariance that StatsBase computed, or NaN in each entry when the divisor of the weights is not positive.

StatsBase divides a weighted sum of squares by the divisor $D$ of the weights. A corrected fit on one observation of positive weight has $D = 0$, as an unweighted corrected fit on one observation does. But the weighted mean of one observation can differ from it by one unit in the last place. So the sum of squares is about $10^{-36}$ and not zero, and the fit is Inf in place of the NaN of the unweighted fit. Frequency weights that sum to less than one give $D < 0$, and a variance below zero. A divisor that is not positive gives no variance, so the function returns NaN, as the unweighted fit on one observation does.

Mathematical definition

\[\begin{align} D &= \begin{cases} \sum_{t} w_{t} & \textrm{uncorrected}\\ \sum_{t} w_{t} - \dfrac{\sum_{t} w_{t}^{2}}{\sum_{t} w_{t}} & \textrm{corrected analytic weights}\\ \sum_{t} w_{t} - 1 & \textrm{corrected frequency weights}\\ \dfrac{\left(\sum_{t} w_{t}\right) \left(n_{+} - 1\right)}{n_{+}} & \textrm{corrected probability weights} \end{cases}\,. \end{align}\]

Where:

  • $D$: The divisor of the weighted sum of squares. It is the inverse of StatsBase.varcorrection(w, corrected).
  • $w_{t}$: Observation weight of observation $t$.
  • $n_{+}$: The number of observations of positive weight.

Arguments

  • val: The variance or the covariance matrix that StatsBase computed with w. The array method writes into it. A caller that needs a standard deviation takes the square root after the guard, because a variance below zero has no square root.
  • w: The resolved observation weights of the fit. get_observation_weights resolves them.
  • corrected: The bias correction of the fit.

Returns

  • val: val without change when $D > 0$. Otherwise NaN of the element type of val, in each entry.

Related

source
PortfolioOptimisers.observed_sample — Function
observed_sample(X::VecNum, w::Option{<:Union{ObsWeights, VecNum}},
                obs::AbstractVector{Bool} = isfinite.(X); kwargs...)
    -> (VecNum, Option{<:VecNum})
observed_sample(X::MatNum, w::Option{<:Union{ObsWeights, VecNum}},
                obs::AbstractVector{Bool}; dims::Int = 1, kwargs...)
    -> (MatNum, Option{<:VecNum})
observed_sample(X::MatNum, w::Option{<:Union{ObsWeights, VecNum}}; dims::Int = 1,
                kwargs...) -> (MatNum, Option{<:VecNum})

Drop the observations of X that obs marks false, and drop the observation weight of each of them with it.

A fit that cannot read an observation, such as an observation with a NaN return, drops it and reads the others. This function is the one rule for the weights of the observations that remain. observed_weights states the rule for each type of weights. The weights of the observed sample are the weights of the same observations in the full sample. They keep their type, and they are not scaled.

The mask of the observed sample

With no obs, the function reads the mask from X: an observation is observed when every entry of it is finite. A return series and the columns of one pair of assets use this form.

Give obs when the reason for the drop is outside X. The rows of a portfolio drop where the portfolio return X * w is not finite, and a NaN in an asset of zero weight does not always give one. A gap in a Shared Series drops the row, but a gap of one asset in X only removes that asset from the Coverage Universe. An inactive cell of the active mask can hold a finite return.

The returned values are borrowed when obs keeps every observation

When obs is true at every observation, the function returns X itself, and the weights of get_observation_weights, with no copy. The caller must not mutate them, as get_observation_weights states. When obs drops an observation, X_obs and w_obs are new arrays.

Algorithm

  1. With no obs, check dims for a matrix. Take obs as true at each observation whose entries are all finite.
  2. Check that obs has one entry for each observation of X. For a matrix, check dims first, and read the observations along dims.
  3. Keep X when obs keeps every observation. Otherwise index X with obs along the observations.
  4. Get the weights of the observed sample with observed_weights.

Arguments

  • X: The sample. A vector holds one observation in each entry. A matrix holds the observations along dims.
  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • obs: true at each observation that the fit reads, and false at each observation that it drops. With no obs, the observations whose entries are all finite.
  • dims: The dimension of a matrix X that holds the observations.
  • kwargs...: More keyword arguments for get_observation_weights, where a DynamicAbstractWeights can read them.

Validation

  • dims in (1, 2), for a matrix X, through assert_dims.
  • length(obs) is the number of observations of X. A DimensionMismatch is thrown otherwise.
  • The rules of observed_weights.

Returns

  • X_obs: The observations of X at which obs is true.
  • w_obs::Option{<:VecNum}: The observation weights of X_obs, or nothing when w is nothing.

Examples

julia> X, w = PortfolioOptimisers.observed_sample([0.01, NaN, -0.02, 0.03], fweights([2, 1, 1, 3]),                                                  [true, false, true, true]);julia> X3-element Vector{Float64}:  0.01 -0.02  0.03julia> w3-element FrequencyWeights{Int64, Int64, Vector{Int64}}: 2 1 3

Related

source
PortfolioOptimisers.assert_observation_mask — Function
assert_observation_mask(obs::AbstractVector{Bool}, T::Integer) -> Nothing

Check that the mask obs of the observed sample has one entry for each of the T observations of the sample.

Arguments

  • obs: true at each observation that the fit reads.
  • T: The number of observations of the sample.

Validation

  • length(obs) == T. A DimensionMismatch is thrown otherwise.

Returns

  • nothing.

Related

source
PortfolioOptimisers.observed_weights — Function
observed_weights(w::Nothing, obs::AbstractVector{Bool},
                 X_obs::Option{<:VecNum_MatNum} = nothing; kwargs...) -> Nothing
observed_weights(w::DynamicAbstractWeights, obs::AbstractVector{Bool},
                 X_obs::Nothing = nothing) -> DynamicAbstractWeights
observed_weights(w::DynamicAbstractWeights, obs::AbstractVector{Bool},
                 X_obs::VecNum_MatNum; kwargs...) -> VecNum
observed_weights(w::VecNum, obs::AbstractVector{Bool},
                 X_obs::Option{<:VecNum_MatNum} = nothing; kwargs...) -> VecNum

Return the observation weights of the observed sample X_obs, the observations of a sample at which obs is true. With no X_obs, return the weights that an estimator holds, cut to the same observations, before the fit.

The rule is one for every type of weights. An observation that the fit drops takes its weight with it, and the other observations keep their weights.

  • Every type drops the observation. A StatsBase.FrequencyWeights counts the copies of one observation, so the drop removes every copy.
  • The weights keep their type. The function indexes w with obs, and StatsBase returns weights of the type of w. StatsBase.varcorrection then gives the bias correction of that type on the observed sample: $\sum_t w_t - 1$ for frequency weights, $\sum_t w_t - \sum_t w_t^2 / \sum_t w_t$ for analytic weights, and $\left(\sum_t w_t\right) (n_{+} - 1) / n_{+}$ for probability weights, where $n_{+}$ counts the observed weights that are not zero. A view of w is a SubArray, which loses the type, so the function never takes one.
  • The weights are not scaled. The analytic, probability and plain corrections do not change when the weights scale, and every weighted kernel divides by $\sum_t w_t$. A frequency weight is a count, so a scale makes it wrong.
  • A DynamicAbstractWeights resolves on the observed sample. It is a rule that computes the weights from the sample, so it reads the sample that the fit reads. A decay then ages each observation by the observations that the fit reads, as ExpWeightedCovariance ages each asset on the clock of its own valid observations.

A sample with no observation of positive weight left has no value. The consumer answers NaN, and documents it. has_positive_weight answers the test.

Algorithm

The method that Julia selects on the type of w is the algorithm.

  1. nothing: return nothing. The fit stays unweighted.
  2. A DynamicAbstractWeights with an X_obs: resolve it on X_obs with get_observation_weights, and pass kwargs... to it.
  3. A DynamicAbstractWeights with no X_obs: return w. It resolves later on the sample that the fit reads, which is the observed sample.
  4. A weight vector: check that it has one entry for each entry of obs. Return w itself when obs keeps every observation, and the weights of the same type at the kept observations otherwise, through nothing_scalar_array_getindex. That function also keeps the type of weights that store a range. The method ignores X_obs.

The form with no X_obs is the rule for weights that an estimator holds in a field, before it reads a sample. shared_series_estimator calls it on each weights field of an estimator tree, through obs_weights_index, when a Shared Series gap drops observations from the returns data.

Arguments

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • obs: true at each observation of the full sample that the fit reads.
  • X_obs: The observed sample, or nothing before the fit. A dynamic weight reads it. The other methods ignore it.
  • kwargs...: More keyword arguments for get_observation_weights. A matrix sample passes dims here.

Validation

  • A weight vector has length(w) == length(obs). A DimensionMismatch that names w is thrown otherwise.

Returns

  • w_obs::Option{<:ObsWeights}: The weights of the observed sample, of the type of w, or nothing. A DynamicAbstractWeights with no X_obs returns itself.

Examples

julia> PortfolioOptimisers.observed_weights(aweights(1.0:4.0), [true, false, true, true])3-element AnalyticWeights{Float64, Float64, Vector{Float64}}: 1.0 3.0 4.0

Related

source
PortfolioOptimisers.has_positive_weight — Function
has_positive_weight(w::Nothing, T::Integer) -> Bool
has_positive_weight(w::AbstractVector, T::Integer) -> Bool

Answer whether a sample of T observations under the weights w holds an observation of positive weight.

A sample with no such observation has no weighted value: a weighted mean divides zero by zero. A fit on an observed sample, see observed_sample, answers NaN when this function answers false, and documents it.

Algorithm

The method that Julia selects on the type of w is the algorithm.

  1. nothing: every observation weighs one, so answer T > 0.
  2. A weight vector: answer whether an entry of w is above zero.

Arguments

  • w: The resolved observation weights of the sample, or nothing. observed_weights resolves them.
  • T: The number of observations of the sample.

Returns

  • flag::Bool: true when an observation of the sample has a positive weight.

Related

source

References

[9]
L. Kish. Survey Sampling (John Wiley & Sons, New York, 1965).