Observation weights: private API
PortfolioOptimisers.ObsWeights — Type
const ObsWeights = Union{DynamicAbstractWeights, StatsBase.AbstractWeights}Union type for the observation weights that an estimator accepts.
It holds a subtype of DynamicAbstractWeights, which computes the weights from the data when the estimator fits. It also holds a StatsBase.AbstractWeights, which holds weights that the caller computed before.
Related
PortfolioOptimisers.ScenarioProbabilities — Type
const ScenarioProbabilities = Union{StatsBase.AnalyticWeights,
StatsBase.ProbabilityWeights}Union type for the probabilities of the scenarios of a prior that reweights its scenarios.
The type of the weights fixes the bias correction of a weighted variance, and corrected = true is the default of the library. Probabilities of scenarios sum to one. Under that sum, StatsBase.AnalyticWeights divides by $1 - \sum_t p_t^2$, the reliability correction $n_{eff} / (n_{eff} - 1)$ of the Kish count $n_{eff} = 1 / \sum_t p_t^2$. StatsBase.ProbabilityWeights gives $T / (T - 1)$ whatever the weights are. The other kinds give no usable correction: StatsBase.FrequencyWeights divides by $\sum_t p_t - 1 = 0$, and StatsBase.Weights refuses corrected = true.
Related
PortfolioOptimisers.scenario_probabilities — Function
scenario_probabilities(w::VecNum)
scenario_probabilities(w, x::VecNum)Wrap the probabilities x of the scenarios in the kind of the weights w. The one-argument method reads w as the probabilities, and returns it unchanged when it already has a kind of ScenarioProbabilities.
A prior that reweights its scenarios returns posterior probabilities of the same kind as its prior probabilities. A caller who gives StatsBase.ProbabilityWeights thus keeps that correction. With no such weights, the probabilities are StatsBase.AnalyticWeights, whose correction is the correction of reliability weights. ScenarioProbabilities states the two corrections.
Algorithm
The method that Julia selects is the algorithm.
- When
wis aScenarioProbabilities, return weights of the kind ofwoverx. With nox, returnw. - When
wis a different value, which includesnothingand a plain vector, returnStatsBase.aweights(x). With nox, returnStatsBase.aweights(w).
Arguments
w: The weights whose kind the result takes, ornothing.x: The probabilities of the scenarios. It iswwhen it is not given.
Returns
p::ScenarioProbabilities: The weights overx.
Examples
julia> PortfolioOptimisers.scenario_probabilities(nothing, [0.25, 0.75])2-element AnalyticWeights{Float64, Float64, Vector{Float64}}: 0.25 0.75julia> PortfolioOptimisers.scenario_probabilities(pweights([0.5, 0.5]), [0.25, 0.75])2-element ProbabilityWeights{Float64, Float64, Vector{Float64}}: 0.25 0.75Related
PortfolioOptimisers.observation_weight_count — Function
observation_weight_count(w::Nothing, T::Integer) -> Integer
observation_weight_count(w::StatsBase.FrequencyWeights, T::Integer) -> Number
observation_weight_count(w::AbstractVector, T::Integer) -> NumberReturn the number of observations that a sample of T rows under the weights w is worth.
The function dispatches on the weight type, as simple_variance_count and glm_weights do. Frequency weights count replicated observations, so their count is their sum. Every other weight vector states the relative weight of each observation and no count, so its count is the Kish count. With no weights the count is T.
Mathematical definition
\[\begin{align} T_{e} &= \dfrac{\left(\sum_{t} w_{t}\right)^{2}}{\sum_{t} w_{t}^{2}}\,. \end{align}\]
Where:
- $T_{e}$: Effective sample size, which is Kish's when the observation weights are stated.
- $w_{t}$: Observation weight of observation $t$.
Arguments
w: The resolved observation weights, ornothing.get_observation_weightsresolves them.T: The number of rows of the sample.
Returns
T_e::Number: The effective number of observations.
Related
References
- [9] L. Kish. Survey Sampling (John Wiley & Sons, 1965). Whole book. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.weighted_moment_guard — Function
weighted_moment_guard(val::Number, w::StatsBase.AbstractWeights, corrected::Bool) -> Number
weighted_moment_guard(val::AbstractArray, w::StatsBase.AbstractWeights,
corrected::Bool) -> AbstractArrayReturn a weighted variance or covariance that StatsBase computed, or NaN in each entry when the divisor of the weights is not positive.
StatsBase divides a weighted sum of squares by the divisor $D$ of the weights. A corrected fit on one observation of positive weight has $D = 0$, as an unweighted corrected fit on one observation does. But the weighted mean of one observation can differ from it by one unit in the last place. So the sum of squares is about $10^{-36}$ and not zero, and the fit is Inf in place of the NaN of the unweighted fit. Frequency weights that sum to less than one give $D < 0$, and a variance below zero. A divisor that is not positive gives no variance, so the function returns NaN, as the unweighted fit on one observation does.
Mathematical definition
\[\begin{align} D &= \begin{cases} \sum_{t} w_{t} & \textrm{uncorrected}\\ \sum_{t} w_{t} - \dfrac{\sum_{t} w_{t}^{2}}{\sum_{t} w_{t}} & \textrm{corrected analytic weights}\\ \sum_{t} w_{t} - 1 & \textrm{corrected frequency weights}\\ \dfrac{\left(\sum_{t} w_{t}\right) \left(n_{+} - 1\right)}{n_{+}} & \textrm{corrected probability weights} \end{cases}\,. \end{align}\]
Where:
- $D$: The divisor of the weighted sum of squares. It is the inverse of
StatsBase.varcorrection(w, corrected). - $w_{t}$: Observation weight of observation $t$.
- $n_{+}$: The number of observations of positive weight.
Arguments
val: The variance or the covariance matrix thatStatsBasecomputed withw. The array method writes into it. A caller that needs a standard deviation takes the square root after the guard, because a variance below zero has no square root.w: The resolved observation weights of the fit.get_observation_weightsresolves them.corrected: The bias correction of the fit.
Returns
val:valwithout change when $D > 0$. OtherwiseNaNof the element type ofval, in each entry.
Related
PortfolioOptimisers.observed_sample — Function
observed_sample(X::VecNum, w::Option{<:Union{ObsWeights, VecNum}},
obs::AbstractVector{Bool} = isfinite.(X); kwargs...)
-> (VecNum, Option{<:VecNum})
observed_sample(X::MatNum, w::Option{<:Union{ObsWeights, VecNum}},
obs::AbstractVector{Bool}; dims::Int = 1, kwargs...)
-> (MatNum, Option{<:VecNum})
observed_sample(X::MatNum, w::Option{<:Union{ObsWeights, VecNum}}; dims::Int = 1,
kwargs...) -> (MatNum, Option{<:VecNum})Drop the observations of X that obs marks false, and drop the observation weight of each of them with it.
A fit that cannot read an observation, such as an observation with a NaN return, drops it and reads the others. This function is the one rule for the weights of the observations that remain. observed_weights states the rule for each type of weights. The weights of the observed sample are the weights of the same observations in the full sample. They keep their type, and they are not scaled.
The mask of the observed sample
With no obs, the function reads the mask from X: an observation is observed when every entry of it is finite. A return series and the columns of one pair of assets use this form.
Give obs when the reason for the drop is outside X. The rows of a portfolio drop where the portfolio return X * w is not finite, and a NaN in an asset of zero weight does not always give one. A gap in a Shared Series drops the row, but a gap of one asset in X only removes that asset from the Coverage Universe. An inactive cell of the active mask can hold a finite return.
The returned values are borrowed when obs keeps every observation
When obs is true at every observation, the function returns X itself, and the weights of get_observation_weights, with no copy. The caller must not mutate them, as get_observation_weights states. When obs drops an observation, X_obs and w_obs are new arrays.
Algorithm
- With no
obs, checkdimsfor a matrix. Takeobsastrueat each observation whose entries are all finite. - Check that
obshas one entry for each observation ofX. For a matrix, checkdimsfirst, and read the observations alongdims. - Keep
Xwhenobskeeps every observation. Otherwise indexXwithobsalong the observations. - Get the weights of the observed sample with
observed_weights.
Arguments
X: The sample. A vector holds one observation in each entry. A matrix holds the observations alongdims.w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.obs:trueat each observation that the fit reads, andfalseat each observation that it drops. With noobs, the observations whose entries are all finite.dims: The dimension of a matrixXthat holds the observations.kwargs...: More keyword arguments forget_observation_weights, where aDynamicAbstractWeightscan read them.
Validation
dims in (1, 2), for a matrixX, throughassert_dims.length(obs)is the number of observations ofX. ADimensionMismatchis thrown otherwise.- The rules of
observed_weights.
Returns
X_obs: The observations ofXat whichobsistrue.w_obs::Option{<:VecNum}: The observation weights ofX_obs, ornothingwhenwisnothing.
Examples
julia> X, w = PortfolioOptimisers.observed_sample([0.01, NaN, -0.02, 0.03], fweights([2, 1, 1, 3]), [true, false, true, true]);julia> X3-element Vector{Float64}: 0.01 -0.02 0.03julia> w3-element FrequencyWeights{Int64, Int64, Vector{Int64}}: 2 1 3Related
PortfolioOptimisers.assert_observation_mask — Function
assert_observation_mask(obs::AbstractVector{Bool}, T::Integer) -> NothingCheck that the mask obs of the observed sample has one entry for each of the T observations of the sample.
Arguments
obs:trueat each observation that the fit reads.T: The number of observations of the sample.
Validation
length(obs) == T. ADimensionMismatchis thrown otherwise.
Returns
nothing.
Related
PortfolioOptimisers.observed_weights — Function
observed_weights(w::Nothing, obs::AbstractVector{Bool},
X_obs::Option{<:VecNum_MatNum} = nothing; kwargs...) -> Nothing
observed_weights(w::DynamicAbstractWeights, obs::AbstractVector{Bool},
X_obs::Nothing = nothing) -> DynamicAbstractWeights
observed_weights(w::DynamicAbstractWeights, obs::AbstractVector{Bool},
X_obs::VecNum_MatNum; kwargs...) -> VecNum
observed_weights(w::VecNum, obs::AbstractVector{Bool},
X_obs::Option{<:VecNum_MatNum} = nothing; kwargs...) -> VecNumReturn the observation weights of the observed sample X_obs, the observations of a sample at which obs is true. With no X_obs, return the weights that an estimator holds, cut to the same observations, before the fit.
The rule is one for every type of weights. An observation that the fit drops takes its weight with it, and the other observations keep their weights.
- Every type drops the observation. A
StatsBase.FrequencyWeightscounts the copies of one observation, so the drop removes every copy. - The weights keep their type. The function indexes
wwithobs, andStatsBasereturns weights of the type ofw.StatsBase.varcorrectionthen gives the bias correction of that type on the observed sample: $\sum_t w_t - 1$ for frequency weights, $\sum_t w_t - \sum_t w_t^2 / \sum_t w_t$ for analytic weights, and $\left(\sum_t w_t\right) (n_{+} - 1) / n_{+}$ for probability weights, where $n_{+}$ counts the observed weights that are not zero. Aviewofwis aSubArray, which loses the type, so the function never takes one. - The weights are not scaled. The analytic, probability and plain corrections do not change when the weights scale, and every weighted kernel divides by $\sum_t w_t$. A frequency weight is a count, so a scale makes it wrong.
- A
DynamicAbstractWeightsresolves on the observed sample. It is a rule that computes the weights from the sample, so it reads the sample that the fit reads. A decay then ages each observation by the observations that the fit reads, asExpWeightedCovarianceages each asset on the clock of its own valid observations.
A sample with no observation of positive weight left has no value. The consumer answers NaN, and documents it. has_positive_weight answers the test.
Algorithm
The method that Julia selects on the type of w is the algorithm.
nothing: returnnothing. The fit stays unweighted.- A
DynamicAbstractWeightswith anX_obs: resolve it onX_obswithget_observation_weights, and passkwargs...to it. - A
DynamicAbstractWeightswith noX_obs: returnw. It resolves later on the sample that the fit reads, which is the observed sample. - A weight vector: check that it has one entry for each entry of
obs. Returnwitself whenobskeeps every observation, and the weights of the same type at the kept observations otherwise, throughnothing_scalar_array_getindex. That function also keeps the type of weights that store a range. The method ignoresX_obs.
The form with no X_obs is the rule for weights that an estimator holds in a field, before it reads a sample. shared_series_estimator calls it on each weights field of an estimator tree, through obs_weights_index, when a Shared Series gap drops observations from the returns data.
Arguments
w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.obs:trueat each observation of the full sample that the fit reads.X_obs: The observed sample, ornothingbefore the fit. A dynamic weight reads it. The other methods ignore it.kwargs...: More keyword arguments forget_observation_weights. A matrix sample passesdimshere.
Validation
- A weight vector has
length(w) == length(obs). ADimensionMismatchthat nameswis thrown otherwise.
Returns
w_obs::Option{<:ObsWeights}: The weights of the observed sample, of the type ofw, ornothing. ADynamicAbstractWeightswith noX_obsreturns itself.
Examples
julia> PortfolioOptimisers.observed_weights(aweights(1.0:4.0), [true, false, true, true])3-element AnalyticWeights{Float64, Float64, Vector{Float64}}: 1.0 3.0 4.0Related
PortfolioOptimisers.has_positive_weight — Function
has_positive_weight(w::Nothing, T::Integer) -> Bool
has_positive_weight(w::AbstractVector, T::Integer) -> BoolAnswer whether a sample of T observations under the weights w holds an observation of positive weight.
A sample with no such observation has no weighted value: a weighted mean divides zero by zero. A fit on an observed sample, see observed_sample, answers NaN when this function answers false, and documents it.
Algorithm
The method that Julia selects on the type of w is the algorithm.
nothing: every observation weighs one, so answerT > 0.- A weight vector: answer whether an entry of
wis above zero.
Arguments
w: The resolved observation weights of the sample, ornothing.observed_weightsresolves them.T: The number of observations of the sample.
Returns
flag::Bool:truewhen an observation of the sample has a positive weight.
Related
References
- [9]
- L. Kish. Survey Sampling (John Wiley & Sons, New York, 1965).