GitHub

Partial fit: private API

PortfolioOptimisers.assert_mergeable_states — Function
assert_mergeable_states(
    a::AbstractPartialFitState,
    b::AbstractPartialFitState
)

Refuses two partial-fit states that cannot be merged, naming the reason.

Two states merge only when they describe the same estimate over the same assets, so this check holds for every state family: the two operands are the same struct, and every array field they hold has the same shape. A family whose merge needs more than that adds a method of its own that calls this one first. An exponentially weighted pair is the case that needs it, because two states of different decay weight the same observation differently.

Algorithm

  1. Refuse the pair when the two operands are not the same struct, whatever their type parameters.
  2. Walk the fields of the pair in declaration order.
  3. Refuse the pair when a field holds an array in both operands and the two shapes differ.

Arguments

  • a: The state of the first block of observations.
  • b: The state of the second block of observations.

Validation

  • a and b are the same struct. An ArgumentError is thrown otherwise.
  • Every array field of a has the shape of its counterpart in b. A DimensionMismatch is thrown otherwise.

Returns

  • nothing.

Related

source
PortfolioOptimisers.chan_merge — Function
chan_merge(
    n_a::Number,
    mu_a::Union{Number, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}},
    M_a::Union{Number, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}},
    n_b::Number,
    mu_b::Union{Number, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}},
    M_b::Union{Number, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}}
) -> Tuple{Any, Any, Any}

Folds the observation count, the mean and the second-moment accumulator of two blocks into those of the concatenated block.

This is the merge of [6], and it is what makes an incremental fit parallel and associative: a sample split into any set of disjoint blocks gives the state of the whole sample, whatever order the blocks are folded in. The second moment is the accumulator $M$, not the variance, so a caller divides by the count or by the count less one after the last merge.

Mathematical definition

\[\begin{align} n &= n_A + n_B\\ \boldsymbol{\delta} &= \boldsymbol{\mu}_B - \boldsymbol{\mu}_A\\ \boldsymbol{\mu} &= \boldsymbol{\mu}_A + \boldsymbol{\delta} \frac{n_B}{n}\\ M &= M_A + M_B + \boldsymbol{\delta} \boldsymbol{\delta}^{\intercal} \frac{n_A n_B}{n}\, . \end{align}\]

Where:

  • $n_A$, $n_B$: observation counts of the two blocks.
  • $\boldsymbol{\mu}_A$, $\boldsymbol{\mu}_B$: means of the two blocks.
  • $M_A$, $M_B$: second-moment accumulators of the two blocks.
  • $\boldsymbol{\delta}$: difference of the two means.

The outer product $\boldsymbol{\delta} \boldsymbol{\delta}^{\intercal}$ becomes the elementwise square $\boldsymbol{\delta}^2$ when $M$ is per-asset rather than a co-moment matrix.

Algorithm

  1. Add the two counts, giving n.
  2. Subtract the two means, giving delta.
  3. Move the first mean along delta by the share of the total that the second block holds, giving mu.
  4. Add the two accumulators, and add the outer product of delta scaled by n_a * n_b / n, giving M.

Arguments

  • n_a: Observation count of the first block.
  • mu_a: Mean of the first block.
  • M_a: Second-moment accumulator of the first block.
  • n_b: Observation count of the second block.
  • mu_b: Mean of the second block.
  • M_b: Second-moment accumulator of the second block.

Validation

  • n_a + n_b > 0. A DomainError is thrown otherwise, because the merge divides by the total count.

Returns

  • (n, mu, M)::Tuple: The count, the mean and the accumulator of the concatenated block.

Examples

julia> PortfolioOptimisers.chan_merge(1, [1.0, 2.0], [0.0, 0.0], 1, [3.0, 6.0], [0.0, 0.0])(2, [2.0, 4.0], [2.0, 8.0])

References

  • [6] T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983).

Related

source
PortfolioOptimisers.assert_partial_fit_state — Function
assert_partial_fit_state(
    n::Integer,
    mu::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
)
assert_partial_fit_state(
    n::Integer,
    mu::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M::Union{Nothing, AbstractArray{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}}
)

Refuses a partial-fit state whose count, mean or accumulator cannot describe a sample.

Every second-order state carries the same three quantities, so one check serves the whole family: a non-negative count, a non-empty finite mean, and an accumulator whose every axis has the length of the mean. The accumulator is per-asset in a variance state and a co-moment matrix in a covariance state, and the axis rule reads both.

Algorithm

  1. Refuse a negative n.
  2. Refuse an empty or non-finite mu.
  3. Return when M is nothing, which is the mean-only state.
  4. Refuse a non-finite M, and refuse an M whose any axis does not have the length of mu.

Arguments

  • n: Number of observations folded into the state.
  • mu: Running mean of the observations folded into the state, assets × 1.
  • M: The second-moment accumulator, or nothing for a state that carries a mean alone.

Validation

  • n >= 0. A DomainError is thrown otherwise.
  • !isempty(mu). An IsEmptyError is thrown otherwise.
  • Every entry of mu is finite. An IsNonFiniteError is thrown otherwise.
  • Every entry of M is finite. An IsNonFiniteError is thrown otherwise.
  • Every axis of M has the length of mu. A DimensionMismatch is thrown otherwise.

Returns

  • nothing.

Related

source
PortfolioOptimisers.partial_fit_cache — Function
partial_fit_cache(
    est::Union{AbstractEstimator, CovarianceEstimator}
) -> Any

Returns the partial-fit state an estimator carries, and refuses an estimator that carries none.

A read-out verb called on the estimator alone reads the state out of the cache field, and that field holds nothing until the first fold. The refusal names the verb that fills it, so a caller who reached the one-argument form too early is told what to call rather than meeting a MethodError.

Arguments

  • est: The estimator whose cache field carries the state.

Validation

  • est.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

  • state::AbstractPartialFitState: The state the estimator carries.

Related

source
PortfolioOptimisers.observation_count — Function
observation_count(
    est::Union{AbstractEstimator, CovarianceEstimator}
) -> Any

Returns the number of observations an estimator has folded into its partial-fit state.

Every state of the seam records the count of observations it has seen in its n field, whatever else it accumulates, because the count is what turns a running sum into a moment. This verb reads it, so a read-out that needs the sample size — and not the sample — asks the state rather than the matrix that is no longer there. Statistics.cov of a PortfolioOptimisersCovariance is the case that motivated it: its denoising step reads size(X, 1) alone, and the folded count is exactly that number, NaN rows included.

Arguments

  • est: The estimator whose state carries the count.

Validation

  • est.cache is not nothing. An ArgumentError is thrown otherwise.

Returns

  • n::Integer: Number of observations folded so far.

Related

source

References

[6]
T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983).