Higher-Moment Partial Fit: private API

struct CoskewnessPartialFitState{__T_n, __T_mu, __T_M2, __T_M3} <: AbstractPartialFitState

Partial-fit state of a Coskewness estimator under FullMoment.

The state carries the running count, the running mean and the running second and third central co-moment accumulators. The third accumulator is the answer the estimator reports, and the second one is there because the update of the third reads it: a new observation moves the centre, and every past third co-moment is corrected by a term in the second one.

This type is an implementation detail and is not intended for direct use. partial_fit! writes it, coskewness reads it, and merge_states folds two of them.

Fields

• n: Number of observations folded into the state.

• mu: Running mean of the observations folded into the state, assets × 1.

• M2: Running second central co-moment accumulator, assets × assets. It is the sum over the observations and not the covariance, so a read-out divides it by n.

• M3: Running third central co-moment accumulator, assets × assets². It is the sum over the observations and not the coskewness, so a read-out divides it by n.

Related

• Coskewness
• AbstractPartialFitState
• CokurtosisPartialFitState
• partial_fit!
• merge_states

struct CokurtosisPartialFitState{__T_n, __T_mu, __T_M2, __T_M3, __T_M4} <: AbstractPartialFitState

Partial-fit state of a Cokurtosis estimator under FullMoment.

The state carries the running count, the running mean and the running second, third and fourth central co-moment accumulators. The fourth accumulator is the answer the estimator reports. The second and the third are there because the update of the fourth reads both: a new observation moves the centre, and every past fourth co-moment is corrected by terms in the third and in the second.

M4 is assets² × assets², so the state is the largest object in the incremental seam. At 100 assets it holds 10⁸ entries, which is 800 MB in double precision. The seam trades that memory for the recomputation it removes, and a caller who cannot pay it runs the batch verb instead.

This type is an implementation detail and is not intended for direct use. partial_fit! writes it, cokurtosis reads it, and merge_states folds two of them.

Fields

• n: Number of observations folded into the state.

• mu: Running mean of the observations folded into the state, assets × 1.

• M2: Running second central co-moment accumulator, assets × assets. It is the sum over the observations and not the covariance, so a read-out divides it by n.

• M3: Running third central co-moment accumulator, assets × assets². It is the sum over the observations and not the coskewness, so a read-out divides it by n.

• M4: M4: Running fourth central co-moment accumulator, assets² × assets². It is the sum over the observations and not the cokurtosis, so a read-out divides it by n.

Related

• Cokurtosis
• AbstractPartialFitState
• CoskewnessPartialFitState
• partial_fit!
• merge_states

comoment_block(
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> NTuple{5, Any}

Builds the count, the mean and the central co-moment accumulators of one block of observations.

The accumulators are the numerators of the batch verbs: M2 divided by n is the covariance the batch route computes, M3 divided by n is the coskewness tensor, and transpose(z) * z divided by n is the cokurtosis matrix. The block of one observation has a mean equal to that observation and three accumulators of zero, which is what makes one partial_fit! per row the Welford update.

A pair of assets $(p, q)$ is one column of the third accumulator and one row or column of the fourth, at position $(p - 1) N + q$, with p the slow index and q the fast one. That is the ordering kron(o, Y) ⊙ kron(Y, o) produces, so nothing is transposed between the incremental route and the batch one.

Algorithm

1. Take the mean of X over its observations, giving mu.
2. Subtract mu from every observation, giving the deviation matrix Y.
3. Form z, whose column $(p - 1) N + q$ is the elementwise product of columns p and q of Y.
4. Return n, mu, transpose(Y) * Y, transpose(Y) * z and z.

Arguments

• X: Block of observations (observations × assets).

Returns

• (n, mu, M2, M3, z)::Tuple: The count, the mean, the second and third accumulators, and the pairwise deviation product z, which the fourth accumulator squares.

Related

• shift_comoment3
• shift_comoment4
• partial_fit!

shift_comoment3(
    m::Number,
    a::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M2::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M3::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Any

Moves the third central co-moment accumulator of one block from its own mean to a shifted centre.

This is the whole mathematics of the higher-moment incremental fit, and both partial_fit! and merge_states reach it. A block records its co-moments about its own mean. Two blocks folded into one have a common mean that is neither of theirs, so each block's accumulator is first moved to the common centre and the two are then added. The one-observation update is the same operation with a block of one row, whose accumulators are zero.

Mathematical definition

Write $y_{ti} = x_{ti} - \mu_i$ for the deviations of the block about its own mean, and let the centre move to $\mu + a$. The deviations become $y_{ti} - a_i$, and the sum of the deviations about the block's own mean is zero. Expanding the product of three shifted deviations and dropping every term that carries a lone $\sum_t y_t$ gives

\[\begin{align} M_{3, kpq}(\mu + a) &= M_{3, kpq} - a_k M_{2, pq} - a_p M_{2, kq} - a_q M_{2, kp} - m\, a_k a_p a_q\, . \end{align}\]

Where:

• $m$: number of observations in the block.
• $a$: displacement of the centre.
• $M_2$, $M_3$: second and third central co-moment accumulators of the block about its own mean.

The three correction terms read the old accumulator $M_2$, so a caller shifts the third accumulator before it overwrites the second one. The scalar case $N = 1$ is the univariate Welford recursion $M_3 \leftarrow M_3 - 3 a M_2 - m a^3$.

Algorithm

1. Form the pairwise product az of the displacement with itself, ordered as in comoment_block.
2. Take the three correction terms in the second accumulator, one for each index of the third.
3. Subtract them, and subtract the displacement cubed scaled by the block count.

Arguments

• m: Number of observations in the block.
• a: Displacement of the centre, assets × 1.
• M2: Second central co-moment accumulator of the block, about its own mean.
• M3: Third central co-moment accumulator of the block, about its own mean.

Returns

• M3::MatNum: The third accumulator of the same block about the shifted centre.

Related

• shift_comoment4
• comoment_block
• merge_states

shift_comoment4(
    m::Number,
    a::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M2::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M3::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    M4::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Any

Moves the fourth central co-moment accumulator of one block from its own mean to a shifted centre.

The companion of shift_comoment3, derived from the same expansion carried one order further. It reads the third and the second accumulator of the block, so a caller shifts the fourth one first.

Mathematical definition

With the notation of shift_comoment3, expanding the product of four shifted deviations and dropping every term that carries a lone $\sum_t y_t$ gives

\[\begin{align} M_{4, pqrs}(\mu + a) &= M_{4, pqrs} - \left(a_p M_{3, qrs} + a_q M_{3, prs} + a_r M_{3, spq} + a_s M_{3, rpq}\right)\\ &+ \left(a_p a_q M_{2, rs} + a_p a_r M_{2, qs} + a_p a_s M_{2, qr} + a_q a_r M_{2, ps} + a_q a_s M_{2, pr} + a_r a_s M_{2, pq}\right)\\ &+ m\, a_p a_q a_r a_s\, . \end{align}\]

Where:

• $m$: number of observations in the block.
• $a$: displacement of the centre.
• $M_2$, $M_3$, $M_4$: second, third and fourth central co-moment accumulators of the block about its own mean.

The four third-order terms are one per index, and the six second-order terms are one per pair of indices. Both accumulators are fully symmetric in their indices, so each term is one Kronecker product of the displacement with the accumulator. The scalar case $N = 1$ is the univariate Welford recursion $M_4 \leftarrow M_4 - 4 a M_3 + 6 a^2 M_2 + m a^4$.

Algorithm

1. Form the pairwise product az of the displacement with itself, and the outer product aa.
2. Subtract the four third-order terms, one for each index of the fourth accumulator.
3. Add the six second-order terms, one for each pair of indices.
4. Add the displacement to the fourth power, scaled by the block count.

Arguments

• m: Number of observations in the block.
• a: Displacement of the centre, assets × 1.
• M2: Second central co-moment accumulator of the block, about its own mean.
• M3: Third central co-moment accumulator of the block, about its own mean.
• M4: Fourth central co-moment accumulator of the block, about its own mean.

Returns

• M4::MatNum: The fourth accumulator of the same block about the shifted centre.

Related

• shift_comoment3
• comoment_block
• merge_states

assert_partial_fittable(
    me::AbstractExpectedReturnsEstimator,
    w::Union{Nothing, DynamicAbstractWeights, AbstractWeights},
    name::AbstractString
)

Refuses an estimator whose configuration no incremental fit reproduces, naming the field that refuses it.

A partial fit keeps the running sample mean and the central co-moments about it, and nothing else of the sample. Two configuration choices break that, and both are refused rather than approximated.

• Observation weights. A weights vector describes a sample of a fixed length, and a DynamicAbstractWeights derives every weight from the length of the sample. Either way a new observation reweights every past one, so no state written before it is still valid.
• A mean estimator that is not the plain sample mean. The state carries the running sample mean, so an estimator whose centre is a shrunk, a median or a custom value would be centred on a quantity the state does not hold.

An estimator that carries no centring field hands over nothing, which the Nothing method reads as SimpleExpectedReturns, because that is the centre such an estimator takes.

Arguments

• me: Expected returns estimator of the estimator being fitted.
• w: Observation weights of the estimator being fitted.
• name: Name of the estimator, used in the message.

Validation

• w is nothing. An ArgumentError is thrown otherwise.
• me is a SimpleExpectedReturns whose own w is nothing. An ArgumentError is thrown otherwise.

Returns

• nothing.

Related

• partial_fit!
• SimpleExpectedReturns

assert_partial_fittable(
    _::Nothing,
    w::Union{Nothing, DynamicAbstractWeights, AbstractWeights},
    name::AbstractString
)

Nothing method of assert_partial_fittable. An estimator that carries no centring field, and one whose nothing selects SimpleExpectedReturns, both centre on the plain sample mean, so the pair is checked against that estimator and only the weights can refuse it.

copy(
    x::CoskewnessPartialFitState
) -> CoskewnessPartialFitState

Copies a CoskewnessPartialFitState, so the copy shares no array with the original.

The copy method of the AbstractPartialFitState interface. The count is a scalar and passes through, and the running mean and the two accumulators are copied. This family overrides partial_fit, so the seam's own value form never calls this method on it, and a caller who copies a state by hand still gets one.

Arguments

• x: The state to copy.

Returns

• state::CoskewnessPartialFitState: A fresh state, equal to x, whose mu, M2 and M3 are fresh arrays.

Related

• CoskewnessPartialFitState
• partial_fit
• AbstractPartialFitState

copy(
    x::CokurtosisPartialFitState
) -> CokurtosisPartialFitState

Copies a CokurtosisPartialFitState, so the copy shares no array with the original.

The copy method of the AbstractPartialFitState interface. The count is a scalar and passes through, and the running mean and the three accumulators are copied. M4 is assets² × assets², so this method is the expensive one of the seam, and it is why the family overrides partial_fit rather than paying the copy on every fold.

Arguments

• x: The state to copy.

Returns

• state::CokurtosisPartialFitState: A fresh state, equal to x, whose mu, M2, M3 and M4 are fresh arrays.

Related

• CokurtosisPartialFitState
• partial_fit
• AbstractPartialFitState