Partial fit
An incremental fit folds one observation into an estimate without reading the sample again. partial_fit! is the verb each family writes, partial_fit is the value form that folds a copy of the state, its running quantities live in a AbstractPartialFitState, and merge_states combines the states of two disjoint blocks of observations into the state of the concatenated block.
PortfolioOptimisers.partial_fit! — Function
partial_fit!(est, X)Folds observations into an estimator's partial-fit state, and returns the estimator.
An incremental fit reads each observation once and keeps what it needs in an AbstractPartialFitState, so a later call continues where the last one stopped instead of reading the sample again. The state lives in the estimator's cache field, which holds nothing until the first call. That field is the one Result an estimator holds.
Reading the sample once is the advertised benefit; numerical accuracy is the real one. Every state accumulates by Welford's recursion at the observation and by the merge of [6] at the block, and both centre each increment on the running mean rather than differencing two large sums. On a sample whose mean dwarfs its spread — prices rather than returns — a textbook accumulator of squares loses most of its significant digits to cancellation, while these recursions hold machine precision. test/test_08r_partial_fit.jl pins that on a sample of mean 1000 and unit spread, so a later simplification to the textbook formula fails loudly instead of quietly returning a worse answer.
This is the method each family writes, and it is the family's cheapest exact fold. It writes into the array fields of the state where it can, and it rebinds the scalar fields with Accessors.@reset. So it returns a new estimator, and the caller must rebind it.
The verb promises nothing about an estimator the caller kept from before the call, and that is what the ! in the name says. Three pitfalls follow from it. partial_fit is the verb that has none of them, because it folds a copy of the state.
- A kept estimator holds a state whose arrays moved and whose count did not, so it reads neither the old sample nor the new one.
- Two folds that start from one warm estimator write into the same arrays, so they contaminate each other.
- A family whose fold builds a fresh state leaves the kept estimator valid by accident, and no caller may rely on that.
A batch verb ignores the state. var(ce, X) fits X alone, so an estimator carrying a state still answers any input it is given.
Interfaces
A family that answers this verb implements two methods:
partial_fit!(est, X::MatNum; dims::Int = 1, kwargs...) -> est: Folds every observation ofX, in order.partial_fit!(est, x::VecNum; kwargs...) -> est: Folds one observation, whose entries are the assets.
Arguments
est: Estimator whose state is folded forward.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets.
Returns
est: The estimator, with itscachefield rebound to the state after the last observation.
References
- [6] T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983).
Related
PortfolioOptimisers.partial_fit — Function
partial_fit(est, args...; kwargs...)Folds observations into a copy of an estimator's partial-fit state, and returns the estimator that carries the copy.
This is the value form of partial_fit!, and it is the pair matrix_processing makes with matrix_processing!. The estimator handed over is untouched, and so is the state it carries, so two folds that start from one warm estimator cannot contaminate each other.
One generic method serves the whole seam, so a family writes no method for it. A family whose fold builds a fresh state in either verb overrides it, to skip a copy that nothing reads.
Interfaces
A family that overrides this verb implements the two methods the family already answers under partial_fit!:
partial_fit(est, X::MatNum; dims::Int = 1, kwargs...) -> est: Folds every observation ofX, in order.partial_fit(est, x::VecNum; kwargs...) -> est: Folds one observation, whose entries are the assets.
Arguments
est: Estimator whose state is folded forward.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets.
Returns
est: A new estimator, whose states carry the fold after the last observation.
Related
PortfolioOptimisers.partial_fit — Method
partial_fit(
est::Union{AbstractEstimator, CovarianceEstimator},
args...;
kwargs...
) -> Any
Generic method of partial_fit. Copies every state the estimator tree carries, and folds the observations into the copies.
Every family of the seam reaches this method, because the copy and the fold are the same two steps whatever the state holds. The copy is copy_states, the walk Resume takes at entry: it copies the estimator's own cache when it holds a state, descends into every estimator-valued field, and rebuilds each host whose fields moved. So a host that holds no cache field of its own and folds through the states of its members — a HighOrderPriorEstimator folds its pe, ske and kte, a hierarchical optimiser folds its opt.pe — is served by the same method as a leaf, and the kept host's states are as untouched as a leaf's. The cost is one copy per state the tree carries, which is the order of the update itself for every second-order family.
An estimator with no incremental fit of its own carries no state, so the walk returns it as it is, and partial_fit! gives the refusal that names the wrapper.
Algorithm
- Rebuild the estimator tree around a copy of every state it carries, with
copy_states. A tree that carries none is returned as it is, because there is no state to protect and the fold seeds one of its own. - Fold the observations into the copies with
partial_fit!, and return the estimator it gives.
Arguments
est: Estimator whose state is folded forward.args...: The observations, forwarded topartial_fit!.kwargs...: Additional keyword arguments, forwarded topartial_fit!.
Returns
est: A new estimator, whose states carry the fold after the last observation.
Related
PortfolioOptimisers.merge_states — Function
merge_states(
a::AbstractPartialFitState,
b::AbstractPartialFitState
) -> ReturnsBufferState{__T_nx, __T_X, __T_nf, __T_F, __T_nb, __T_B, __T_ts, __T_pnl, __T_max_history} where {__T_nx<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_X<:Union{Nothing, SampleBufferState}, __T_nf<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_F<:Union{Nothing, SampleBufferState}, __T_nb<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_B<:Union{Nothing, SampleBufferState, AbstractVector}, __T_ts<:Union{Nothing, AbstractVector}, __T_pnl<:Union{Nothing, AssetPanel}, __T_max_history<:Union{Nothing, Integer}}
Combines two partial-fit states fitted on disjoint blocks of observations into the state of the concatenated block.
Each state family implements its own method, so this generic method is reached only by a pair no family answers. It refuses the pair that cannot merge at all, and names the method the family still owes. The verb is deliberately not Base.merge, whose contract on a Dict and on a NamedTuple is that the right operand wins a key conflict, where this merge is a sum.
Algorithm
- Refuse the pair with
assert_mergeable_states, which names a type mismatch and an asset-count mismatch. - Throw an
ArgumentErrornaming themerge_statesmethod the state's own family must implement.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.- The family of
aimplementsmerge_states. AnArgumentErroris thrown otherwise.
Returns
state::AbstractPartialFitState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::SampleBufferState,
b::SampleBufferState
) -> SampleBufferState{_A, Int64} where _A
Folds two SampleBufferState fitted on disjoint blocks into the buffer of the concatenated block.
Concatenation, which is exact: a buffer holds its observations verbatim, so the buffer of two blocks is the buffer of the rows of one followed by the rows of the other. It is the one buffer state that merges. A cap is applied to the result, which keeps the last max_history rows of the concatenation. The masks and the factor rows concatenate with the rows they explain, and two buffers that disagree about which of them they record are refused, on the reasoning assert_buffer_presence_agreement states for a fold.
assert_mergeable_states is deliberately not called. Its array-shape rule reads every array field on every axis, and a buffer's backing matrix carries the observation axis as well as the asset axis, so two buffers over the same assets and different numbers of observations would be refused by it. The width, the cap and the masks are checked here instead.
Algorithm
- Refuse two buffers of different widths, and two buffers of different caps.
- Refuse two buffers that do not record the same masks, or that do not agree on recording factor rows, or whose factor rows are of different widths.
- Concatenate the valid region of the first with the valid region of the second, and each mask and the factor rows with their own.
- Keep the last
max_historyrows when a cap is set.
Arguments
a: The buffer of the first block of observations.b: The buffer of the second block of observations.
Validation
aandbdescribe the same number of assets. ADimensionMismatchis thrown otherwise.aandbcarry the same cap. AnArgumentErroris thrown otherwise.aandbrecord the same masks, and both record factor rows or neither does. AnArgumentErroris thrown otherwise.aandbdescribe the same number of factors, when they record factor rows. ADimensionMismatchis thrown otherwise.
Returns
state::SampleBufferState: The buffer the two blocks give when they are folded as one block.
Related
merge_states(
a::ReturnsBufferState,
b::ReturnsBufferState
) -> ReturnsBufferState{__T_nx, __T_X, __T_nf, __T_F, __T_nb, __T_B, __T_ts, __T_pnl, __T_max_history} where {__T_nx<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_X<:Union{Nothing, SampleBufferState}, __T_nf<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_F<:Union{Nothing, SampleBufferState}, __T_nb<:Union{Nothing, AbstractVector{<:AbstractString}}, __T_B<:Union{Nothing, SampleBufferState, AbstractVector}, __T_ts<:Union{Nothing, AbstractVector}, __T_pnl<:Union{Nothing, AssetPanel}, __T_max_history<:Union{Nothing, Integer}}
Folds two ReturnsBufferState fitted on disjoint blocks into the state of the concatenated block.
Each column merges as its buffer merges — concatenation — and the pinned context must agree, because two runs over different universes describe no single run.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
- The pinned context of
aandbagree. AnArgumentErroris thrown otherwise.
Returns
state::ReturnsBufferState: The state the two blocks give when they are folded as one block.
Related
merge_states(
a::SimpleExpectedReturnsState,
b::SimpleExpectedReturnsState
) -> Union{SimpleExpectedReturnsState{_A, _B, Nothing} where {_A, _B}, SimpleExpectedReturnsState{_A, _B, __T_cvg} where {_A, _B, __T_cvg<:(CoverageCounts{_A, Nothing} where _A)}}
Folds two SimpleExpectedReturnsState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the counts and the means with
chan_merge, whose accumulator argument isfalse, the zero of a state that carries no accumulator. The accumulator it returns is discarded.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::SimpleExpectedReturnsState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::CovarianceState,
b::CovarianceState
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}
Folds two CovarianceState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the counts, the means and the accumulators with
chan_merge, whose outer-product method reads a co-moment accumulator.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CovarianceState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::SimpleVarianceState,
b::SimpleVarianceState
) -> Union{SimpleVarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, SimpleVarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:(CoverageCounts{_A, Nothing} where _A)}}
Folds two SimpleVarianceState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the counts, the means and the accumulators with
chan_merge, whose elementwise method reads a per-asset accumulator.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::SimpleVarianceState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::ExpWeightedExpectedReturnsState,
b::ExpWeightedExpectedReturnsState
)
Refuses to merge two ExpWeightedExpectedReturnsState.
An exponentially weighted state folds forward exactly, S = λ^{n_b} S_a + S_b, but only while no asset resets inside the second block. The state records the count that a reset zeroed and not the reset itself, so the two cases are indistinguishable after the fact and a merge would silently keep a history the reset discarded. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The pair is refused with an
ArgumentError.
Related
merge_states(
a::ExpWeightedVarianceState,
b::ExpWeightedVarianceState
)
Refuses to merge two ExpWeightedVarianceState.
An exponentially weighted state folds forward exactly, S = λ^{n_b} S_a + S_b, but only while no asset resets inside the second block. The state records the count that a reset zeroed and not the reset itself, so the two cases are indistinguishable after the fact and a merge would silently keep a history the reset discarded. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The pair is refused with an
ArgumentError.
Related
merge_states(
a::ExpWeightedCovarianceState,
b::ExpWeightedCovarianceState
)
Refuses to merge two ExpWeightedCovarianceState.
An exponentially weighted state folds forward exactly, S = λ^{n_b} S_a + S_b, but only while no asset resets inside the second block. The state records the count that a reset zeroed and not the reset itself, so the two cases are indistinguishable after the fact and a merge would silently keep a history the reset discarded. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The pair is refused with an
ArgumentError.
Related
merge_states(
a::RegimeAdjustedVarianceState,
b::RegimeAdjustedVarianceState
) -> Union{}Refuses a pair of regime-adjusted states, because this family does not merge.
A merge needs the state of a block to be a sufficient statistic for what that block contributes, and this one is not. The regime state reads each observation's standardised squared innovation, which divides by the running variance and is gated by the running observation count, so a block fitted from a cold start weighs its own first observations differently from the same block fitted after another. The exponentially weighted accumulator itself does fold, as $\lambda^{n_B} v_A + v_B$, but the regime state that scales it does not, and an uncentred fit also carries its running location while a HAC fit carries its buffer of recent returns.
Fold the second block into the first with partial_fit! instead. A sequential fit is exact, and it is the route this family gives.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states, so a mismatched pair is named as such.- The pair is then refused with an
ArgumentError, whatever it holds.
Returns
- Nothing is returned. The method always throws.
Examples
julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(RegimeAdjustedExpWeightedVariance(; decay = 0.9, min_obs = 2, regime_min_obs = 2), X);julia> try PortfolioOptimisers.merge_states(ce.cache, ce.cache) catch err err isa ArgumentError endtrueRelated
merge_states(
a::RegimeAdjustedCovarianceState,
b::RegimeAdjustedCovarianceState
)
Refuses a pair of regime-adjusted covariance states, because this family does not merge.
A block fitted from a cold start is not what the same block contributes after another one. The regime statistic scores each observation against the state that stands before it, so a cold block loses every comparison its first min_obs observations would have made, and a correlation state that is normalised by a running variance carries that variance with it.
Fold the second block into the first with partial_fit! instead. A sequential fit is exact, and it is the route this family gives.
Algorithm
- Refuse the pair with
assert_mergeable_states, which names a type mismatch and an asset-count mismatch first, as theAbstractPartialFitStateinterface asks of every family. - Throw an
ArgumentErrornaming the reason this family does not merge.
Arguments
a: The first state.b: The second state.
Validation
aandbpassassert_mergeable_states.
Returns
- Never returns. An
ArgumentErroris thrown.
Related
merge_states(
a::CoskewnessPartialFitState,
b::CoskewnessPartialFitState
) -> CoskewnessPartialFitState
Folds two CoskewnessPartialFitState fitted on disjoint blocks into the state of the concatenated block.
The count, the mean and the second accumulator take Chan's merge. The third accumulator is moved to the common mean one block at a time, with shift_comoment3, and the two shifted accumulators are added.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the count, the mean and the second accumulator with
chan_merge. - Take the displacement of each block's own mean to the common mean.
- Shift each block's third accumulator to the common mean, and add the two.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CoskewnessPartialFitState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::CokurtosisPartialFitState,
b::CokurtosisPartialFitState
) -> CokurtosisPartialFitState
Folds two CokurtosisPartialFitState fitted on disjoint blocks into the state of the concatenated block.
The companion of the CoskewnessPartialFitState merge, carried one order further. The fourth accumulator is shifted before the third, because shift_comoment4 reads the third accumulator of the block about its own mean.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the count, the mean and the second accumulator with
chan_merge. - Take the displacement of each block's own mean to the common mean.
- Shift each block's fourth accumulator to the common mean, and add the two.
- Shift each block's third accumulator to the common mean, and add the two.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CokurtosisPartialFitState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::PriorCarryState,
b::PriorCarryState
) -> PriorCarryState
Folds two PriorCarryState fitted on disjoint blocks into the state of the concatenated block.
The buffer's merge, which is concatenation, and the union of the two named-asset sets: an asset either half has already told the caller about does not need telling again.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Returns
state::PriorCarryState: The state the two blocks give when they are folded as one block.
Related
PortfolioOptimisers.obs_weights_view — Method
obs_weights_view(_::AbstractPartialFitState, _)
Drops a partial-fit state when its estimator is viewed on the observation axis.
obs_weights_view selects observations, and a state describes the observations it was fitted on. No slice of a state exists on that axis: removing an observation from a running accumulator has no numerically stable inverse, which is the reason the seam refuses a windowed estimator in the first place. So the channel drops the state rather than carrying one that answers over observations the view excluded, and the viewed estimator's read-out refuses instead of answering wrongly.
The asset axis is the other case, and it slices. port_opt_view restricts the estimator to a subset of assets, and a family whose state has an exact sub-state over that subset returns it by index copy.
Arguments
::AbstractPartialFitState: The state the estimator carries, read for its type alone.::Any: Index of the observations to keep, which no slice of a state reads.
Returns
nothing.
Related
References
- [6]
- T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983).