Partial fit
An incremental fit adds new observations to an estimate without reading the earlier observations again. partial_fit! adds a block of observations to an estimator and returns the updated estimator. partial_fit does the same to a copy of the state, so the estimator you pass in keeps its old state. The running quantities of the fit are stored in an AbstractPartialFitState. merge_states combines the states of two separate blocks of observations into the state of the two blocks placed end to end.
PortfolioOptimisers.partial_fit! — Function
partial_fit!(est, X)Folds observations into an estimator's partial-fit state, and returns the estimator.
An incremental fit reads each observation once. It keeps the data that the estimate needs in an AbstractPartialFitState, so a later call starts from that state and does not read the sample again. The state is in the cache field of the estimator, which holds nothing until the first call. That field is the one Result that an estimator holds.
The states of the sample mean, variance, covariance, coskewness and cokurtosis also keep the accuracy of the estimate. Each one adds an observation by Welford's recursion and a block by chan_merge, and both centre each increment on the running mean. The textbook one-pass formula subtracts two large sums instead. Table 1 of [13] bounds the relative error of the variance by $N \kappa^2 u$ for the textbook formula, and by $N \kappa u$ for the updating formula. $N$ is the number of observations, $u$ is the unit roundoff, and $\kappa$ is about the mean divided by the standard deviation. So a price series whose mean is 1000 times its standard deviation gives the textbook formula a bound 1000 times larger.
Each family writes a method of this verb. The method writes into the array fields of the state where it can. It builds a new state with the new scalar fields through the positional constructor of the state. So it returns a new estimator, and the caller must rebind it.
The verb promises nothing about an estimator that the caller kept from before the call. The ! in the name shows this, and three faults follow from it. partial_fit has none of them, because it folds a copy of each state.
- A kept estimator holds a state whose arrays changed and whose count did not, so it describes neither the old sample nor the new one.
- Two folds that start from one estimator write into the same arrays, so each fold changes the result of the other.
- A family whose fold builds a new state leaves the kept estimator correct. No caller can rely on that.
A batch verb ignores the state. var(ce, X) fits X alone, so an estimator that holds a state still gives the estimate of any X.
Interfaces
A family that answers this verb implements two methods:
partial_fit!(est, X::MatNum; dims::Int = 1, kwargs...) -> est: Folds every observation ofX, in order.partial_fit!(est, x::VecNum; kwargs...) -> est: Folds one observation, whose entries are the assets.
A family that folds also states it through supports_partial_fit:
supports_partial_fit(est::MyEstimator) -> Bool: Returnstrue.
The fallback returns true only for an estimator whose cache holds a SampleBufferState. The carry fold of a CrossSectionalFactorPrior reads this answer. At each step, it fits a member that returns false again, over every row. Under FoldOnly, the constructor of the prior refuses such a ve.
Arguments
est: The estimator whose state the verb folds.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets.
Returns
est: The estimator, with itscachefield rebound to the state after the last observation.
Related
References
- [13] T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983). Table 1. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.partial_fit — Function
partial_fit(est, args...; kwargs...)Folds observations into a copy of the partial-fit state of an estimator, and returns the estimator that holds the copy.
This is the value form of partial_fit!, as matrix_processing is the value form of matrix_processing!. The verb does not change the estimator that it gets, or the state of that estimator. So two folds that start from one estimator do not change each other.
One generic method applies to every family of the partial_fit! interface, so a family writes no method for it. A family whose fold always builds a new state can add a method, to skip a copy that nothing reads. Coskewness and Cokurtosis add one.
Interfaces
A family that overrides this verb implements the two methods the family already answers under partial_fit!:
partial_fit(est, X::MatNum; dims::Int = 1, kwargs...) -> est: Folds every observation ofX, in order.partial_fit(est, x::VecNum; kwargs...) -> est: Folds one observation, whose entries are the assets.
Arguments
est: The estimator whose state the verb folds.X: Observations to fold. A matrix holds one observation per row whendims == 1, and one per column whendims == 2. A vector is a single observation across the assets.
Returns
est: A new estimator, whose states hold the fold after the last observation.
Related
PortfolioOptimisers.partial_fit — Method
partial_fit(
est::Union{AbstractEstimator, CovarianceEstimator},
args...;
kwargs...
) -> Any
Generic method of partial_fit. Copies every state that the estimator tree holds, and folds the observations into the copies.
Every family of the partial_fit! interface uses this method, because the copy and the fold are the same two steps for each state. The copy is copy_states, which Resume also calls at entry. It copies the cache of the estimator when that field holds a state. It goes into every field that holds an estimator, and it rebuilds each outer estimator whose fields changed.
So the method also applies to an outer estimator that folds through the states of its members. A HighOrderPriorEstimator folds its pe, ske and kte, and a hierarchical optimiser folds its opt.pe. The states of the kept outer estimator do not change, as for a leaf. The cost is one copy of each state in the tree. For a second-order family, that copy costs as much as the fold of one observation.
An estimator with no incremental fit holds no state, so the walk returns it with no change. partial_fit! then refuses it, and the message names the type of the estimator.
Algorithm
- Copy every state of the estimator tree with
copy_states, giving a rebuilt tree. A tree that holds no state comes back with no change, because the fold makes a new state. - Fold the observations into the copies with
partial_fit!, and return the estimator that it gives.
Arguments
est: The estimator whose state the verb folds.args...: The observations, forwarded topartial_fit!.kwargs...: Additional keyword arguments, forwarded topartial_fit!.
Returns
est: A new estimator, whose states hold the fold after the last observation.
Related
PortfolioOptimisers.merge_states — Function
merge_states(
a::AbstractPartialFitState,
b::AbstractPartialFitState
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}
Combines two partial-fit states fitted on disjoint blocks of observations into the state of the concatenated block.
Each state family implements its own method, so a call reaches this generic method only for a pair that no family answers. The method refuses a pair that cannot merge, and it names the method that the family must implement. The verb is not Base.merge. On a Dict and on a NamedTuple, Base.merge keeps the value of the right operand when a key is in both, and this merge adds the two states.
Algorithm
- Refuse the pair with
assert_mergeable_states, which names a type mismatch and a mismatch of the asset count. - Throw an
ArgumentErrorthat names themerge_statesmethod that the family of the state must implement.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.- The family of
aimplementsmerge_states. The function throws anArgumentErrorotherwise.
Returns
state::AbstractPartialFitState: The state that the fit of the two blocks as one block gives.
Related
merge_states(
a::SampleBufferState,
b::SampleBufferState
) -> SampleBufferState
Folds two SampleBufferState fitted on disjoint blocks into the buffer of the concatenated block.
The merge is a concatenation, and it is exact. A buffer holds its observations verbatim, so the buffer of two blocks holds the rows of the first block followed by the rows of the second. When a cap is set, the result keeps the last max_history rows of the concatenation. The masks, the factor rows, the Exogenous Series and the Panel Fields concatenate with their rows. Two buffers that disagree about what they record, or that name their series differently, are refused, for the reason that assert_buffer_presence_agreement states for a fold. A buffer that holds no observations records nothing, so the merge returns a copy of the other buffer.
The method does not call assert_mergeable_states. That check compares every array field on every axis. The backing matrix of a buffer has the observation axis as well as the asset axis, so the check refuses two buffers over the same assets with different numbers of observations. This method checks the cap, the width, the masks and the factor rows itself.
Algorithm
- Refuse two buffers of different caps.
- Return a copy of the other buffer when one buffer holds no observations.
- Refuse two buffers of different widths.
- Write the masks of a buffer that records neither mask where the other records masks, with
buffer_universe_masks. Refuse two buffers that do not record the same masks, that disagree about factor rows, a series or Panel Fields, whose factor rows have different widths, or whose series have different names. - Concatenate the valid region of the first with the valid region of the second, and each mask, the factor rows and the series with their own. Concatenate the marks of the Shared Series gaps with
merge_buffer_gaps. - Keep the last
max_historyrows when a cap is set. - Join the Panel Fields of the two buffers over the rows that the result keeps, with
panel_fields_append.
Arguments
a: The buffer of the first block of observations.b: The buffer of the second block of observations.
Validation
aandbcarry the same cap. AnArgumentErroris thrown otherwise.aandbdescribe the same number of assets, when both hold observations. ADimensionMismatchis thrown otherwise.aandbrecord the same masks, or one of them records no mask, both record factor rows or neither does, both record a series under the same names or neither does, and both record Panel Fields or neither does, when both hold observations. AnArgumentErroris thrown otherwise.aandbdescribe the same number of factors, when they record factor rows. ADimensionMismatchis thrown otherwise.
Returns
state::SampleBufferState: The buffer the two blocks give when they are folded as one block.
Related
merge_states(
a::ReturnsBufferState,
b::ReturnsBufferState
) -> ReturnsBufferState
Folds two ReturnsBufferState fitted on disjoint blocks into the state of the concatenated block.
The two states must describe one run, so their pinned context, their cap and the set of columns they keep must agree. Each column then merges as its own buffer merges, by concatenation, and the cap keeps the last max_history rows of every column, so the merged columns hold one number of observations.
Algorithm
- Refuse two states whose names or static panels differ.
- Refuse two states whose caps differ.
- Refuse two states that do not keep the same columns.
- Merge each column with
merge_column, and return the new state.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
- The pinned context of
aandbagrees. AnArgumentErroris thrown otherwise. a.max_history == b.max_history. AnArgumentErroris thrown otherwise.aandbkeep the same columns. AnArgumentErroris thrown otherwise.- Every refusal of
merge_stateson aSampleBufferStateholds for the matrix columns.
Returns
state::ReturnsBufferState: The state that the two blocks give when they are folded as one block.
Related
merge_states(
a::SimpleExpectedReturnsState,
b::SimpleExpectedReturnsState
) -> Union{SimpleExpectedReturnsState{_A, _B, Nothing} where {_A, _B}, SimpleExpectedReturnsState{_A, _B, __T_cvg} where {_A, _B, __T_cvg<:(CoverageCounts{_A, Nothing} where _A)}}
Folds two SimpleExpectedReturnsState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - If one of the two states holds no coverage counts, fold the counts and the means with
chan_merge. Its accumulator argument isfalse, which is the zero of a state that has no accumulator. Discard the accumulator that it returns. - Otherwise, fold the mean of each asset with the per-asset counts
nuof the two states. An asset with no observation in the two blocks keeps a mean of zero. Merge the coverage counts too.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::SimpleExpectedReturnsState: The state of the two blocks, as if one fit read them as one block.
Related
merge_states(
a::CovarianceState,
b::CovarianceState
) -> Union{CovarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, CovarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:CoverageCounts}}
Folds two CovarianceState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - If one of the two states holds no coverage counts, fold the counts, the means and the accumulators with
chan_merge. Its outer-product method reads a co-moment accumulator. - Otherwise, fold the centre and the accumulator of each pair with the pairwise counts
nuof the two states. A pair with no shared observation in the two blocks keeps a centre and an accumulator of zero. Read the mean of each asset from the diagonal of the centre, and merge the coverage counts too.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CovarianceState: The state of the two blocks, as if one fit read them as one block.
Related
merge_states(
a::SimpleVarianceState,
b::SimpleVarianceState
) -> Union{SimpleVarianceState{_A, _B, _C, Nothing} where {_A, _B, _C}, SimpleVarianceState{_A, _B, _C, __T_cvg} where {_A, _B, _C, __T_cvg<:(CoverageCounts{_A, Nothing} where _A)}}
Folds two SimpleVarianceState fitted on disjoint blocks into the state of the concatenated block.
Algorithm
- Refuse the pair with
assert_mergeable_states. - If one of the two states holds no coverage counts, fold the counts, the means and the accumulators with
chan_merge. Its element-wise method reads a per-asset accumulator. - Otherwise, fold the mean and the accumulator of each asset with the per-asset counts
nuof the two states. An asset with no observation in the two blocks keeps a mean and an accumulator of zero. Merge the coverage counts too.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::SimpleVarianceState: The state of the two blocks, as if one fit read them as one block.
Related
merge_states(
a::ExpWeightedExpectedReturnsState,
b::ExpWeightedExpectedReturnsState
)
Refuses to merge two ExpWeightedExpectedReturnsState.
The state of two consecutive blocks is $S = \lambda^{n_b} S_a + S_b$, where $S_a$ is the state of the first block, $S_b$ the state of the second block from a cold start, and $n_b$ the count of valid observations of the asset in the second block. That holds only when no asset resets inside the second block. The state records the count after a reset and not the reset itself, so the two cases cannot be told apart, and a merge would keep a history that the reset removed. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The pair is refused with an
ArgumentError.
Related
merge_states(
a::ExpWeightedVarianceState,
b::ExpWeightedVarianceState
)
Refuses to merge two ExpWeightedVarianceState.
A state under PreCentred folds forward exactly, S = λ^{n_b} S_a + S_b with n_b the valid observations of the second block, but only while no asset resets inside that block. The state records the count that a reset zeroed and not the reset itself, so a merge cannot tell the two cases apart, and it would keep a history that the reset discarded. A state under EstimatedCentring does not fold at all, because each deviation of the second block reads a location that carries the first block. Fold the second block into the first with partial_fit! instead.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The method throws an
ArgumentErrorfor every pair.
Related
merge_states(
a::ExpWeightedCovarianceState,
b::ExpWeightedCovarianceState
)
Refuses to merge two ExpWeightedCovarianceState.
The fold $S = \lambda^{n_b} S_a + S_b$ is exact only under PreCentred over a second block in which every asset is valid at every observation. Under EstimatedCentring a block reads the location and the overlap that the first block carries. A holiday makes the decay of an entry depend on the common valid observations of its two assets, and a reset discards the first block for its asset. The state records neither the holidays nor the resets, so a merge cannot tell the cases apart. Fold the second block into the first with partial_fit!.
Arguments
a: State of the first block.b: State of the second block.
Validation
- The method throws an
ArgumentErrorfor every pair.
Related
merge_states(
a::RegimeAdjustedVarianceState,
b::RegimeAdjustedVarianceState
) -> Union{}Refuses a pair of regime-adjusted states, because this family does not merge.
A merge needs the state of a block to be a sufficient statistic for what that block contributes, and this state is not one. The regime state reads the standardised squared return of each observation. That score divides by the running variance, and the running observation count gates it. So a block fitted from a cold start weighs its first observations differently from the same block fitted after another block. The exponentially weighted accumulator folds as $\lambda^{n_B} v_A + v_B$, but the regime state that scales it does not fold. A fit under EstimatedCentring also holds its running location, and a HAC fit holds its buffer of recent returns.
Fold the second block into the first with partial_fit!. A sequential fit is exact.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states, so a mismatched pair is named as such.- The method then refuses the pair with an
ArgumentError, whatever it holds.
Returns
- The method always throws, so it returns no value.
Examples
julia> X = [0.01 -0.02; -0.015 0.03; 0.02 -0.01; -0.005 0.012];julia> ce = partial_fit!(RegimeAdjustedExpWeightedVariance(; decay = 0.9, min_obs = 2, regime_min_obs = 2), X);julia> try PortfolioOptimisers.merge_states(ce.cache, ce.cache) catch err err isa ArgumentError endtrueRelated
merge_states(
a::RegimeAdjustedCovarianceState,
b::RegimeAdjustedCovarianceState
)
Refuses a pair of regime-adjusted covariance states, because this family does not merge.
A block fitted from a cold start is not what the same block contributes after another one. The regime statistic scores each observation against the state that stands before it, so a cold block does not score its first min_obs observations, which the same block after another one scores. A correlation state that is normalised by a running variance carries that variance with it.
Fold the second block into the first with partial_fit! instead. A sequential fit is exact, and it is the route this family gives.
Algorithm
- Refuse the pair with
assert_mergeable_states, which names a type mismatch and an asset-count mismatch first, as theAbstractPartialFitStateinterface asks of every family. - Throw an
ArgumentErrornaming the reason this family does not merge.
Arguments
a: The first state.b: The second state.
Validation
aandbpassassert_mergeable_states.
Returns
- Never returns. An
ArgumentErroris thrown.
Related
merge_states(
a::CoskewnessPartialFitState,
b::CoskewnessPartialFitState
) -> CoskewnessPartialFitState
Folds two CoskewnessPartialFitState fitted on disjoint blocks into the state of the concatenated block.
The count, the mean and the second accumulator take Chan's merge. The third accumulator is moved to the common mean one block at a time, with shift_comoment3, and the two shifted accumulators are added.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the count, the mean and the second accumulator with
chan_merge. - Take the displacement of each block's own mean to the common mean.
- Shift each block's third accumulator to the common mean, and add the two.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CoskewnessPartialFitState: The state the two blocks give when they are fitted as one block.
Related
merge_states(
a::CokurtosisPartialFitState,
b::CokurtosisPartialFitState
) -> CokurtosisPartialFitState
Folds two CokurtosisPartialFitState fitted on disjoint blocks into the state of the concatenated block.
The companion of the CoskewnessPartialFitState merge, carried one order further. The fourth accumulator is shifted before the third, because shift_comoment4 reads the third accumulator of the block about its own mean.
Algorithm
- Refuse the pair with
assert_mergeable_states. - Fold the count, the mean and the second accumulator with
chan_merge. - Take the displacement of each block's own mean to the common mean.
- Shift each block's fourth accumulator to the common mean, and add the two.
- Shift each block's third accumulator to the common mean, and add the two.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbpassassert_mergeable_states.
Returns
state::CokurtosisPartialFitState: The state the two blocks give when they are fitted as one block.
Related
merge_states(_::EWBetaState, _::EWBetaState)
Refuses to merge two EWBetaState: the recursion of the second block starts from the state after the first one, so two states fitted on disjoint blocks do not give the state of their union to the last bit.
Validation
- Always throws an
ArgumentError.
Related
merge_states(
a::LagDescriptorState,
b::LagDescriptorState
) -> LagDescriptorState{__T_q} where __T_q<:DataStructures.CircularBuffer
Merges two LagDescriptorState fitted on consecutive blocks into the state of the concatenated block.
The state of a block is the lagged quantity of its last lag observations. So the state of the concatenated block is the last lag rows of the rows of a followed by the rows of b, and the merge is exact.
Arguments
a: The state of the first block.b: The state of the second block.
Validation
aandbhave the same capacity, the lag. ADimensionMismatchis thrown otherwise.- The rule of
LagDescriptorStateon the merged rows.
Returns
state::LagDescriptorState: The state of the concatenated block.
Related
merge_states(_::EWMeanState, _::EWMeanState)
Refuses to merge two EWMeanState: the recursion of the second block starts from the state after the first one, so two states fitted on disjoint blocks do not give the state of their union to the last bit.
Validation
- Always throws an
ArgumentError.
Related
merge_states(_::EWVolatilityState, _::EWVolatilityState)
Refuses to merge two EWVolatilityState: the recursion of the second block starts from the state after the first one, so two states fitted on disjoint blocks do not give the state of their union to the last bit.
Validation
- Always throws an
ArgumentError.
Related
merge_states(
_::EWMacroSensitivityState,
_::EWMacroSensitivityState
)
Refuses to merge two EWMacroSensitivityState: the recursion of the second block starts from the state after the first one, so two states fitted on disjoint blocks do not give the state of their union to the last bit.
Validation
- Always throws an
ArgumentError.
Related
merge_states(_::EWDownsideBetaState, _::EWDownsideBetaState)
Refuses to merge two EWDownsideBetaState: the recursion of the second block starts from the state after the first one, so two states fitted on disjoint blocks do not give the state of their union to the last bit.
Validation
- Always throws an
ArgumentError.
Related
merge_states(
_::RollingLogReturnState,
_::RollingLogReturnState
)
Refuses to merge two RollingLogReturnState fitted on disjoint blocks.
The cumulative sums of the second block start from the last row of the first block, so the state of the second block alone does not give its Descriptors. The two blocks fold in sequence.
Arguments
a: The state of the first block.b: The state of the second block.
Validation
- Always throws an
ArgumentError.
Related
merge_states(
a::PriorCarryState,
b::PriorCarryState
) -> PriorCarryState
Merges two PriorCarryState fitted on disjoint blocks into the state of the concatenated block.
The buffers merge by concatenation. The named-asset sets merge by union, because an asset that either state has already named needs no second notice.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Returns
state::PriorCarryState: The state that a fold of the two blocks as one block gives.
Related
merge_states(
_::CrossSectionalCarryState,
_::CrossSectionalCarryState
)
Refuses to merge two CrossSectionalCarryState fitted on disjoint blocks.
The state is not a sufficient statistic of its own block. The first regression of the second block reads the exposures of the last rows of the first block, the Descriptors of its first rows read the panel rows of the first block, and the idiosyncratic variance and the factor prior fold in the order of the observations. So the two blocks fold in sequence.
Arguments
a: The state of the first block.b: The state of the second block.
Validation
- Always throws an
ArgumentError.
Related
merge_states(
_::OnlinePortfolioSelectionState,
_::OnlinePortfolioSelectionState
)
Refuses to merge two online selection states.
An Online Update is not a sufficient statistic for its block. The state of a rule can be one, such as a Gram sum or a prefix of rows, but the allocation beside it is not. So two states folded on disjoint blocks describe no single run. To fold the rows in one pass, run the Causal Pass, which costs O(N) a row.
Validation
- The function always throws an
ArgumentError.
Related
PortfolioOptimisers.obs_weights_view — Method
obs_weights_view(_::AbstractPartialFitState, _)
Removes the partial-fit state from an estimator whose observations a view selects.
obs_weights_view selects observations, and a state describes the observations of its fit. A state has no slice on that axis, because no numerically stable operation removes an observation from a running accumulator. For the same reason, partial_fit! refuses an estimator under a RollingWindow. So the method returns nothing in place of the state. The viewed estimator then refuses a call with no data, and it gives no estimate over observations that the view excluded.
The asset axis is different, and a state has a slice on it. port_opt_view keeps a subset of the assets. A family whose state has an exact sub-state over that subset returns it by an index copy.
Arguments
::AbstractPartialFitState: The state the estimator carries, read for its type alone.::Any: Index of the observations to keep, which no slice of a state reads.
Returns
nothing.
Related
PortfolioOptimisers.assert_mergeable_states — Function
assert_mergeable_states(
a::AbstractPartialFitState,
b::AbstractPartialFitState
)
Refuses two partial-fit states that cannot be merged, naming the reason.
Two states merge only when they describe the same estimate over the same assets. So the two operands must be the same struct, and each array field must have the same shape in both. The check applies to a state whose array sizes depend on the assets alone. A SampleBufferState holds one row for each observation, so its merge does not call this check.
A family whose merge needs more adds a method that calls this check first. An exponentially weighted state is one. After this check, its method refuses every pair, because the state does not record a reset of an asset inside the second block.
Algorithm
- Refuse the pair when the two operands are not the same struct, whatever their type parameters.
- Walk the fields of the pair in declaration order.
- Refuse the pair when a field holds an array in both operands and the two shapes differ.
Arguments
a: The state of the first block of observations.b: The state of the second block of observations.
Validation
aandbare the same struct. The function throws anArgumentErrorotherwise.- Every array field of
ahas the shape of its counterpart inb. The function throws aDimensionMismatchotherwise.
Returns
nothing.
Related
PortfolioOptimisers.chan_merge — Function
chan_merge(
n_a::Number,
mu_a::Union{Number, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
M_a::Union{Number, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
n_b::Number,
mu_b::Union{Number, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
M_b::Union{Number, AbstractVector{<:Union{Number, AbstractJuMPScalar}}}
) -> Tuple{Any, Any, Any}
Folds the observation count, the mean and the second-moment accumulator of two blocks into those of the concatenated block.
Equation 2.1 of [14] states the merge for one series, and Equation 3.1 of [15] states it for a co-moment matrix. The merge is associative and symmetric in the two blocks. So any split of a sample into disjoint blocks gives the state of the whole sample, in any order of the merges. $M$ is the sum of the products of the deviations, not the variance. So a caller divides it by the count, or by the count minus one, after the last merge.
Mathematical definition
\[\begin{align} n &= n_A + n_B\\ \boldsymbol{\delta} &= \boldsymbol{\mu}_B - \boldsymbol{\mu}_A\\ \boldsymbol{\mu} &= \boldsymbol{\mu}_A + \boldsymbol{\delta} \frac{n_B}{n}\\ M &= M_A + M_B + \boldsymbol{\delta} \boldsymbol{\delta}^{\intercal} \frac{n_A n_B}{n}\, . \end{align}\]
Where:
- $n_A$, $n_B$: The observation counts of the two blocks.
- $\boldsymbol{\mu}_A$, $\boldsymbol{\mu}_B$: The means of the two blocks.
- $M_A$, $M_B$: The second-moment accumulators of the two blocks, each the sum of the products of the deviations from the mean of its block.
- $\boldsymbol{\delta}$: The difference of the two means.
When $M$ holds one accumulator for each asset, and not a co-moment matrix, the outer product $\boldsymbol{\delta} \boldsymbol{\delta}^{\intercal}$ becomes its diagonal, the elementwise square $\boldsymbol{\delta}^2$. For one series with the block sums $T_A = n_A \mu_A$ and $T_B = n_B \mu_B$, the last term equals $\frac{n_A}{n_B n} \left( \frac{n_B}{n_A} T_A - T_B \right)^2$, which is the form of Equation 2.1b of [14].
Algorithm
- Add the two counts, giving
n. - Subtract the two means, giving
delta. - Move the first mean along
deltaby the share of the total that the second block holds, givingmu. - Add the two accumulators, and add the outer product of
deltascaled byn_a * n_b / n, givingM.
Arguments
n_a: Observation count of the first block.mu_a: Mean of the first block.M_a: Second-moment accumulator of the first block.n_b: Observation count of the second block.mu_b: Mean of the second block.M_b: Second-moment accumulator of the second block.
Validation
n_a + n_b > 0. The function throws aDomainErrorotherwise, because the merge divides by the total count.
Returns
(n, mu, M)::Tuple: The count, the mean and the accumulator of the concatenated block.
Examples
julia> PortfolioOptimisers.chan_merge(1, [1.0, 2.0], [0.0, 0.0], 1, [3.0, 6.0], [0.0, 0.0])(2, [2.0, 4.0], [2.0, 8.0])Related
References
- [14] T. F. Chan, G. H. Golub and R. J. LeVeque. Updating formulae and a pairwise algorithm for computing sample variances. Technical report STAN-CS-79-773, Department of Computer Science, Stanford University (1979). Equation 2.1, page 4.
- [15] P. Pébay. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments. Technical report SAND2008-6212, Sandia National Laboratories (2008). Equation 3.1, page 12.
References
- [13]
- T. F. Chan, G. H. Golub and R. J. LeVeque. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician 37, 242–247 (1983).
- [14]
- T. F. Chan, G. H. Golub and R. J. LeVeque. Updating Formulae and a Pairwise Algorithm for Computing Sample Variances. Technical report STAN-CS-79-773 (Department of Computer Science, Stanford University, 1979).
- [15]
- P. Pébay. Formulas for Robust, One-Pass Parallel Computation of Covariances and Arbitrary-Order Statistical Moments. Technical report SAND2008-6212 (Sandia National Laboratories, 2008).