Online (a)

Some estimators have no exact incremental fit. Such an estimator stores the observations it has seen in a PortfolioOptimisers.SampleBufferState, and runs its ordinary batch fit over those rows when you read the estimate. Wrap the estimator in Online to give it a buffer. The max_history keyword of Online caps the buffer, and the estimate is then the batch fit over the last max_history observations.

PortfolioOptimisers.update_online_estimator replaces each Online with the estimator it wraps, which now carries an empty buffer. It runs once, before the first block of observations, so no Online remains in the estimator that you update. The buffer also stores the per-observation masks of a CoveragePolicy and the factor returns of a factor prior next to the asset returns. The first block of observations fixes which of them the buffer stores. A prior holds its rows in one buffer, and every member of the prior that has no incremental fit refits from those rows.

PortfolioOptimisers.partial_fit! — Function

partial_fit!(
    state::SampleBufferState,
    X::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}};
    ...
) -> Any
partial_fit!(
    state::SampleBufferState,
    X::AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    F::Union{Nothing, AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    ne,
    E,
    iv,
    ivpa,
    panel_fields,
    gaps
) -> Any

Folds every observation of a block into a SampleBufferState.

This is the block arm of the partial_fit! interface, and all of the fold of the buffer. The single-observation arm reshapes its argument and calls this arm. A cap truncates the incoming rows before the copy, so the fold never copies the whole of a block longer than the cap. reserve_sample_buffer grows the backing matrices, and its docstring states why an append costs amortised O(1).

The masks, the factor rows, the Exogenous Series, the implied volatilities and the Panel Fields stay with their rows through each step: the orientation, the truncation, the growth and the copy. The first append fixes whether the buffer records a mask, whether it records factor rows, whether it records the Exogenous Series, whether it records implied volatilities and whether it records Panel Fields, as it fixes the width and the element type. It also fixes the names of the series and the adjustment ivpa. assert_buffer_presence_agreement refuses a later fold that disagrees. A buffer that knows the activity of some of its rows and not of others answers neither question. A buffer with factor rows for some observations and not for others describes no regression. A buffer that holds no observations has nothing to disagree with, so the fold seeds it again from the block.

The Panel Fields are observations first whatever dims is, as the Panel Fields of an AssetPanel always are. They keep the rows of the buffer that the cap keeps, and panel_fields_append joins them to the rows of the block.

The marks gaps of the Shared Series gaps are one entry for each observation whatever dims is. They travel with their rows as the masks do, but they have no presence rule: a buffer with no marks has no gap, so append_buffer_gaps writes the marks of a later fold onto false marks for the rows before it, and a fold with no marks gives false to a buffer that has them.

Algorithm

Orient X, the masks, iv, F and E to observations × assets, observations × factors and observations × series. Transpose them when dims == 2.
Refuse a mask or an iv that does not have the shape of the block, a factor block whose row count is not the row count of the block, a series that assert_buffer_series_shape refuses, and marks that assert_buffer_gap_shape refuses.
Return the state unchanged when the block is empty.
Refuse Panel Fields that do not make a time-varying Asset Panel with the masks of the block, over the observations and the assets of the block.
When a cap is set, truncate the incoming rows, their masks, their implied volatilities, their factor rows, their series, their Panel Fields and their marks to the last max_history of them.
When the buffer holds no observations, seed it from the block with seed_sample_buffer, seed iv with seed_buffer_array, seed G with seed_buffer_gaps, pin ivpa, and return it. The seed fixes the width, the element type, the masks, the implied volatilities with ivpa, the factor rows, the series with its names, and the Panel Fields.
Refuse a block whose width is not the fixed width. Refuse masks, implied volatilities, factor rows, a series or Panel Fields that are absent where the buffer records them or present where it does not, factor rows of a different width, a series under other names with assert_buffer_series_names, and an ivpa that is not the pinned one with assert_pinned_value.
Make room for the block with reserve_sample_buffer, which drops the rows that the cap pushes out and grows the backing matrices.
Copy the block, its masks, its implied volatilities, its factor rows and its series after the valid region, and write its marks with append_buffer_gaps. Join the Panel Fields that the buffer keeps to the Panel Fields of the block with panel_fields_append, and rebind P and n with Accessors.@reset.

Arguments

state: The buffer to fold into.
X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
F: Factor observations to fold, oriented as X is, or nothing.
dims: Dimension along which to perform the computation.
active_mask: The active mask of the block, of the shape of X, or nothing. A buffer records it for every observation it holds or for none of them.
estimation_mask: The estimation mask of the block, of the shape of X, or nothing. It is carried on the same terms as active_mask.
ne: The names of the Exogenous Series of the block, or nothing.
E: The Exogenous Series of the block, oriented as X is, or nothing.
iv: The implied volatilities of the block, oriented as X is, or nothing.
ivpa: The implied volatility risk premium adjustment, or nothing.
panel_fields: The time-varying Panel Fields of the block, observations first, or nothing.
gaps: One mark for each observation of the block, true at an observation with a Shared Series gap, or nothing when the block has none.

Validation

dims in (1, 2).
gaps, when it is not nothing, has one entry for each observation of the block. A DimensionMismatch is thrown otherwise.
X has the width the first append fixed. A DimensionMismatch is thrown otherwise.
active_mask, estimation_mask and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
F, when it is not nothing, has as many rows as X, and the width that the first append with factor rows fixed. A DimensionMismatch is thrown otherwise.
ne and E obey the rules of assert_buffer_series_shape, and ne equals the names that the first append with a series fixed. An ArgumentError is thrown otherwise.
panel_fields, when it is not nothing, obeys the rules of assert_buffer_panel_block, and it holds the Panel Fields that the first append with Panel Fields fixed, as vcat of two Asset Panels states them.
A buffer that holds observations receives the masks that it records, and it receives implied volatilities, factor rows, a series and Panel Fields if and only if it records them. An ArgumentError is thrown otherwise.

Returns

state::SampleBufferState: The buffer after the last row.

Related

source

PortfolioOptimisers.partial_fit! — Function

partial_fit!(
    state::SampleBufferState,
    x::AbstractVector{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}};
    ...
) -> Any
partial_fit!(
    state::SampleBufferState,
    x::AbstractVector{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}},
    f::Union{Nothing, AbstractVector{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}};
    active_mask,
    estimation_mask,
    ne,
    E,
    iv,
    ivpa
) -> Any

Folds one observation into a SampleBufferState.

This is the single-observation arm of the partial_fit! interface. It reshapes the observation into a one-row block with no copy, and it folds the block through the block arm. It reshapes the masks, the factor observation, the observation of the Exogenous Series and the implied volatilities in the same way, so one observation with a policy, with factors, with a series or with implied volatilities folds through the code of the block.

Arguments

state: The buffer to fold into.
x: One observation, whose entries are the assets.
f: The contemporaneous factor observation, whose entries are the factors, or nothing.
active_mask: The active mask of the observation, one entry per asset, or nothing.
estimation_mask: The estimation mask of the observation, one entry per asset, or nothing.
ne: The names of the Exogenous Series, or nothing.
E: The observation of the Exogenous Series, one entry per series, or nothing.
iv: The implied volatilities of the observation, one entry per asset, or nothing.
ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

x has the width the first append fixed. A DimensionMismatch is thrown otherwise.
active_mask and estimation_mask, when they are not nothing, have one entry per asset. A DimensionMismatch is thrown otherwise.
f, when it is not nothing, has the width that the first append with factor rows fixed. A DimensionMismatch is thrown otherwise.
A buffer that holds observations receives f if and only if it records factor rows. An ArgumentError is thrown otherwise.

Returns

state::SampleBufferState: The buffer after the observation.

Related

source

PortfolioOptimisers.port_opt_view — Method

port_opt_view(
    x::SampleBufferState,
    i,
    args...
) -> SampleBufferState

Slices a SampleBufferState to the selected assets.

A buffer holds its observations verbatim, so the slice of the buffer is the buffer of the sliced universe, column for column. The observation axis does not change. The masks and the implied volatilities are per cell, so they slice on the asset axis by the same indices, and a vector ivpa slices with them. The factor rows and the Exogenous Series are copied and not sliced, because the selection indexes assets, a factor and a series belong to no asset, and a fit over a subset of the universe reads the same factors and the same series. The Panel Fields are viewed over the selected assets and not copied, because nothing writes into the arrays of a panel. A tensor Panel Field keeps its whole label axis, because the buffer holds no asset names to find the square case with. The slice copies by index and does not take a view. A later partial_fit! on the viewed estimator writes into its backing matrices, and with a view it would change the buffer of the original estimator. A buffer that holds no observations has no width to slice, so its view is the empty seed with the same cap.

Algorithm

Return the empty seed with the same cap when the buffer holds no observations.
Copy the columns i of X, of each mask and of iv, view a vector ivpa over the entries i, copy F, E and the marks G whole, and view each Panel Field over the columns i.

Arguments

x: The buffer to slice.
i: Index or indices of the assets to keep.
args...: Additional positional arguments (ignored).

Returns

state::SampleBufferState: The buffer of the same observations over the selected assets.

Related

source

PortfolioOptimisers.partial_fit! — Method

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s152", var"#s69"} where {var"#s152"<:Number, var"#s69"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask,
    iv,
    ivpa
) -> Any

Folds observations into the sample buffer an estimator carries.

This is the buffering arm of partial_fit!. Each estimator that carries a SampleBufferState reaches this method. A family that folds exactly has methods of its own, and each of them narrows the cache type parameter of its estimator to the state of that fold. So a buffer never reaches them, and it reaches this method. The type of the state selects the route, and no method refuses the step.

The method covers both arms of the interface. The families that refuse the step also declare one method over both arms, and two narrower methods here would be ambiguous against each of them. So the body selects the arm by the type of X, and the compiler resolves that branch at each call site.

A buffer carries the per-observation masks beside the observations. So a CoveragePolicy mask goes through the wrapper as it goes through the accumulator of an estimator, and the call with no data gives it back to the batch verb. A wrapped estimator folded under a policy gives the answer of a batch fit over the same window under the same policy. The implied volatilities and ivpa go through the wrapper in the same way, so a wrapped estimator that reads them, such as an ImpliedVolatility inside a PortfolioOptimisersCovariance, gives the answer of a batch fit over the same rows.

Algorithm

Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
Fold a matrix, its masks and its implied volatilities through the block arm of partial_fit!, and a vector, its masks and its implied volatilities through the single-observation arm.
Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

est: Estimator whose buffer is folded forward.
X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
dims: Dimension along which to perform the computation.
active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
estimation_mask: The estimation mask, on the same terms as active_mask.
iv: The implied volatilities, on the same terms as active_mask.
ivpa: The implied volatility risk premium adjustment, or nothing.

Validation

est carries a SampleBufferState. An ArgumentError is thrown otherwise.
The masks and iv, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
A buffer that holds observations receives the masks that it records, and it receives iv if and only if it records implied volatilities. An ArgumentError is thrown otherwise.
ivpa equals the adjustment that the first append pinned. An ArgumentError is thrown otherwise.
dims in (1, 2).

Returns

est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

source

PortfolioOptimisers.merge_states — Method

merge_states(
    a::SampleBufferState,
    b::SampleBufferState
) -> SampleBufferState

Folds two SampleBufferState fitted on disjoint blocks into the buffer of the concatenated block.

The merge is a concatenation, and it is exact. A buffer holds its observations verbatim, so the buffer of two blocks holds the rows of the first block followed by the rows of the second. When a cap is set, the result keeps the last max_history rows of the concatenation. The masks, the factor rows, the Exogenous Series and the Panel Fields concatenate with their rows. Two buffers that disagree about what they record, or that name their series differently, are refused, for the reason that assert_buffer_presence_agreement states for a fold. A buffer that holds no observations records nothing, so the merge returns a copy of the other buffer.

The method does not call assert_mergeable_states. That check compares every array field on every axis. The backing matrix of a buffer has the observation axis as well as the asset axis, so the check refuses two buffers over the same assets with different numbers of observations. This method checks the cap, the width, the masks and the factor rows itself.

Algorithm

Refuse two buffers of different caps.
Return a copy of the other buffer when one buffer holds no observations.
Refuse two buffers of different widths.
Refuse two buffers that do not record the same masks, that disagree about factor rows, a series or Panel Fields, whose factor rows have different widths, or whose series have different names.
Concatenate the valid region of the first with the valid region of the second, and each mask, the factor rows and the series with their own. Concatenate the marks of the Shared Series gaps with merge_buffer_gaps.
Keep the last max_history rows when a cap is set.
Join the Panel Fields of the two buffers over the rows that the result keeps, with panel_fields_append.

Arguments

a: The buffer of the first block of observations.
b: The buffer of the second block of observations.

Validation

a and b carry the same cap. An ArgumentError is thrown otherwise.
a and b describe the same number of assets, when both hold observations. A DimensionMismatch is thrown otherwise.
a and b record the same masks, both record factor rows or neither does, both record a series under the same names or neither does, and both record Panel Fields or neither does, when both hold observations. An ArgumentError is thrown otherwise.
a and b describe the same number of factors, when they record factor rows. A DimensionMismatch is thrown otherwise.

Returns

state::SampleBufferState: The buffer the two blocks give when they are folded as one block.

Related

source