The online step

struct Online{T1, T2} <: AbstractEstimator

Declares that an estimator takes the online step from a buffer of the observations it has seen.

An Online is stored directly in the estimator field it wraps — e.g. HighOrderPriorEstimator(; pe = Online(EmpiricalPrior())), or the pe of an optimiser, JuMPOptimiser(; pe = Online(EmpiricalPrior()), slv = …) — and it is transient, in the sense TimeDependent is: update_online_estimator walks the fields that hold one, seeds each wrapped estimator's cache with a SampleBufferState, and rebuilds the host through its keyword constructor. What comes out is an ordinary estimator carrying a state, and no Online exists from that point on, so every verb downstream meets a plain estimator.

Only a field whose bound is Onl admits a wrapper: the pe slot of a wrapping prior and of an optimiser. A moment slot inside a prior — ce, me, ske, kte — takes none, so EmpiricalPrior(; ce = Online(…)) is refused by the field's bound before any warm-up runs. The prior is the row owner: it holds its observations once, at the bottom of its chain, and refits every member that does not fold from those rows, so a buffer under one of its moments would hold the same rows a second time. Wrap the prior instead. A moment estimator is wrapped on its own only when it is itself the estimator under study, update_online_estimator(Online(Coskewness())).

It differs from TimeDependent in when it resolves, and the difference is deliberate. A schedule re-resolves every fold, because its value changes every fold. An Online resolves once, at warm-up, because after warm-up the state is what is threaded from step to step and a second resolution would throw the buffer away.

A wrapper replaces an exact fold; it does not add to one. Every family that folds exactly narrows the cache type parameter of its own partial_fit! methods to the state that fold reads, so a wrapped estimator never meets them: it buffers its observations and answers every read-out verb by running the batch verb over the rows the buffer holds. The state's type is the whole route, and it is decided by whether the caller wrapped the estimator.

So wrap an estimator whose estimate has no exact incremental fold of its own, one that folds exactly and carries its observations for a consumer downstream, or one whose estimate you want fitted over a window rather than over every observation. An estimator that folds exactly and carries nothing needs no wrapper: unwrapped it already answers partial_fit!, seeding its own state on the first call, and it folds in the memory one state costs rather than the memory a buffer costs.

A wrapper and a TimeDependent schedule do not wrap each other, and the difference in when they resolve is the whole reason. Neither Online(TimeDependent(…)) nor a schedule whose entry or default is an Online is admissible: a wrapper reached through a schedule entry would be resolved at no fold at all, or re-seeded at every fold, throwing away the buffer the step threads. They compose the other way round, and both ways are ordinary. An estimator an Online wraps may hold schedules of its own, which survive the seeding untouched and resolve per fold afterwards; and one host may hold a wrapper in one field and a schedule in another, each resolving at its own time. The two field scans are disjoint by construction — a field holding one is invisible to the other's candidate list — so neither resolution can reach the other's wrapper.

max_history caps that buffer, and the cap is the window. An uncapped buffer answers exactly what a batch fit over every observation folded so far answers, and a capped one answers exactly what a batch fit over the last max_history observations answers — for the estimate itself and for every consumer that reads the observations, the scenario risk measures among them, so CVaR, EVaR and CDaR. There is one rule and no special case: a buffer means the batch verb over the buffer's rows. An estimator left unwrapped is unaffected, folds exactly, and stays fitted over every observation.

Fields

• est: Estimator the buffer is seeded on. It is the value the field takes once the wrapper has resolved.

• max_history: Optional cap on the number of observations the carry buffer keeps. nothing keeps every observation folded so far. A cap bounds memory and nothing else: it does not make the estimator windowed, because an estimate that folds exactly is fitted over every observation whatever the buffer holds.

Constructors

Online(est::Union{<:AbstractEstimator, <:StatsBase.CovarianceEstimator};       max_history::Option{<:Integer} = nothing)Online(; est::Union{<:AbstractEstimator, <:StatsBase.CovarianceEstimator},       max_history::Option{<:Integer} = nothing)

Validation

• est is not an Online. An ArgumentError is thrown otherwise.
• est has a cache field. An ArgumentError is thrown otherwise.
• max_history > 0 when it is not nothing. A DomainError is thrown otherwise.

Examples

julia> Online(Covariance(; alg = SemiMoment()); max_history = 250)Online          est ┼ Covariance              │    me ┼ SimpleExpectedReturns              │       │   w ┴ nothing              │    ce ┼ GeneralCovariance              │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)              │       │    w ┴ nothing              │   alg ┼ SemiMoment()              │     w ┴ nothing  max_history ┴ Int64: 250

Related

• SampleBufferState
• Online_Option
• update_online_estimator
• online_fields
• TimeDependent

partial_fit!(
    state::SampleBufferState,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    ...
) -> Any
partial_fit!(
    state::SampleBufferState,
    X::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    F::Union{Nothing, AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds every observation of a block into a SampleBufferState.

The block arm of the partial_fit! interface, and the whole implementation of the buffer's fold: the single-observation arm reshapes its argument and calls this one. A cap truncates the incoming rows before they are copied, so a block longer than the cap never allocates the whole of it. reserve_sample_buffer carries the growth, and its docstring states what makes an append amortised O(1).

The masks and the factor rows travel with the rows they explain, row for row, through every step: the orientation, the truncation, the growth and the copy. Whether the buffer records a mask, and whether it records factor rows, is fixed by its first append, exactly as the width and the element type are, and a later fold that disagrees is refused by assert_buffer_presence_agreement — a buffer whose activity is known for some of its rows and not for others answers neither question, and a buffer whose factor rows exist for some observations and not for others describes no regression. A buffer holding no observations has nothing to disagree with, so reset_empty_buffer empties it and the fold seeds it afresh.

Algorithm

1. Orient X, the masks and F to observations × assets and observations × factors, transposing them when dims == 2, refuse a mask that is not of the shape of the block and a factor block whose rows are not the block's, and return the state unchanged when the block is empty.
2. Truncate the incoming rows, their masks and their factor rows to the last max_history of them when a cap is set.
3. Empty a buffer that holds no observations and disagrees with the fold about the masks or the factor rows, so that the seed below fixes them afresh.
4. Seed the buffer with seed_sample_buffer when it is the empty seed, which fixes the width, the element type, the masks and the factor rows; and refuse a block whose width is not the width already fixed, whose masks disagree with the ones the buffer holds, or whose factor rows are absent where the buffer records them, present where it does not, or of a width the buffer did not fix.
5. Make room for the block with reserve_sample_buffer, which drops what the cap pushes out and grows the backing matrices.
6. Copy the block, its masks and its factor rows in after the valid region, and rebind n with Accessors.@reset.

Arguments

• state: The buffer to fold into.
• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• F: Factor observations to fold, oriented as X is, or nothing.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or nothing. A buffer records it for every observation it holds or for none of them.
• estimation_mask: The estimation mask of the block, of the shape of X, or nothing. It is carried on the same terms as active_mask.

Validation

• dims in (1, 2).
• X has the width the first append fixed. A DimensionMismatch is thrown otherwise.
• active_mask and estimation_mask, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• F, when it is not nothing, has as many rows as X, and the width the first append that carried one fixed. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records, and factor rows exactly when it already records them. An ArgumentError is thrown otherwise.

Returns

• state::SampleBufferState: The buffer after the last row.

Related

• SampleBufferState
• reserve_sample_buffer
• seed_sample_buffer
• assert_buffer_presence_agreement
• partial_fit!
• sample_buffer
• factor_buffer

partial_fit!(
    state::SampleBufferState,
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}};
    ...
) -> Any
partial_fit!(
    state::SampleBufferState,
    x::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}},
    f::Union{Nothing, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    active_mask,
    estimation_mask
) -> Any

Folds one observation into a SampleBufferState.

The single-observation arm of the partial_fit! interface. It reshapes the observation into a one-row block, which costs no copy, and folds it through the block arm. The masks and the factor observation are reshaped on the same terms, so one observation under a policy, or beside its factors, folds through the same code the block does.

Arguments

• state: The buffer to fold into.
• x: One observation, whose entries are the assets.
• f: The contemporaneous factor observation, whose entries are the factors, or nothing.
• active_mask: The active mask of the observation, one entry per asset, or nothing.
• estimation_mask: The estimation mask of the observation, one entry per asset, or nothing.

Validation

• x has the width the first append fixed. A DimensionMismatch is thrown otherwise.
• active_mask and estimation_mask, when they are not nothing, have one entry per asset. A DimensionMismatch is thrown otherwise.
• f, when it is not nothing, has the width the first factor append fixed, and is given exactly when the buffer records factor rows. A DimensionMismatch or an ArgumentError is thrown otherwise.

Returns

• state::SampleBufferState: The buffer after the observation.

Related

• SampleBufferState
• partial_fit!

port_opt_view(
    x::SampleBufferState,
    i,
    args...
) -> SampleBufferState

Slices a SampleBufferState to the selected assets.

A buffer holds its observations verbatim, so the slice of the buffer is the buffer of the sliced universe, column for column, and the observation axis passes through. The masks are per cell, so they slice on the same axis and by the same indices, and the viewed buffer answers the same question over the selected assets that the whole one answers over all of them. The factor rows are copied, never sliced: the selection indexes assets, a factor is not an asset, and a fit over a subset of the universe reads the same factors. The slice copies by index and does not view: a later partial_fit! on the viewed estimator would otherwise write through into the backing matrices of the estimator the view was taken from.

Arguments

• x: The buffer to slice.
• i: Index or indices of the assets to keep.
• args...: Additional positional arguments (ignored).

Returns

• state::SampleBufferState: The buffer of the same observations over the selected assets.

Related

• SampleBufferState
• port_opt_view
• partial_fit!

partial_fit!(
    est::Union{AbstractEstimator, CovarianceEstimator},
    X::Union{AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}};
    dims,
    active_mask,
    estimation_mask
) -> Any

Folds observations into the sample buffer an estimator carries.

The buffering arm of partial_fit!, and the method every estimator carrying a SampleBufferState reaches. A family that folds exactly writes methods of its own, and each of them narrows the cache type parameter of its own estimator to the state that fold reads, so a buffer never meets them and this method is what remains. The state's type is therefore the whole route, and nothing refuses the step.

It is one method over both arms of the interface rather than two, because the families that refuse the step declare one method over both arms too, and a pair of narrower methods here would be ambiguous against each of them. So the arm is chosen by the type of X inside the body, which is statically resolved at every call site.

A buffer carries the per-observation masks beside the observations, so a CoveragePolicy mask threads through the wrapper as it does through an estimator's own accumulator, and the read-out hands it back to the batch verb. A wrapped estimator folded under a policy therefore answers what a batch fit over the same window under the same policy answers, and the unwrapped and wrapped paths agree.

Algorithm

1. Read the buffer out of the cache field with assert_sample_buffer, which refuses an estimator that was never wrapped in Online.
2. Fold a matrix and its masks through the block arm of partial_fit!, and a vector and its masks through the single-observation arm.
3. Rebind est.cache with Accessors.@reset, and return the estimator.

Arguments

• est: Estimator whose buffer is folded forward.
• X: Observations to fold. A matrix holds one observation per row when dims == 1, and one per column when dims == 2. A vector is a single observation across the assets, and dims is ignored.
• dims: Dimension along which to perform the computation.
• active_mask: The active mask of the block, of the shape of X, or of one entry per asset when X is one observation, or nothing.
• estimation_mask: The estimation mask, on the same terms as active_mask.

Validation

• est carries a SampleBufferState. An ArgumentError is thrown otherwise.
• The masks, when they are not nothing, have the shape of X. A DimensionMismatch is thrown otherwise.
• A buffer holding observations is given the masks it already records. An ArgumentError is thrown otherwise.
• dims in (1, 2).

Returns

• est: The estimator, with its cache field rebound to the buffer after the last observation.

Related

• SampleBufferState
• Online
• sample_buffer_kwargs
• CoveragePolicy
• partial_fit!

merge_states(
    a::SampleBufferState,
    b::SampleBufferState
) -> SampleBufferState{_A, Int64} where _A

Folds two SampleBufferState fitted on disjoint blocks into the buffer of the concatenated block.

Concatenation, which is exact: a buffer holds its observations verbatim, so the buffer of two blocks is the buffer of the rows of one followed by the rows of the other. It is the one buffer state that merges. A cap is applied to the result, which keeps the last max_history rows of the concatenation. The masks and the factor rows concatenate with the rows they explain, and two buffers that disagree about which of them they record are refused, on the reasoning assert_buffer_presence_agreement states for a fold.

assert_mergeable_states is deliberately not called. Its array-shape rule reads every array field on every axis, and a buffer's backing matrix carries the observation axis as well as the asset axis, so two buffers over the same assets and different numbers of observations would be refused by it. The width, the cap and the masks are checked here instead.

Algorithm

1. Refuse two buffers of different widths, and two buffers of different caps.
2. Refuse two buffers that do not record the same masks, or that do not agree on recording factor rows, or whose factor rows are of different widths.
3. Concatenate the valid region of the first with the valid region of the second, and each mask and the factor rows with their own.
4. Keep the last max_history rows when a cap is set.

Arguments

• a: The buffer of the first block of observations.
• b: The buffer of the second block of observations.

Validation

• a and b describe the same number of assets. A DimensionMismatch is thrown otherwise.
• a and b carry the same cap. An ArgumentError is thrown otherwise.
• a and b record the same masks, and both record factor rows or neither does. An ArgumentError is thrown otherwise.
• a and b describe the same number of factors, when they record factor rows. A DimensionMismatch is thrown otherwise.

Returns

• state::SampleBufferState: The buffer the two blocks give when they are folded as one block.

Related

• SampleBufferState
• merge_states
• assert_buffer_presence_agreement
• sample_buffer