Online Pipeline
A Pipeline updates incrementally. partial_fit! passes each block of observations through the steps in order, up to the step that stores the rows: the prior step, or the optimiser step when the pipeline has no prior. fit(pipe) with no data returns the fitted PipelineResult. It rebuilds the returns from the rows that the step stored, and refits over them every step that chooses assets. It then restricts the state of the step that stores the rows to the assets that remain, and runs the steps after it as a batch fit.
A step that transforms each row on its own updates with each block. A step that only chooses assets waits until you read the result. A step with no incremental form makes the pipeline throw an error that names the step, before the first update. To run such a step, wrap the pipeline as Online(pipe). Its PortfolioOptimisers.PipelineBufferState then stores the input data, and reading the result runs a batch fit over it.
Online(pipe) and Resume(res) of the Result of a pipeline go through the method of a pipeline, cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER). Its docstring states what each of them adds.
PortfolioOptimisers.partial_fit! — Method
partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{PipelineBufferState, ReturnsBufferState}}}, data::Prices_RR)Folds a block of observations into a Pipeline, without a fit.
This is the online step of the Pipeline, and the pipeline takes it for its steps. The verb hands each step before the row owner the block that the step before it emitted, and folds the rows into the owner, which is the prior step, else the optimisation step (pipeline_row_owner). The verb leaves every step after the owner untouched, and fit(pipe) with no data fits it as batch fits it. Under Online(pipe) the pipeline carries a PipelineBufferState instead. The verb then appends the block to the buffer, no step folds, and fit(pipe) with no data is a batch fit over the buffer.
The arity mirrors the batch verb. fit(pipe, data) takes a PricesResult or a ReturnsResult at the input level of the pipeline, so the online step takes one too, with one observation or a block of them.
Algorithm
On the fold route:
- Find the row owner, giving
k. - Walk the steps before
kin order. Skip each step that writes no data slot (is_data_step) and each universe-only step (is_universe_step). A universe-only step passes the rows through untouched, andfit(pipe)with no data applies its universe. - Fold each remaining step through
partial_fit_transform, giving the folded step and the nextblock. - Fold
blockinto the owner throughfold_pipeline_owner, giving the folded owner andcache. A prior owner folds throughfold_prior, andcacheis aReturnsBufferStateof the Pipeline that records the rest of theReturnsResult, so thatfit(pipe)can rebuild theReturnsResultthat the batch path reads. An optimisation owner folds through its ownpartial_fit!and keeps its own Fold Context, socacheisnothing. - Return the pipeline rebuilt from the folded steps and
cache.
On the refit route, append data to the buffer through partial_fit!, and return the pipeline with the new state. The refit route is the method on a pipeline whose cache is a PipelineBufferState, which is more specific than this one, so this method holds the fold route alone.
Arguments
pipe: The pipeline to fold into, with its wrappers resolved byupdate_online_estimator.data: The block of observations, at the price level or at the returns level, as the input of the pipeline.
Validation
On the fold route:
- The pipeline has a row owner. An
ArgumentErroris thrown otherwise. - No step holds an
Onlinedeclaration (pipeline_online_member), because the warm-up resolves each one. A wrapper that no warm-up resolved would fold the estimator that it wraps with no buffer, and its cap would have no effect. AnArgumentErrorthat names the step is thrown otherwise. - The rows reach the owner as a
ReturnsResult. AnArgumentErroris thrown otherwise. - Everything the folds of the steps refuse.
Returns
pipe: The pipeline, with the owner and every step before it folded, and its own state recorded.
Related
StatsAPI.fit — Method
fit(pipe::Pipeline)Reads a folded Pipeline out, with the result that the batch fit over the observations folded so far gives.
This is the call with no data of the online step of the Pipeline, as optimise(opt) with no data is for an optimiser. It returns an ordinary PipelineResult, so predict, assert_universe_aligned and the search take it unchanged.
Algorithm
- Rebuild the
ReturnsResultof the observations folded so far from the row owner throughpipeline_returns_result, givingrd₀. A prior owner gives its rows throughprior_returns_buffer, and the Pipeline adds the rest from its ownReturnsBufferState. An optimisation owner gives itsReturnsResultthroughreturns_result. TheReturnsResultspans the input universe of the pipeline at warm-up width. - Walk the data steps before the owner in order, through
fit_data_step_from_state. A row-local step reads its fitted Result out of its state, restricted to the assets that survive so far. A universe-only step runs its batch verb overrd₀viewed to the surviving assets, and narrows the surviving set. The indexidxof the surviving assets intord₀is the column map, andrd = port_opt_view(rd₀, :, idx)is the context's returns. - Run every other step before the owner over that context, as batch runs it. Such a step is a phylogeny step, an uncertainty-set step or a constraint step.
- Read the owner out over the surviving assets through
fit_owner_from_state. A prior owner reads out throughprior(port_opt_view(pe, idx)), which views its state by asset and never slices its rows again. An optimisation owner reads out throughoptimise(opt)on the viewed estimator, with the context injected. - Run every step after the owner as batch, the optimisation step with the context injected, and return the result.
So a selection that moves between two steps is a view of a state fitted over the whole universe, which is the batch fit over those columns. The selector ranks the assets again at every step, as the batch loop refits it at every fold.
Under Online(pipe), fit(pipe) with no data is fit(pipe, buffer), the batch fit over the observations that the PipelineBufferState holds.
Validation
- The pipeline has a row owner. An
ArgumentErroris thrown otherwise. - The pipeline has taken a step. An
ArgumentErroris thrown otherwise. - A prior owner reads out over as many assets as survive the universe steps before it (
fit_owner_from_state). AnArgumentErroris thrown otherwise.
Returns
res::PipelineResult: The result the batch fit over the observations folded so far gives.
Related
PortfolioOptimisers.partial_fit! — Method
partial_fit!(state::PipelineBufferState, data::Prices_RR) -> PipelineBufferState
partial_fit!(state::PipelineBufferState, ::Nothing) -> PipelineBufferStateAppends a block of observations to a PipelineBufferState, and drops the oldest rows past the cap.
The state is immutable, so the method returns a new state and leaves state as it was. A data of nothing holds no row, so that method returns state itself. merge_states passes the data of a second state, which is nothing before its first block.
Algorithm
- Concatenate the held data and
databy rows throughvcat_observations, givingheld. When the state holds no data yet,heldisdata. - Count the rows of
held, givingn. - When
max_historyis set andn > max_history, view the lastmax_historyrows ofheld. - Return a new state that holds
heldand carries the same cap.
Validation
- Everything
vcat_observationsrefuses.
Related