GitHub

The Pipeline's online step

A Pipeline is a host of the online step (ADR 0142). partial_fit! walks the steps in order and folds each block of observations through them into the row owner — the prior step, else the optimiser step — and fit(pipe) with no data reads the fitted PipelineResult out: it reconstitutes the carrier from the owner's rows, refits every universe step over it, views the owner's state to the surviving assets, and runs the tail as batch. A row-local step folds, a universe-only step defers to a view, and a step with no online form is refused at warm-up by name unless the pipeline declares a refit with Online(pipe), whose PortfolioOptimisers.PipelineBufferState holds the input carrier and reads out by a batch fit.

PortfolioOptimisers.partial_fit! — Method
partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{<:PipelineBufferState, <:ReturnsBufferState}}}, data::Prices_RR)

Folds a block of observations into a Pipeline, without fitting.

The Pipeline's online step: the pipeline is a host, and the verb walks the steps in order, handing each the block the step before it emitted, until the rows reach the row owner — the prior step, else the optimisation step (pipeline_row_owner). A row-local step folds and emits through partial_fit_transform; a universe-only step (is_universe_step) passes the rows through untouched, its universe deferred to the read-out; every step after the owner is untouched, and is fitted at the read-out exactly as batch fits it. A prior owner is folded through fold_prior, and the Pipeline records the rest of the carrier — the benchmark, the timestamps, the pinned names and a static panel — in a ReturnsBufferState of its own, taking the owner's cap, so that fit(pipe) can rebuild the carrier the batch path reads. An optimisation owner is folded through its own partial_fit!, and keeps its own Fold Context; the Pipeline then holds none.

Under Online(pipe) the pipeline carries a PipelineBufferState instead, and the block is appended to it: no step folds, and the read-out is a batch fit over the buffer.

The arity mirrors the batch verb: fit(pipe, data) takes a carrier of the pipeline's input level, so the step takes one, holding one observation or a block of them.

Arguments

  • pipe: The pipeline to fold into, its wrappers resolved by update_online_estimator.
  • data: The block of observations, price- or returns-level as the pipeline's input.

Validation

  • The pipeline has a row owner, and the rows reach it at the returns level. An ArgumentError is thrown otherwise.
  • Everything the steps' own folds refuse.

Returns

  • pipe: The pipeline, with every step before the owner and the owner itself folded, and its own context recorded.

Related

source
StatsAPI.fit — Method
fit(pipe::Pipeline)

Reads a folded Pipeline out: reconstitutes the carrier from the row owner's rows, refits every universe step over it, views the owner's state to the surviving assets, and runs the tail as batch.

The read-out verb of the Pipeline's online step, mirroring optimise(opt). It returns an ordinary PipelineResult, so predict, assert_universe_aligned and the search consume it unchanged.

Algorithm

  1. Rebuild the carrier of the observations folded so far, rd₀, from the row owner: a prior owner's rows through prior_returns_buffer and the Pipeline's own ReturnsBufferState, an optimisation owner's through its returns_result. The carrier is over the pipeline's input universe at warm-up width.
  2. Walk the data steps before the owner in order, through readout_data_step. A row-local step reads its fitted Result out of its state, restricted to the assets that survive so far. A universe-only step runs its batch verb over rd₀ viewed to the surviving assets, and narrows the surviving set. The index idx of the surviving assets into rd₀ is the column map, and rd = port_opt_view(rd₀, :, idx) is the context's returns.
  3. Run every other step before the owner — a phylogeny, an uncertainty-set or a constraint step — over that context, as batch runs it.
  4. Read the owner out over the surviving assets: a prior owner through prior(port_opt_view(pe, idx)), whose state is viewed by asset and never re-sliced; an optimisation owner through optimise(opt) on the viewed and injected estimator.
  5. Run every step after the owner as batch, the optimisation step with the context injected, and return the result.

So a selection that moves between two steps is expressed as a view of a state fitted over the whole universe, which is the batch fit over those columns, and the selector re-ranks per step exactly as the batch loop refits it per fold.

Under Online(pipe) the read-out is fit(pipe, buffer), the batch fit over the observations the PipelineBufferState holds.

Validation

  • The pipeline has taken a step. An ArgumentError is thrown otherwise.

Returns

  • res::PipelineResult: The result the batch fit over the observations folded so far gives.

Related

source
PortfolioOptimisers.cross_val_predict — Method
cross_val_predict(o::Online{<:Pipeline}, data::Prices_RR, cv::CVER; ex = FLoops.ThreadedEx(), id = nothing)
cross_val_predict(o::Online{<:Pipeline}, data::Prices_RR, cv::MultipleRandomised; ex = FLoops.ThreadedEx(), kwargs...)

Run a walk-forward over Online(pipe), the declared refit of a Pipeline from an input-carrier buffer.

The same doors as the pipeline's, with the wrapper threaded through the fold loop: the online arm resolves it at warm-up into a pipeline carrying a PipelineBufferState, every fold appends its new rows to the buffer, and the read-out is fit(pipe, buffer). So the run is exact for every configuration at batch cost — the window-valued steps the host route refuses included — and with max_history = w it equals the rolling batch walk-forward with warm-up w + purged_size. A scheme with no Fold Fit is refused by name, because the wrapper resolves only at the online arm's warm-up.

Related

source
PortfolioOptimisers.cross_val_predict — Method
cross_val_predict(r::PipelineResume, data::Prices_RR, cv::CVER; ex = FLoops.ThreadedEx(), id = nothing)

Continue an online walk-forward over a Pipeline from its Result, over the full history extended.

The pipeline door of Resume: the scheme is checked (assert_resume_scheme), the holdout refused as the one-shot door refuses it, and the fold loop takes its resumed arm through pipeline_cross_val_predict. A host route and an Online(pipe) refit route resume alike, because the entry reads the state off the pipeline generically.

Related

source