Online Pipeline

A Pipeline updates incrementally. partial_fit! passes each block of observations through the steps in order, up to the step that stores the rows: the prior step, or the optimiser step when the pipeline has no prior. fit(pipe) with no data returns the fitted PipelineResult. It rebuilds the returns from the rows that the step stored, and refits over them every step that chooses assets. It then restricts the state of the step that stores the rows to the assets that remain, and runs the steps after it as a batch fit.

A step that transforms each row on its own updates with each block. A step that only chooses assets waits until you read the result. A step with no incremental form makes the pipeline throw an error that names the step, before the first update. To run such a step, wrap the pipeline as Online(pipe). Its PortfolioOptimisers.PipelineBufferState then stores the input data, and reading the result runs a batch fit over it.

Online(pipe) and Resume(res) of the Result of a pipeline go through the method of a pipeline, cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER). Its docstring states what each of them adds.

PortfolioOptimisers.partial_fit! — Method
partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{PipelineBufferState, ReturnsBufferState}}}, data::Prices_RR)

Folds a block of observations into a Pipeline, without a fit.

This is the online step of the Pipeline, and the pipeline takes it for its steps. The verb hands each step before the row owner the block that the step before it emitted, and folds the rows into the owner, which is the prior step, else the optimisation step (pipeline_row_owner). The verb leaves every step after the owner untouched, and fit(pipe) with no data fits it as batch fits it. Under Online(pipe) the pipeline carries a PipelineBufferState instead. The verb then appends the block to the buffer, no step folds, and fit(pipe) with no data is a batch fit over the buffer.

The arity mirrors the batch verb. fit(pipe, data) takes a PricesResult or a ReturnsResult at the input level of the pipeline, so the online step takes one too, with one observation or a block of them.

Algorithm

On the fold route:

  1. Find the row owner, giving k.
  2. Walk the steps before k in order. Skip each step that writes no data slot (is_data_step) and each universe-only step (is_universe_step). A universe-only step passes the rows through untouched, and fit(pipe) with no data applies its universe.
  3. Fold each remaining step through partial_fit_transform, giving the folded step and the next block.
  4. Fold block into the owner through fold_pipeline_owner, giving the folded owner and cache. A prior owner folds through fold_prior, and cache is a ReturnsBufferState of the Pipeline that records the rest of the ReturnsResult, so that fit(pipe) can rebuild the ReturnsResult that the batch path reads. An optimisation owner folds through its own partial_fit! and keeps its own Fold Context, so cache is nothing.
  5. Return the pipeline rebuilt from the folded steps and cache.

On the refit route, append data to the buffer through partial_fit!, and return the pipeline with the new state. The refit route is the method on a pipeline whose cache is a PipelineBufferState, which is more specific than this one, so this method holds the fold route alone.

Arguments

  • pipe: The pipeline to fold into, with its wrappers resolved by update_online_estimator.
  • data: The block of observations, at the price level or at the returns level, as the input of the pipeline.

Validation

On the fold route:

  • The pipeline has a row owner. An ArgumentError is thrown otherwise.
  • No step holds an Online declaration (pipeline_online_member), because the warm-up resolves each one. A wrapper that no warm-up resolved would fold the estimator that it wraps with no buffer, and its cap would have no effect. An ArgumentError that names the step is thrown otherwise.
  • The rows reach the owner as a ReturnsResult. An ArgumentError is thrown otherwise.
  • Everything the folds of the steps refuse.

Returns

  • pipe: The pipeline, with the owner and every step before it folded, and its own state recorded.

Related

source
StatsAPI.fit — Method
fit(pipe::Pipeline)

Reads a folded Pipeline out, with the result that the batch fit over the observations folded so far gives.

This is the call with no data of the online step of the Pipeline, as optimise(opt) with no data is for an optimiser. It returns an ordinary PipelineResult, so predict, assert_universe_aligned and the search take it unchanged.

Algorithm

  1. Rebuild the ReturnsResult of the observations folded so far from the row owner through pipeline_returns_result, giving rd₀. A prior owner gives its rows through prior_returns_buffer, and the Pipeline adds the rest from its own ReturnsBufferState. An optimisation owner gives its ReturnsResult through returns_result. The ReturnsResult spans the input universe of the pipeline at warm-up width.
  2. Walk the data steps before the owner in order, through fit_data_step_from_state. A row-local step reads its fitted Result out of its state, restricted to the assets that survive so far. A universe-only step runs its batch verb over rd₀ viewed to the surviving assets, and narrows the surviving set. The index idx of the surviving assets into rd₀ is the column map, and rd = port_opt_view(rd₀, :, idx) is the context's returns.
  3. Run every other step before the owner over that context, as batch runs it. Such a step is a phylogeny step, an uncertainty-set step or a constraint step.
  4. Read the owner out over the surviving assets through fit_owner_from_state. A prior owner reads out through prior(port_opt_view(pe, idx)), which views its state by asset and never slices its rows again. An optimisation owner reads out through optimise(opt) on the viewed estimator, with the context injected.
  5. Run every step after the owner as batch, the optimisation step with the context injected, and return the result.

So a selection that moves between two steps is a view of a state fitted over the whole universe, which is the batch fit over those columns. The selector ranks the assets again at every step, as the batch loop refits it at every fold.

Under Online(pipe), fit(pipe) with no data is fit(pipe, buffer), the batch fit over the observations that the PipelineBufferState holds.

Validation

  • The pipeline has a row owner. An ArgumentError is thrown otherwise.
  • The pipeline has taken a step. An ArgumentError is thrown otherwise.
  • A prior owner reads out over as many assets as survive the universe steps before it (fit_owner_from_state). An ArgumentError is thrown otherwise.

Returns

  • res::PipelineResult: The result the batch fit over the observations folded so far gives.

Related

source
PortfolioOptimisers.partial_fit! — Method
partial_fit!(state::PipelineBufferState, data::Prices_RR) -> PipelineBufferState
partial_fit!(state::PipelineBufferState, ::Nothing) -> PipelineBufferState

Appends a block of observations to a PipelineBufferState, and drops the oldest rows past the cap.

The state is immutable, so the method returns a new state and leaves state as it was. A data of nothing holds no row, so that method returns state itself. merge_states passes the data of a second state, which is nothing before its first block.

Algorithm

  1. Concatenate the held data and data by rows through vcat_observations, giving held. When the state holds no data yet, held is data.
  2. Count the rows of held, giving n.
  3. When max_history is set and n > max_history, view the last max_history rows of held.
  4. Return a new state that holds held and carries the same cap.

Validation

Related

source