GitHub

The Pipeline's online step: private API

PortfolioOptimisers.PipelineBufferState — Type
struct PipelineBufferState{__T_data, __T_max_history} <: AbstractPartialFitState

The input-carrier buffer Online(pipe) seeds: every block of observations a Pipeline is handed, concatenated, so the read-out is the batch fit over them.

The declared refit. No step folds under it — the buffer holds the pipeline's input as it was given, price- or returns-level, and fit(pipe) runs fit(pipe, data) over the buffer — so it is exact for every configuration at batch cost, and with a cap it is a rolling Pipeline, equal to the rolling batch walk-forward. The state's type is the route: a ReturnsBufferState in pipe.cache is the host route's Fold Context, and this is the refit route.

Fields

  • data: The observations folded so far, as one carrier of the pipeline's input level, or nothing before the first block.
  • max_history: Optional cap on the number of observations the carry buffer keeps. nothing keeps every observation folded so far. A cap bounds memory and nothing else: it does not make the estimator windowed, because an estimate that folds exactly is fitted over every observation whatever the buffer holds.

Constructors

PipelineBufferState(; data::Option{<:Prices_RR} = nothing, max_history::Option{<:Integer} = nothing)

Validation

  • max_history > 0 when it is not nothing. A DomainError is thrown otherwise.

Related

source
PortfolioOptimisers.fold_pipeline — Function
fold_pipeline(pipe::Pipeline, cache::PipelineBufferState, data::Prices_RR)
fold_pipeline(pipe::Pipeline, cache::Option{<:ReturnsBufferState}, data::Prices_RR)

The two routes of partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{<:PipelineBufferState, <:ReturnsBufferState}}}, data::Prices_RR), chosen by dispatch on the state the Pipeline carries: an input-carrier buffer appends the block, and a Fold Context, or none, walks the steps.

Related

source
PortfolioOptimisers.fold_pipeline_owner — Function
fold_pipeline_owner(pe::AbstractPriorEstimator, cache, rd::ReturnsResult)
fold_pipeline_owner(opt::OptimisationEstimator, cache, rd::ReturnsResult)

Folds a block into the row owner of a Pipeline and records the Pipeline's own context where the owner keeps none.

A prior owner is folded through fold_prior, first, because the Pipeline's context takes its cap from the buffer the prior seeds, exactly as an optimiser's does (fold_returns). The factor column is owned once, on the same terms: the context keeps it only when the prior's tree never reads it. An optimisation owner is itself a host and folds through its own step, and the Pipeline records nothing beside it.

Related

source
PortfolioOptimisers.readout_data_step — Function
readout_data_step(est, rd0::ReturnsResult, idx::AbstractVector{<:Integer}) -> (fitted, idx′)

Reads one data step of a folded Pipeline out, and narrows the surviving assets where the step's universe does.

The method Julia selects is the step's class. A PricesToReturns reads itself out. A PriceGapFill reads the PriceGapFillResult of the whole history out of its state, restricted to the assets surviving so far, because the batch fit was made over the columns an earlier filter left. A MissingDataFilter reads its MissingDataFilterResult out the same way, and narrows the surviving set to the assets it keeps, in the order they hold. An AbstractAssetSelector runs its batch verb over the owner's rows viewed to the surviving assets, and narrows the set to the fitted universe in fitted order, as apply_preprocessing orders it. Any other row-local step reads its own read-out out.

Arguments

  • est: The step's estimator, carrying its state.
  • rd0: The carrier of every observation folded, over the pipeline's input universe.
  • idx: The indices into rd0 of the assets surviving the steps before this one.

Returns

  • (fitted, idx′): The step's fitted Result, and the surviving indices after it.

Related

source
PortfolioOptimisers.readout_owner — Function
readout_owner(pe::AbstractPriorEstimator, idx, n::Integer, ctx::PipelineContext)
readout_owner(opt::OptimisationEstimator, idx, n::Integer, ctx::PipelineContext)

Reads the row owner of a folded Pipeline out over the surviving assets, and writes its slot.

The owner's state is viewed to the surviving assets through port_opt_view before it is read out — a slice by asset: a state fitted over the full universe, viewed to a column set and read out, is the batch fit over that column set. A prior owner reads out through prior(pe); an optimisation owner is injected with the context and read out through optimise(opt), so the clustering, the constraints and every uncertainty set are fitted from the reconstituted carrier exactly as batch fits them.

Related

source
PortfolioOptimisers.pipeline_fold_fit — Function
pipeline_fold_fit(pipe::Pipeline, data::Prices_RR, train_idx::VecInt, cols = :)
pipeline_fold_fit(pipe::Pipeline, data::Prices_RR, ::Nothing, cols = :)

The fit of one fold of a Pipeline, by whether the fold carries a training window.

The pipeline's twin of fit_fold_result. A window fits the workflow over it, fit(pipe, data[train_idx]), which is the refit every fold of the batch arms runs. nothing says the pipeline holds its window — the online arm of fold_loop has folded every row of it — so the fold reads the pipeline out through fit(pipe) with no data.

Related

source
PortfolioOptimisers.is_row_owner — Function
is_row_owner(est) -> Bool

Answers whether a step can own the rows of a Pipeline's online step: a prior estimator, or an optimisation step of any form, a schedule included, so that the walk names the one it meets and the refusals below state why a schedule cannot fold.

Related

source
PortfolioOptimisers.pipeline_row_owner — Function
pipeline_row_owner(p::Pipeline) -> Any

Finds the row owner of a Pipeline by a walk: the first prior step, else the optimisation step, else 0.

The prior step owns the rows when there is one, because inject_context overrides the optimiser's pe with it and the optimiser's own prior is never fitted; else the optimisation step, which is then itself a host and keeps its own Fold Context; and a pipeline with neither has nothing to fold into.

Related

source
PortfolioOptimisers.step_online_cap — Function
step_online_cap(step)

Reads the max_history of the first Online declaration one Pipeline step carries, walking the paths step_online_member names: the step's own wrapper, the first field of its estimator holding one, or the bundle a JuMP or hierarchical head holds. Answers nothing when the step carries no wrapper, or when its wrapper carries no cap.

The read the host route's refusal of a capped owner runs: a cap on the owner is a window counted in the owner's rows, and a row-local step before it folds a state across that window's front edge.

Related

source
PortfolioOptimisers.assert_online_entry — Method
assert_online_entry(p::Pipeline)
assert_online_entry(o::Online{<:Pipeline})

Refuses a Pipeline that cannot take the online step, at the entry of the fold loop's online arm and before any fit, by name.

The walk goes over the steps of the route. On the host route, six refusals. A state anywhere — the Pipeline's own cache or a step's — because the loop starts cold. No row owner: a pipeline with neither a prior nor an optimisation step has nothing to fold into. A TimeDependent schedule as the row owner, which is the case only when no prior step precedes the optimisation step: a schedule swaps the estimator that carries the state; with a prior step before it, the schedule swaps a stateless step and composes. An optimisation step that owns the rows is held to assert_online_entry's own refusals. And every data step before the owner must fold or defer: a window-valued configuration (PriceGapFill with a statistic fill, MissingDataFilter with row_thr < 1), a callable PipelineStep writing :prices or :returns, a nested Pipeline, and a caller's preprocessing estimator with no online form are refused, and the message names both routes: give the step a partial_fit_transform, or declare a refit with Online(pipe). And a capped owner — an Online on the owner's path carrying max_history, read through step_online_cap — is refused when a row-local step folds before it: the owner's window is counted in its own rows, and the step's carry reaches across the window's front edge, so the run equals no batch scheme. The rolling window through a Pipeline is Online(pipe; max_history = w). A capped owner behind universe-only steps alone, or on returns input, stays on the host route, because the read-out refits a universe-only step over the owner's capped rows.

On the refit route, two: a state anywhere, as above, and an Online member below the Online(pipe), because no member folds under a refit and its cap would be silently ignored.

Validation

  • Everything above. An ArgumentError naming the step is thrown otherwise.

Related

source
PortfolioOptimisers.update_online_estimator — Method
update_online_estimator(p::Pipeline)

Resolves the Online declarations a Pipeline carries in its steps, at warm-up.

On the host route every step is resolved through update_online_step, so an Online(EmpiricalPrior(); max_history = w) step becomes the prior carrying its buffer, and an optimisation step resolves the wrappers under its own prior. On the refit route — the pipeline carries a PipelineBufferState, which Online(pipe) seeded — no member is seeded, because no member folds under a refit; assert_online_entry has refused any wrapper below the root by name already.

Related

source
PortfolioOptimisers.is_time_dependent — Method
is_time_dependent(o::Online{<:Pipeline})
needs_previous_weights(o::Online{<:Pipeline})
assert_time_dependent_fold_count(o::Online{<:Pipeline}, n::Integer, all_binds::Bool = true)

The fold loop reads its three traits off the root it is handed, and an Online(pipe) root answers for the pipeline it wraps: the wrapper resolves at the online arm's warm-up, and the per-fold copy is made of the pipeline.

Related

source
PortfolioOptimisers.held_timestamps — Method
held_timestamps(p::Pipeline)

The timestamps a stepped Pipeline holds, through the state its row owner keeps, or nothing.

Three arms, by the state the pipeline carries. Under Online(pipe) the PipelineBufferState holds the input carrier itself, and its timestamps are the answer. A prior owner leaves the Pipeline its own ReturnsBufferState, which holds them. An optimisation owner keeps its own Fold Context, and the pipeline holds none, so the owner answers (held_timestamps). At the price level a PricesToReturns step drops the first row, and the returns' timestamps are the prices' from the second row on, so the held span still equals its rows of the price carrier.

Related

source
PortfolioOptimisers.pipeline_held_timestamps — Function
pipeline_held_timestamps(p::Pipeline, cache::PipelineBufferState)
pipeline_held_timestamps(p::Pipeline, cache::ReturnsBufferState)
pipeline_held_timestamps(p::Pipeline, ::Nothing)

The three arms of held_timestamps(p::Pipeline), chosen by dispatch on the state the pipeline carries: the input-carrier buffer's timestamps, the Fold Context's, or the optimisation owner's through pipeline_row_owner.

Related

source