Online Pipeline: private API

PortfolioOptimisers.PipelineBufferState — Type
struct PipelineBufferState{__T_data, __T_max_history} <: AbstractPartialFitState

Holds every block of observations that Online(pipe) folds, concatenated, so fit(pipe) with no data is a batch fit over them.

This state is the declared refit of a Pipeline, and no step folds under it. The buffer holds the input of the pipeline as the caller gave it, at the price level or at the returns level, and fit(pipe) runs fit(pipe, data) over the buffer. So fit(pipe) with no data is exact for every configuration, at the cost of a batch fit. With a cap the buffer keeps the last max_history rows, and the run equals the rolling batch walk-forward.

The type of the state in pipe.cache selects the route. A ReturnsBufferState is the Fold Context of the fold route, and a PipelineBufferState is the refit route.

Fields

  • data: The observations folded so far, as one PricesResult or ReturnsResult at the pipeline's input level, or nothing before the first block.
  • max_history: Optional cap on the number of observations the buffer keeps. nothing keeps every observation folded so far. A capped buffer drops its oldest observations as new ones arrive and holds the last max_history of them. The call with no data reads those rows only. The cap is the window of the fit. An estimator wrapped in Online returns the batch fit over the last max_history observations, also when its statistic has an exact update.

Constructors

PipelineBufferState(; data::Option{<:Prices_RR} = nothing, max_history::Option{<:Integer} = nothing)

Validation

  • max_history > 0 when it is not nothing. A DomainError is thrown otherwise.

Related

source
PortfolioOptimisers.fold_pipeline_owner — Function
fold_pipeline_owner(pe::AbstractPriorEstimator, cache, rd::ReturnsResult,
                    gsrc::Option{<:ExcessSource})
fold_pipeline_owner(opt::OptimisationEstimator, cache, rd::ReturnsResult,
                    gsrc::Option{<:ExcessSource})
fold_pipeline_owner(opt::OnlinePortfolioSelection, cache, rd::ReturnsResult,
                    gsrc::Option{<:ExcessSource})

Folds a block into the row owner of a Pipeline, and records the own context of the Pipeline where the owner keeps none.

An optimisation owner takes the online step itself. It folds through fold_optimiser, the step of partial_fit! under the source of the group of the Pipeline, and the Pipeline records nothing beside it. The method refuses an OnlinePortfolioSelection head. This is the check of a hand-driven run, as assert_online_owner is the check of the fold loop, and that docstring states the reason.

A prior owner folds from the source that its own x_src states, as the batch step fits from it (run_step). The marks of a gap of a vector benchmark follow the group of the optimiser that receives the prior, gsrc, as the batch step drops the gaps of that group (run_group_step). A prior that no optimiser receives marks by its own source alone. Under ExcessSource() the buffer of the prior holds the excess returns X - B, so the context of the Pipeline keeps the raw rows, and fit(pipe) with no data rebuilds the raw ReturnsResult, as an optimiser does (fold_returns).

Algorithm

For a prior owner:

  1. Fold rd into the prior through fold_prior, which reads the source of the prior, giving pe. The marks of a gap of a vector benchmark read gsrc when it is ExcessSource(), and the source of the prior otherwise, see gap_source. This goes first, because the context takes its cap from the buffer that the prior seeds, as the context of an optimiser does (fold_returns).
  2. Read the buffer of the prior through prior_returns_buffer, giving rows.
  3. Set own_factors when needs_factor_returns answers false for pe. The context keeps the factor columns only then, so the rebuilt ReturnsResult holds each factor column once. Set own_exogenous when reads_exogenous_series answers false for pe, on the same terms for the Exogenous Series.
  4. Fold rd into the context of the Pipeline through fold_context under the cap rows.max_history, giving cache. The context keeps the raw returns when the prior states x_src = ExcessSource().
  5. Return pe and cache.

Arguments

  • pe, opt: The row owner, without its PipelineStep wrapper.
  • cache: The Fold Context of the Pipeline, or nothing before the first step.
  • rd: The block of returns that reaches the owner.
  • gsrc: The source of the gap drop of the owner, see pipeline_gap_sources.

Validation

Related

source
PortfolioOptimisers.fit_pipeline_from_state — Function
fit_pipeline_from_state(pipe::Pipeline, cache::PipelineBufferState)
fit_pipeline_from_state(pipe::Pipeline, cache::Option{<:ReturnsBufferState})

The two routes of fit(pipe::Pipeline), selected by dispatch on the state that the Pipeline carries.

A buffer of the input data takes a batch fit. A Fold Context, or no state, takes the walk of the steps. The steps and the refusals are those of the verb.

Related

source
PortfolioOptimisers.pipeline_returns_result — Function
pipeline_returns_result(pe::AbstractPriorEstimator, cache::ReturnsBufferState, name)
pipeline_returns_result(opt::OptimisationEstimator, ::Nothing, name)
pipeline_returns_result(owner, ::Nothing, name)

Rebuilds the ReturnsResult of the observations that a Pipeline folded, from its row owner.

A prior owner rebuilds it from the Fold Context of the Pipeline and the buffer of the prior. An optimisation owner rebuilds it from its own Fold Context (returns_result).

Validation

  • The pipeline has taken a step, so it carries a Fold Context or its optimisation owner carries a state. An ArgumentError that names the owner is thrown otherwise.

Related

source
PortfolioOptimisers.fit_data_step_from_state — Function
fit_data_step_from_state(est, rd0::ReturnsResult, idx::AbstractVector{<:Integer}) -> (fitted, idx′)

Reads one data step of a folded Pipeline out, and narrows the surviving assets where the universe of the step does.

Dispatch on the estimator of the step selects the method.

Arguments

  • est: The step's estimator, carrying its state.
  • rd0: The ReturnsResult of every observation folded, over the pipeline's input universe.
  • idx: The indices into rd0 of the assets surviving the steps before this one.

Returns

  • (fitted, idx′): The step's fitted Result, and the surviving indices after it.

Related

source
PortfolioOptimisers.fit_owner_from_state — Function
fit_owner_from_state(pe::AbstractPriorEstimator, idx, n::Integer, ctx::PipelineContext)
fit_owner_from_state(opt::OptimisationEstimator, idx, n::Integer, ctx::PipelineContext)

Reads the row owner of a folded Pipeline out over the surviving assets, and writes its slot.

The method views the state of the owner to the surviving assets through view_owner before it reads the owner out. The view is a slice by asset. A state fitted over the full universe, viewed to a set of columns and read out, is the batch fit over that set. A prior owner reads out through prior(pe). The method injects the context into an optimisation owner and reads it out through optimise(opt), so the clustering, the constraints and every uncertainty set come from the rebuilt ReturnsResult, as batch fits them.

Validation

  • A prior owner reads out over as many assets as survive. Otherwise its state has no asset view, and the method throws an ArgumentError that asks for the universe steps after the prior.

Related

source
PortfolioOptimisers.pipeline_fold_fit — Function
pipeline_fold_fit(pipe::AbstractPipelineEstimator, data::Prices_RR, train_idx::VecInt, cols = :)
pipeline_fold_fit(pipe::AbstractPipelineEstimator, data::Prices_RR, ::Nothing, cols = :)

The fit of one fold of a pipeline, by whether the fold carries a training window.

This is the pipeline form of fit_fold_result. A window fits the workflow over it, fit(pipe, data[train_idx]), which is the refit that every fold of the batch arms runs. nothing means that the pipeline holds its window, because the online arm of fold_loop folded every row of it. The fold then reads the pipeline out through fit(pipe) with no data. The two methods call only fit, so they serve every AbstractPipelineEstimator. The second method needs fit(pipe), which only a pipeline that holds a partial-fit state implements.

Related

source
PortfolioOptimisers.rewrap_step — Function
rewrap_step(step, est)

Puts a folded estimator back in the wrapper that its step came in.

A PipelineStep keeps its reads, its writes and its target, and a bare step is the estimator itself. An Online wrapper never comes back, because the warm-up resolved it before the first fold.

Related

source
PortfolioOptimisers.replace_step — Function
replace_step(steps::Tuple, step, i::Integer) -> Tuple
replace_step(steps::AbstractVector, step, i::Integer) -> Vector

Return the steps of a Pipeline with step at position i.

A fold can change the type of a step, so the new container takes the type of step. The Tuple method calls Base.setindex. The AbstractVector method copies the steps into a vector whose element type joins the old element type and the type of step, through Base.promote_typejoin. Neither method changes steps, so the pipeline that holds them keeps its state.

Arguments

  • steps: The step estimators, a Tuple or an AbstractVector, as the user gave them.
  • step: The new step.
  • i: The position of the new step.

Returns

  • The steps, in the container form of steps, with step at position i.

Related

source
PortfolioOptimisers.is_universe_step — Function
is_universe_step(est) -> Bool

Answers whether a data step is universe-only.

A universe-only step folds nothing. In fit(pipe) with no data its batch verb runs over the rows of the row owner, and the call applies its universe as a view. The asset selectors are universe-only, and every other step answers false.

Related

source
PortfolioOptimisers.pipeline_row_owner — Function
pipeline_row_owner(p::Pipeline) -> Any

Finds the index of the row owner of a Pipeline, or answers 0 when the pipeline has none.

A prior step owns the rows when one exists, because inject_context replaces the pe of the optimiser with it, and the prior of the optimiser is never fitted. Without a prior step, the optimisation step owns the rows. It then takes the online step itself, and it keeps its own Fold Context. A pipeline with neither has nothing to fold the rows into.

Algorithm

  1. Find the first step whose estimator (step_estimator) is a prior, giving k. Return k when it exists.
  2. Otherwise find the first step whose estimator is a row owner (is_row_owner), giving k. Return k, or 0 when no step is one.

Related

source
PortfolioOptimisers.pipeline_online_member — Function
pipeline_online_member(p::Pipeline) -> Any

Names the first Online declaration under the steps of a Pipeline, prefixed with the name of the step, or answers nothing.

Two refusals read this walk. Under Online(pipe) no member folds, so a wrapper below the root has no fold to seed and its cap would have no effect, and assert_online_entry refuses it with an error that names it. On the fold route the warm-up resolves every wrapper, so the fold refuses a wrapper that it still meets.

Related

source
PortfolioOptimisers.step_online_member — Function
step_online_member(step)

Names the first Online declaration that one Pipeline step carries, relative to the step.

The answer is "" when the step is itself a wrapper, and the dotted path of the wrapper when the estimator of the step holds one at any depth (online_wrapper_path). Otherwise it is nothing. A PipelineStep answers for its estimator, and a nested Pipeline answers through pipeline_online_member. The walk goes to every depth because the warm-up does too. update_online_estimator resolves a wrapper under the prior of an optimiser, and under the inner prior of a prior that holds one.

Related

source
PortfolioOptimisers.step_online_cap — Function
step_online_cap(step)

Reads the first max_history that an Online declaration of one Pipeline step carries, or answers nothing when no wrapper of the step carries a cap.

The refusal of a capped owner on the fold route reads this cap. A cap on the owner is a window counted in the rows of the owner, and a row-local step before the owner folds a state across the front edge of that window. The walk goes to every depth, as step_online_member does, so a cap on the inner prior of a prior under an optimiser counts.

Algorithm

  1. A wrapper answers its own max_history. An Online never wraps another, so the walk does not enter it.
  2. A PipelineStep answers for its estimator.
  3. An estimator walks its estimator-valued fields (estimator_fields) in order, and answers the first cap that a field answers, or nothing.
  4. Any other value answers nothing.

Related

source
PortfolioOptimisers.assert_online_entry — Method
assert_online_entry(p::Pipeline)
assert_online_entry(o::Online{<:Pipeline})

Refuses a Pipeline that cannot take the online step, with an error that names the step.

The fold loop calls it at the entry of its online arm, before any fit. The checks depend on the route. The fold route folds the rows through the steps into a row owner, and it has the most checks. The refit route, Online(pipe), folds no member, so it has two.

The rolling window through a Pipeline is Online(pipe; max_history = w). A capped owner stays on the fold route only when every data step before it is universe-only, because fit(pipe) with no data refits a universe-only step over the capped rows of the owner.

Validation

On the fold route, in this order:

  • The pipeline carries no state, in its own cache or under a step (online_entry_state), because the loop starts cold.
  • The pipeline has a row owner (pipeline_row_owner), a prior step or an optimisation step.
  • The row owner passes assert_online_owner. It is not a TimeDependent schedule and not an OnlinePortfolioSelection head, and an optimisation step that owns the rows passes the checks of assert_online_entry for an optimiser. A schedule owns the rows only when no prior step precedes it. With a prior step before it, the schedule swaps a stateless step and composes.
  • Every data step before the owner can fold or defer. The check refuses a nested Pipeline, a data step that is not a preprocessing estimator, and a preprocessing estimator with no online form (supports_partial_fit). A callable PipelineStep that writes :prices or :returns is not a preprocessing estimator. PriceGapFill with a statistic fill and MissingDataFilter with row_thr < 1 are window-valued, and have no online form. The message names both ways out, a partial_fit_transform for the step or a refit with Online(pipe).
  • When a wrapper on the owner carries a cap (step_online_cap), every data step before the owner is universe-only (is_universe_step). The owner counts its window in its own rows, and the carry of a row-local step reads rows that the window dropped, so the run equals no batch scheme. This holds on returns input too.

On the refit route:

  • The pipeline carries no state, as on the fold route.
  • No step carries an Online declaration of its own (pipeline_online_member), because no member folds under a refit and its cap would have no effect.

Each refusal throws an ArgumentError that names the step.

Related

source
PortfolioOptimisers.assert_online_owner — Function
assert_online_owner(owner, name::AbstractString)

Refuses a row owner that cannot fold, with an error that names the step.

A prior passes. An optimisation step passes when it passes the checks of assert_online_entry for an optimiser. A precomputed result never reaches this check. A result is not a step, so it enters a pipeline only as an entry of a schedule, and the check refuses the schedule first.

The check refuses an OnlinePortfolioSelection head because of its call with no data, not because of its fold. optimise(opt) with no data on the head answers a Recursion Result, so it rebuilds no ReturnsResult (batch_from_state). fit(pipe) with no data rebuilds a ReturnsResult from the row owner, and then refits the universe steps over it. Two things are missing, and a ReturnsResult would supply only the first. A rule whose tree reads no rows holds no rows, so nothing exists to rebuild the ReturnsResult from. And fit(pipe) with no data expresses a selection as a view of the state of the owner. For a recursion that view is not the run over those columns, because the allocation is a path, the projection couples the columns, and the wealth factor reads every column.

Validation

Each refusal throws an ArgumentError that names the step.

Related

source
PortfolioOptimisers.assert_pipeline_entry — Function
assert_pipeline_entry(pipe::Pipeline, _)

Runs the entry checks of the cross_val_predict methods of a Pipeline.

Validation

Each refusal throws an ArgumentError.

Related

source
PortfolioOptimisers.run_step — Function
run_step(o::Online, ::PipelineContext)
run_step(
    o::Online,
    ::PipelineContext,
    ::Union{Nothing, PipelineStep}
)

Refuses an Online step that reaches a fit with no folds, with an error that names it.

The warm-up of the online arm resolves a wrapper, and fit(pipe, data) runs no warm-up. A wrapper that reaches it has no buffer to seed and no step to fold. A fit of the wrapped estimator as a plain one would return a batch answer under an online declaration, so the method refuses the step instead.

Validation

  • The method always throws an ArgumentError. The message names the wrapped estimator and the two ways out: an Online Scheme, or the plain estimator.

Related

source
PortfolioOptimisers.update_online_estimator — Method
update_online_estimator(p::Pipeline)

Resolves the Online declarations in the steps of a Pipeline, at warm-up.

On the fold route update_online_estimator resolves every step: a bare Online step, an estimator, and a PipelineStep through update_online_estimator(ps::PipelineStep). A callable and a result pass through its fallback unchanged. So an Online(EmpiricalPrior(); max_history = w) step becomes the prior with its buffer, and an optimisation step resolves the wrappers under its own prior. On the refit route the pipeline carries a PipelineBufferState, which Online(pipe) seeded, and the method returns the pipeline unchanged. No member folds under a refit, and assert_online_entry already refused any wrapper below the root with an error that names it.

Related

source
PortfolioOptimisers.update_online_estimator — Method
update_online_estimator(ps::PipelineStep)

Resolves the Online declarations of the estimator that a PipelineStep wraps, at warm-up.

The method rebuilds the wrapper around the resolved estimator with rebuild_struct, so the step keeps its reads, its writes and its target. A wrapped Online becomes the estimator that it wraps, with a seeded buffer, and a wrapped estimator resolves the wrappers in its own fields. A wrapped callable passes through the fallback of update_online_estimator unchanged.

Related

source
PortfolioOptimisers.is_time_dependent — Method
is_time_dependent(o::Online{<:Pipeline})
needs_previous_weights(o::Online{<:Pipeline})
assert_time_dependent_fold_count(o::Online{<:Pipeline}, n::Integer, all_binds::Bool = true)

Answers the three traits of the fold loop for an Online(pipe) root, as the pipeline that it wraps.

The fold loop reads the traits off the root that it receives. The wrapper resolves at the warm-up of the online arm, and the loop makes its per-fold copy of the pipeline, so the pipeline answers.

Related

source
PortfolioOptimisers.show_fields — Method
show_fields(
    p::Pipeline
) -> Union{Tuple{Symbol, Symbol}, Tuple{Symbol, Symbol, Symbol}}

Lists the fields that show renders for a Pipeline: the names, the steps, and the cache only when it holds a state.

A pipeline that took no step therefore renders no cache line.

Related

source
PortfolioOptimisers.copy_states — Method
copy_states(p::Pipeline)

Copies every partial-fit state that a Pipeline carries, and rebuilds the pipeline around the copies, for Resume.

The states are the own cache of the pipeline and the states under every step, which copy_states copies. A PipelineStep copies through copy_states(ps::PipelineStep), and a callable and a result pass through the fallback unchanged. A fold of the copy leaves the held rows of the original unchanged.

Related

source
PortfolioOptimisers.held_timestamps — Method
held_timestamps(p::Pipeline)

Returns the timestamps that a stepped Pipeline holds, through the state of its row owner, or nothing.

The state that the pipeline carries selects the answer.

  • Under Online(pipe) the PipelineBufferState holds the input data, and the timestamps of that data are the answer.
  • A prior owner leaves the Pipeline its own ReturnsBufferState, which holds them.
  • An optimisation owner keeps its own Fold Context, and the pipeline holds none, so the owner answers (held_timestamps).
  • A pipeline whose owner carries no state, or that has no owner, has taken no step, and answers nothing.

At the price level the answer of the fold route starts one row after the answer of the refit route. A PricesToReturns step drops the first price row, so the returns carry the timestamps of the prices from the second row on. Both answers are a run of the timestamps of the price data that ends at the last row folded, which is what the alignment check of Resume reads.

Related

source
PortfolioOptimisers.held_timestamps — Method
held_timestamps(p::Pipeline, cache::PipelineBufferState)
held_timestamps(p::Pipeline, cache::ReturnsBufferState)
held_timestamps(p::Pipeline, ::Nothing)

The three arms of held_timestamps(p::Pipeline), selected by dispatch on the state that the pipeline carries, which the one-argument method passes.

A buffer of the input data answers with the timestamps of that data, and a Fold Context answers with its own. With no state, the row owner (pipeline_row_owner) answers when it carries a state (online_entry_state), and the method answers nothing otherwise.

Related

source