Online Pipeline: private API
PortfolioOptimisers.PipelineResume — Type
PipelineResume = Resume{<:Any, <:Pipeline}Alias for a Resume whose Result carries a Pipeline.
Pipeline_OnlPipe holds it, so a resumed pipeline takes the cross_val_predict methods of the pipeline, with its own entry check, and not those of an optimiser. The second type parameter of Resume is the type of the estimator that the Result carries.
Related
PortfolioOptimisers.PipelineBufferState — Type
struct PipelineBufferState{__T_data, __T_max_history} <: AbstractPartialFitStateHolds every block of observations that Online(pipe) folds, concatenated, so fit(pipe) with no data is a batch fit over them.
This state is the declared refit of a Pipeline, and no step folds under it. The buffer holds the input of the pipeline as the caller gave it, at the price level or at the returns level, and fit(pipe) runs fit(pipe, data) over the buffer. So fit(pipe) with no data is exact for every configuration, at the cost of a batch fit. With a cap the buffer keeps the last max_history rows, and the run equals the rolling batch walk-forward.
The type of the state in pipe.cache selects the route. A ReturnsBufferState is the Fold Context of the fold route, and a PipelineBufferState is the refit route.
Fields
data: The observations folded so far, as onePricesResultorReturnsResultat the pipeline's input level, ornothingbefore the first block.
max_history: Optional cap on the number of observations the buffer keeps.nothingkeeps every observation folded so far. A capped buffer drops its oldest observations as new ones arrive and holds the lastmax_historyof them. The call with no data reads those rows only. The cap is the window of the fit. An estimator wrapped inOnlinereturns the batch fit over the lastmax_historyobservations, also when its statistic has an exact update.
Constructors
PipelineBufferState(; data::Option{<:Prices_RR} = nothing, max_history::Option{<:Integer} = nothing)Validation
max_history > 0when it is notnothing. ADomainErroris thrown otherwise.
Related
PortfolioOptimisers.fold_pipeline_owner — Function
fold_pipeline_owner(pe::AbstractPriorEstimator, cache, rd::ReturnsResult,
gsrc::Option{<:ExcessSource})
fold_pipeline_owner(opt::OptimisationEstimator, cache, rd::ReturnsResult,
gsrc::Option{<:ExcessSource})
fold_pipeline_owner(opt::OnlinePortfolioSelection, cache, rd::ReturnsResult,
gsrc::Option{<:ExcessSource})Folds a block into the row owner of a Pipeline, and records the own context of the Pipeline where the owner keeps none.
An optimisation owner takes the online step itself. It folds through fold_optimiser, the step of partial_fit! under the source of the group of the Pipeline, and the Pipeline records nothing beside it. The method refuses an OnlinePortfolioSelection head. This is the check of a hand-driven run, as assert_online_owner is the check of the fold loop, and that docstring states the reason.
A prior owner folds from the source that its own x_src states, as the batch step fits from it (run_step). The marks of a gap of a vector benchmark follow the group of the optimiser that receives the prior, gsrc, as the batch step drops the gaps of that group (run_group_step). A prior that no optimiser receives marks by its own source alone. Under ExcessSource() the buffer of the prior holds the excess returns X - B, so the context of the Pipeline keeps the raw rows, and fit(pipe) with no data rebuilds the raw ReturnsResult, as an optimiser does (fold_returns).
Algorithm
For a prior owner:
- Fold
rdinto the prior throughfold_prior, which reads the source of the prior, givingpe. The marks of a gap of a vector benchmark readgsrcwhen it isExcessSource(), and the source of the prior otherwise, seegap_source. This goes first, because the context takes its cap from the buffer that the prior seeds, as the context of an optimiser does (fold_returns). - Read the buffer of the prior through
prior_returns_buffer, givingrows. - Set
own_factorswhenneeds_factor_returnsanswersfalseforpe. The context keeps the factor columns only then, so the rebuiltReturnsResultholds each factor column once. Setown_exogenouswhenreads_exogenous_seriesanswersfalseforpe, on the same terms for the Exogenous Series. - Fold
rdinto the context of the Pipeline throughfold_contextunder the caprows.max_history, givingcache. The context keeps the raw returns when the prior statesx_src = ExcessSource(). - Return
peandcache.
Arguments
pe,opt: The row owner, without itsPipelineStepwrapper.cache: The Fold Context of the Pipeline, ornothingbefore the first step.rd: The block of returns that reaches the owner.gsrc: The source of the gap drop of the owner, seepipeline_gap_sources.
Validation
- The owner is not an
OnlinePortfolioSelectionhead. AnArgumentErroris thrown otherwise.
Related
PortfolioOptimisers.fit_pipeline_from_state — Function
fit_pipeline_from_state(pipe::Pipeline, cache::PipelineBufferState)
fit_pipeline_from_state(pipe::Pipeline, cache::Option{<:ReturnsBufferState})The two routes of fit(pipe::Pipeline), selected by dispatch on the state that the Pipeline carries.
A buffer of the input data takes a batch fit. A Fold Context, or no state, takes the walk of the steps. The steps and the refusals are those of the verb.
Related
PortfolioOptimisers.pipeline_returns_result — Function
pipeline_returns_result(pe::AbstractPriorEstimator, cache::ReturnsBufferState, name)
pipeline_returns_result(opt::OptimisationEstimator, ::Nothing, name)
pipeline_returns_result(owner, ::Nothing, name)Rebuilds the ReturnsResult of the observations that a Pipeline folded, from its row owner.
A prior owner rebuilds it from the Fold Context of the Pipeline and the buffer of the prior. An optimisation owner rebuilds it from its own Fold Context (returns_result).
Validation
- The pipeline has taken a step, so it carries a Fold Context or its optimisation owner carries a state. An
ArgumentErrorthat names the owner is thrown otherwise.
Related
PortfolioOptimisers.fit_data_step_from_state — Function
fit_data_step_from_state(est, rd0::ReturnsResult, idx::AbstractVector{<:Integer}) -> (fitted, idx′)Reads one data step of a folded Pipeline out, and narrows the surviving assets where the universe of the step does.
Dispatch on the estimator of the step selects the method.
- A
PriceGapFillreads thePriceGapFillResultof the whole history out of its state, restricted to the assets that survive so far, because the batch fit ran over the columns that an earlier filter left. - A
MissingDataFilterreads itsMissingDataFilterResultout the same way, and narrows the surviving assets to the ones that it keeps, in the order that they hold. - An
AbstractAssetSelectorruns its batch verb over the rows of the owner, viewed to the surviving assets. It narrows them to the fitted universe, in the fitted order, asapply_preprocessingorders it. - Any other row-local step, such as a
PricesToReturns, reads out its own state throughfit_preprocessingand narrows nothing.
Arguments
est: The step's estimator, carrying its state.rd0: TheReturnsResultof every observation folded, over the pipeline's input universe.idx: The indices intord0of the assets surviving the steps before this one.
Returns
(fitted, idx′): The step's fitted Result, and the surviving indices after it.
Related
PortfolioOptimisers.fit_owner_from_state — Function
fit_owner_from_state(pe::AbstractPriorEstimator, idx, n::Integer, ctx::PipelineContext)
fit_owner_from_state(opt::OptimisationEstimator, idx, n::Integer, ctx::PipelineContext)Reads the row owner of a folded Pipeline out over the surviving assets, and writes its slot.
The method views the state of the owner to the surviving assets through view_owner before it reads the owner out. The view is a slice by asset. A state fitted over the full universe, viewed to a set of columns and read out, is the batch fit over that set. A prior owner reads out through prior(pe). The method injects the context into an optimisation owner and reads it out through optimise(opt), so the clustering, the constraints and every uncertainty set come from the rebuilt ReturnsResult, as batch fits them.
Validation
- A prior owner reads out over as many assets as survive. Otherwise its state has no asset view, and the method throws an
ArgumentErrorthat asks for the universe steps after the prior.
Related
PortfolioOptimisers.view_owner — Function
view_owner(
est,
idx,
n::Integer,
ctx::PipelineContext
) -> Any
Views the row owner's state to the surviving assets, and leaves it alone when every one of the n assets survives in its own order.
Related
PortfolioOptimisers.pipeline_fold_fit — Function
pipeline_fold_fit(pipe::AbstractPipelineEstimator, data::Prices_RR, train_idx::VecInt, cols = :)
pipeline_fold_fit(pipe::AbstractPipelineEstimator, data::Prices_RR, ::Nothing, cols = :)The fit of one fold of a pipeline, by whether the fold carries a training window.
This is the pipeline form of fit_fold_result. A window fits the workflow over it, fit(pipe, data[train_idx]), which is the refit that every fold of the batch arms runs. nothing means that the pipeline holds its window, because the online arm of fold_loop folded every row of it. The fold then reads the pipeline out through fit(pipe) with no data. The two methods call only fit, so they serve every AbstractPipelineEstimator. The second method needs fit(pipe), which only a pipeline that holds a partial-fit state implements.
Related
PortfolioOptimisers.step_estimator — Function
step_estimator(step)Returns the estimator that a Pipeline step stands for, unwrapped from a PipelineStep and from an Online declaration.
The online step classifies each step by this estimator. A callable step comes back as it is.
Related
PortfolioOptimisers.rewrap_step — Function
rewrap_step(step, est)Puts a folded estimator back in the wrapper that its step came in.
A PipelineStep keeps its reads, its writes and its target, and a bare step is the estimator itself. An Online wrapper never comes back, because the warm-up resolved it before the first fold.
Related
PortfolioOptimisers.replace_step — Function
replace_step(steps::Tuple, step, i::Integer) -> Tuple
replace_step(steps::AbstractVector, step, i::Integer) -> VectorReturn the steps of a Pipeline with step at position i.
A fold can change the type of a step, so the new container takes the type of step. The Tuple method calls Base.setindex. The AbstractVector method copies the steps into a vector whose element type joins the old element type and the type of step, through Base.promote_typejoin. Neither method changes steps, so the pipeline that holds them keeps its state.
Arguments
steps: The step estimators, aTupleor anAbstractVector, as the user gave them.step: The new step.i: The position of the new step.
Returns
- The steps, in the container form of
steps, withstepat positioni.
Related
PortfolioOptimisers.is_data_step — Function
is_data_step(step) -> BoolAnswers whether a Pipeline step writes a data slot, :prices or :returns.
A data step changes the rows that the row owner folds.
Related
PortfolioOptimisers.is_universe_step — Function
is_universe_step(est) -> BoolAnswers whether a data step is universe-only.
A universe-only step folds nothing. In fit(pipe) with no data its batch verb runs over the rows of the row owner, and the call applies its universe as a view. The asset selectors are universe-only, and every other step answers false.
Related
PortfolioOptimisers.is_row_owner — Function
is_row_owner(est) -> BoolAnswers whether a step can own the rows of the online step of a Pipeline.
A prior estimator can, and so can an optimisation step of any form, a schedule included. A schedule answers true so that the walk names the step that it meets, and assert_online_owner then states why a schedule cannot fold.
Related
PortfolioOptimisers.pipeline_row_owner — Function
pipeline_row_owner(p::Pipeline) -> Any
Finds the index of the row owner of a Pipeline, or answers 0 when the pipeline has none.
A prior step owns the rows when one exists, because inject_context replaces the pe of the optimiser with it, and the prior of the optimiser is never fitted. Without a prior step, the optimisation step owns the rows. It then takes the online step itself, and it keeps its own Fold Context. A pipeline with neither has nothing to fold the rows into.
Algorithm
- Find the first step whose estimator (
step_estimator) is a prior, givingk. Returnkwhen it exists. - Otherwise find the first step whose estimator is a row owner (
is_row_owner), givingk. Returnk, or0when no step is one.
Related
PortfolioOptimisers.online_entry_state — Method
online_entry_state(p::Pipeline)Names the first state that a Pipeline carries at the entry of the online arm of the fold loop, or answers nothing.
The answer is "cache" when the pipeline holds a state of its own. Otherwise the method walks the steps in order, and answers the path of the first state under a step, prefixed with the name of the step.
Related
PortfolioOptimisers.pipeline_online_member — Function
pipeline_online_member(p::Pipeline) -> Any
Names the first Online declaration under the steps of a Pipeline, prefixed with the name of the step, or answers nothing.
Two refusals read this walk. Under Online(pipe) no member folds, so a wrapper below the root has no fold to seed and its cap would have no effect, and assert_online_entry refuses it with an error that names it. On the fold route the warm-up resolves every wrapper, so the fold refuses a wrapper that it still meets.
Related
PortfolioOptimisers.step_online_member — Function
step_online_member(step)Names the first Online declaration that one Pipeline step carries, relative to the step.
The answer is "" when the step is itself a wrapper, and the dotted path of the wrapper when the estimator of the step holds one at any depth (online_wrapper_path). Otherwise it is nothing. A PipelineStep answers for its estimator, and a nested Pipeline answers through pipeline_online_member. The walk goes to every depth because the warm-up does too. update_online_estimator resolves a wrapper under the prior of an optimiser, and under the inner prior of a prior that holds one.
Related
PortfolioOptimisers.step_online_cap — Function
step_online_cap(step)Reads the first max_history that an Online declaration of one Pipeline step carries, or answers nothing when no wrapper of the step carries a cap.
The refusal of a capped owner on the fold route reads this cap. A cap on the owner is a window counted in the rows of the owner, and a row-local step before the owner folds a state across the front edge of that window. The walk goes to every depth, as step_online_member does, so a cap on the inner prior of a prior under an optimiser counts.
Algorithm
- A wrapper answers its own
max_history. AnOnlinenever wraps another, so the walk does not enter it. - A
PipelineStepanswers for its estimator. - An estimator walks its estimator-valued fields (
estimator_fields) in order, and answers the first cap that a field answers, ornothing. - Any other value answers
nothing.
Related
PortfolioOptimisers.assert_online_entry — Method
assert_online_entry(p::Pipeline)
assert_online_entry(o::Online{<:Pipeline})Refuses a Pipeline that cannot take the online step, with an error that names the step.
The fold loop calls it at the entry of its online arm, before any fit. The checks depend on the route. The fold route folds the rows through the steps into a row owner, and it has the most checks. The refit route, Online(pipe), folds no member, so it has two.
The rolling window through a Pipeline is Online(pipe; max_history = w). A capped owner stays on the fold route only when every data step before it is universe-only, because fit(pipe) with no data refits a universe-only step over the capped rows of the owner.
Validation
On the fold route, in this order:
- The pipeline carries no state, in its own
cacheor under a step (online_entry_state), because the loop starts cold. - The pipeline has a row owner (
pipeline_row_owner), a prior step or an optimisation step. - The row owner passes
assert_online_owner. It is not aTimeDependentschedule and not anOnlinePortfolioSelectionhead, and an optimisation step that owns the rows passes the checks ofassert_online_entryfor an optimiser. A schedule owns the rows only when no prior step precedes it. With a prior step before it, the schedule swaps a stateless step and composes. - Every data step before the owner can fold or defer. The check refuses a nested
Pipeline, a data step that is not a preprocessing estimator, and a preprocessing estimator with no online form (supports_partial_fit). A callablePipelineStepthat writes:pricesor:returnsis not a preprocessing estimator.PriceGapFillwith a statistic fill andMissingDataFilterwithrow_thr < 1are window-valued, and have no online form. The message names both ways out, apartial_fit_transformfor the step or a refit withOnline(pipe). - When a wrapper on the owner carries a cap (
step_online_cap), every data step before the owner is universe-only (is_universe_step). The owner counts its window in its own rows, and the carry of a row-local step reads rows that the window dropped, so the run equals no batch scheme. This holds on returns input too.
On the refit route:
- The pipeline carries no state, as on the fold route.
- No step carries an
Onlinedeclaration of its own (pipeline_online_member), because no member folds under a refit and its cap would have no effect.
Each refusal throws an ArgumentError that names the step.
Related
PortfolioOptimisers.assert_online_owner — Function
assert_online_owner(owner, name::AbstractString)Refuses a row owner that cannot fold, with an error that names the step.
A prior passes. An optimisation step passes when it passes the checks of assert_online_entry for an optimiser. A precomputed result never reaches this check. A result is not a step, so it enters a pipeline only as an entry of a schedule, and the check refuses the schedule first.
The check refuses an OnlinePortfolioSelection head because of its call with no data, not because of its fold. optimise(opt) with no data on the head answers a Recursion Result, so it rebuilds no ReturnsResult (batch_from_state). fit(pipe) with no data rebuilds a ReturnsResult from the row owner, and then refits the universe steps over it. Two things are missing, and a ReturnsResult would supply only the first. A rule whose tree reads no rows holds no rows, so nothing exists to rebuild the ReturnsResult from. And fit(pipe) with no data expresses a selection as a view of the state of the owner. For a recursion that view is not the run over those columns, because the allocation is a path, the projection couples the columns, and the wealth factor reads every column.
Validation
- The owner is not a
TimeDependentschedule. A schedule swaps the estimator that carries the state at every fold, so the optimiser of a fold never saw the rows folded before it. - The owner is not an
OnlinePortfolioSelectionhead. - An optimisation owner passes
assert_online_entry.
Each refusal throws an ArgumentError that names the step.
Related
PortfolioOptimisers.assert_pipeline_entry — Function
assert_pipeline_entry(pipe::Pipeline, _)
Runs the entry checks of the cross_val_predict methods of a Pipeline.
Validation
- The pipeline holds no holdout step (
assert_no_holdout). - An
Online(pipe)root runs only under an Online Scheme (folds_are_stepped). The wrapper resolves at the warm-up of the online arm, and a batch scheme has none. - A
Resumeruns only under a scheme thatassert_resume_schemeaccepts.
Each refusal throws an ArgumentError.
Related
PortfolioOptimisers.pipe_writes — Method
pipe_writes(o::Online) = pipe_writes(o.est)
pipe_reads(o::Online) = pipe_reads(o.est)An Online step writes and reads the slots of the estimator that it wraps.
Online(EmpiricalPrior(); max_history = w) is a prior step whose window is capped. The warm-up of the online arm resolves it to a plain prior that carries a buffer.
Related
PortfolioOptimisers.run_step — Function
run_step(o::Online, ::PipelineContext)
run_step(
o::Online,
::PipelineContext,
::Union{Nothing, PipelineStep}
)
Refuses an Online step that reaches a fit with no folds, with an error that names it.
The warm-up of the online arm resolves a wrapper, and fit(pipe, data) runs no warm-up. A wrapper that reaches it has no buffer to seed and no step to fold. A fit of the wrapped estimator as a plain one would return a batch answer under an online declaration, so the method refuses the step instead.
Validation
- The method always throws an
ArgumentError. The message names the wrapped estimator and the two ways out: an Online Scheme, or the plain estimator.
Related
PortfolioOptimisers.update_online_estimator — Method
update_online_estimator(p::Pipeline)Resolves the Online declarations in the steps of a Pipeline, at warm-up.
On the fold route update_online_estimator resolves every step: a bare Online step, an estimator, and a PipelineStep through update_online_estimator(ps::PipelineStep). A callable and a result pass through its fallback unchanged. So an Online(EmpiricalPrior(); max_history = w) step becomes the prior with its buffer, and an optimisation step resolves the wrappers under its own prior. On the refit route the pipeline carries a PipelineBufferState, which Online(pipe) seeded, and the method returns the pipeline unchanged. No member folds under a refit, and assert_online_entry already refused any wrapper below the root with an error that names it.
Related
PortfolioOptimisers.update_online_estimator — Method
update_online_estimator(ps::PipelineStep)Resolves the Online declarations of the estimator that a PipelineStep wraps, at warm-up.
The method rebuilds the wrapper around the resolved estimator with rebuild_struct, so the step keeps its reads, its writes and its target. A wrapped Online becomes the estimator that it wraps, with a seeded buffer, and a wrapped estimator resolves the wrappers in its own fields. A wrapped callable passes through the fallback of update_online_estimator unchanged.
Related
PortfolioOptimisers.is_time_dependent — Method
is_time_dependent(o::Online{<:Pipeline})
needs_previous_weights(o::Online{<:Pipeline})
assert_time_dependent_fold_count(o::Online{<:Pipeline}, n::Integer, all_binds::Bool = true)Answers the three traits of the fold loop for an Online(pipe) root, as the pipeline that it wraps.
The fold loop reads the traits off the root that it receives. The wrapper resolves at the warm-up of the online arm, and the loop makes its per-fold copy of the pipeline, so the pipeline answers.
Related
PortfolioOptimisers.show_fields — Method
show_fields(
p::Pipeline
) -> Union{Tuple{Symbol, Symbol}, Tuple{Symbol, Symbol, Symbol}}
Lists the fields that show renders for a Pipeline: the names, the steps, and the cache only when it holds a state.
A pipeline that took no step therefore renders no cache line.
Related
PortfolioOptimisers.copy_states — Method
copy_states(p::Pipeline)Copies every partial-fit state that a Pipeline carries, and rebuilds the pipeline around the copies, for Resume.
The states are the own cache of the pipeline and the states under every step, which copy_states copies. A PipelineStep copies through copy_states(ps::PipelineStep), and a callable and a result pass through the fallback unchanged. A fold of the copy leaves the held rows of the original unchanged.
Related
PortfolioOptimisers.copy_states — Method
copy_states(ps::PipelineStep)Copies the partial-fit states under the estimator that a PipelineStep wraps.
The method rebuilds the wrapper around the copied estimator with rebuild_struct, so the step keeps its reads, its writes and its target. A wrapped callable passes through the fallback of copy_states unchanged.
Related
PortfolioOptimisers.held_timestamps — Method
held_timestamps(p::Pipeline)Returns the timestamps that a stepped Pipeline holds, through the state of its row owner, or nothing.
The state that the pipeline carries selects the answer.
- Under
Online(pipe)thePipelineBufferStateholds the input data, and the timestamps of that data are the answer. - A prior owner leaves the Pipeline its own
ReturnsBufferState, which holds them. - An optimisation owner keeps its own Fold Context, and the pipeline holds none, so the owner answers (
held_timestamps). - A pipeline whose owner carries no state, or that has no owner, has taken no step, and answers
nothing.
At the price level the answer of the fold route starts one row after the answer of the refit route. A PricesToReturns step drops the first price row, so the returns carry the timestamps of the prices from the second row on. Both answers are a run of the timestamps of the price data that ends at the last row folded, which is what the alignment check of Resume reads.
Related
PortfolioOptimisers.held_timestamps — Method
held_timestamps(p::Pipeline, cache::PipelineBufferState)
held_timestamps(p::Pipeline, cache::ReturnsBufferState)
held_timestamps(p::Pipeline, ::Nothing)The three arms of held_timestamps(p::Pipeline), selected by dispatch on the state that the pipeline carries, which the one-argument method passes.
A buffer of the input data answers with the timestamps of that data, and a Fold Context answers with its own. With no state, the row owner (pipeline_row_owner) answers when it carries a state (online_entry_state), and the method answers nothing otherwise.
Related