Pipeline cross-validation: private API
The pipeline fold loop
cross_val_predict over a Pipeline fits the whole workflow per fold and predicts on each test window. It is also the fold loop that consumes TimeDependent schedules in a pipeline (ADR 0030, "swap, then inject"): schedules are swapped for their per-fold values before fit runs, so injection never sees a schedule and fit/run_step never learn about folds.
A scheme that declares a Fold Fit sends the loop down its online arm, where the pipeline is warmed up once, folded fold by fold through partial_fit!, and read out through fit(pipe); Online(pipe) takes the same doors as the declared refit. See the Pipeline's online step.
PortfolioOptimisers.Pipeline_OnlPipe — Type
const Pipeline_OnlPipe = Union{<:Pipeline, <:Online{<:Pipeline}, <:Resume{<:MultiPeriodPredictionResult{<:Any, <:Any, <:Any, <:Pipeline}}}Alias for what a Pipeline's cross-validation door threads through the fold loop: the Pipeline itself, taking the host route, Online(pipe), the declared refit from an input-carrier buffer, or Resume(res), the continuation of an online run whose Result carries a pipeline.
Related
PortfolioOptimisers.pipeline_cross_val_predict — Function
pipeline_cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::MultipleRandomised; ex = FLoops.ThreadedEx())
pipeline_cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::CVER; ex = FLoops.ThreadedEx(), id = nothing)The body of a Pipeline's cross-validation door, shared by the Pipeline and by Online(pipe): the door checks its entry through assert_pipeline_door, splits the data, and runs the folds through fold_loop, whose online arm resolves the wrapper at warm-up and threads the pipeline from fold to fold.
Related
Combinatorial and asset-resampling over a returns-level pipeline
A returns-level pipeline runs the multi-path schemes like the plain-optimiser loops: combinatorial fits each split on its (possibly non-contiguous) training rows and predicts its test groups; multiple-randomised runs each path's inner walk-forward over an asset-subset view of the input, so the pipeline fits fresh on the sub-universe and never sub-selects fitted state.
PortfolioOptimisers.pipeline_path_fit_and_predict — Function
pipeline_path_fit_and_predict(pipe::Pipeline, data::Prices_RR, folds, path_id; ex) -> MultiPeriodPredictionResultRun one MultipleRandomised path of a price- or returns-level Pipeline: fit and predict the path's inner walk-forward folds over the path's asset subset.
folds is the path's (train_idx, test_idx, asset_idx) tuples in split-enumeration order. Each fold takes the asset-subset view of data (via pipeline_asset_view, which indexes assets consistently at either level) — the pipeline fits fresh on the sub-universe, so it never sub-selects fitted state — then fits on the training window and predicts on the test window. Time-dependent steps resolve per fold (the context carries path_id); when the pipeline needs_previous_weights, fold_loop runs the path sequentially and threads the previous fold's weights. Predictions are sorted by test index for reporting.
Related
Time-dependent traits and the swap over steps
The per-step legs of the time-dependent machinery: the traits recurse over a pipeline's steps, the swap maps over them (unwrapping PipelineStep-wrapped schedules), the fold-less reset resolves schedule steps to their explicit default, and the previous-weights factory delivers w_prev to the optimisation steps after the swap.
PortfolioOptimisers.pipeline_step_is_time_dependent — Function
pipeline_step_is_time_dependent(x)Return true if a Pipeline step carries time-dependent constraints.
Steps that participate in the time-dependent machinery — optimisation estimators and results, TimeDependent schedules, nested Pipelines, and PipelineStep wrappers around any of them — delegate to is_time_dependent; every other step (preprocessing, prior, phylogeny, uncertainty, constraint estimators, and custom callables) contributes false, because no non-optimiser family hosts schedules (they are spelled as fields of the optimisation step instead).
Related
PortfolioOptimisers.is_time_dependent — Method
is_time_dependent(p::Pipeline) -> Any
Return true if any step of the Pipeline carries time-dependent constraints: a TimeDependent schedule standing in for the optimisation step, an optimisation step whose fields hold schedules, or a nested pipeline containing either.
Related
PortfolioOptimisers.assert_pipeline_step_fold_count — Function
assert_pipeline_step_fold_count(x, n::Integer, all_binds::Bool)Assert the fold count of one Pipeline step: optimisation steps, TimeDependent schedule steps, and nested pipelines delegate to assert_time_dependent_fold_count (schedule entries are sized to the loop, the default is not); every other step is a no-op.
Related
PortfolioOptimisers.update_time_dependent_step — Function
update_time_dependent_step(est, ctx::TimeDependentContext, all_binds::Bool)Resolve one Pipeline step for the fold described by ctx — the per-step leg of the pipeline swap.
A TimeDependent schedule step resolves to its fold-i entry (an estimator or a precomputed result) and is recursed into with the same context; an optimisation step resolves its own scheduled fields; a nested pipeline recurses. A PipelineStep wrapping a schedule is unwrapped to the resolved value — the wrapper existed only to declare the slots dispatch could not infer, and after the swap run_step dispatches on the resolved value directly (which a wrapper could not hold anyway when the entry is a result). Every other step passes through unchanged.
Related
PortfolioOptimisers.update_time_dependent_estimator — Function
update_time_dependent_estimator(
p::Pipeline,
ctx::TimeDependentContext
) -> Pipeline
update_time_dependent_estimator(
p::Pipeline,
ctx::TimeDependentContext,
all_binds::Bool
) -> Pipeline
Resolve the time-dependent steps of a Pipeline for the fold described by ctx.
The swap happens in the fold loop, outside fit entirely: it maps update_time_dependent_step over the steps (names preserved), so by the time fit runs on the fold's training window every schedule step is already a plain optimiser or precomputed result and injection (inject_context / maybe_inject_step) never sees a schedule — fit and run_step never learn about folds.
Related
PortfolioOptimisers.reset_time_dependent_step — Function
reset_time_dependent_step(est)Replace one Pipeline step with its fold-less value.
A TimeDependent schedule step resolves to its explicit default (there is no static default an optimisation step could fall back to), throwing a TimeDependentDefaultError that points at cross_val_predict when it has none; optimisation steps and nested pipelines recurse through reset_time_dependent_estimator; a PipelineStep wrapping a schedule is unwrapped to the fold-less optimiser. Every other step passes through unchanged.
Related
PortfolioOptimisers.reset_time_dependent_estimator — Method
reset_time_dependent_estimator(p::Pipeline) -> Pipeline
Replace every time-dependent step of a Pipeline with its fold-less value (see reset_time_dependent_step). Called at the top of the fold-less fit; per-fold pipelines produced by update_time_dependent_estimator contain no schedules, so they pass through unchanged.
Related
PortfolioOptimisers.pipeline_step_factory — Function
pipeline_step_factory(est, w)Deliver the previous fold's weights w to one Pipeline step: optimisation steps and nested pipelines go through factory (turnover, fees and tracking pick the weights up; everything else passes through), PipelineStep-wrapped optimisers are rebuilt around the updated estimator, and non-optimiser steps pass through unchanged.
Related