Pipeline cross-validation
The price-level restriction (the rolling-window rule)
Combinatorial and multiple-randomised cross-validation recombine non-contiguous groups / resampled paths. A pipeline that starts from prices runs a rolling, order-dependent transform (a PricesToReturns, or any windowed preprocessing) that needs contiguous input rows, so those schemes are rejected at the price-level split. They are supported for a returns-level pipeline (below), which has no such transform.
PortfolioOptimisers.port_opt_view Method
port_opt_view(x, i, args...; kwargs...) -> nothing_scalar_array_view(x, i)Sub-select an estimator, result, or algorithm to the asset/observation index i.
port_opt_view is the index-selection counterpart of factory: where factory threads runtime values down a composed struct tree, port_opt_view threads an index selection — restricting every data-bearing field and composed child to the subset i. It is the mechanism that makes meta-optimisers (NestedClustered, SubsetResampling) and cross-validation variants operate on subproblems with identical struct shapes.
Callers do not normally call port_opt_view directly; it is driven by meta-optimisers and cross-validation internals. It is public (not exported) because extension authors who implement a new composed estimator may need to define a method. Use @vprop on data-bearing fields to have the method generated automatically.
This universal fallback handles leaf values: arrays are sliced via nothing_scalar_array_view; scalars, nothing, estimators without data fields, and algorithms pass through unchanged. Composed structs that recurse into children define their own (more specific) method — emitted by @vprop or hand-written.
The threaded tail args... (typically the returns matrix X for the JuMP families) and any kwargs are accepted and dropped here, so a macro-threaded port_opt_view(child, i, X) never MethodErrors on a leaf field.
Related
sourceport_opt_view(pipe::Pipeline, i, args...; kwargs...)Deliberately unsupported: a Pipeline cannot be sub-selected by asset view.
Meta-optimisers (NestedClustered, Stacking, SubsetResampling) build asset sub-portfolios by taking a port_opt_view of their inner estimator. A pipeline's asset universe is fitted state — the missing-data filter decides it from the training window — so an asset view taken before fitting is not well defined. Wrapping a Pipeline in a meta-optimiser is therefore unsupported in v1 (ADR 0028, "Future expansion"); a meta-optimiser may still be the optimisation step of a pipeline.
Related
sourceThe pipeline fold loop
cross_val_predict over a Pipeline fits the whole workflow per fold and predicts on each test window. It is also the fold loop that consumes TimeDependent schedules in a pipeline (ADR 0030, "swap, then inject"): schedules are swapped for their per-fold values before fit runs, so injection never sees a schedule and fit/run_step never learn about folds.
PortfolioOptimisers.cross_val_predict Method
cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.
The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.
This is the fold loop that consumes TimeDependent schedules in a pipeline (ADR 0030): when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. When the pipeline needs_previous_weights, run_folds runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory.
Arguments
pipe: The pipeline.data: Price- or returns-level input data (Prices_RR).cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults toKFold().ex: FLoops executor controlling parallelism. Defaults toFLoops.ThreadedEx().id: Identifier stored on the result.
Returns
MultiPeriodPredictionResult: One prediction per fold, in fold order.
Related
sourceCombinatorial and asset-resampling over a returns-level pipeline
A returns-level pipeline runs the multi-path schemes like the plain-optimiser loops: combinatorial fits each split on its (possibly non-contiguous) training rows and predicts its test groups; multiple-randomised runs each path's inner walk-forward over an asset-subset view of the input, so the pipeline fits fresh on the sub-universe and never sub-selects fitted state.
PortfolioOptimisers.cross_val_predict Method
cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation; ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResultRun combinatorial cross-validation over a price- or returns-level Pipeline.
Each split fits the whole workflow on its (possibly non-contiguous) training rows and predicts each of the split's disjoint test groups; sort_predictions! then recombines the per-split test-group predictions into the scheme's paths, exactly like the plain-optimiser combinatorial loop. Time-dependent steps resolve per split against the fold's TimeDependentContext before fit.
At the returns level the training rows are order-independent for moment-style fitted steps, so this is exact. At the price level a split's training rows are non-contiguous — there are gaps where the held-out test groups sit — so the fold's rolling transform (PricesToReturns) produces one spurious return per gap boundary (a boundary return spanning a gap). That is the rolling-window approximation: combinatorial paths at the price level cost a few boundary returns in each fold's training window. Test groups are contiguous, so predictions are unaffected. Use MultipleRandomised if you need contiguous training rows at the price level.
Related
sourcecross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.
The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.
This is the fold loop that consumes TimeDependent schedules in a pipeline (ADR 0030): when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. When the pipeline needs_previous_weights, run_folds runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory.
Arguments
pipe: The pipeline.data: Price- or returns-level input data (Prices_RR).cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults toKFold().ex: FLoops executor controlling parallelism. Defaults toFLoops.ThreadedEx().id: Identifier stored on the result.
Returns
MultiPeriodPredictionResult: One prediction per fold, in fold order.
Related
sourcePortfolioOptimisers.cross_val_predict Method
cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised; ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResultRun asset-resampling (multiple-randomised) cross-validation over a price- or returns-level Pipeline.
Each resampled path is an inner walk-forward over a random asset subset; the subset is applied to the input data (an asset view via pipeline_asset_view), and the pipeline is fitted fresh on the sub-universe — so the pipeline never needs to sub-select its fitted universe. Paths are run by pipeline_path_fit_and_predict and returned as a PopulationPredictionResult. Asset resampling keeps every observation window contiguous (it draws over assets, not rows), so — unlike combinatorial cross-validation — multiple-randomised is admissible at the price level too: a price-starting pipeline fits fresh on the asset-subset prices per fold, with no rolling-window violation.
Related
sourcecross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.
The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.
This is the fold loop that consumes TimeDependent schedules in a pipeline (ADR 0030): when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. When the pipeline needs_previous_weights, run_folds runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory.
Arguments
pipe: The pipeline.data: Price- or returns-level input data (Prices_RR).cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults toKFold().ex: FLoops executor controlling parallelism. Defaults toFLoops.ThreadedEx().id: Identifier stored on the result.
Returns
MultiPeriodPredictionResult: One prediction per fold, in fold order.
Related
sourcePortfolioOptimisers.pipeline_path_fit_and_predict Function
pipeline_path_fit_and_predict(pipe::Pipeline, data::Prices_RR, folds, path_id; ex) -> MultiPeriodPredictionResultRun one MultipleRandomised path of a price- or returns-level Pipeline: fit and predict the path's inner walk-forward folds over the path's asset subset.
folds is the path's (train_idx, test_idx, asset_idx) tuples in split-enumeration order. Each fold takes the asset-subset view of data (via pipeline_asset_view, which indexes assets consistently at either level) — the pipeline fits fresh on the sub-universe, so it never sub-selects fitted state — then fits on the training window and predicts on the test window. Time-dependent steps resolve per fold (the context carries path_id); when the pipeline needs_previous_weights, run_folds runs the path sequentially and threads the previous fold's weights. Predictions are sorted by test index for reporting.
Related
sourceTime-dependent traits and the swap over steps
The per-step legs of the time-dependent machinery: the traits recurse over a pipeline's steps, the swap maps over them (unwrapping PipelineStep-wrapped schedules), the fold-less reset resolves schedule steps to their explicit default, and the previous-weights factory delivers w_prev to the optimisation steps after the swap.
PortfolioOptimisers.pipeline_step_is_time_dependent Function
pipeline_step_is_time_dependent(x)Return true if a Pipeline step carries time-dependent constraints.
Steps that participate in the time-dependent machinery — optimisation estimators and results, TimeDependent schedules, nested Pipelines, and PipelineStep wrappers around any of them — delegate to is_time_dependent; every other step (preprocessing, prior, phylogeny, uncertainty, constraint estimators, and custom callables) contributes false, because no non-optimiser family hosts schedules (they are spelled as fields of the optimisation step instead).
Related
sourcePortfolioOptimisers.is_time_dependent Method
is_time_dependent(p::Pipeline) -> AnyReturn true if any step of the Pipeline carries time-dependent constraints: a TimeDependent schedule standing in for the optimisation step, an optimisation step whose fields hold schedules, or a nested pipeline containing either.
Related
sourcePortfolioOptimisers.needs_previous_weights Method
needs_previous_weights(p::Pipeline) -> AnyReturn true if any step of the Pipeline requires the previous fold's portfolio weights, forcing sequential fold execution and a populated w_prev in the TimeDependentContext. Recurses through PipelineStep wrappers and nested pipelines; schedule entries are inspected per the needs_previous_weights conventions (entries yes, default no).
Related
sourcePortfolioOptimisers.assert_pipeline_step_fold_count Function
assert_pipeline_step_fold_count(x, n::Integer, all_binds::Bool)Assert the fold count of one Pipeline step: optimisation steps, TimeDependent schedule steps, and nested pipelines delegate to assert_time_dependent_fold_count (schedule entries are sized to the loop, the default is not); every other step is a no-op.
Related
sourcePortfolioOptimisers.update_time_dependent_step Function
update_time_dependent_step(est, ctx::TimeDependentContext, all_binds::Bool)Resolve one Pipeline step for the fold described by ctx — the per-step leg of the pipeline swap.
A TimeDependent schedule step resolves to its fold-i entry (an estimator or a precomputed result) and is recursed into with the same context; an optimisation step resolves its own scheduled fields; a nested pipeline recurses. A PipelineStep wrapping a schedule is unwrapped to the resolved value — the wrapper existed only to declare the slots dispatch could not infer, and after the swap run_step dispatches on the resolved value directly (which a wrapper could not hold anyway when the entry is a result). Every other step passes through unchanged.
Related
sourcePortfolioOptimisers.update_time_dependent_estimator Function
update_time_dependent_estimator(
p::Pipeline,
ctx::TimeDependentContext
) -> Pipeline
update_time_dependent_estimator(
p::Pipeline,
ctx::TimeDependentContext,
all_binds::Bool
) -> PipelineResolve the time-dependent steps of a Pipeline for the fold described by ctx — the swap of ADR 0030's pipeline integration.
The swap happens in the fold loop, outside fit entirely: it maps update_time_dependent_step over the steps (names preserved), so by the time fit runs on the fold's training window every schedule step is already a plain optimiser or precomputed result and injection (inject_context / maybe_inject_step) never sees a schedule — fit and run_step never learn about folds.
Related
sourcePortfolioOptimisers.reset_time_dependent_step Function
reset_time_dependent_step(est)Replace one Pipeline step with its fold-less value.
A TimeDependent schedule step resolves to its explicit default (there is no static default an optimisation step could fall back to), throwing a TimeDependentDefaultError that points at cross_val_predict when it has none; optimisation steps and nested pipelines recurse through reset_time_dependent_estimator; a PipelineStep wrapping a schedule is unwrapped to the fold-less optimiser. Every other step passes through unchanged.
Related
sourcePortfolioOptimisers.reset_time_dependent_estimator Method
reset_time_dependent_estimator(p::Pipeline) -> PipelineReplace every time-dependent step of a Pipeline with its fold-less value (see reset_time_dependent_step). Called at the top of the fold-less fit; per-fold pipelines produced by update_time_dependent_estimator contain no schedules, so they pass through unchanged.
Related
sourcePortfolioOptimisers.pipeline_step_factory Function
pipeline_step_factory(est, w)Deliver the previous fold's weights w to one Pipeline step: optimisation steps and nested pipelines go through factory (turnover, fees and tracking pick the weights up; everything else passes through), PipelineStep-wrapped optimisers are rebuilt around the updated estimator, and non-optimiser steps pass through unchanged.
Related
sourcePortfolioOptimisers.factory Method
factory(
p::Pipeline,
w::AbstractVector{<:Union{var"#s29", var"#s28"} where {var"#s29"<:Number, var"#s28"<:AbstractJuMPScalar}}
) -> PipelineRebuild a Pipeline with the previous fold's weights delivered to every optimisation step (see pipeline_step_factory). Applied by the fold loop after the swap, so freshly swapped-in per-fold optimisers receive the previous weights too.
Related
source