Pipeline cross-validation

port_opt_view(x, i, args...; kwargs...) -> nothing_scalar_array_view(x, i)

Sub-select an estimator, result, or algorithm to the asset/observation index i.

port_opt_view is the index-selection counterpart of factory: where factory threads runtime values down a composed struct tree, port_opt_view threads an index selection — restricting every data-bearing field and composed child to the subset i. It is the mechanism that makes meta-optimisers (NestedClustered, SubsetResampling) and cross-validation variants operate on subproblems with identical struct shapes.

Callers do not normally call port_opt_view directly; it is driven by meta-optimisers and cross-validation internals. It is public (not exported) because extension authors who implement a new composed estimator may need to define a method. Use @vprop on data-bearing fields to have the method generated automatically.

This universal fallback handles leaf values: arrays are sliced via nothing_scalar_array_view; scalars, nothing, estimators without data fields, and algorithms pass through unchanged. Composed structs that recurse into children define their own (more specific) method — emitted by @vprop or hand-written.

The threaded tail args... (typically the returns matrix X for the JuMP families) and any kwargs are accepted and dropped here, so a macro-threaded port_opt_view(child, i, X) never MethodErrors on a leaf field.

Algorithm

1. Drop args... and kwargs.... This method is the leaf of the recursion, so it threads nothing further.
2. Return nothing_scalar_array_view of x at i, whose own algorithm names the rule for each leaf type.

Related

• factory
• @vprop
• nothing_scalar_array_view

port_opt_view(pipe::Pipeline, i, args...; kwargs...)

Deliberately unsupported: a Pipeline cannot be sub-selected by asset view.

Meta-optimisers (NestedClustered, Stacking, SubsetResampling) build asset sub-portfolios by taking a port_opt_view of their inner estimator. A pipeline's asset universe is fitted state — the missing-data filter decides it from the training window — so an asset view taken before fitting is not well defined. Wrapping a Pipeline in a meta-optimiser is therefore unsupported for now; a meta-optimiser may still be the optimisation step of a pipeline.

Related

• Pipeline
• optimise(::Pipeline)

needs_previous_weights(p::Pipeline) -> Any

Return true if any step of the Pipeline requires the previous fold's portfolio weights, forcing sequential fold execution and a populated w_prev in the TimeDependentContext. Recurses through PipelineStep wrappers and nested pipelines; schedule entries are inspected per the needs_previous_weights conventions (entries yes, default no).

Related

• needs_previous_weights
• fold_loop
• run_folds

cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)

Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.

Return type by cross-validation scheme

cross_val_predict is the single entry point for every scheme, but the result type depends on cv: single-path schemes return one series, multi-path schemes return a per-path collection. The type flips with the scheme, so result-navigation code written against one shape (e.g. a KFold run) breaks when the scheme is swapped — branch on the scheme, not on the run.

The combinatorial and asset-resampling schemes are dispatched by their own methods (see Related); the rest of this docstring describes the contiguous, single-path (CVER) method.

The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.

This is the fold loop that consumes TimeDependent schedules in a pipeline: when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. The loop is fold_loop, shared with the optimiser-level schemes. The scheme states whether its folds are a timeline through folds_are_time_ordered. A walk-forward answers true, so a pipeline that needs_previous_weights runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory. A KFold answers false, because its folds are independent of each other. Its folds run in parallel, w_prev is nothing, and no factory pass runs — the same behaviour the optimiser-level KFold path already has.

A walk-forward that declares a Fold Fit (ff = OnlineStep()) sends the loop down its online arm: the pipeline is warmed up once on the first training window, each fold's new rows are folded through its steps into the row owner by partial_fit!, and the fold reads the pipeline out through fit(pipe) where a refit would have run, through pipeline_fold_fit. The run reaches the weights of the batch expanding walk-forward fold for fold, and Online(pipe) takes the same door as the declared refit from an input-carrier buffer.

Arguments

• pipe: The pipeline.
• data: Price- or returns-level input data (Prices_RR).
• cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults to KFold(). folds_are_time_ordered decides whether its folds thread the previous fold's weights, and fold_fit whether the folds refit or fold.
• ex: FLoops executor controlling parallelism. Defaults to FLoops.ThreadedEx().
• id: Identifier stored on the result.

Returns

• MultiPeriodPredictionResult: One prediction per fold, in fold order.

Related

• Pipeline
• fit
• TimeDependent
• folds_are_time_ordered
• search_cross_validation
• MultiPeriodPredictionResult / PopulationPredictionResult (the two return shapes)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised)

cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation; ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResult

Run combinatorial cross-validation over a price- or returns-level Pipeline.

Each split fits the whole workflow on its (possibly non-contiguous) training rows and predicts each of the split's disjoint test groups; sort_predictions! then recombines the per-split test-group predictions into the scheme's paths, exactly like the plain-optimiser combinatorial loop. Time-dependent steps resolve per split against the fold's TimeDependentContext before fit.

At the returns level the training rows are order-independent for moment-style fitted steps, so this is exact. At the price level a split's training rows are non-contiguous — there are gaps where the held-out test groups sit — so the fold's rolling transform (PricesToReturns) produces one spurious return per gap boundary (a boundary return spanning a gap). That is the rolling-window approximation: combinatorial paths at the price level cost a few boundary returns in each fold's training window. Test groups are contiguous, so predictions are unaffected. Use MultipleRandomised if you need contiguous training rows at the price level.

Related

• CombinatorialCrossValidation
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER)

cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)

Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.

Return type by cross-validation scheme

cross_val_predict is the single entry point for every scheme, but the result type depends on cv: single-path schemes return one series, multi-path schemes return a per-path collection. The type flips with the scheme, so result-navigation code written against one shape (e.g. a KFold run) breaks when the scheme is swapped — branch on the scheme, not on the run.

The combinatorial and asset-resampling schemes are dispatched by their own methods (see Related); the rest of this docstring describes the contiguous, single-path (CVER) method.

The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.

This is the fold loop that consumes TimeDependent schedules in a pipeline: when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. The loop is fold_loop, shared with the optimiser-level schemes. The scheme states whether its folds are a timeline through folds_are_time_ordered. A walk-forward answers true, so a pipeline that needs_previous_weights runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory. A KFold answers false, because its folds are independent of each other. Its folds run in parallel, w_prev is nothing, and no factory pass runs — the same behaviour the optimiser-level KFold path already has.

A walk-forward that declares a Fold Fit (ff = OnlineStep()) sends the loop down its online arm: the pipeline is warmed up once on the first training window, each fold's new rows are folded through its steps into the row owner by partial_fit!, and the fold reads the pipeline out through fit(pipe) where a refit would have run, through pipeline_fold_fit. The run reaches the weights of the batch expanding walk-forward fold for fold, and Online(pipe) takes the same door as the declared refit from an input-carrier buffer.

Arguments

• pipe: The pipeline.
• data: Price- or returns-level input data (Prices_RR).
• cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults to KFold(). folds_are_time_ordered decides whether its folds thread the previous fold's weights, and fold_fit whether the folds refit or fold.
• ex: FLoops executor controlling parallelism. Defaults to FLoops.ThreadedEx().
• id: Identifier stored on the result.

Returns

• MultiPeriodPredictionResult: One prediction per fold, in fold order.

Related

• Pipeline
• fit
• TimeDependent
• folds_are_time_ordered
• search_cross_validation
• MultiPeriodPredictionResult / PopulationPredictionResult (the two return shapes)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised)

cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised; ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResult

Run asset-resampling (multiple-randomised) cross-validation over a price- or returns-level Pipeline.

Each resampled path is an inner walk-forward over a random asset subset; the subset is applied to the input data (an asset view via pipeline_asset_view), and the pipeline is fitted fresh on the sub-universe — so the pipeline never needs to sub-select its fitted universe. Paths are run by pipeline_path_fit_and_predict and returned as a PopulationPredictionResult. Asset resampling keeps every observation window contiguous (it draws over assets, not rows), so — unlike combinatorial cross-validation — multiple-randomised is admissible at the price level too: a price-starting pipeline fits fresh on the asset-subset prices per fold, with no rolling-window violation.

Related

• MultipleRandomised
• pipeline_path_fit_and_predict
• pipeline_asset_view
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER)

cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CVER = KFold(); ex = FLoops.ThreadedEx(), id = nothing)

Run cross-validated prediction over an entire Pipeline workflow and return a MultiPeriodPredictionResult.

Return type by cross-validation scheme

cross_val_predict is the single entry point for every scheme, but the result type depends on cv: single-path schemes return one series, multi-path schemes return a per-path collection. The type flips with the scheme, so result-navigation code written against one shape (e.g. a KFold run) breaks when the scheme is swapped — branch on the scheme, not on the run.

The combinatorial and asset-resampling schemes are dispatched by their own methods (see Related); the rest of this docstring describes the contiguous, single-path (CVER) method.

The input is split at its own level — price-level data by the prices-aware split methods (contiguous windows, so stateful preprocessing stays inside the fold), returns-level data as usual — and for each fold the whole workflow is fitted on the training window and predicts on the test window, exactly as fit/predict do for a holdout. This method covers the contiguous, single-path schemes (KFold and the walk-forwards). Combinatorial and asset-resampling schemes have their own methods for a returns-level pipeline (see cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::AbstractReturnsResult, cv::MultipleRandomised)); for a price-starting pipeline they are rejected at split by the rolling-window rule.

This is the fold loop that consumes TimeDependent schedules in a pipeline: when the pipeline is time-dependent, fold i builds a TimeDependentContext — with rd the raw, pre-preprocessing input data, so pipeline-level callables see the fold's data before any step has transformed it — and swaps every schedule for its fold-i value via update_time_dependent_estimator before fit runs. A schedule step may resolve to an estimator (the fold optimises) or a precomputed result (the fold predicts only); injection never sees a schedule. The loop is fold_loop, shared with the optimiser-level schemes. The scheme states whether its folds are a timeline through folds_are_time_ordered. A walk-forward answers true, so a pipeline that needs_previous_weights runs sequentially and threads the previous fold's weights into the context's w_prev and, post-swap, into the optimisation steps via factory. A KFold answers false, because its folds are independent of each other. Its folds run in parallel, w_prev is nothing, and no factory pass runs — the same behaviour the optimiser-level KFold path already has.

A walk-forward that declares a Fold Fit (ff = OnlineStep()) sends the loop down its online arm: the pipeline is warmed up once on the first training window, each fold's new rows are folded through its steps into the row owner by partial_fit!, and the fold reads the pipeline out through fit(pipe) where a refit would have run, through pipeline_fold_fit. The run reaches the weights of the batch expanding walk-forward fold for fold, and Online(pipe) takes the same door as the declared refit from an input-carrier buffer.

Arguments

• pipe: The pipeline.
• data: Price- or returns-level input data (Prices_RR).
• cv::CVER: Cross-validation scheme with contiguous, non-combinatorial folds. Defaults to KFold(). folds_are_time_ordered decides whether its folds thread the previous fold's weights, and fold_fit whether the folds refit or fold.
• ex: FLoops executor controlling parallelism. Defaults to FLoops.ThreadedEx().
• id: Identifier stored on the result.

Returns

• MultiPeriodPredictionResult: One prediction per fold, in fold order.

Related

• Pipeline
• fit
• TimeDependent
• folds_are_time_ordered
• search_cross_validation
• MultiPeriodPredictionResult / PopulationPredictionResult (the two return shapes)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation)
• cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised)

factory(
    p::Pipeline,
    w::AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Pipeline

Rebuild a Pipeline with the previous fold's weights delivered to every optimisation step (see pipeline_step_factory). Applied by the fold loop after the swap, so freshly swapped-in per-fold optimisers receive the previous weights too.

Related

• pipeline_step_factory
• cross_val_predict