Pipeline

The Pipeline estimator reifies an end-to-end workflow — data preparation, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps fitted as a single unit. Computed slots override the terminal optimiser's internal configuration; absent steps fall back to what the optimiser computes internally.

Pipeline symbols

PortfolioOptimisers.Pipeline Type

julia

struct Pipeline{__T_names, __T_steps} <: AbstractPipelineEstimator

A reified end-to-end portfolio workflow: an ordered list of steps executed left-to-right over a PipelineContext.

Steps are ordinary estimators — preprocessing, prior, phylogeny, uncertainty-set, constraint-generation, and optimisation estimators, nested Pipelines, or PipelineStep wrappers — mapped to context slots by their family via pipe_writes/pipe_reads. Fitting a pipeline with fit walks the steps in order; computed slots override the terminal optimiser's internal configuration (see inject_context), and absent steps fall back to whatever the optimiser computes internally, so every stage is optional.

A terminal optimiser is not required: a prior-only pipeline is legal; prediction is what needs weights.

See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Fields

names: Step names, aligned with steps.
steps: The step estimators, in execution order.

Constructors

julia

Pipeline(; steps::Union{<:Tuple, <:AbstractVector}) -> Pipeline

Steps are given in execution order. Each element is either a step estimator or a "name" => estimator pair; unnamed steps are auto-named from the slot they write ("prior"), suffixed in order of appearance when a slot repeats ("prices_1", "prices_2").

Validation

!isempty(steps).
Every step must be steppable (pipe_writes must be defined for it).
Every slot a step reads must be written by an earlier step or fillable by the pipeline input (prices or returns).
No step may write a slot that invalidates a slot an earlier step already wrote (see PIPELINE_INVALIDATES). A step that rewrites :returns after a prior, phylogeny, uncertainty, or constraint step would leave that result computed on a stale asset universe.
Step names must be unique.

Examples

julia

julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));

julia> pipe.names
("returns", "prior", "opt")

Related

PortfolioOptimisers.PipelineResult Type

julia

struct PipelineResult{__T_names, __T_results, __T_ctx} <: AbstractPipelineResult

Fitted result of a Pipeline.

Carries the fitted per-step results (named, in step order), and the final PipelineContext whose slots hold the computed data, prior, phylogeny, uncertainty, constraints, and terminal optimisation result.

Step results are accessed by name with getindex (res["prior"]) or by position through the results field (res.results[2]); integer indexing keeps the package-wide length-1 container semantics. The w property forwards to the terminal optimisation result's weights (res.ctx.opt.w) and throws a PropertyPathError when the pipeline produced no optimisation result.

Fields

names: Step names, aligned with results.
results: Fitted per-step results, in step order.
ctx: The final PipelineContext.

Related

StatsAPI.fit Method

julia

StatsAPI.fit(pipe::Pipeline, data::Prices_RR) -> PipelineResult

Fit a Pipeline on price- or returns-level data.

The context slot matching the input type is filled (PricesResult → prices, ReturnsResult → returns, so passing returns-level data skips the price stages), then the steps run left-to-right via run_step. Immediately before an optimisation step runs, the computed slots override its internal configuration via inject_context.

fit is a fold-less entry point, so TimeDependent schedule steps are inert here: each resolves to its explicit default (see reset_time_dependent_estimator) before the steps run, and a schedule with no default throws a TimeDependentDefaultError — backtest the pipeline with cross_val_predict, whose folds the schedule resolves against. Inside a fold loop this reset is a no-op, because the loop swaps every schedule for its per-fold value first.

Arguments

pipe: The pipeline.
data: The input data (PricesResult or ReturnsResult).

Returns

res::PipelineResult: Named per-step fitted results and the final context.

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4),
                     [100.0 101.0; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);

julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));

julia> res = fit(pipe, PricesResult(; X = X));

julia> res.w
2-element Vector{Float64}:
 0.5
 0.5

Related

StatsAPI.predict Function

julia

predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult


predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idxs::VecVecInt, cols = Colon()) -> PredictionResult

predict(res::PipelineResult, data::AbstractReturnsResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult

Apply a fitted pipeline to an unseen data test_idx and produce the same PredictionResult the weights-level machinery consumes.

The test_idx selects observation rows of data (integer indices, timestamps, or : for all rows). The test_idx is transformed by replaying the fitted preprocessing steps in step order — the training universe subset, the training imputation parameters, then the returns conversion — so no statistics of the test test_idx leak into the transformation. The result is then handed to the existing weights-level predict, so scorers and risk measures carry over untouched.

Price-level data requires the pipeline to contain a PricesToReturns step; a pipeline that produced no optimisation result cannot predict.

Arguments

res: The fitted PipelineResult.
data: Price- or returns-level data containing the test_idx (PricesResult or ReturnsResult).
test_idx: Observation test_idx into the rows of data. Integer indices, timestamps, or : (all rows).

Returns

pred::PredictionResult: The weights-level prediction on the transformed test_idx.

Related

PortfolioOptimisers.fit_predict Method

julia

fit_predict(opt::Pipeline, data::Prices_RR)

Fit pipeline estimator opt on data data and immediately produce a PredictionResult.

The prediction is made on data itself — in-sample — unless the pipeline begins with a TrainTestSplit, in which case it is made on the held-out window that step reserved and no fitted step has seen. That is the one-line holdout evaluation: fit on the training rows, score on the test rows.

Arguments

opt: Optimisation estimator or result.
data::Prices_RR: Price- or returns-level data.

Returns

PredictionResult: On the held-out window when the pipeline splits, on data otherwise.

Related

PortfolioOptimisers.port_opt_view Method

julia

port_opt_view(pipe::Pipeline, i, args...; kwargs...)

Deliberately unsupported: a Pipeline cannot be sub-selected by asset view.

Meta-optimisers (NestedClustered, Stacking, SubsetResampling) build asset sub-portfolios by taking a port_opt_view of their inner estimator. A pipeline's asset universe is fitted state — the missing-data filter decides it from the training window — so an asset view taken before fitting is not well defined. Wrapping a Pipeline in a meta-optimiser is therefore unsupported in v1 (ADR 0028, "Future expansion"); a meta-optimiser may still be the optimisation step of a pipeline.

Related

Holdout splitting

A TrainTestSplit step reserves a held-out test window before any other step runs. It is pinned to the first position — a stateful step fitted before it would have seen the held-out rows — and excludes cross-validation, which defines its own train/test windows. fit_predict(pipe, data) predicts on the window the split reserved.

PortfolioOptimisers.assert_split_position Function

julia

assert_split_position(ests)

Validate that a TrainTestSplit appears only as the first step of a Pipeline, and never inside a nested one.

The holdout exists to keep the test window away from every fitted step. A stateful step fitted before the split — a MissingDataFilter choosing the universe, an Imputer computing fill values — would have read the held-out rows, so its fitted state leaks test data into the training workflow. Position one is the only place that cannot happen, and a nested pipeline is never step one of itself.

Validation

At most one TrainTestSplit, and only at index 1.
No TrainTestSplit inside a nested Pipeline or a PipelineStep.

Related

PortfolioOptimisers.has_split Function

julia

has_split(_) -> Bool

Return whether a step is, or contains, a TrainTestSplit.

A nested Pipeline is searched recursively: a split hidden inside one would be fitted on data an outer step had already touched, which is exactly what pinning it to the first position prevents. The same recursion answers whether a whole pipeline carries a holdout, which is what the cross-validation entry points check before running.

Related

PortfolioOptimisers.assert_no_holdout Function

julia

assert_no_holdout(pipe::Pipeline)

Reject a Pipeline carrying a TrainTestSplit from the cross-validation machinery.

A holdout split and a cross-validator are two evaluation protocols, and cross-validation already defines the train/test windows of every fold. A split left in the pipeline would shave a second, redundant holdout off each fold's training window and stash a test window nobody reads — a silent loss of training data. One protocol per call: this throws instead.

Related

PortfolioOptimisers.holdout_window Function

julia

holdout_window(res::PipelineResult) -> Any

Return the held-out window stashed by a pipeline's TrainTestSplit step, or nothing when it has none.

Related

Injection

PortfolioOptimisers.inject_context Function

julia

inject_context(
    opt::OptimisationEstimator,
    ctx::PipelineContext
) -> Any

Override an optimisation step's internal configuration with the computed slots of the pipeline context, immediately before the step runs.

Estimators carrying an opt configuration field (JuMPOptimiser or HierarchicalOptimiser) are rebuilt via inject_config, and a covariance uncertainty set is routed into their risk measures via inject_sigma_ucs. Estimators without an injectable configuration (naive and meta-optimisers) ignore the prior and phylogeny slots — they compute internally — but populated uncertainty or constraints slots throw an ArgumentError rather than being silently dropped.

Arguments

opt: The optimisation step estimator.
ctx: The pipeline context.

Returns

opt′: The (possibly rebuilt) estimator actually run.

Related

PortfolioOptimisers.inject_config Function

julia

inject_config(cfg::Union{<:JuMPOptimiser, <:HierarchicalOptimiser}, ctx::PipelineContext)

Override an optimiser configuration's internal estimators with the computed slots of the pipeline context.

Routing:

prior slot → pe (both configurations; the result-in-place-of-estimator idiom, cf. prior(::AbstractPriorResult)).
phylogeny slot → cle on HierarchicalOptimiser when it is a hierarchical clustering result; ignored by JuMPOptimiser, whose phylogeny information enters as constraint results.
uncertainty slot, mean half → ret.ucs on JuMPOptimiser (requires ArithmeticReturn); unroutable into a HierarchicalOptimiser. The covariance half is routed separately into the risk measure by inject_sigma_ucs.
constraints slot elements, by result type: WeightBounds → wb; LinearConstraint → lcse and phylogeny constraint results → ple (JuMPOptimiser only).

Unroutable elements throw an ArgumentError naming the offending type.

Arguments

cfg: The optimiser configuration.
ctx: The pipeline context.

Returns

cfg′: The updated configuration.

Related

PortfolioOptimisers.inject_sigma_ucs Function

julia

inject_sigma_ucs(
    opt::OptimisationEstimator,
    sig::AbstractUncertaintySetResult
) -> Any

Route a covariance uncertainty set into the UncertaintySetVariance risk measure(s) of an optimisation estimator.

The optimiser must expose a risk-measure field r containing at least one UncertaintySetVariance (possibly inside a vector); each one's ucs is replaced with sig. Anything else throws an ArgumentError.

Arguments

opt: The optimisation estimator.
sig: The covariance uncertainty set result.

Returns

opt′: The updated estimator.

Related

PortfolioOptimisers.constraint_results Function

julia

constraint_results(_::Nothing) -> Tuple{}

Iterate the elements of the constraints slot uniformly.

Arguments

x: nothing, a single AbstractConstraintResult, or a vector of them.

Returns

An iterable of constraint results (empty for nothing).

Related

inject_config

PortfolioOptimisers.maybe_inject_step Function

julia

maybe_inject_step(est, ::PipelineContext) = est
maybe_inject_step(opt::OptimisationEstimator, ctx::PipelineContext)
maybe_inject_step(ps::PipelineStep, ctx::PipelineContext)

Either return the step estimator unchanged, inject the context into the optimiser, or inject the context into the optimiser and create a pipeline step.

Arguments

est: A step estimator.
opt: An optimisation step estimator.
ps: A PipelineStep wrapping an optimisation step estimator.
ctx: The pipeline context.

Returns

est′: The step estimator to run.
opt: The optimiser with its configuration overridden by the context.
ps: The pipeline step with its optimiser overridden by the context.

julia

maybe_inject_step(
    res::NonFiniteAllocationOptimisationResult,
    ctx::PipelineContext
) -> NonFiniteAllocationOptimisationResult

Injection rules for a precomputed optimisation result standing in the optimisation step — the predict-only fold of a mixed TimeDependent schedule.

A result is already solved, so it has no configuration to override; this reuses the non-injectable pattern of inject_context: computed prior and phylogeny slots pass by (the result was fitted with its own), but populated uncertainty or constraints slots throw an ArgumentError rather than being silently dropped — a computed constraint that never reaches a solve is a fail-closed error, not a no-op.

Related

Prediction

Predicting with a fitted pipeline replays the fitted preprocessing steps — the training universe, the training imputation parameters, the returns conversion — on an unseen data window, then delegates to the existing weights-level prediction machinery. Cross-validation folds can be computed directly on price-level data (Prices_RR), so the whole workflow is fitted per fold with no test-window leakage into stateful preprocessing.

PortfolioOptimisers.apply_fitted_step Function

julia

apply_fitted_step(fitted, data) -> data′

Replay one fitted pipeline step on a data window during prediction.

Preprocessing steps transform the window at the data level they apply to: price-level fitted objects (AbstractPricesPreprocessingResult, AbstractPricesPreprocessingEstimator) transform price-level windows, returns-level ones transform returns-level windows, and PricesToReturns converts the window from prices to returns. A fitted object whose data level does not match the current window passes it through unchanged — mirroring fit, where such a step cannot affect the data that reaches the optimiser. Non-preprocessing fitted results (priors, phylogeny, uncertainty, constraints, optimisation) pass the window through untouched, and a nested PipelineResult replays its own steps recursively.

Arguments

fitted: A fitted per-step result from a PipelineResult.
data: The current data window (AbstractPricesResult or AbstractReturnsResult).

Returns

data′: The transformed (or untouched) data window.

Related

PortfolioOptimisers.apply_fitted_steps Function

julia

apply_fitted_steps(
    results::Tuple,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> Any

Replay the fitted preprocessing steps of a pipeline on a data window, in step order.

Arguments

results: The fitted per-step results of a PipelineResult.
data: The data window to transform.

Returns

data′: The transformed data window (returns-level when the steps include a PricesToReturns conversion).

Related

PortfolioOptimisers.assert_universe_aligned Function

julia

assert_universe_aligned(
    res::PipelineResult,
    rd::AbstractReturnsResult
)

Assert that replaying a pipeline's fitted steps on a test window reproduces the training asset universe.

The terminal weights are indexed by the training universe, so a test window whose transformed returns carry a different asset set (or a different asset order) would silently misalign weights and returns. This is the failure the fit/apply contract exists to prevent, so it is reported as an error naming both universes rather than surfacing as a dimension mismatch inside the risk calculation.

The usual cause is relying on PricesToReturns alone to define the universe: it is stateless, and the underlying prices_to_returns drops assets that are entirely missing in the window being converted, which differs between train and test. Pin the universe with a MissingDataFilter step, and fill the remaining gaps with an Imputer step, before converting.

Arguments

res: The fitted PipelineResult.
rd: The transformed test-window returns.

Returns

nothing.

Related