GitHub

PortfolioOptimisers pipeline

The Pipeline estimator reifies an end-to-end workflow — data preparation, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps fitted as a single unit. Computed slots override the terminal optimiser's internal configuration; absent steps fall back to what the optimiser computes internally.

PortfolioOptimisers.Pipeline — Type
struct Pipeline{__T_names, __T_steps, __T_cache} <: AbstractPipelineEstimator

A reified end-to-end portfolio workflow: an ordered list of steps executed left-to-right over a PipelineContext.

Steps are ordinary estimators — preprocessing, prior, phylogeny, uncertainty-set, constraint-generation, and optimisation estimators, nested Pipelines, or PipelineStep wrappers — mapped to context slots by their family via pipe_writes/pipe_reads. Fitting a pipeline with fit walks the steps in order; computed slots override the terminal optimiser's internal configuration (see inject_context), and absent steps fall back to whatever the optimiser computes internally, so every stage is optional.

A terminal optimiser is not required: a prior-only pipeline is legal; prediction is what needs weights.

See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Fields

  • names: Step names, aligned with steps.
  • steps: The step estimators, in execution order.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one, and the estimator's read-out verb reads it when the caller gives no data matrix. Each propagation channel does one thing with it: factory carries it unchanged, because a factory call resolves configuration rather than the sample; port_opt_view slices it to the selected assets by index copy, so the viewed estimator answers over those assets alone; and obs_weights_view drops it, because no slice of a state exists on the observation axis. A family whose state has no exact asset slice drops it on both axes and names the reason.

Constructors

Pipeline(; steps::Union{<:Tuple, <:AbstractVector},           cache::Option{<:AbstractPartialFitState} = nothing) -> Pipeline

Steps are given in execution order. Each element is either a step estimator or a "name" => estimator pair; unnamed steps are auto-named from the slot they write ("prior"), suffixed in order of appearance when a slot repeats ("prices_1", "prices_2").

Validation

  • !isempty(steps).
  • Every step must be steppable (pipe_writes must be defined for it).
  • Every slot a step reads must be written by an earlier step or fillable by the pipeline input (prices or returns).
  • No step may write a slot that invalidates a slot an earlier step already wrote (see PIPELINE_INVALIDATES). A step that rewrites :returns after a prior, phylogeny, uncertainty, or constraint step would leave that result computed on a stale asset universe.
  • An optimisation step, if present, must be the last step (see assert_opt_last): it writes the terminal :opt slot, and no step may run after it.
  • Step names must be unique.

Online form

A Pipeline is a host of the online step: partial_fit! walks the steps in order, folding each block of observations through them into the row owner — the prior step, else the optimiser step — and fit(pipe) with no data reads the fitted PipelineResult out. Every step before the owner belongs to one of three classes. A row-local step (PricesToReturns, PriceGapFill with a CarriedPrice, MissingDataFilter at row_thr = 1) folds and emits the transformed rows. A universe-only step (an AbstractAssetSelector, and the column filter of a MissingDataFilter) folds nothing and is refitted at the read-out over the owner's rows, its universe applied as a view. A window-valued step, and any other step that writes a data slot, is refused at warm-up by name, and Online(pipe) is the declared refit that admits it. A cap on the row owner alone, Online(pe; max_history = w), is a window counted in the owner's rows, and a row-local step before it folds a state across that window's front edge, so the pair is refused at warm-up by name too: the rolling window through a Pipeline is Online(pipe; max_history = w), which refits every step over the window. cache is the Fold Context the Pipeline keeps when a prior step owns the rows, or the input-carrier buffer Online(pipe) seeds; it is nothing until a step writes one. See partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{<:PipelineBufferState, <:ReturnsBufferState}}}, data::Prices_RR) and fit(pipe::Pipeline).

Examples

julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> pipe.names("returns", "prior", "opt")

Related

source
PortfolioOptimisers.PipelineResult — Type
struct PipelineResult{__T_names, __T_results, __T_ctx} <: AbstractPipelineResult

Fitted result of a Pipeline.

Carries the fitted per-step results (named, in step order), and the final PipelineContext whose slots hold the computed data, prior, phylogeny, uncertainty, constraints, and terminal optimisation result.

Step results are accessed by name with getindex (res["prior"]) or by position through the results field (res.results[2]); integer indexing keeps the package-wide length-1 container semantics. The w property forwards to the terminal optimisation result's weights (res.ctx.opt.w) and throws a PropertyPathError when the pipeline produced no optimisation result.

Fields

  • names: Step names, aligned with results.
  • results: Fitted per-step results, in step order.

Related

source
StatsAPI.fit — Method
StatsAPI.fit(pipe::Pipeline, data::Prices_RR) -> PipelineResult

Fit a Pipeline on price- or returns-level data.

The context slot matching the input type is filled (PricesResult → prices, ReturnsResult → returns, so passing returns-level data skips the price stages), then the steps run left-to-right via run_step. Immediately before an optimisation step runs, the computed slots override its internal configuration via inject_context.

fit is a fold-less entry point, so TimeDependent schedule steps are inert here: each resolves to its explicit default (see reset_time_dependent_estimator) before the steps run, and a schedule with no default throws a TimeDependentDefaultError — backtest the pipeline with cross_val_predict, whose folds the schedule resolves against. Inside a fold loop this reset is a no-op, because the loop swaps every schedule for its per-fold value first.

Arguments

Returns

  • res::PipelineResult: Named per-step fitted results and the final context.

Examples

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4),                     [100.0 101.0; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> res = fit(pipe, PricesResult(; X = X));julia> res.w2-element Vector{Float64}: 0.5 0.5

Related

source
StatsAPI.predict — Function
predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult


predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idxs::VecVecInt, cols = Colon()) -> PredictionResult

predict(res::PipelineResult, data::AbstractReturnsResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult

Apply a fitted pipeline to an unseen window of data and produce the same PredictionResult the weights-level machinery consumes.

test_idx selects the observation rows of the window and cols selects its asset columns. The window is transformed by replaying the fitted preprocessing steps in step order — the training universe subset, the training imputation parameters, then the returns conversion — so no statistic of the test window leaks into the transformation. The result is then handed to the existing weights-level predict, so scorers and risk measures carry over untouched.

Price-level data requires the pipeline to contain a PricesToReturns step; a pipeline that produced no optimisation result cannot predict.

A vector of index vectors predicts on each window in turn and returns one result per window, which is the shape the cross-validation machinery consumes.

Arguments

  • res: The fitted PipelineResult.
  • data: Price- or returns-level data containing the window (PricesResult or ReturnsResult).
  • test_idx: Observation window into the rows of data. Integer indices, timestamps, or : (all rows).
  • test_idxs: Several such windows, as a vector of index vectors.
  • cols: Asset window into the columns of data. Integer indices, or : (all assets).

Returns

  • pred::PredictionResult: The weights-level prediction on the transformed window, or one such result per window when several are given.

Related

source
PortfolioOptimisers.fit_predict — Method
fit_predict(opt::Pipeline, data::Prices_RR)

Fit pipeline estimator opt on data data and immediately produce a PredictionResult.

The prediction is made on data itself — in-sample — unless the pipeline begins with a TrainTestSplit, in which case it is made on the held-out window that step reserved and no fitted step has seen. That is the one-line holdout evaluation: fit on the training rows, score on the test rows.

Arguments

  • opt: Optimisation estimator or result.
  • data::Prices_RR: Price- or returns-level data.

Returns

  • PredictionResult: On the held-out window when the pipeline splits, on data otherwise.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(pipe::Pipeline, i, args...; kwargs...)

Deliberately unsupported: a Pipeline cannot be sub-selected by asset view.

Meta-optimisers (NestedClustered, Stacking, SubsetResampling) build asset sub-portfolios by taking a port_opt_view of their inner estimator. A pipeline's asset universe is fitted state — the missing-data filter decides it from the training window — so an asset view taken before fitting is not well defined. Wrapping a Pipeline in a meta-optimiser is therefore unsupported for now; a meta-optimiser may still be the optimisation step of a pipeline.

Related

source
PortfolioOptimisers.implicit_constraint_target — Function
implicit_constraint_target(_::WeightBounds) -> Symbol

The routing target a constraint result names by its type alone, or nothing.

Four result types name exactly one optimiser field, so a value of one of those types places itself: a WeightBounds can only be :wb, a LinearConstraint only :lcse, a phylogeny constraint result only :ple, a RiskBudget only :rkb. Everything else answers nothing, and needs a target carried alongside it — see TargetedConstraint.

This is the only type-driven half of the fan-out, and it is also what decides whether a step's value needs a wrapper at all: add_constraint_result wraps exactly when the value cannot name its own destination, so the constraints slot holds a bare result wherever it can.

Arguments

  • c: A constraint result.

Returns

Related

source

Holdout splitting

A TrainTestSplit step reserves a held-out test window before any other step runs. It is pinned to the first position — a stateful step fitted before it would have seen the held-out rows — and excludes cross-validation, which defines its own train/test windows. fit_predict(pipe, data) predicts on the window the split reserved.

PortfolioOptimisers.assert_split_position — Function
assert_split_position(ests)

Validate that a TrainTestSplit appears only as the first step of a Pipeline, and never inside a nested one.

The holdout exists to keep the test window away from every fitted step. A stateful step fitted before the split — a MissingDataFilter choosing the universe, a PriceGapFill computing fill values — would have read the held-out rows, so its fitted state leaks test data into the training workflow. Position one is the only place that cannot happen, and a nested pipeline is never step one of itself.

Validation

  • At most one TrainTestSplit, and only at index 1.
  • No TrainTestSplit inside a nested Pipeline or a PipelineStep.

Related

source
PortfolioOptimisers.has_split — Function
has_split(_) -> Bool

Return whether a step is, or contains, a TrainTestSplit.

A nested Pipeline is searched recursively: a split hidden inside one would be fitted on data an outer step had already touched, which is exactly what pinning it to the first position prevents. The same recursion answers whether a whole pipeline carries a holdout, which is what the cross-validation entry points check before running.

Related

source
PortfolioOptimisers.assert_no_holdout — Function
assert_no_holdout(pipe::Pipeline)

Reject a Pipeline carrying a TrainTestSplit from the cross-validation machinery.

A holdout split and a cross-validator are two evaluation protocols, and cross-validation already defines the train/test windows of every fold. A split left in the pipeline would shave a second, redundant holdout off each fold's training window and stash a test window nobody reads — a silent loss of training data. One protocol per call: this throws instead.

Related

source