PortfolioOptimisers pipeline
The Pipeline estimator reifies an end-to-end workflow — data preparation, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps fitted as a single unit. Computed slots override the terminal optimiser's internal configuration; absent steps fall back to what the optimiser computes internally.
Pipeline symbols
PortfolioOptimisers.Pipeline — Type
struct Pipeline{__T_names, __T_steps} <: AbstractPipelineEstimatorA reified end-to-end portfolio workflow: an ordered list of steps executed left-to-right over a PipelineContext.
Steps are ordinary estimators — preprocessing, prior, phylogeny, uncertainty-set, constraint-generation, and optimisation estimators, nested Pipelines, or PipelineStep wrappers — mapped to context slots by their family via pipe_writes/pipe_reads. Fitting a pipeline with fit walks the steps in order; computed slots override the terminal optimiser's internal configuration (see inject_context), and absent steps fall back to whatever the optimiser computes internally, so every stage is optional.
A terminal optimiser is not required: a prior-only pipeline is legal; prediction is what needs weights.
See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.
Fields
names: Step names, aligned withsteps.
steps: The step estimators, in execution order.
Constructors
Pipeline(; steps::Union{<:Tuple, <:AbstractVector}) -> PipelineSteps are given in execution order. Each element is either a step estimator or a "name" => estimator pair; unnamed steps are auto-named from the slot they write ("prior"), suffixed in order of appearance when a slot repeats ("prices_1", "prices_2").
Validation
!isempty(steps).- Every step must be steppable (
pipe_writesmust be defined for it). - Every slot a step reads must be written by an earlier step or fillable by the pipeline input (
pricesorreturns). - No step may write a slot that invalidates a slot an earlier step already wrote (see
PIPELINE_INVALIDATES). A step that rewrites:returnsafter a prior, phylogeny, uncertainty, or constraint step would leave that result computed on a stale asset universe. - An optimisation step, if present, must be the last step (see
assert_opt_last): it writes the terminal:optslot, and no step may run after it. - Step names must be unique.
Examples
julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> pipe.names("returns", "prior", "opt")Related
PortfolioOptimisers.PipelineResult — Type
struct PipelineResult{__T_names, __T_results, __T_ctx} <: AbstractPipelineResultFitted result of a Pipeline.
Carries the fitted per-step results (named, in step order), and the final PipelineContext whose slots hold the computed data, prior, phylogeny, uncertainty, constraints, and terminal optimisation result.
Step results are accessed by name with getindex (res["prior"]) or by position through the results field (res.results[2]); integer indexing keeps the package-wide length-1 container semantics. The w property forwards to the terminal optimisation result's weights (res.ctx.opt.w) and throws a PropertyPathError when the pipeline produced no optimisation result.
Fields
names: Step names, aligned withresults.
results: Fitted per-step results, in step order.
ctx: The finalPipelineContext.
Related
StatsAPI.fit — Method
StatsAPI.fit(pipe::Pipeline, data::Prices_RR) -> PipelineResultFit a Pipeline on price- or returns-level data.
The context slot matching the input type is filled (PricesResult → prices, ReturnsResult → returns, so passing returns-level data skips the price stages), then the steps run left-to-right via run_step. Immediately before an optimisation step runs, the computed slots override its internal configuration via inject_context.
fit is a fold-less entry point, so TimeDependent schedule steps are inert here: each resolves to its explicit default (see reset_time_dependent_estimator) before the steps run, and a schedule with no default throws a TimeDependentDefaultError — backtest the pipeline with cross_val_predict, whose folds the schedule resolves against. Inside a fold loop this reset is a no-op, because the loop swaps every schedule for its per-fold value first.
Arguments
pipe: The pipeline.data: The input data (PricesResultorReturnsResult).
Returns
res::PipelineResult: Named per-step fitted results and the final context.
Examples
julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4), [100.0 101.0; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> res = fit(pipe, PricesResult(; X = X));julia> res.w2-element Vector{Float64}: 0.5 0.5Related
StatsAPI.predict — Function
predict(res::PipelineResult, data::AbstractPricesResult,
test_idx = Colon(), cols = Colon()) -> PredictionResult
predict(res::PipelineResult, data::AbstractPricesResult,
test_idxs::VecVecInt, cols = Colon()) -> PredictionResult
predict(res::PipelineResult, data::AbstractReturnsResult,
test_idx = Colon(), cols = Colon()) -> PredictionResultApply a fitted pipeline to an unseen window of data and produce the same PredictionResult the weights-level machinery consumes.
test_idx selects the observation rows of the window and cols selects its asset columns. The window is transformed by replaying the fitted preprocessing steps in step order — the training universe subset, the training imputation parameters, then the returns conversion — so no statistic of the test window leaks into the transformation. The result is then handed to the existing weights-level predict, so scorers and risk measures carry over untouched.
Price-level data requires the pipeline to contain a PricesToReturns step; a pipeline that produced no optimisation result cannot predict.
A vector of index vectors predicts on each window in turn and returns one result per window, which is the shape the cross-validation machinery consumes.
Arguments
res: The fittedPipelineResult.data: Price- or returns-level data containing the window (PricesResultorReturnsResult).test_idx: Observation window into the rows ofdata. Integer indices, timestamps, or:(all rows).test_idxs: Several such windows, as a vector of index vectors.cols: Asset window into the columns ofdata. Integer indices, or:(all assets).
Returns
pred::PredictionResult: The weights-level prediction on the transformed window, or one such result per window when several are given.
Related
PortfolioOptimisers.fit_predict — Method
fit_predict(opt::Pipeline, data::Prices_RR)Fit pipeline estimator opt on data data and immediately produce a PredictionResult.
The prediction is made on data itself — in-sample — unless the pipeline begins with a TrainTestSplit, in which case it is made on the held-out window that step reserved and no fitted step has seen. That is the one-line holdout evaluation: fit on the training rows, score on the test rows.
Arguments
opt: Optimisation estimator or result.data::Prices_RR: Price- or returns-level data.
Returns
PredictionResult: On the held-out window when the pipeline splits, ondataotherwise.
Related
PortfolioOptimisers.port_opt_view — Method
port_opt_view(pipe::Pipeline, i, args...; kwargs...)Deliberately unsupported: a Pipeline cannot be sub-selected by asset view.
Meta-optimisers (NestedClustered, Stacking, SubsetResampling) build asset sub-portfolios by taking a port_opt_view of their inner estimator. A pipeline's asset universe is fitted state — the missing-data filter decides it from the training window — so an asset view taken before fitting is not well defined. Wrapping a Pipeline in a meta-optimiser is therefore unsupported in v1 (ADR 0028, "Future expansion"); a meta-optimiser may still be the optimisation step of a pipeline.
Related
PortfolioOptimisers.first_duplicate — Function
first_duplicate(xs) -> Any
Return the first element that repeats an earlier one, for a name-uniqueness error that names the offending token without dumping the whole collection (ADR 0026 boundary discipline). Only ever called on the failing path.
Arguments
xs: A collection of names.
Returns
- The first element that repeats an earlier one, or
nothingwhen every element is unique.
Related
Holdout splitting
A TrainTestSplit step reserves a held-out test window before any other step runs. It is pinned to the first position — a stateful step fitted before it would have seen the held-out rows — and excludes cross-validation, which defines its own train/test windows. fit_predict(pipe, data) predicts on the window the split reserved.
PortfolioOptimisers.assert_split_position — Function
assert_split_position(ests)
Validate that a TrainTestSplit appears only as the first step of a Pipeline, and never inside a nested one.
The holdout exists to keep the test window away from every fitted step. A stateful step fitted before the split — a MissingDataFilter choosing the universe, an Imputer computing fill values — would have read the held-out rows, so its fitted state leaks test data into the training workflow. Position one is the only place that cannot happen, and a nested pipeline is never step one of itself.
Validation
- At most one
TrainTestSplit, and only at index 1. - No
TrainTestSplitinside a nestedPipelineor aPipelineStep.
Related
PortfolioOptimisers.has_split — Function
has_split(_) -> Bool
Return whether a step is, or contains, a TrainTestSplit.
A nested Pipeline is searched recursively: a split hidden inside one would be fitted on data an outer step had already touched, which is exactly what pinning it to the first position prevents. The same recursion answers whether a whole pipeline carries a holdout, which is what the cross-validation entry points check before running.
Related
PortfolioOptimisers.assert_no_holdout — Function
assert_no_holdout(pipe::Pipeline)
Reject a Pipeline carrying a TrainTestSplit from the cross-validation machinery.
A holdout split and a cross-validator are two evaluation protocols, and cross-validation already defines the train/test windows of every fold. A split left in the pipeline would shave a second, redundant holdout off each fold's training window and stash a test window nobody reads — a silent loss of training data. One protocol per call: this throws instead.
Related
PortfolioOptimisers.holdout_window — Function
holdout_window(res::PipelineResult) -> Any
Return the held-out window stashed by a pipeline's TrainTestSplit step, or nothing when it has none.
Related
Injection
The pipeline resolves its computed slots into routing targets and hands each one to the optimiser, which owns the decision of where it lands. See pipe_route for the optimiser-owned half of the seam.
PortfolioOptimisers.inject_context — Function
inject_context(
opt::OptimisationEstimator,
ctx::PipelineContext
) -> Any
Override an optimisation step's internal configuration with the computed slots of the pipeline context, immediately before the step runs.
This is the pipeline-owned half of the injection seam. It resolves everything that depends on the slots — which halves of the uncertainty pair are populated, which result types the constraints slot holds, how many of each — into a flat sequence of routing targets, then hands each one to pipe_route without knowing where it lands. Which optimiser field receives a target is the optimiser's business, so a field rename is a local edit rather than a break here.
Targets an optimiser has no home for are handled by unroutable_target: :pe and :cle pass by, everything else throws rather than being silently dropped. This is why a naive or meta-optimiser accepts a computed prior it can use while still rejecting an uncertainty set it cannot.
Arguments
opt: The optimisation step estimator.ctx: The pipeline context.
Returns
opt′: The (possibly rebuilt) estimator actually run.
Related
PortfolioOptimisers.constraint_results — Function
constraint_results(_::Nothing) -> Tuple{}
Iterate the elements of the constraints slot uniformly.
Arguments
x:nothing, a singleAbstractConstraintResult, or a vector of them.
Returns
- An iterable of constraint results (empty for
nothing).
Related
PortfolioOptimisers.implicit_constraint_target — Function
implicit_constraint_target(_::WeightBounds) -> Symbol
The routing target a constraint result names by its type alone, or nothing.
Four result types name exactly one optimiser field, so a value of one of those types places itself: a WeightBounds can only be :wb, a LinearConstraint only :lcse, a phylogeny constraint result only :ple, a RiskBudget only :rkb. Everything else answers nothing, and needs a target carried alongside it — see TargetedConstraint.
This is the only type-driven half of the fan-out, and it is also what decides whether a step's value needs a wrapper at all: add_constraint_result wraps exactly when the value cannot name its own destination, so the constraints slot holds a bare result wherever it can.
Arguments
c: A constraint result.
Returns
target::Union{Nothing, Symbol}: One ofPIPELINE_ROUTING_TARGETS, ornothing.
Related
PortfolioOptimisers.constraint_target_of — Function
constraint_target_of(c) -> Symbol
The routing target one element of the constraints slot lands in.
An element a constraint step could not place by type carries its target — run_constraint_step paired the two — and it is read straight off. Everything else is placed by implicit_constraint_target.
Two cases throw. A Threshold names six optimiser fields, so its type cannot place it; the error names the declaration that would. A result of any other unplaceable type has no target at all, and is rejected here rather than at an optimiser.
Arguments
c: One element of theconstraintsslot.
Returns
target::Symbol: One ofPIPELINE_ROUTING_TARGETS.
Related
PortfolioOptimisers.constraint_value_of — Function
constraint_value_of(c) -> Any
The value one element of the constraints slot delivers, with the routing wrapper removed.
Arguments
c: One element of theconstraintsslot.
Returns
- The value to route.
Related
PortfolioOptimisers.accumulate_constraint_values — Function
accumulate_constraint_values(
_::Val,
vals
) -> LinearConstraint
Combine the several values that reached one accumulating routing target.
The default packs them into a vector in write order, which is the shape every field holding one result per estimator expects.
:cte is the exception, and it is what this seam exists for. Its field takes a vector of CentralityConstraint estimators, and centrality_constraints appends every row of every estimator into one LinearConstraint. Separate steps therefore merge rather than pack, so n centrality steps in a Pipeline reach the optimiser with the value one cte field holding n estimators would have produced.
Only ever called with more than one value; a single value is unwrapped by constraint_targets before it gets here.
Arguments
::Val{target}: The routing target the values reached.vals: The values, in write order.
Returns
- The combined value.
Related
PortfolioOptimisers.constraint_targets — Function
constraint_targets(cs) -> Vector{Pair{Symbol, Any}}
Fan the constraints slot out into routing targets.
Each element is placed by constraint_target_of and unwrapped by constraint_value_of. Several results reaching one accumulating target are combined by accumulate_constraint_values — packed into a vector in write order, or, for :cte, merged into the one constraint that holds all their rows. A group of one is unwrapped, matching the scalar-or-vector shape those fields accept everywhere else. A second result reaching any other target is refused, because that field holds one value and the second would silently replace the first.
Arguments
cs: Theconstraintsslot.
Returns
- A vector of
target => valuepairs, in the order the results were written.
Related
PortfolioOptimisers.maybe_inject_step — Function
maybe_inject_step(est, ::PipelineContext) = est
maybe_inject_step(opt::OptimisationEstimator, ctx::PipelineContext)
maybe_inject_step(ps::PipelineStep, ctx::PipelineContext)Either return the step estimator unchanged, inject the context into the optimiser, or inject the context into the optimiser and create a pipeline step.
Arguments
est: A step estimator.opt: An optimisation step estimator.ps: APipelineStepwrapping an optimisation step estimator.ctx: The pipeline context.
Returns
est′: The step estimator to run.opt: The optimiser with its configuration overridden by the context.ps: The pipeline step with its optimiser overridden by the context.
maybe_inject_step(
res::NonFiniteAllocationOptimisationResult,
ctx::PipelineContext
) -> NonFiniteAllocationOptimisationResult
Injection rules for a precomputed optimisation result standing in the optimisation step — the predict-only fold of a mixed TimeDependent schedule.
A result is already solved, so it has no configuration to override; this reuses the non-injectable pattern of inject_context: computed prior and phylogeny slots pass by (the result was fitted with its own), but populated uncertainty or constraints slots throw an ArgumentError rather than being silently dropped — a computed constraint that never reaches a solve is a fail-closed error, not a no-op.
Related
PortfolioOptimisers.pipe_required_targets — Function
pipe_required_targets(
ps::PipelineStep
) -> Union{Tuple{}, Tuple{Symbol}, Tuple{Symbol, Symbol}}
The routing targets a step is known at construction to produce.
An uncertainty-set step qualifies. It must declare which parameters it bounds through its PipelineStep wrapper, and that declaration is a field of the step rather than a property of a computed result, so the targets it will write are known before anything runs.
A constraint step qualifies for the same reason, one step removed: its target is declared by its family through pipe_constraint_targets, and where the family names several, by the step's own target field. Both are known before anything runs, and run_constraint_step resolves the destination from the same declaration, so the target checked here is the target the step will write.
Everything else returns an empty tuple. A callable step writing :constraints declares no family, and a precomputed result carried in by the pipeline input names its target only by its type.
Arguments
est: A step estimator.
Returns
- A tuple of routing targets, empty when nothing is statically known.
Related
PortfolioOptimisers.assert_routable — Function
assert_routable(ests)
Reject at construction a pipeline whose terminal optimiser cannot receive a target an earlier step will write.
Without this, an unroutable uncertainty set is discovered by inject_context at injection time — which, under cross_val_predict, is after the fold loop has already fitted every earlier step of the first fold. The check asks the optimiser directly via pipe_accepts, so it stays honest as optimisers gain or lose fields.
It is deliberately structural: it establishes that the optimiser family can receive the target at all, not that this particular configuration will accept the value. A JuMPOptimiser always accepts :mu_ucs, but one carrying a non-ArithmeticReturn estimator still fails at injection — that condition belongs to pipe_route and is not duplicated here.
Skipped when the terminal step is a TimeDependent schedule or a precomputed result, since the optimiser is then not known until the fold loop resolves it.
Arguments
ests: The step estimators, optimisation step last (seeassert_opt_last).
Returns
nothing.
Related
PortfolioOptimisers.assert_constraint_targets — Function
assert_constraint_targets(ests)
Validate that every constraint step of a Pipeline resolves to exactly one routing target.
Runs resolve_constraint_target on each constraint step, which is the same call run_constraint_step makes when the step runs. Doing it here moves three failures from the fold loop to the constructor: a family that computes nothing for the constraints slot and is therefore not a step, a family that names several targets and was not told which, and a declared target that belongs to another family.
Validation
- Each constraint step's family declares at least one target (see
pipe_constraint_targets). - A family declaring several has a
PipelineSteptargetnaming one of them.
Arguments
ests: The step estimators.
Returns
nothing.
Related
Prediction
Predicting with a fitted pipeline replays the fitted preprocessing steps — the training universe, the training imputation parameters, the returns conversion — on an unseen data window, then delegates to the existing weights-level prediction machinery. Cross-validation folds can be computed directly on price-level data (Prices_RR), so the whole workflow is fitted per fold with no test-window leakage into stateful preprocessing.
PortfolioOptimisers.apply_fitted_step — Function
apply_fitted_step(fitted, data) -> data′Replay one fitted pipeline step on a data window during prediction.
Preprocessing steps transform the window at the data level they apply to: price-level fitted objects (AbstractPricesPreprocessingResult, AbstractPricesPreprocessingEstimator) transform price-level windows, returns-level ones transform returns-level windows, and PricesToReturns converts the window from prices to returns. A fitted object whose data level does not match the current window passes it through unchanged — mirroring fit, where such a step cannot affect the data that reaches the optimiser. Non-preprocessing fitted results (priors, phylogeny, uncertainty, constraints, optimisation) pass the window through untouched, and a nested PipelineResult replays its own steps recursively.
Arguments
fitted: A fitted per-step result from aPipelineResult.data: The current data window (AbstractPricesResultorAbstractReturnsResult).
Returns
data′: The transformed (or untouched) data window.
Related
PortfolioOptimisers.apply_fitted_steps — Function
apply_fitted_steps(
results::Tuple,
data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> Any
Replay the fitted preprocessing steps of a pipeline on a data window, in step order.
Arguments
results: The fitted per-step results of aPipelineResult.data: The data window to transform.
Returns
data′: The transformed data window (returns-level when the steps include aPricesToReturnsconversion).
Related
PortfolioOptimisers.assert_universe_aligned — Function
assert_universe_aligned(
res::PipelineResult,
rd::AbstractReturnsResult
)
Assert that replaying a pipeline's fitted steps on a test window reproduces the training asset universe.
The terminal weights are indexed by the training universe, so a test window whose transformed returns carry a different asset set (or a different asset order) would silently misalign weights and returns. This is the failure the fit/apply contract exists to prevent, so it is reported as an error naming both universes rather than surfacing as a dimension mismatch inside the risk calculation.
The usual cause is relying on PricesToReturns alone to define the universe: it is stateless, and the underlying prices_to_returns drops assets that are entirely missing in the window being converted, which differs between train and test. Pin the universe with a MissingDataFilter step, and fill the remaining gaps with an Imputer step, before converting.
Arguments
res: The fittedPipelineResult.rd: The transformed test-window returns.
Returns
nothing.
Related