The source files can be found in examples/.
Pipelines
Every example so far has started from a returns matrix. But a returns matrix is already the output of a series of decisions: which assets had enough data to keep, how the gaps in their price history were filled, whether returns are simple or logarithmic. Those decisions have hyperparameters, and until now they have been made once, on the full sample, before any cross-validation loop began.
That is a leak. If the imputation fill value for an asset is computed from the whole price history, then every "out-of-sample" test fold has already seen the future. If the surviving asset universe is chosen by looking at missingness across all five years, the backtest is quietly conditioned on knowing which assets survive.
A Pipeline fixes this by making the entire workflow — price cleaning, prices-to-returns conversion, prior estimation, phylogeny, constraint generation, and optimisation — the unit that gets fitted. Fit a pipeline on a training window and it learns its preprocessing state there; predict on a test window and that state is replayed, never recomputed. Cross-validate a pipeline and the split happens on the input rows, so the cleaning steps live inside the fold.
This example builds a pipeline stage by stage, shows what each stage contributes, and then tunes preprocessing hyperparameters jointly with optimiser hyperparameters under walk-forward cross-validation.
using PortfolioOptimisers, PrettyTables
# Format for pretty tables.
tsfmt = (v, i, j) -> begin
if j == 1
return Date(v)
else
return v
end
end;
resfmt = (v, i, j) -> begin
if j == 1
return v
else
return isa(v, Number) ? "$(round(v*100, digits=3)) %" : v
end
end;1. Setting up
We use four years of daily data so that walk-forward cross-validation has enough history for several folds. Crucially, we start from prices, not returns — the pipeline is what turns one into the other.
using CSV, TimeSeries, DataFrames, Clarabel, Statistics, StableRNGs
X = TimeArray(CSV.File(joinpath(@__DIR__, "..", "SP500.csv.gz")); timestamp = :Date)[(end - 252 * 4):end]
X = X[Symbol.(["AAPL", "MSFT", "JNJ", "JPM", "XOM", "PG", "KO", "WMT", "PEP", "MRK"])...]
pretty_table(X[(end - 5):end]; formatters = [tsfmt])
slv = Solver(; name = :clarabel, solver = Clarabel.Optimizer,
settings = Dict("verbose" => false),
check_sol = (; allow_local = true, allow_almost = true));┌────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬──────
│ timestamp │ AAPL │ MSFT │ JNJ │ JPM │ XOM │ PG │ ⋯
│ Date │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Flo ⋯
├────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼──────
│ 2022-12-20 │ 131.916 │ 240.67 │ 173.109 │ 127.844 │ 104.964 │ 147.661 │ 61 ⋯
│ 2022-12-21 │ 135.057 │ 243.287 │ 175.09 │ 129.282 │ 106.312 │ 149.015 │ 62 ⋯
│ 2022-12-22 │ 131.846 │ 237.077 │ 174.45 │ 127.814 │ 104.168 │ 149.359 │ 62 ⋯
│ 2022-12-23 │ 131.477 │ 237.614 │ 174.893 │ 128.421 │ 106.922 │ 149.781 │ 62 ⋯
│ 2022-12-27 │ 129.652 │ 235.852 │ 174.844 │ 128.871 │ 108.408 │ 151.086 │ 6 ⋯
│ 2022-12-28 │ 125.674 │ 233.434 │ 174.085 │ 129.575 │ 106.627 │ 149.133 │ 62 ⋯
└────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴──────
4 columns omitted1.1 PricesResult, the price-level container
PricesResult is the prices-level mirror of ReturnsResult: asset prices plus optional factor, benchmark, and implied-volatility series. It is the input a pipeline expects when it starts at the price level.
To make the cleaning steps do visible work, we punch some holes in the data: one asset loses most of its early history, and two others lose isolated observations.
vals = Matrix{Float64}(values(X))
vals[1:(end ÷ 2), 3] .= NaN ## JNJ: missing for the first half of the sample
vals[10, 1] = NaN ## AAPL: an isolated gap
vals[25, 5] = NaN ## XOM: an isolated gap
Xm = TimeArray(timestamp(X), vals, colnames(X))
pr = PricesResult(; X = Xm)
# How much is missing, per asset?
miss = DataFrame(; asset = string.(colnames(Xm)),
missing_frac = vec(count(isnan, vals; dims = 1)) ./ size(vals, 1))
pretty_table(miss; formatters = [resfmt])┌────────┬──────────────┐
│ asset │ missing_frac │
│ String │ Float64 │
├────────┼──────────────┤
│ AAPL │ 0.099 % │
│ MSFT │ 0.0 % │
│ JNJ │ 49.95 % │
│ JPM │ 0.0 % │
│ XOM │ 0.099 % │
│ PG │ 0.0 % │
│ KO │ 0.0 % │
│ WMT │ 0.0 % │
│ PEP │ 0.0 % │
│ MRK │ 0.0 % │
└────────┴──────────────┘2. Building a pipeline stage by stage
A pipeline is an ordered list of steps. Each step is an ordinary estimator; its family decides which slot of the pipeline context it reads and writes, so there is no wrapper ceremony. Steps may be named with "name" => estimator; unnamed steps are auto-named after the slot they write.
2.1 The minimal pipeline: prices to returns
PricesToReturns is the step form of prices_to_returns. It is stateless: applying it to any window simply runs the conversion.
pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()))
pipe.names("returns", "prior", "opt")Fitting walks the steps left to right. The result carries every step's fitted output, the final context, and the terminal weights.
res = fit(pipe, pr)
pretty_table(DataFrame(; asset = res.ctx.returns.nx, weight = res.w); formatters = [resfmt])┌────────┬─────────┐
│ asset │ weight │
│ String │ Float64 │
├────────┼─────────┤
│ AAPL │ 10.0 % │
│ MSFT │ 10.0 % │
│ JNJ │ 10.0 % │
│ JPM │ 10.0 % │
│ XOM │ 10.0 % │
│ PG │ 10.0 % │
│ KO │ 10.0 % │
│ WMT │ 10.0 % │
│ PEP │ 10.0 % │
│ MRK │ 10.0 % │
└────────┴─────────┘Note the row count: T prices become T - 1 returns. The pipeline tracks this contraction for you, which matters when cross-validation windows are sized in input rows.
size(values(pr.X), 1), size(res.ctx.returns.X, 1)(1009, 504)2.2 Universe selection is fitted state
MissingDataFilter drops assets whose missing fraction exceeds col_thr. The surviving universe is fitted state: the training window decides it, and applying the fitted result to an unseen window subsets that window to the same assets. This is what keeps train weights and test returns aligned.
With col_thr = 0.4, JNJ (missing half its history) is dropped.
pipe = Pipeline(;
steps = ("filter" => MissingDataFilter(; col_thr = 0.4), PricesToReturns(),
EmpiricalPrior(), EqualWeighted()))
res = fit(pipe, pr)
res["filter"].nx9-element Vector{Symbol}:
:AAPL
:MSFT
:JPM
:XOM
:PG
:KO
:WMT
:PEP
:MRKRaise the threshold and JNJ survives — the universe is a hyperparameter, and in §4 we will tune it rather than guess it.
Pin the universe before converting to returns
PricesToReturns is stateless, and the underlying prices_to_returns silently drops assets that are entirely missing in the window it converts. A training window in which an asset has no history at all therefore produces fewer assets than a clean test window, and the fitted weights would not line up with the test returns.
Any pipeline you intend to predict or cross-validate with should pin the universe with a MissingDataFilter step and fill the remaining gaps with an Imputer step before converting. If you forget, predict refuses to guess and reports the two universes it found.
res_lax = fit(Pipeline(;
steps = ("filter" => MissingDataFilter(; col_thr = 0.9),
PricesToReturns(), EmpiricalPrior(), EqualWeighted())), pr)
res_lax["filter"].nx10-element Vector{Symbol}:
:AAPL
:MSFT
:JNJ
:JPM
:XOM
:PG
:KO
:WMT
:PEP
:MRK2.3 Imputation parameters are fitted state
Imputer fills gaps with a per-asset statistic computed on the training window. The fill values are fitted state: a test window is filled with training statistics, never with its own. This is the leakage-prevention exemplar.
To see why that matters, fit the same imputer on two different windows and compare what it learns.
pipe_imp = Pipeline(; steps = ("impute" => Imputer(), PricesToReturns(), EmpiricalPrior()))
res_train = fit(pipe_imp, PricesResult(; X = Xm[1:500]))
res_test = fit(pipe_imp, PricesResult(; X = Xm[501:end]))
j = findfirst(==(:AAPL), res_train["impute"].nx)
k = findfirst(==(:AAPL), res_test["impute"].nx)
fills = DataFrame(; window = ["train (1:500)", "test (501:end)"],
aapl_fill = [res_train["impute"].v[j], res_test["impute"].v[k]])
pretty_table(fills)┌────────────────┬───────────┐
│ window │ aapl_fill │
│ String │ Float64 │
├────────────────┼───────────┤
│ train (1:500) │ 64.901 │
│ test (501:end) │ 145.653 │
└────────────────┴───────────┘The two windows disagree — AAPL's price level is very different across them. A pipeline fitted on the training window carries the train number, and predict replays exactly that number on the test window. Had we imputed on the full sample before splitting, the fill would have been contaminated by the test period, and every subsequent "out-of-sample" score would be optimistic.
The statistic itself is configurable — Imputer(; stat = MeanValue()) versus MedianValue() — and is another hyperparameter to tune in §4.
3. The full workflow
Now the whole chain from the ADR: prices → filter → impute → returns → prior → phylogeny constraints → weight bounds → MeanRisk.
Two things make this work without any plumbing:
Slot routing. Each step writes the context slot its family owns. A prior estimator writes
:prior; a phylogeny-constraint estimator writes:constraints.Injection. Immediately before the optimisation step runs, the computed slots override the optimiser's internal configuration. The pipeline's prior replaces the optimiser's default
pe; the constraint results are routed by type intowb,lcse, andple. Every stage is optional — an absent step simply lets the optimiser compute that quantity internally, exactly as it does today.
pipe = Pipeline(;
steps = ("filter" => MissingDataFilter(; col_thr = 0.4),
"impute" => Imputer(), PricesToReturns(), EmpiricalPrior(),
SemiDefinitePhylogenyEstimator(),
"wb" => WeightBoundsEstimator(; ub = 0.4),
MeanRisk(; opt = JuMPOptimiser(; slv = slv))))
pipe.names
res = fit(pipe, pr)
pretty_table(DataFrame(; asset = res.ctx.returns.nx, weight = res.w); formatters = [resfmt])┌────────┬──────────┐
│ asset │ weight │
│ String │ Float64 │
├────────┼──────────┤
│ AAPL │ 0.006 % │
│ MSFT │ 0.001 % │
│ JPM │ 0.0 % │
│ XOM │ 6.269 % │
│ PG │ 15.693 % │
│ KO │ 19.96 % │
│ WMT │ 31.428 % │
│ PEP │ 0.0 % │
│ MRK │ 26.642 % │
└────────┴──────────┘The weight bound is respected, and the prior was computed once and shared with the optimiser rather than recomputed inside it.
maximum(res.w) <= 0.4 + 1e-8true3.1 Predicting on an unseen window
predict replays the fitted preprocessing on a test window — universe subset, then train-fitted imputation, then the returns conversion — and hands the result to the ordinary weights-level prediction machinery. Scorers and risk measures carry over untouched.
T = size(values(Xm), 1)
res_train = fit(pipe, PricesResult(; X = Xm[1:800]))
pred = PortfolioOptimisers.predict(res_train, pr, 801:T)
expected_risk(ConditionalValueatRisk(), pred)0.0259633328420141344. Tuning the whole workflow
This is the point of the whole exercise. search_cross_validation splits the input rows into contiguous windows, and for each candidate fits the entire pipeline on the training window and scores it on the test window. Preprocessing hyperparameters are searched jointly with optimiser hyperparameters, and no candidate ever sees the test window during preprocessing.
Lens keys address steps three ways:
by name with a trailing property path —
"filter.col_thr";by name alone (or by integer position), which swaps the whole step —
"impute";by raw property path, exactly as for plain optimisers —
"steps[1].col_thr".
pipe = Pipeline(;
steps = ("filter" => MissingDataFilter(; col_thr = 0.4),
"impute" => Imputer(), PricesToReturns(), EmpiricalPrior(),
"opt" => MeanRisk(; opt = JuMPOptimiser(; slv = slv))))
p = ["filter.col_thr" => [0.4, 0.9],
"impute" => [Imputer(; stat = MeanValue()), Imputer(; stat = MedianValue())]]
gscv = GridSearchCrossValidation(p; cv = IndexWalkForward(500, 250),
r = ConditionalValueatRisk())
tuned = search_cross_validation(pipe, gscv, pr)
# Mean test score per candidate (bigger is better after the sign convention).
scores = DataFrame(; candidate = 1:length(tuned.val_grid),
col_thr = [v[1] for v in tuned.val_grid],
imputer = [string(nameof(typeof(v[2].stat))) for v in tuned.val_grid],
mean_score = vec(mean(tuned.test_scores; dims = 1)))
pretty_table(scores)┌───────────┬─────────┬─────────────┬────────────┐
│ candidate │ col_thr │ imputer │ mean_score │
│ Int64 │ Float64 │ String │ Float64 │
├───────────┼─────────┼─────────────┼────────────┤
│ 1 │ 0.4 │ MeanValue │ -0.0218836 │
│ 2 │ 0.9 │ MeanValue │ -0.0200198 │
│ 3 │ 0.4 │ MedianValue │ -0.021892 │
│ 4 │ 0.9 │ MedianValue │ -0.0200315 │
└───────────┴─────────┴─────────────┴────────────┘tuned.opt is the winning pipeline, ready to fit on the full sample.
tuned.idx, tuned.opt.steps[1].col_thr, nameof(typeof(tuned.opt.steps[2].stat))
final = fit(tuned.opt, pr)
pretty_table(DataFrame(; asset = final.ctx.returns.nx, weight = final.w);
formatters = [resfmt])┌────────┬──────────┐
│ asset │ weight │
│ String │ Float64 │
├────────┼──────────┤
│ AAPL │ 0.014 % │
│ MSFT │ 0.002 % │
│ JNJ │ 73.075 % │
│ JPM │ 0.0 % │
│ XOM │ 4.452 % │
│ PG │ 0.002 % │
│ KO │ 5.403 % │
│ WMT │ 11.686 % │
│ PEP │ 0.001 % │
│ MRK │ 5.366 % │
└────────┴──────────┘4.1 Structural search: swapping whole estimators
Because lens values are arbitrary objects, swapping an entire step is just another grid value. Here we search over the prior estimator itself rather than one of its fields.
pipe_struct = Pipeline(;
steps = (MissingDataFilter(; col_thr = 0.4), Imputer(),
PricesToReturns(), "prior" => EmpiricalPrior(),
EqualWeighted()))
p_struct = ["prior" => [EmpiricalPrior(),
EmpiricalPrior(;
ce = PortfolioOptimisersCovariance(;
ce = GerberCovariance()))]]
tuned_struct = search_cross_validation(pipe_struct,
GridSearchCrossValidation(p_struct;
cv = IndexWalkForward(500,
250),
r = ConditionalValueatRisk()),
pr)
tuned_struct.idx, vec(mean(tuned_struct.test_scores; dims = 1))(1, [-0.02016834266414066, -0.02016834266414066])4.2 Randomised search
RandomisedSearchCrossValidation samples the grid and then delegates to the grid form, exactly as it does for plain optimisers.
rscv = RandomisedSearchCrossValidation(p; cv = IndexWalkForward(500, 250),
r = ConditionalValueatRisk(), rng = StableRNG(42),
n_iter = 2, seed = 1)
tuned_rand = search_cross_validation(pipe, rscv, pr)
size(tuned_rand.test_scores)(2, 4)5. Many paths: combinatorial and asset-resampling cross-validation
The walk-forward above produces one backtest path. CombinatorialCrossValidation and MultipleRandomised produce many — a distribution of out-of-sample outcomes rather than a single number. There is one rule: they need contiguous input rows, which recombined groups and resampled paths do not guarantee, so a pipeline that starts from prices — with a PricesToReturns or any rolling, order-dependent step — rejects them (the rolling-window rule). A returns-level pipeline has no such step, so it runs them exactly as a plain optimiser would.
So we start from returns and drop the price-level cleaning:
rd = prices_to_returns(X)
rpipe = Pipeline(;
steps = (EmpiricalPrior(),
MeanRisk(; obj = MinimumRisk(), opt = JuMPOptimiser(; slv = slv))))Pipeline
names ┼ Tuple{String, String}: ("prior", "opt")
steps ┴ Tuple{EmpiricalPrior{PortfolioOptimisersCovariance{Covariance{SimpleExpectedReturns{Nothing}, GeneralCovariance{SimpleCovariance, Nothing}, FullMoment}, MatrixProcessing{Posdef{UnionAll, @NamedTuple{}}, Nothing, Nothing, Nothing, NTuple{4, Symbol}}}, SimpleExpectedReturns{Nothing}, Nothing}, MeanRisk{JuMPOptimiser{EmpiricalPrior{PortfolioOptimisersCovariance{Covariance{SimpleExpectedReturns{Nothing}, GeneralCovariance{SimpleCovariance, Nothing}, FullMoment}, MatrixProcessing{Posdef{UnionAll, @NamedTuple{}}, Nothing, Nothing, Nothing, NTuple{4, Symbol}}}, SimpleExpectedReturns{Nothing}, Nothing}, Solver{Symbol, UnionAll, Dict{String, Bool}, @NamedTuple{allow_local::Bool, allow_almost::Bool}, Bool}, WeightBounds{Float64, Float64}, Float64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, ArithmeticReturn{Nothing, Nothing, Nothing}, SumScalariser, Nothing, Nothing, Int64, Int64, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Bool, Bool, Bool}, Variance{RiskMeasureSettings{Float64, Nothing, Bool}, Nothing, Nothing, Nothing, SquaredSOCRiskExpr}, MinimumRisk, Nothing, Nothing}}: (EmpiricalPrior
ce ┼ PortfolioOptimisersCovariance
│ ce ┼ Covariance
│ │ me ┼ SimpleExpectedReturns
│ │ │ w ┴ nothing
│ │ ce ┼ GeneralCovariance
│ │ │ ce ┼ SimpleCovariance: SimpleCovariance(true)
│ │ │ w ┴ nothing
│ │ alg ┴ FullMoment()
│ mp ┼ MatrixProcessing
│ │ pdm ┼ Posdef
│ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton
│ │ │ kwargs ┴ @NamedTuple{}: NamedTuple()
│ │ dn ┼ nothing
│ │ dt ┼ nothing
│ │ alg ┼ nothing
│ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)
me ┼ SimpleExpectedReturns
│ w ┴ nothing
horizon ┴ nothing
, MeanRisk
opt ┼ JuMPOptimiser
│ pe ┼ EmpiricalPrior
│ │ ce ┼ PortfolioOptimisersCovariance
│ │ │ ce ┼ Covariance
│ │ │ │ me ┼ SimpleExpectedReturns
│ │ │ │ │ w ┴ nothing
│ │ │ │ ce ┼ GeneralCovariance
│ │ │ │ │ ce ┼ SimpleCovariance: SimpleCovariance(true)
│ │ │ │ │ w ┴ nothing
│ │ │ │ alg ┴ FullMoment()
│ │ │ mp ┼ MatrixProcessing
│ │ │ │ pdm ┼ Posdef
│ │ │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton
│ │ │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple()
│ │ │ │ dn ┼ nothing
│ │ │ │ dt ┼ nothing
│ │ │ │ alg ┼ nothing
│ │ │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)
│ │ me ┼ SimpleExpectedReturns
│ │ │ w ┴ nothing
│ │ horizon ┴ nothing
│ slv ┼ Solver
│ │ name ┼ Symbol: :clarabel
│ │ solver ┼ UnionAll: Clarabel.MOIwrapper.Optimizer
│ │ settings ┼ Dict{String, Bool}: Dict{String, Bool}("verbose" => 0)
│ │ check_sol ┼ @NamedTuple{allow_local::Bool, allow_almost::Bool}: (allow_local = true, allow_almost = true)
│ │ add_bridges ┴ Bool: true
│ wb ┼ WeightBounds
│ │ lb ┼ Float64: 0.0
│ │ ub ┴ Float64: 1.0
│ bgt ┼ Float64: 1.0
│ sbgt ┼ nothing
│ lt ┼ nothing
│ st ┼ nothing
│ lcse ┼ nothing
│ cte ┼ nothing
│ gcarde ┼ nothing
│ sgcarde ┼ nothing
│ smtx ┼ nothing
│ sgmtx ┼ nothing
│ slt ┼ nothing
│ sst ┼ nothing
│ sglt ┼ nothing
│ sgst ┼ nothing
│ tn ┼ nothing
│ fees ┼ nothing
│ sets ┼ nothing
│ tr ┼ nothing
│ ple ┼ nothing
│ ret ┼ ArithmeticReturn
│ │ ucs ┼ nothing
│ │ lb ┼ nothing
│ │ mu ┴ nothing
│ sca ┼ SumScalariser()
│ ccnt ┼ nothing
│ cobj ┼ nothing
│ sc ┼ Int64: 1
│ so ┼ Int64: 1
│ ss ┼ nothing
│ card ┼ nothing
│ scard ┼ nothing
│ wn2 ┼ nothing
│ wnp ┼ nothing
│ wninf ┼ nothing
│ l1 ┼ nothing
│ l2 ┼ nothing
│ linf ┼ nothing
│ lp ┼ nothing
│ brt ┼ Bool: false
│ cle_pr ┼ Bool: true
│ strict ┴ Bool: false
r ┼ Variance
│ settings ┼ RiskMeasureSettings
│ │ scale ┼ Float64: 1.0
│ │ ub ┼ nothing
│ │ rke ┴ Bool: true
│ sigma ┼ nothing
│ chol ┼ nothing
│ rc ┼ nothing
│ alg ┴ SquaredSOCRiskExpr()
obj ┼ MinimumRisk()
wi ┼ nothing
fb ┴ nothing
)5.1 Combinatorial paths
Each split trains on its (possibly non-contiguous) groups and predicts the held-out ones; the per-split test groups recombine into paths. expected_risk over the population gives one realised risk per path — the spread is the robustness picture the scheme exists to produce.
comb = CombinatorialCrossValidation(; n_folds = 5, n_test_folds = 2)
pp = cross_val_predict(rpipe, rd, comb)
cvar = ConditionalValueatRisk()
pretty_table(DataFrame(; path = [p.id for p in pp.pred],
folds = [length(p.pred) for p in pp.pred],
cvar = expected_risk(cvar, pp)); formatters = [resfmt])┌───────┬─────────┬─────────┐
│ path │ folds │ cvar │
│ Int64 │ Int64 │ Float64 │
├───────┼─────────┼─────────┤
│ 1 │ 500.0 % │ 2.679 % │
│ 2 │ 500.0 % │ 2.662 % │
│ 3 │ 500.0 % │ 2.669 % │
│ 4 │ 500.0 % │ 2.678 % │
└───────┴─────────┴─────────┘5.2 Asset-resampling paths
MultipleRandomised draws a random asset subset per path and runs an inner walk-forward over it. The subset is applied to the input, so the pipeline fits fresh on each sub-universe — it never sub-selects fitted state (the restriction that used to rule this out).
mr = MultipleRandomised(IndexWalkForward(500, 250); subset_size = 6, n_subsets = 4,
rng = StableRNG(20240607))
pm = cross_val_predict(rpipe, rd, mr)
pretty_table(DataFrame(; path = [p.id for p in pm.pred],
folds = [length(p.pred) for p in pm.pred],
assets = [length(p.pred[1].res.w) for p in pm.pred]);
formatters = [resfmt])┌───────┬─────────┬─────────┐
│ path │ folds │ assets │
│ Int64 │ Int64 │ Int64 │
├───────┼─────────┼─────────┤
│ 1 │ 200.0 % │ 600.0 % │
│ 2 │ 200.0 % │ 600.0 % │
│ 3 │ 200.0 % │ 600.0 % │
│ 4 │ 200.0 % │ 600.0 % │
└───────┴─────────┴─────────┘6. Boundaries
A few things a pipeline deliberately will not do, each with an explanatory error rather than a silent wrong answer:
Combinatorial and multiple-randomised cross-validation are price-level-restricted. A price-starting pipeline rejects them by the rolling-window rule of §5; run them on a returns-level pipeline, as above.
A pipeline cannot be wrapped in a meta-optimiser (
NestedClustered,Stacking,SubsetResampling). Those build asset sub-portfolios via an asset view of their inner estimator, and a pipeline's universe is fitted state, so the view is not well defined before fitting. The reverse is supported: a meta-optimiser makes a perfectly good optimisation step of a pipeline.Predicting without weights — a pipeline with no terminal optimisation step is legal (a prior-only pipeline is useful), but it has nothing to predict with.
try
split(CombinatorialCrossValidation(), pr)
catch e
println(e.msg)
end
try
optimise(pipe, res.ctx.returns)
catch e
println(e.msg)
enda Pipeline is a workflow, not an OptimisationEstimator: fit it with fit(pipeline, data). Wrapping a Pipeline inside a meta-optimiser is not supported.7. Summary
A Pipeline turns an implicit, hand-tuned data-preparation prologue into an explicit, fitted, tunable part of the model.
Steps are ordinary estimators, routed to context slots by their family.
Preprocessing steps have a fit/apply contract: the training window learns the universe and the imputation parameters, and unseen windows replay them.
Computed slots are injected into the optimiser's configuration, so a prior is computed once and shared, and constraints are routed by result type.
Cross-validation splits the input rows, so the cleaning steps are refitted inside every fold — which is precisely the leakage the pipeline exists to remove.
Hyperparameters of the preprocessing and of the optimiser are searched in one grid, and whole estimators can be swapped as grid values.
See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale and the list of deliberately deferred features.
This page was generated using Literate.jl.