The source files can be found in examples/.

Train/test splitting

A holdout split trains on the first part of the history and scores on the rest. Most cells of this page hold out the last 20 %. Cross-validation scores many folds instead, at the cost of one fit per fold.

Every step before the optimiser must also respect the split. As the pipelines example shows, a fill value or an asset filter fitted on the whole history already uses the prices of the test rows. If you split a returns matrix built that way, the test rows have already entered the training data.

TrainTestSplit, with the alias TTS, makes the split a step of a pipeline. It is the first step and runs before any other step sees the data. Every fitted step after it sees the training window and nothing else.

This example covers these subjects.

• The function train_test_split, its sizes and the embargo.
• The pipeline step, and why it must come first.
• Scoring on the held-out window with fit_predict.
• Why cross-validation throws an error for a pipeline with a split.

docs/adr/0031-holdout-split-as-a-pipeline-step.md gives the reasons for the design.

using PortfolioOptimisers, PrettyTablesresfmt = (v, i, j) -> begin    if j == 1        return v    else        return isa(v, Number) ? "$(round(v*100, digits=3)) %" : v    endend;

1. Setting up

We use four years of daily prices for ten S&P 500 stocks. As in the pipelines example, we start from prices, not returns, because the split must happen before the pipeline cleans and converts the data.

using CSV, TimeSeries, DataFrames, Clarabel, Statistics, DatesX = TimeArray(CSV.File(joinpath(@__DIR__, "..", "SP500.csv.gz")); timestamp = :Date)[(end - 252 * 4):end]X = X[Symbol.(["AAPL", "MSFT", "JNJ", "JPM", "XOM", "PG", "KO", "WMT", "PEP", "MRK"])...]pr = PricesResult(; X = X)slv = Solver(; name = :clarabel, solver = Clarabel.Optimizer,             settings = Dict("verbose" => false),             check_sol = (; allow_local = true, allow_almost = true));size(values(X))

(1009, 10)

2. The free function

train_test_split cuts price or returns data into two windows. The training window holds the first rows and the test window the last rows. The split keeps the time order, and the test window is always the most recent data. A random shuffle would let the model train on later rows and test on earlier ones.

train, test = train_test_split(pr; test_size = 0.2)DataFrame(; window = ["train", "test"],          rows = [size(values(train.X), 1), size(values(test.X), 1)],          from = [first(timestamp(train.X)), first(timestamp(test.X))],          to = [last(timestamp(train.X)), last(timestamp(test.X))])

2.1 Sizes as counts or fractions

A size is an Integer count of rows, or an AbstractFloat fraction of the rows, strictly between 0 and 1. test_size = 250 means the last 250 rows, and test_size = 0.25 means the last quarter. A fraction rounds down to a whole number of rows.

If you give one size, the other window takes the remaining rows. If you give neither, the training window takes 75 % of the rows. The cell prints the window sizes of four calls. A fraction rounds down on the side you give, so test_size = 0.2 and train_size = 0.8 differ by one row.

N = size(values(X), 1)spellings = ["train_test_split(pr)" => (nothing, nothing),             "train_test_split(pr; test_size = 0.2)" => (nothing, 0.2),             "train_test_split(pr; train_size = 0.8)" => (0.8, nothing),             "train_test_split(pr; test_size = 202)" => (nothing, 202)]sizes = DataFrame(; spelling = String[], train = Int[], test = Int[])for (label, (tr_s, te_s)) in spellings    a, b = train_test_split(pr; train_size = tr_s, test_size = te_s)    push!(sizes, (label, size(values(a.X), 1), size(values(b.X), 1)))endsizes

2.2 The embargo

If you give both sizes, the two windows need not cover all the rows. The first rows go to the training window, the last rows go to the test window, and the rows between them belong to neither. Those rows are the embargo.

The embargo keeps the last training rows away from the first test rows. Many inputs to a model are autocorrelated. A 20-day average return computed on a window that crosses the boundary puts test information into the last training rows, and an embargo of 20 rows removes it. KFold applies the same idea to many folds with its purged_size and embargo_size fields.

train_test_split throws an error when the two windows overlap, and when a window is empty. The cell leaves an embargo between a training window of 60 % and a test window of 20 %. Then it asks for two windows that overlap and prints the error.

tr_e, te_e = train_test_split(pr; train_size = 0.6, test_size = 0.2)DataFrame(; window = ["train", "embargoed", "test"],          rows = [size(values(tr_e.X), 1),                  N - size(values(tr_e.X), 1) - size(values(te_e.X), 1),                  size(values(te_e.X), 1)])try    train_test_split(pr; train_size = 0.9, test_size = 0.2)catch e    println(e.msg)end

the training and test windows of a train/test split must not overlap, but 908 training and 201 test observations exceed the 1009 available; rows between the two windows are embargoed, so their sizes may sum to less than 1009 but never to more

3. The mistake that a manual split allows

Nothing stops you from writing this:

#! This fits the pipeline on every row of pr, the test window included.res  = fit(pipe, pr)pred = predict(res, pr, test_window)

The code runs and gives a number, but the number is not out of sample. The missing-data filter of the pipeline used the test rows to choose its assets, and the price fill computed its values from them. The call looks the same as a correct one. Only the rows that the fit saw differ, and nothing warns you.

To prevent this, make the split part of the workflow, so that the fit cannot see the test rows.

4. The split as the first pipeline step

TrainTestSplit cuts the data that you give the pipeline, here the prices, and passes the training window to every later step. Its fitted result also keeps the held-out window, which fit_predict uses in section 5.

The step takes the name "split" when you give it none. The filter and the gap fill both change the prices, so they take the names "prices_1" and "prices_2".

pipe = Pipeline(;                steps = (TrainTestSplit(; test_size = 0.2), MissingDataFilter(),                         PriceGapFill(), PricesToReturns(), EmpiricalPrior(),                         MeanRisk(; r = Variance(),                                  opt = JuMPOptimiser(; slv = slv, pe = EmpiricalPrior()))))pipe.names

("split", "prices_1", "prices_2", "returns", "prior", "opt")

We fit the pipeline and count the rows at each stage.

res = fit(pipe, pr)split_res = res["split"]DataFrame(;          quantity = ["prices in the sample", "prices in the training window",                      "prices held out", "returns the optimiser gets"],          rows = [N, size(values(split_res.train.X), 1), size(values(split_res.test.X), 1),                  size(res.ctx.returns.X, 1)])

4.1 The split must be the first step

A TrainTestSplit must be the first step, so that every other step is fitted on the training window alone. A MissingDataFilter fitted before the split would use the held-out rows to choose the assets. A PriceGapFill fitted before it would compute its fill values from them. Their fitted state would then depend on the test rows.

The Pipeline constructor therefore throws an error when a split is not the first step. It does the same for a split inside a nested pipeline, even when the nested pipeline is the first step.

try    Pipeline(;             steps = (MissingDataFilter(), PriceGapFill(),                      TrainTestSplit(; test_size = 0.2), PricesToReturns(),                      EqualWeighted()))catch e    println(e.msg)endtry    inner = Pipeline(; steps = (TrainTestSplit(; test_size = 0.2), PricesToReturns()))    Pipeline(; steps = (inner, EqualWeighted()))catch e    println(e.msg)end

a TrainTestSplit step must be the first step of a Pipeline, but one appears at step 3; a stateful step fitted before the split would have seen the held-out test rows, leaking them into the fitted workflow
a TrainTestSplit step is nested inside a Pipeline step of a Pipeline; the holdout must be the first step of the outermost Pipeline, where no step has yet touched the data

5. fit_predict scores on the held-out window

For a pipeline with a split, fit_predict fits on the training rows and predicts on the rows that the split held out.

For a pipeline without a split, fit_predict predicts on the same data that it fits, so the result is in sample. For comparison, we also predict on the training window, in sample.

pred_test = fit_predict(pipe, pr)pred_train = predict(res, split_res.train)

PredictionResult
  res ┼ MeanRiskResult
      │   jr ┼ JuMPOptimisationResult
      │      │        pa ┼ ProcessedJuMPOptimiserAttributes
      │      │           │        pr ┼ LowOrderPrior
      │      │           │           │         X ┼ 807×10 Matrix{Float64}
      │      │           │           │       o_X ┼ nothing
      │      │           │           │        mu ┼ 10-element Vector{Float64}
      │      │           │           │     sigma ┼ 10×10 Matrix{Float64}
      │      │           │           │      chol ┼ nothing
      │      │           │           │         w ┼ nothing
      │      │           │           │       ens ┼ nothing
      │      │           │           │       kld ┼ nothing
      │      │           │           │        ow ┼ nothing
      │      │           │           │        rr ┼ nothing
      │      │           │           │       fpr ┼ nothing
      │      │           │           │     x_src ┼ nothing
      │      │           │           │   ss_gaps ┼ nothing
      │      │           │           │        pc ┴ nothing
      │      │           │        wb ┼ WeightBounds
      │      │           │           │   lb ┼ 10-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
      │      │           │           │   ub ┴ 10-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
      │      │           │        lt ┼ nothing
      │      │           │        st ┼ nothing
      │      │           │      lcsr ┼ nothing
      │      │           │       ctr ┼ nothing
      │      │           │    gcardr ┼ nothing
      │      │           │   sgcardr ┼ nothing
      │      │           │      smtx ┼ nothing
      │      │           │     sgmtx ┼ nothing
      │      │           │       slt ┼ nothing
      │      │           │       sst ┼ nothing
      │      │           │      sglt ┼ nothing
      │      │           │      sgst ┼ nothing
      │      │           │        tn ┼ nothing
      │      │           │      fees ┼ nothing
      │      │           │       plr ┼ nothing
      │      │           │       ret ┼ ArithmeticReturn
      │      │           │           │   settings ┼ JuMPReturnsSettings
      │      │           │           │            │   scale ┼ Float64: 1.0
      │      │           │           │            │      lb ┼ nothing
      │      │           │           │            │     rte ┼ Bool: true
      │      │           │           │            │     fee ┼ Bool: true
      │      │           │           │            │     mic ┴ Bool: true
      │      │           │           │        ucs ┼ nothing
      │      │           │           │         mu ┼ 10-element Vector{Float64}
      │      │           │           │   mtx_sqrt ┴ EigenFallbackSquareRoot()
      │      │           │       sca ┼ SumScalariser()
      │      │           │      imsk ┴ nothing
      │      │   retcode ┼ OptimisationSuccess
      │      │           │   res ┴ Dict{Any, Any}: Dict{Any, Any}()
      │      │       sol ┼ JuMPOptimisationSolution
      │      │           │   w ┴ 10-element Vector{Float64}
      │      │     model ┼ A JuMP Model
      │      │           │ ├ solver: Clarabel
      │      │           │ ├ objective_sense: MIN_SENSE
      │      │           │ │ └ objective_function_type: QuadExpr
      │      │           │ ├ num_variables: 11
      │      │           │ ├ num_constraints: 4
      │      │           │ │ ├ AffExpr in MOI.EqualTo{Float64}: 1
      │      │           │ │ ├ Vector{AffExpr} in MOI.Nonnegatives: 1
      │      │           │ │ ├ Vector{AffExpr} in MOI.Nonpositives: 1
      │      │           │ │ └ Vector{AffExpr} in MOI.SecondOrderCone: 1
      │      │           │ └ Names registered in the model
      │      │           │   └ :G, :T, :bgt, :cdev_soc_1, :dev_1, :k, :lw, :obj_expr, :ret, :ret_1, :ret_vec, :risk, :risk_minimised, :risk_vec, :sc, :so, :variance_flag, :variance_risk_1, :w, :w_gross_ub, :w_lb, :w_ub
      │    r ┼ Variance
      │      │    settings ┼ RiskMeasureSettings
      │      │             │   scale ┼ Float64: 1.0
      │      │             │      ub ┼ nothing
      │      │             │     rke ┴ Bool: true
      │      │       sigma ┼ 10×10 Matrix{Float64}
      │      │        chol ┼ nothing
      │      │          rc ┼ nothing
      │      │         alg ┼ SquaredSOCRiskExpr()
      │      │    mtx_sqrt ┼ EigenFallbackSquareRoot()
      │      │   ratio_alg ┼ SquareRootRatio()
      │      │      corner ┴ nothing
      │   fb ┴ nothing
   rd ┼ PredictionReturnsResult
      │     nx ┼ 10-element Vector{String}
      │      X ┼ 807-element Vector{Float64}
      │     nf ┼ nothing
      │      F ┼ nothing
      │     nb ┼ nothing
      │      B ┼ nothing
      │     ne ┼ nothing
      │      E ┼ nothing
      │     ts ┼ 807-element Vector{Date}
      │     iv ┼ nothing
      │   ivpa ┴ nothing
   hw ┼ nothing
  idx ┴ nothing

We score both predictions on the return series of the portfolio. expected_risk on a prediction sees only that series. Variance and StandardDeviation need the weights, so they throw an error here, as the cross-validation example explains. SCM() is the second central moment of the return series, which is its variance.

DataFrame(; window = ["train (in-sample)", "test (held out)"],          observations = [length(pred_train.rd.X), length(pred_test.rd.X)],          variance = [expected_risk(SCM(), pred_train), expected_risk(SCM(), pred_test)],          cvar = [expected_risk(ConditionalValueatRisk(), pred_train),                  expected_risk(ConditionalValueatRisk(), pred_test)])

The difference between the two rows is what a holdout measures. The in-sample risk comes from the rows on which the optimiser minimised the variance. The held-out risk comes from rows that the fit never saw. Here both held-out values are lower than the in-sample values, so on this window the in-sample risk did not understate the held-out risk. A holdout gives you one such comparison, and a different window can give the opposite result.

The next table gives the weights. They cover the assets of the training window, and the test prediction uses the same assets.

DataFrame(; asset = res.ctx.returns.nx, weight = res.w)

6. A fitted split returns new data unchanged

The rows of a fitted holdout belong to the window that it was fitted on. They mean nothing on another window, so apply_preprocessing on a TrainTestSplitResult returns the data unchanged.

You can therefore still fit on the history and predict on new rows. Every other fitted step still applies its state, such as the training assets, the training fill values and the conversion to returns. Only the split does nothing. We treat the last 41 prices as data that arrived after the fit.

future = PricesResult(; X = X[(end - 40):end])pred_future = predict(res, future)DataFrame(; source = ["held-out window (fit_predict)", "fresh data (predict)"],          observations = [length(pred_test.rd.X), length(pred_future.rd.X)],          variance = [expected_risk(SCM(), pred_test), expected_risk(SCM(), pred_future)])

7. One evaluation protocol per call

A holdout and cross-validation both choose the rows to test on, and you cannot use both at once. Cross-validation already defines the training window and the test window of every fold. A split left in the pipeline would cut a second holdout off the training data of every fold, and it would keep a test window that nothing uses.

The pipeline would lose training data with no warning. search_cross_validation therefore throws an error for a pipeline with a split.

gscv = GridSearchCrossValidation(Dict("returns" =>                                          [PricesToReturns(; ret_method = :simple),                                           PricesToReturns(; ret_method = :log)]);                                 cv = KFold(; n = 3), r = SCM())try    search_cross_validation(pipe, gscv, pr)catch e    println(e.msg)end

this Pipeline contains a TrainTestSplit step, so it cannot also be cross-validated: cross-validation already defines the train and test window of every fold, and the split would shave a second holdout off each fold's training data. Remove the TrainTestSplit step, or evaluate the pipeline with fit_predict instead of cross-validating it.

Without the split, the same pipeline tunes with no error, and cross-validation chooses the test rows.

pipe_cv = Pipeline(;                   steps = (MissingDataFilter(), PriceGapFill(), PricesToReturns(),                            EmpiricalPrior(),                            MeanRisk(; r = Variance(),                                     opt = JuMPOptimiser(; slv = slv,                                                         pe = EmpiricalPrior()))))scv_res = search_cross_validation(pipe_cv, gscv, pr)scv_res.idx

2

8. Choosing between them

Tune with cross-validation, and do not look at the held-out score until the model is fixed. If you tune against the held-out score, the held-out rows become part of the fit, and the score is in sample again.

Summary

• train_test_split cuts data into its first rows for training and its last rows for testing. If you give one size, the other window takes the rest. If you give both, the rows between the two windows are an embargo.
• TrainTestSplit is the first step of a Pipeline, so the asset filter, the gap fill, the prior and the optimiser after it see only the training window. The Pipeline constructor throws an error for a split in any other position.
• fit_predict on a pipeline with a split scores on the held-out window. The fitted result of the split keeps both windows, res["split"].train and res["split"].test.
• A fitted split returns new data unchanged, and a prediction on new rows still works.
• Cross-validation throws an error for a pipeline with a split. Use one of the two per call.

This page was generated using Literate.jl.