Train/test split

Train/test splitting

A holdout split keeps the last observations as a test window and trains on the observations before them. It has two forms. train_test_split cuts data into a training part and a test part. TrainTestSplit, alias TTS, is an estimator that does the same as the first step of a Pipeline. Every fitted step after it then sees the training window only, and fit_predict(pipe, data) scores the pipeline on the test window.

A size is a number of rows, an Integer, or a fraction of the observations, any other Real in (0, 1). If you give one size, the other window takes the remaining rows. If you give both, the rows between the two windows go to neither, which is an embargo.

Called with keywords, train_test_split returns a (train, test) tuple. Called with a TrainTestSplit, as train_test_split(tts, data), it returns the TrainTestSplitResult that the split step of a pipeline makes. You can use one TrainTestSplit both inside and outside a pipeline.

Types

PortfolioOptimisers.TrainTestSplit — Type
struct TrainTestSplit{__T_train_size, __T_test_size} <: AbstractPreprocessingEstimator

Preprocessing estimator that holds out the last observations as a test window.

It is the estimator form of train_test_split, and it adds a holdout to a Pipeline. It must be the first step of the pipeline. It passes the training window to every later step, and it keeps the test window in its fitted TrainTestSplitResult. fit_predict(pipe, data) then predicts on the test window, which no fitted step has seen.

It is the one preprocessing estimator that takes both data levels. It splits the price or returns data that the pipeline input holds, because a holdout selects rows and reads no column.

A fitted split passes new data through unchanged. Its rows index the fitting data alone, so predict(res, future_data) predicts on every new observation.

Warning

A pipeline that holds a TrainTestSplit cannot also be cross-validated. Cross-validation makes its own train and test windows, so a split inside each fold removes a second test window from every training window. cross_val_predict and search_cross_validation throw an ArgumentError for such a pipeline.

Fields

  • train_size: The training observations, as a CountOf or a FractionOf. nothing takes the complement of test_size.
  • test_size: The test observations, as a CountOf or a FractionOf. nothing takes the complement of train_size.

Constructors

TrainTestSplit(;    train_size::Option{<:Real_CountFraction} = nothing,    test_size::Option{<:Real_CountFraction} = nothing,) -> TrainTestSplit

Keywords correspond to the struct's fields. A size is a CountOf of rows or a FractionOf of the observations, and the constructor reads a plain number through count_or_fraction. One size makes the other window its complement, two sizes embargo the rows between the windows, and no size gives a 75/25 split. safe_index checks the windows when the split runs on data.

Validation

Examples

julia> pipe = Pipeline(;                       steps = (TrainTestSplit(; test_size = 0.2), PricesToReturns(),                                EmpiricalPrior(), EqualWeighted()));julia> pipe.names("split", "returns", "prior", "opt")

Related

source
PortfolioOptimisers.TrainTestSplitResult — Type
struct TrainTestSplitResult{__T_train, __T_test} <: AbstractResult

Fitted holdout split that holds the training window and the test window.

It is the result of a TrainTestSplit. The test window is the data that the fitted pipeline has not seen, and fit_predict(pipe, data) predicts on it. The result also keeps the train window, so a caller can read the training data from the result without a second split.

Both windows are port_opt_views of the input, at the data level of the input.

Fields

  • train: The training window. It is the first rows of the observations, and every later step fits on it.
  • test: The test window. It is the last rows of the observations, and no fitted step has seen it.

Related

source

Functions

PortfolioOptimisers.train_test_split — Function
train_test_split(rd::ReturnsResult; train_size, test_size) -> (train, test)
train_test_split(pr::PricesResult; train_size, test_size) -> (train, test)

Cut price or returns data into a training window of the first rows and a test window of the last rows.

This is the function form of TrainTestSplit. port_opt_view makes each window. At the returns level each window holds views of the input arrays, so the split copies no data. At the price level port_opt_view indexes a TimeArray by date, and that copies the rows. safe_index states the sizing rules.

Algorithm

  1. Read the observation count N, as size(rd.X, 1) at the returns level and as size(TimeSeries.values(pr.X), 1) at the price level.
  2. Resolve the two row ranges train and test with safe_index.
  3. Return a port_opt_view of each range. The returns-level method gives : as the asset index after the row range, because the three-argument form of port_opt_view takes the observations first and the assets second.

Arguments

  • rd, pr: The data to split.
  • train_size: The training rows, as a CountOf, a FractionOf or a plain number that count_or_fraction reads. nothing takes the complement of test_size.
  • test_size: The test rows, in the same forms. nothing takes the complement of train_size. If neither size is given, the split is 75/25.

Validation

  • The two sizes pass the checks of safe_index for N rows.

Returns

  • (train, test): The two windows. Each is a ReturnsResult for a ReturnsResult input, and a PricesResult for a PricesResult input.

Examples

julia> rd = ReturnsResult(; nx = ["A"], X = reshape(collect(0.1:0.1:1.0), 10, 1));julia> train, test = train_test_split(rd; test_size = 0.2);julia> size(train.X, 1), size(test.X, 1)(8, 2)

Related

source
train_test_split(
    tts::TrainTestSplit,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult

Split data under a TrainTestSplit, and return both windows as a TrainTestSplitResult.

This is the estimator form of the keyword method. train_test_split(rd; test_size = 0.2) returns a (train, test) tuple, and this method returns the result that the split step of a Pipeline makes. So one TrainTestSplit gives the same windows inside and outside a pipeline.

Algorithm

  1. Call the keyword method of train_test_split with tts.train_size and tts.test_size, giving train and test.
  2. Return TrainTestSplitResult(train, test).

Validation

  • The sizes of tts pass the checks of safe_index for the rows of data.

Related

source