Validation

PortfolioOptimisers.cross_val_predict — Function
cross_val_predict(opt, rd::ReturnsResult, cv::CVER = KFold(); cols = :, ex = FLoops.ThreadedEx())
cross_val_predict(opt, rd::ReturnsResult, cv::CVER, cv_res::CrossValidationResult; ex = FLoops.ThreadedEx())

Run cross-validated portfolio optimisation and return predictions over all folds.

Accepts either an optimisation estimator or an optimisation result. When cols is provided, restricts the optimisation to that subset of assets. Parallel fold execution is controlled by ex.

The form with cv_res runs the folds of a split that the caller made already. A caller that runs many estimators over one scheme splits once, and every estimator then scores the same folds. That form takes no cols, because the split belongs to the data as the caller passes it. View rd by the columns before the split.

Arguments

  • opt: Optimisation estimator or result.
  • rd::ReturnsResult: Returns data used for fitting and prediction.
  • cv::CVER: Cross-validation scheme. Defaults to KFold().
  • cv_res: The split of the data by cv, a CrossValidationResult. The default splits the data argument by cv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates of cv.
  • cols: Column selector. Defaults to : (all assets).
  • ex: FLoops executor controlling parallelism. Defaults to FLoops.ThreadedEx().

Returns

  • Cross-validation prediction result.

Related

source
cross_val_predict(r::OptimiserResume, rd::ReturnsResult, cv::CVER; cols = :, ex = FLoops.ThreadedEx())
fit_and_predict(r::OptimiserResume, rd::ReturnsResult, cv::CVER, cv_res = split(cv, rd); cols = :, ex = FLoops.ThreadedEx(), id = nothing)

Continues an online walk-forward from a Result, over the full history extended.

The methods that take an optimiser in the estimator slot take a Resume too. cross_val_predict views rd by cols as the one-shot method views it, but it leaves the estimator of the Result as it is. That estimator was threaded over the view of the old run already, and its state refuses a different view at the first step. The Result that the methods return holds the new folds only.

Algorithm

  1. For cross_val_predict only, run assert_internal_optimiser and assert_external_optimiser on r.opt, and view rd by cols when cols is not :.
  2. Refuse a scheme that a Resume cannot re-enter, through assert_resume_scheme.
  3. Read train_idx and test_idx from the split of rd by cv, which fit_and_predict takes as cv_res, and refuse shuffled folds through assert_unshuffled_folds.
  4. Read the evaluation settings of cv through fold_evaluation, giving wd, pws, fa, store_weight_path and strict, and the drift of the held weights through held_weights_drift, giving hwd.
  5. Run fold_loop on r, which takes the resumed arm of online_folds. Each fold calls the one-fold fit_and_predict with its estimator, its test window, its previous weights and the settings of step 4, giving predictions and the threaded est.
  6. Return a MultiPeriodPredictionResult of predictions, with id, and with est as opt.

Arguments

  • r: The declaration, holding the Result to continue.
  • rd: The returns result to use.
  • cv: The scheme that the Result came from, an Online Scheme.
  • cv_res: The split of the data by cv, a CrossValidationResult. The default splits the data argument by cv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates of cv. cross_val_predict takes no cv_res, because it views rd by cols before the split.
  • cols: The asset view. cross_val_predict views rd by it. fit_and_predict hands it to every fold, which views its estimator and its prediction by it.
  • ex: Not read, because the resumed arm runs the folds in order.
  • id: The identifier that the Result carries.

Validation

Returns

  • res::MultiPeriodPredictionResult: The new folds, carrying the threaded estimator.

Related

source
cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombCVER, cv_res = split(cv, data); ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResult

Run combinatorial cross-validation over a Pipeline on price-level or returns-level data.

Each split fits the whole pipeline on its training rows and predicts each of its test groups. sort_predictions then puts the predictions of the test groups into the paths of the scheme, as the combinatorial loop of an optimiser does. A schedule step resolves for each split against the TimeDependentContext of the split, before fit runs.

The training rows of a split can be non-contiguous, because a test group that the split holds out can sit between them. On returns-level data, each fold fits on returns that the data holds, and the method adds no approximation. On price-level data, PricesToReturns converts the training prices of the fold, and it makes one return across each gap. That return spans the prices from the last row before the gap to the first row after it. So a training window in b blocks holds b - 1 returns that span a gap. The test groups are contiguous, so no prediction holds such a return. Use MultipleRandomised for contiguous training rows on price-level data.

cv is the scheme or its split, a CombinatorialCrossValidationResult from split(cv, data). A split result runs the folds it holds, and evaluates them with the defaults of fold_evaluation, as the direct route does. The fourth argument cv_res takes the split of data by the scheme cv, and the default splits data there. A search splits once and passes the split to the call of each candidate, so every candidate scores the same folds and evaluates them with the settings of cv.

ex is the FLoops executor of the splits. The method accepts every other keyword, such as the id of the single-path method, and ignores it.

Algorithm

  1. Check the entry with assert_pipeline_entry.
  2. Read train_idx and test_idx from cv_res, and refuse shuffled folds with assert_unshuffled_folds.
  3. Read the evaluation keywords of cv with fold_evaluation, giving wd, pws, fa, store_weight_path and strict. Read the drift of the held weights with held_weights_drift, giving hwd.
  4. Through fold_loop, fit the pipeline on the training rows of each split with pipeline_fold_fit, and predict each test group of the split. This gives one vector of predictions for each split.
  5. Put the predictions into the paths of the scheme with sort_predictions, and return the paths as a PopulationPredictionResult.

Validation

Related

source
cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::MRCVR, cv_res = split(cv, data); ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResult

Run asset-resampling cross-validation, MultipleRandomised, over a Pipeline on price-level or returns-level data.

pipe is a Pipeline, Online(pipe) or Resume(res), the three roots that Pipeline_OnlPipe names. The entry check of each root runs first, so Online(pipe) is refused here unless cv is an Online Scheme, and Resume(res) is refused by assert_resume_scheme.

Each path is an inner walk-forward over a random subset of the assets. The subset applies to the input data as a view through port_opt_view, and the pipeline fits again on the subset. So the pipeline never selects from its fitted universe. The scheme draws assets and not rows, so each window of observations stays contiguous. Unlike combinatorial cross-validation, the method adds no approximation on price-level data.

cv is the scheme or its split, a MultipleRandomisedResult from split(cv, data). A split result runs the paths and the asset subsets it holds, and evaluates them with the defaults of fold_evaluation, as the direct route does. The fourth argument cv_res takes the split of data by the scheme cv, and the default splits data there. A search splits once and passes the split to the call of each candidate, so every candidate scores the same folds and evaluates them with the settings of cv.

ex is the FLoops executor of the paths, and of the folds in each path. The method accepts every other keyword, such as the id of the single-path method, and ignores it. Each path carries its own id.

Algorithm

  1. Check the entry with assert_pipeline_entry.
  2. Read train_idx, test_idx, asset_idx and path_ids from cv_res, and refuse shuffled folds with assert_unshuffled_folds.
  3. Put the (train, test, asset) tuples of the folds into one vector for each path, giving dict.
  4. Read the evaluation keywords of cv with fold_evaluation, and the drift of the held weights with held_weights_drift, giving hwd.
  5. Run each path with pipeline_path_fit_and_predict through parallel_folds, and return the paths as a PopulationPredictionResult.

Validation

Returns

Related

source
cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::CVER = KFold(), cv_res = split(cv, data); ex = FLoops.ThreadedEx(), id = nothing)

Run cross-validated prediction over a whole Pipeline, and return a MultiPeriodPredictionResult.

pipe is a Pipeline, Online(pipe) or Resume(res), the three roots that Pipeline_OnlPipe names. They share this method, and assert_pipeline_entry holds the check of each root.

Return type by cross-validation scheme

cross_val_predict is the one entry point for every scheme, but the type of the Result depends on cv. A single-path scheme returns one series, and a multi-path scheme returns one series for each path. Code that reads the Result of one scheme, such as a KFold run, fails on the Result of the other kind. Branch on the scheme, not on the run.

cv schemeReturn type.pred holds
KFold / walk-forward (CVER)MultiPeriodPredictionResultone series, one prediction per fold
CombinatorialCrossValidationPopulationPredictionResulta per-path collection
MultipleRandomisedPopulationPredictionResulta per-path collection

A split result of either multi-path scheme, from split(cv, data), returns the same type as its scheme (CombCVER, MRCVR).

The combinatorial scheme and the asset-resampling scheme have their own methods, and both take price-level or returns-level data. See cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised). The rest of this docstring is about the method for the contiguous, single-path schemes, KFold and the walk-forwards.

The method splits the input at its own level. The split methods of price-level data give contiguous windows, so a preprocessing step with state stays inside the fold. For each fold, the method fits the whole pipeline on the training window and predicts the test window, as fit and predict do for a holdout.

When the pipeline is time-dependent, fold i makes a TimeDependentContext, and update_time_dependent_estimator swaps each schedule for its value at fold i before fit runs. The rd of the context is the input data before any step transforms it, so a callable of the pipeline sees the raw data of the fold. A schedule step can resolve to an estimator, and the fold optimises, or to a precomputed result, and the fold only predicts.

The scheme states through folds_are_time_ordered whether its folds are a timeline. A walk-forward answers true. So the folds of a pipeline that needs_previous_weights run in sequence. Each fold receives the weights of the previous fold in w_prev of the context. After the swap, factory also gives them to the optimisation steps. A KFold answers false, because its folds do not depend on each other. Its folds run in parallel with no previous weights and no factory pass, as the KFold path of an optimiser does.

Under an Online Scheme, the loop takes its online arm. The loop warms the pipeline up once, on the first training window. partial_fit! folds the new rows of each fold through the steps into the row owner. Where a refit would run, the fold calls fit(pipe) with no data, through pipeline_fold_fit. At each fold, the run gives the same weights as the batch expanding walk-forward.

Online(pipe) declares a refit of the pipeline from a buffer of its input data. The online arm resolves the wrapper at its warm-up into a pipeline that carries a PipelineBufferState. Every fold appends its new rows to the buffer, and fit(pipe) with no data is fit(pipe, buffer). So the run is exact for every configuration, at the cost of a batch fit per fold. This includes the window-valued steps that the fold route refuses. With max_history = w the buffer keeps the last w input rows, and the run equals the rolling batch walk-forward whose purged training window holds w rows. Build both schemes with the training size w + purged_size and the same purged_size, such as IndexWalkForward(w + p, t; purged_size = p) and OnlineIndexWalkForward(w + p, t; purged_size = p).

Resume(res) continues an online walk-forward over a pipeline from its Result, over the full history extended. The fold loop takes its resumed arm. The fold route and the refit route of Online(pipe) resume alike, because the entry reads the state off the pipeline through walks that take any estimator.

Algorithm

  1. Check the entry with assert_pipeline_entry.
  2. Read train_idx and test_idx from cv_res, and refuse shuffled folds with assert_unshuffled_folds.
  3. Read the evaluation keywords of cv with fold_evaluation, and the drift of the held weights with held_weights_drift, giving hwd.
  4. Run the folds through fold_loop. Each fold fits with pipeline_fold_fit and predicts its test rows. The online arm of the loop resolves Online(pipe) at its warm-up, and passes the pipeline from each fold to the next.
  5. Return the predictions as a MultiPeriodPredictionResult with id and the estimator that the online arm threaded.

Arguments

  • pipe: The pipeline, Online(pipe), or Resume(res) of a Result that holds a pipeline.
  • data: Price-level or returns-level input data (Prices_RR).
  • cv::CVER: The cross-validation scheme, with contiguous test folds and one path. The default is KFold(). folds_are_time_ordered decides whether its folds receive the weights of the previous fold, and folds_are_stepped decides whether the folds refit or fold.
  • cv_res: The split of the data by cv, a CrossValidationResult. The default splits the data argument by cv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates of cv.
  • ex: The FLoops executor of the folds. The default is FLoops.ThreadedEx().
  • id: The identifier that the Result stores.

Validation

Returns

Related

source