Validation
PortfolioOptimisers.cross_val_predict — Function
cross_val_predict(opt, rd::ReturnsResult, cv::CVER = KFold(); cols = :, ex = FLoops.ThreadedEx())
cross_val_predict(opt, rd::ReturnsResult, cv::CVER, cv_res::CrossValidationResult; ex = FLoops.ThreadedEx())Run cross-validated portfolio optimisation and return predictions over all folds.
Accepts either an optimisation estimator or an optimisation result. When cols is provided, restricts the optimisation to that subset of assets. Parallel fold execution is controlled by ex.
The form with cv_res runs the folds of a split that the caller made already. A caller that runs many estimators over one scheme splits once, and every estimator then scores the same folds. That form takes no cols, because the split belongs to the data as the caller passes it. View rd by the columns before the split.
Arguments
opt: Optimisation estimator or result.rd::ReturnsResult: Returns data used for fitting and prediction.cv::CVER: Cross-validation scheme. Defaults toKFold().cv_res: The split of the data bycv, aCrossValidationResult. The default splits the data argument bycv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates ofcv.cols: Column selector. Defaults to:(all assets).ex: FLoops executor controlling parallelism. Defaults toFLoops.ThreadedEx().
Returns
- Cross-validation prediction result.
Related
cross_val_predict(r::OptimiserResume, rd::ReturnsResult, cv::CVER; cols = :, ex = FLoops.ThreadedEx())
fit_and_predict(r::OptimiserResume, rd::ReturnsResult, cv::CVER, cv_res = split(cv, rd); cols = :, ex = FLoops.ThreadedEx(), id = nothing)Continues an online walk-forward from a Result, over the full history extended.
The methods that take an optimiser in the estimator slot take a Resume too. cross_val_predict views rd by cols as the one-shot method views it, but it leaves the estimator of the Result as it is. That estimator was threaded over the view of the old run already, and its state refuses a different view at the first step. The Result that the methods return holds the new folds only.
Algorithm
- For
cross_val_predictonly, runassert_internal_optimiserandassert_external_optimiseronr.opt, and viewrdbycolswhencolsis not:. - Refuse a scheme that a
Resumecannot re-enter, throughassert_resume_scheme. - Read
train_idxandtest_idxfrom the split ofrdbycv, whichfit_and_predicttakes ascv_res, and refuse shuffled folds throughassert_unshuffled_folds. - Read the evaluation settings of
cvthroughfold_evaluation, givingwd,pws,fa,store_weight_pathandstrict, and the drift of the held weights throughheld_weights_drift, givinghwd. - Run
fold_looponr, which takes the resumed arm ofonline_folds. Each fold calls the one-foldfit_and_predictwith its estimator, its test window, its previous weights and the settings of step 4, givingpredictionsand the threadedest. - Return a
MultiPeriodPredictionResultofpredictions, withid, and withestasopt.
Arguments
r: The declaration, holding the Result to continue.rd: The returns result to use.cv: The scheme that the Result came from, an Online Scheme.cv_res: The split of the data bycv, aCrossValidationResult. The default splits the data argument bycv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates ofcv.cross_val_predicttakes nocv_res, because it viewsrdbycolsbefore the split.cols: The asset view.cross_val_predictviewsrdby it.fit_and_predicthands it to every fold, which views its estimator and its prediction by it.ex: Not read, because the resumed arm runs the folds in order.id: The identifier that the Result carries.
Validation
- Everything that
assert_resume_schemeand the resumed arm ofonline_foldsrefuse. - For
cross_val_predict, everything thatassert_internal_optimiserandassert_external_optimiserrefuse onr.opt.
Returns
res::MultiPeriodPredictionResult: The new folds, carrying the threaded estimator.
Related
cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombCVER, cv_res = split(cv, data); ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResultRun combinatorial cross-validation over a Pipeline on price-level or returns-level data.
Each split fits the whole pipeline on its training rows and predicts each of its test groups. sort_predictions then puts the predictions of the test groups into the paths of the scheme, as the combinatorial loop of an optimiser does. A schedule step resolves for each split against the TimeDependentContext of the split, before fit runs.
The training rows of a split can be non-contiguous, because a test group that the split holds out can sit between them. On returns-level data, each fold fits on returns that the data holds, and the method adds no approximation. On price-level data, PricesToReturns converts the training prices of the fold, and it makes one return across each gap. That return spans the prices from the last row before the gap to the first row after it. So a training window in b blocks holds b - 1 returns that span a gap. The test groups are contiguous, so no prediction holds such a return. Use MultipleRandomised for contiguous training rows on price-level data.
cv is the scheme or its split, a CombinatorialCrossValidationResult from split(cv, data). A split result runs the folds it holds, and evaluates them with the defaults of fold_evaluation, as the direct route does. The fourth argument cv_res takes the split of data by the scheme cv, and the default splits data there. A search splits once and passes the split to the call of each candidate, so every candidate scores the same folds and evaluates them with the settings of cv.
ex is the FLoops executor of the splits. The method accepts every other keyword, such as the id of the single-path method, and ignores it.
Algorithm
- Check the entry with
assert_pipeline_entry. - Read
train_idxandtest_idxfromcv_res, and refuse shuffled folds withassert_unshuffled_folds. - Read the evaluation keywords of
cvwithfold_evaluation, givingwd,pws,fa,store_weight_pathandstrict. Read the drift of the held weights withheld_weights_drift, givinghwd. - Through
fold_loop, fit the pipeline on the training rows of each split withpipeline_fold_fit, and predict each test group of the split. This gives one vector of predictions for each split. - Put the predictions into the paths of the scheme with
sort_predictions, and return the paths as aPopulationPredictionResult.
Validation
- Everything
assert_pipeline_entry,assert_unshuffled_foldsandfold_looprefuse.
Related
cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::MRCVR, cv_res = split(cv, data); ex = FLoops.ThreadedEx(), kwargs...) -> PopulationPredictionResultRun asset-resampling cross-validation, MultipleRandomised, over a Pipeline on price-level or returns-level data.
pipe is a Pipeline, Online(pipe) or Resume(res), the three roots that Pipeline_OnlPipe names. The entry check of each root runs first, so Online(pipe) is refused here unless cv is an Online Scheme, and Resume(res) is refused by assert_resume_scheme.
Each path is an inner walk-forward over a random subset of the assets. The subset applies to the input data as a view through port_opt_view, and the pipeline fits again on the subset. So the pipeline never selects from its fitted universe. The scheme draws assets and not rows, so each window of observations stays contiguous. Unlike combinatorial cross-validation, the method adds no approximation on price-level data.
cv is the scheme or its split, a MultipleRandomisedResult from split(cv, data). A split result runs the paths and the asset subsets it holds, and evaluates them with the defaults of fold_evaluation, as the direct route does. The fourth argument cv_res takes the split of data by the scheme cv, and the default splits data there. A search splits once and passes the split to the call of each candidate, so every candidate scores the same folds and evaluates them with the settings of cv.
ex is the FLoops executor of the paths, and of the folds in each path. The method accepts every other keyword, such as the id of the single-path method, and ignores it. Each path carries its own id.
Algorithm
- Check the entry with
assert_pipeline_entry. - Read
train_idx,test_idx,asset_idxandpath_idsfromcv_res, and refuse shuffled folds withassert_unshuffled_folds. - Put the
(train, test, asset)tuples of the folds into one vector for each path, givingdict. - Read the evaluation keywords of
cvwithfold_evaluation, and the drift of the held weights withheld_weights_drift, givinghwd. - Run each path with
pipeline_path_fit_and_predictthroughparallel_folds, and return the paths as aPopulationPredictionResult.
Validation
- Everything
assert_pipeline_entry,assert_unshuffled_foldsandfold_looprefuse.
Returns
PopulationPredictionResult: OneMultiPeriodPredictionResultfor each path.
Related
cross_val_predict(pipe::Pipeline_OnlPipe, data::Prices_RR, cv::CVER = KFold(), cv_res = split(cv, data); ex = FLoops.ThreadedEx(), id = nothing)Run cross-validated prediction over a whole Pipeline, and return a MultiPeriodPredictionResult.
pipe is a Pipeline, Online(pipe) or Resume(res), the three roots that Pipeline_OnlPipe names. They share this method, and assert_pipeline_entry holds the check of each root.
Return type by cross-validation scheme
cross_val_predict is the one entry point for every scheme, but the type of the Result depends on cv. A single-path scheme returns one series, and a multi-path scheme returns one series for each path. Code that reads the Result of one scheme, such as a KFold run, fails on the Result of the other kind. Branch on the scheme, not on the run.
cv scheme | Return type | .pred holds |
|---|---|---|
KFold / walk-forward (CVER) | MultiPeriodPredictionResult | one series, one prediction per fold |
CombinatorialCrossValidation | PopulationPredictionResult | a per-path collection |
MultipleRandomised | PopulationPredictionResult | a per-path collection |
A split result of either multi-path scheme, from split(cv, data), returns the same type as its scheme (CombCVER, MRCVR).
The combinatorial scheme and the asset-resampling scheme have their own methods, and both take price-level or returns-level data. See cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation) and cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised). The rest of this docstring is about the method for the contiguous, single-path schemes, KFold and the walk-forwards.
The method splits the input at its own level. The split methods of price-level data give contiguous windows, so a preprocessing step with state stays inside the fold. For each fold, the method fits the whole pipeline on the training window and predicts the test window, as fit and predict do for a holdout.
When the pipeline is time-dependent, fold i makes a TimeDependentContext, and update_time_dependent_estimator swaps each schedule for its value at fold i before fit runs. The rd of the context is the input data before any step transforms it, so a callable of the pipeline sees the raw data of the fold. A schedule step can resolve to an estimator, and the fold optimises, or to a precomputed result, and the fold only predicts.
The scheme states through folds_are_time_ordered whether its folds are a timeline. A walk-forward answers true. So the folds of a pipeline that needs_previous_weights run in sequence. Each fold receives the weights of the previous fold in w_prev of the context. After the swap, factory also gives them to the optimisation steps. A KFold answers false, because its folds do not depend on each other. Its folds run in parallel with no previous weights and no factory pass, as the KFold path of an optimiser does.
Under an Online Scheme, the loop takes its online arm. The loop warms the pipeline up once, on the first training window. partial_fit! folds the new rows of each fold through the steps into the row owner. Where a refit would run, the fold calls fit(pipe) with no data, through pipeline_fold_fit. At each fold, the run gives the same weights as the batch expanding walk-forward.
Online(pipe) declares a refit of the pipeline from a buffer of its input data. The online arm resolves the wrapper at its warm-up into a pipeline that carries a PipelineBufferState. Every fold appends its new rows to the buffer, and fit(pipe) with no data is fit(pipe, buffer). So the run is exact for every configuration, at the cost of a batch fit per fold. This includes the window-valued steps that the fold route refuses. With max_history = w the buffer keeps the last w input rows, and the run equals the rolling batch walk-forward whose purged training window holds w rows. Build both schemes with the training size w + purged_size and the same purged_size, such as IndexWalkForward(w + p, t; purged_size = p) and OnlineIndexWalkForward(w + p, t; purged_size = p).
Resume(res) continues an online walk-forward over a pipeline from its Result, over the full history extended. The fold loop takes its resumed arm. The fold route and the refit route of Online(pipe) resume alike, because the entry reads the state off the pipeline through walks that take any estimator.
Algorithm
- Check the entry with
assert_pipeline_entry. - Read
train_idxandtest_idxfromcv_res, and refuse shuffled folds withassert_unshuffled_folds. - Read the evaluation keywords of
cvwithfold_evaluation, and the drift of the held weights withheld_weights_drift, givinghwd. - Run the folds through
fold_loop. Each fold fits withpipeline_fold_fitand predicts its test rows. The online arm of the loop resolvesOnline(pipe)at its warm-up, and passes the pipeline from each fold to the next. - Return the predictions as a
MultiPeriodPredictionResultwithidand the estimator that the online arm threaded.
Arguments
pipe: The pipeline,Online(pipe), orResume(res)of a Result that holds a pipeline.data: Price-level or returns-level input data (Prices_RR).cv::CVER: The cross-validation scheme, with contiguous test folds and one path. The default isKFold().folds_are_time_ordereddecides whether its folds receive the weights of the previous fold, andfolds_are_steppeddecides whether the folds refit or fold.cv_res: The split of the data bycv, aCrossValidationResult. The default splits the data argument bycv. A caller that runs many estimators over one scheme splits once and passes the split to each call. Then every estimator scores the same folds, and a scheme that draws at random draws once. The method still reads the evaluation settings and the fold predicates ofcv.ex: The FLoops executor of the folds. The default isFLoops.ThreadedEx().id: The identifier that the Result stores.
Validation
- The pipeline holds no holdout step, an
Online(pipe)runs only under an Online Scheme, and aResume(res)runs only under a scheme thatassert_resume_schemeaccepts (assert_pipeline_entry). - The folds of
cvare not shuffled (assert_unshuffled_folds). - The number of entries of each schedule is the number of folds, and the batch arms hold no
Online(fold_loop). - Everything
assert_online_entryrefuses on the refit route ofOnline(pipe), and everything the resumed arm of the fold loop refuses (Resume).
Returns
MultiPeriodPredictionResult: One prediction for each fold, in the order of the folds.
Related
PipelinefitTimeDependentfolds_are_time_orderedsearch_cross_validationMultiPeriodPredictionResultPopulationPredictionResultcross_val_predict(pipe::Pipeline, data::Prices_RR, cv::CombinatorialCrossValidation)cross_val_predict(pipe::Pipeline, data::Prices_RR, cv::MultipleRandomised)OnlineResumePipelineBufferState