Search cross-validation
A search over the hyperparameters of a pipeline covers the whole workflow. It searches the preprocessing settings, such as the imputation statistic or the threshold for missing data, together with the settings of the prior, the constraints and the optimiser. Each candidate is fitted on each fold, so a fitted preprocessing step never sees the test window. A key names a setting by the name of its step, "impute.stat", by position, where an integer key replaces the whole step, or by property path, "steps[2].stat".
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(pipe::Pipeline, gscv::GridSearchCrossValidation, data::Prices_RR)
search_cross_validation(pipe::Pipeline, rscv::RandomisedSearchCrossValidation,
data::Prices_RR)Score every candidate Pipeline of the grid on every fold of the scheme, and return the winner.
Each candidate runs through cross_val_predict, the fold loop of every cross-validation entry point. Every fold fits the whole workflow on its training window and predicts its test window, so a preprocessing step learns from the training window alone. A walk-forward threads the weights of the previous fold through the pws of the scheme. A TimeDependent schedule resolves per fold, and a key that names its step replaces the whole schedule.
Under an Online Scheme, every candidate warms up once on the first training window, and partial_fit! folds each later block of rows into it. The search then gives the score matrix of the batch search over the expanding walk-forward with the same folds, to the tolerance of the estimators and of the solver. The two searches select the same candidate. The randomised form samples a grid from rscv.p and runs the grid search on it.
Mathematical definition
\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]
Where:
- $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
- $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
- $F$: Fold count, the count of the folds that
splitenumerates. - $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
- $\Theta$: Search grid, the sequence of grid points that
lens_val_gridbuilds fromp. For aPipeline,pipeline_lens_val_gridbuilds it. - $\mathcal{O}$: Oriented score under
r, fromoriented_scorewith the reductionr_scaand the keyword argumentskwargs. Each measure is oriented so that a higher score is always better, andr_scareduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better. - $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores
NaN, so a candidate that failed one is not in $\mathcal{C}$. - $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
- $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
- $\sigma$: Search scorer
scorer, a map from a score matrix to the position of its best column. The defaultHighestMeanScoretakes the column of greatest mean. - $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.
The scorer never reads the column of a candidate that failed a fold, so the search never selects that candidate.
Algorithm
For gscv:
- Refuse a pipeline that carries a partial-fit state, through
assert_search_entry. - Build
lens_gridandval_gridthroughpipeline_lens_val_grid. - Refuse a candidate that holds a
TrainTestSplit, throughassert_search_candidates. - Split
databygscv.cvonce, givingcv. Every candidate scores the folds ofcv, so a scheme whosesplitdraws at random draws once. - Read the rows of the folds through
score_rows, givingrows. - Allocate
test_scores, of sizeM × NforMfolds andNcandidates, with the element type fromcv_data_eltype, throughallocate_search_scores. Whengscv.train_scoreistrue, it allocatestrain_scoresof the same size. - For each candidate
i, in parallel overgscv.ex, do steps 8 to 10. - Build
pipeithroughsearch_candidate. - Run
pipeiover the folds ofcvthroughcross_val_predict, with the folds in sequence, givingpredictions. - Write $\mathcal{O}$ of each fold into column
ioftest_scoresandtrain_scoresthroughwrite_candidate_scores!, with no view of the training returns. Storepredictionsat entryiof the store ofallocate_search_predictionsthroughstore_search_prediction!, which stores nothing whengscv.rank_predictionisfalse. - Order the finite candidates, keep the best
gscv.n_best, build each frompipe, and return theSearchCrossValidationResultthroughsearch_result, with the predictions of the kept candidates. The path of each row isscore_path_idsofcv.
For rscv:
- Resolve the random number generator from
rscv.rngandrscv.seedthroughresolve_rng. - Draw at most
rscv.n_itertrials throughmake_p_grid, each a joint value of every key of one parameter set. - Build a
GridSearchCrossValidationwith that grid and the other fields ofrscv, and run the search of that grid. Under aCombinatorialCrossValidationthis is the combinatorial method.
Arguments
pipe: The pipeline to tune. The search reads it as configuration alone.gscv: The grid search. It gives the gridp, the schemecv, the risk measuresrand their reductionr_sca, thescorer, the executorex, thetrain_scoreflag and the keyword argumentskwargsoforiented_score.rscv: The randomised search. It gives the fields ofgscv, the distributions or vectorspto sample, the sample countn_iter, andrngandseed.data: The price or returns data, aPrices_RR, at the input level ofpipe.
Validation
pipecarries no partial-fit state, throughassert_search_entry. Under an Online Scheme,pipealso passes every check ofassert_online_entry.- No candidate holds a
TrainTestSplit, throughassert_search_candidates. - The grid is not empty, it holds at most
RESOURCE_LIMITS[].max_search_gridcandidates, and every key names a step or a path rooted atsteps, throughpipeline_lens_val_grid. - A
TimeDependentschedule of a candidate holds one entry per fold. The fold loop checks each candidate as it runs it. - At least one candidate finishes every fold. Otherwise
finite_candidate_orderthrows anIsNonFiniteError.
Returns
res::SearchCrossValidationResult: The fieldtest_scoresis $\mathbf{S}$, the raw matrix, withNaNat a failed fold. Row $f$ is fold $f$ in the order ofsplit, and under aMultipleRandomised,score_rowsputs the scores of each path back onto the rows of the split. The fieldtrain_scoresisnothingwhengscv.train_scoreisfalse. Otherwise it holds $\mathcal{O}$ of each fitted result through its prior result, in a matrix of the same size. The fieldoptis candidate $i^{\star}$, andidxis $i^{\star}$.
Related
GridSearchCrossValidationRandomisedSearchCrossValidationpipeline_lenscross_val_predictwrite_candidate_scores!candidate_train_score: the train score reads the prior result, because the steps of a Pipeline change the data of each fold.score_rowsfinite_candidate_orderassert_search_entryPrices_RR
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(pipe::Pipeline,
gscv::GridSearchCrossValidation{<:Any, <:CombinatorialCrossValidation},
data::Prices_RR)Tune a Pipeline over the grid of gscv under a CombinatorialCrossValidation, with one score per candidate and backtest path.
A combinatorial scheme recombines its disjoint test groups into full backtest paths, and a split scored alone mixes the groups of different paths. So this method scores each path. For each candidate, cross_val_predict runs the scheme, sort_predictions recombines the groups into a PopulationPredictionResult, and oriented_score gives one score per path. The candidates run in sequence, and the fold loop of each candidate runs with the executor gscv.ex.
The method takes price data too. The training rows of a split then have gaps where the test groups are, so PricesToReturns makes one return across each gap. The test groups are contiguous, so the predictions have no such return. A MultipleRandomised scheme keeps the training rows contiguous at the price level.
Mathematical definition
\[\begin{align} S_{pi} &= \mathcal{O}\left(\hat{P}_{p}(\theta_i)\right)\,, \quad p = 1, \ldots, n_{p}\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{pi} \text{ is finite for every } p \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]
Where:
- $S_{pi}$: Path score, the test score of candidate $i$ on path $p$, and the entry of the score matrix $\mathbf{S}$, $n_{p} \times |\Theta|$.
- $\hat{P}_{p}(\theta_i)$: Path prediction, the predictions of candidate $i$ for the test groups that path $p$ holds, pooled into one series. The weights that $\theta_i$ fits on the training window of the fold that tests a group predict that group.
- $n_{p}$: Path count, the greatest entry of
path_idsof the split. - $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
- $\Theta$: Search grid, the sequence of grid points that
lens_val_gridbuilds fromp. For aPipeline,pipeline_lens_val_gridbuilds it. - $\mathcal{O}$: Oriented score under
r, fromoriented_scorewith the reductionr_scaand the keyword argumentskwargs. Each measure is oriented so that a higher score is always better, andr_scareduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better. - $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores
NaN, so a candidate that failed one is not in $\mathcal{C}$. - $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
- $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
- $\sigma$: Search scorer
scorer, a map from a score matrix to the position of its best column. The defaultHighestMeanScoretakes the column of greatest mean. - $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.
The scorer never reads the column of a candidate that failed a path, so the search never selects that candidate.
Algorithm
- Refuse a pipeline that carries a partial-fit state, through
assert_search_entry. - Build
lens_gridandval_gridthroughpipeline_lens_val_grid. - Refuse a candidate that holds a
TrainTestSplit, throughassert_search_candidates. - Split
databygscv.cv, givingcv. - Set
M, the count of the paths, to the greatest entry ofcv.path_ids. - Allocate
test_scores, of sizeM × NforNcandidates, with the element type fromcv_data_eltype, throughallocate_search_scores. Whengscv.train_scoreistrue, it allocatestrain_scores, one matrix per path, with one row per fold of the path and one column per candidate. - For each candidate
i, in sequence, do steps 8 to 10. - Build
pipeithroughsearch_candidate. - Run
pipeiover the folds ofcvthroughcross_val_predict, with the executorgscv.ex, givingpredictions, one path per member. - Write column
ithroughwrite_candidate_scores!, with rowpfor pathp. The test score of each path is $\mathcal{O}$ of the path. Whengscv.train_scoreistrue, the train score of each fold of each path goes into columniof the matrix of that path, throughcandidate_train_scorewith no view of the training returns. Storepredictionsat entryiof the store ofallocate_search_predictionsthroughstore_search_prediction!, which stores nothing whengscv.rank_predictionisfalse. - Order the finite candidates, keep the best
gscv.n_best, build each frompipe, and return theSearchCrossValidationResultthroughsearch_result, with the predictions of the kept candidates. Rowpis pathp, so the path of each row is1:M.
Arguments
pipe: The pipeline to tune. The search reads it as configuration alone.gscv: The grid search over aCombinatorialCrossValidation. It gives the gridp, the schemecv, the risk measuresrand their reductionr_sca, thescorer, the executorex, thetrain_scoreflag and the keyword argumentskwargsoforiented_score.data: The price or returns data, aPrices_RR, at the input level ofpipe.
Validation
pipecarries no partial-fit state, throughassert_search_entry.- No candidate holds a
TrainTestSplit, throughassert_search_candidates. - The grid is not empty, it holds at most
RESOURCE_LIMITS[].max_search_gridcandidates, and every key names a step or a path rooted atsteps, throughpipeline_lens_val_grid. - At least one candidate finishes every path. Otherwise
finite_candidate_orderthrows anIsNonFiniteError.
Returns
res::SearchCrossValidationResult: The fieldtest_scoresis $\mathbf{S}$, the raw matrix, withNaNat a failed path. The fieldtrain_scoresisnothingwhengscv.train_scoreisfalse. Otherwise it is a vector of $n_{p}$ matrices, one per path, and each matrix holds one row per fold of the path and one column per candidate. Each entry is $\mathcal{O}$ of the fitted result of the fold, through its prior result. The folds of one path train on different windows, so the train scores stay per fold. The fieldoptis candidate $i^{\star}$, andidxis $i^{\star}$.
Related
CombinatorialCrossValidationGridSearchCrossValidationRandomisedSearchCrossValidation: its search samples a grid and runs this method on it under a combinatorial scheme.cross_val_predictsort_predictionsexpected_riskfinite_candidate_ordercandidate_train_score