Search cross-validation

A search over the hyperparameters of a pipeline covers the whole workflow. It searches the preprocessing settings, such as the imputation statistic or the threshold for missing data, together with the settings of the prior, the constraints and the optimiser. Each candidate is fitted on each fold, so a fitted preprocessing step never sees the test window. A key names a setting by the name of its step, "impute.stat", by position, where an integer key replaces the whole step, or by property path, "steps[2].stat".

PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(pipe::Pipeline, gscv::GridSearchCrossValidation, data::Prices_RR)
search_cross_validation(pipe::Pipeline, rscv::RandomisedSearchCrossValidation,
                        data::Prices_RR)

Score every candidate Pipeline of the grid on every fold of the scheme, and return the winner.

Each candidate runs through cross_val_predict, the fold loop of every cross-validation entry point. Every fold fits the whole workflow on its training window and predicts its test window, so a preprocessing step learns from the training window alone. A walk-forward threads the weights of the previous fold through the pws of the scheme. A TimeDependent schedule resolves per fold, and a key that names its step replaces the whole schedule.

Under an Online Scheme, every candidate warms up once on the first training window, and partial_fit! folds each later block of rows into it. The search then gives the score matrix of the batch search over the expanding walk-forward with the same folds, to the tolerance of the estimators and of the solver. The two searches select the same candidate. The randomised form samples a grid from rscv.p and runs the grid search on it.

Mathematical definition

\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]

Where:

  • $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
  • $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
  • $F$: Fold count, the count of the folds that split enumerates.
  • $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
  • $\Theta$: Search grid, the sequence of grid points that lens_val_grid builds from p. For a Pipeline, pipeline_lens_val_grid builds it.
  • $\mathcal{O}$: Oriented score under r, from oriented_score with the reduction r_sca and the keyword arguments kwargs. Each measure is oriented so that a higher score is always better, and r_sca reduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better.
  • $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores NaN, so a candidate that failed one is not in $\mathcal{C}$.
  • $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
  • $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
  • $\sigma$: Search scorer scorer, a map from a score matrix to the position of its best column. The default HighestMeanScore takes the column of greatest mean.
  • $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.

The scorer never reads the column of a candidate that failed a fold, so the search never selects that candidate.

Algorithm

For gscv:

  1. Refuse a pipeline that carries a partial-fit state, through assert_search_entry.
  2. Build lens_grid and val_grid through pipeline_lens_val_grid.
  3. Refuse a candidate that holds a TrainTestSplit, through assert_search_candidates.
  4. Split data by gscv.cv once, giving cv. Every candidate scores the folds of cv, so a scheme whose split draws at random draws once.
  5. Read the rows of the folds through score_rows, giving rows.
  6. Allocate test_scores, of size M × N for M folds and N candidates, with the element type from cv_data_eltype, through allocate_search_scores. When gscv.train_score is true, it allocates train_scores of the same size.
  7. For each candidate i, in parallel over gscv.ex, do steps 8 to 10.
  8. Build pipei through search_candidate.
  9. Run pipei over the folds of cv through cross_val_predict, with the folds in sequence, giving predictions.
  10. Write $\mathcal{O}$ of each fold into column i of test_scores and train_scores through write_candidate_scores!, with no view of the training returns. Store predictions at entry i of the store of allocate_search_predictions through store_search_prediction!, which stores nothing when gscv.rank_prediction is false.
  11. Order the finite candidates, keep the best gscv.n_best, build each from pipe, and return the SearchCrossValidationResult through search_result, with the predictions of the kept candidates. The path of each row is score_path_ids of cv.

For rscv:

  1. Resolve the random number generator from rscv.rng and rscv.seed through resolve_rng.
  2. Draw at most rscv.n_iter trials through make_p_grid, each a joint value of every key of one parameter set.
  3. Build a GridSearchCrossValidation with that grid and the other fields of rscv, and run the search of that grid. Under a CombinatorialCrossValidation this is the combinatorial method.

Arguments

  • pipe: The pipeline to tune. The search reads it as configuration alone.
  • gscv: The grid search. It gives the grid p, the scheme cv, the risk measures r and their reduction r_sca, the scorer, the executor ex, the train_score flag and the keyword arguments kwargs of oriented_score.
  • rscv: The randomised search. It gives the fields of gscv, the distributions or vectors p to sample, the sample count n_iter, and rng and seed.
  • data: The price or returns data, a Prices_RR, at the input level of pipe.

Validation

Returns

  • res::SearchCrossValidationResult: The field test_scores is $\mathbf{S}$, the raw matrix, with NaN at a failed fold. Row $f$ is fold $f$ in the order of split, and under a MultipleRandomised, score_rows puts the scores of each path back onto the rows of the split. The field train_scores is nothing when gscv.train_score is false. Otherwise it holds $\mathcal{O}$ of each fitted result through its prior result, in a matrix of the same size. The field opt is candidate $i^{\star}$, and idx is $i^{\star}$.

Related

source
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(pipe::Pipeline,
                        gscv::GridSearchCrossValidation{<:Any, <:CombinatorialCrossValidation},
                        data::Prices_RR)

Tune a Pipeline over the grid of gscv under a CombinatorialCrossValidation, with one score per candidate and backtest path.

A combinatorial scheme recombines its disjoint test groups into full backtest paths, and a split scored alone mixes the groups of different paths. So this method scores each path. For each candidate, cross_val_predict runs the scheme, sort_predictions recombines the groups into a PopulationPredictionResult, and oriented_score gives one score per path. The candidates run in sequence, and the fold loop of each candidate runs with the executor gscv.ex.

The method takes price data too. The training rows of a split then have gaps where the test groups are, so PricesToReturns makes one return across each gap. The test groups are contiguous, so the predictions have no such return. A MultipleRandomised scheme keeps the training rows contiguous at the price level.

Mathematical definition

\[\begin{align} S_{pi} &= \mathcal{O}\left(\hat{P}_{p}(\theta_i)\right)\,, \quad p = 1, \ldots, n_{p}\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{pi} \text{ is finite for every } p \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]

Where:

  • $S_{pi}$: Path score, the test score of candidate $i$ on path $p$, and the entry of the score matrix $\mathbf{S}$, $n_{p} \times |\Theta|$.
  • $\hat{P}_{p}(\theta_i)$: Path prediction, the predictions of candidate $i$ for the test groups that path $p$ holds, pooled into one series. The weights that $\theta_i$ fits on the training window of the fold that tests a group predict that group.
  • $n_{p}$: Path count, the greatest entry of path_ids of the split.
  • $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
  • $\Theta$: Search grid, the sequence of grid points that lens_val_grid builds from p. For a Pipeline, pipeline_lens_val_grid builds it.
  • $\mathcal{O}$: Oriented score under r, from oriented_score with the reduction r_sca and the keyword arguments kwargs. Each measure is oriented so that a higher score is always better, and r_sca reduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better.
  • $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores NaN, so a candidate that failed one is not in $\mathcal{C}$.
  • $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
  • $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
  • $\sigma$: Search scorer scorer, a map from a score matrix to the position of its best column. The default HighestMeanScore takes the column of greatest mean.
  • $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.

The scorer never reads the column of a candidate that failed a path, so the search never selects that candidate.

Algorithm

  1. Refuse a pipeline that carries a partial-fit state, through assert_search_entry.
  2. Build lens_grid and val_grid through pipeline_lens_val_grid.
  3. Refuse a candidate that holds a TrainTestSplit, through assert_search_candidates.
  4. Split data by gscv.cv, giving cv.
  5. Set M, the count of the paths, to the greatest entry of cv.path_ids.
  6. Allocate test_scores, of size M × N for N candidates, with the element type from cv_data_eltype, through allocate_search_scores. When gscv.train_score is true, it allocates train_scores, one matrix per path, with one row per fold of the path and one column per candidate.
  7. For each candidate i, in sequence, do steps 8 to 10.
  8. Build pipei through search_candidate.
  9. Run pipei over the folds of cv through cross_val_predict, with the executor gscv.ex, giving predictions, one path per member.
  10. Write column i through write_candidate_scores!, with row p for path p. The test score of each path is $\mathcal{O}$ of the path. When gscv.train_score is true, the train score of each fold of each path goes into column i of the matrix of that path, through candidate_train_score with no view of the training returns. Store predictions at entry i of the store of allocate_search_predictions through store_search_prediction!, which stores nothing when gscv.rank_prediction is false.
  11. Order the finite candidates, keep the best gscv.n_best, build each from pipe, and return the SearchCrossValidationResult through search_result, with the predictions of the kept candidates. Row p is path p, so the path of each row is 1:M.

Arguments

  • pipe: The pipeline to tune. The search reads it as configuration alone.
  • gscv: The grid search over a CombinatorialCrossValidation. It gives the grid p, the scheme cv, the risk measures r and their reduction r_sca, the scorer, the executor ex, the train_score flag and the keyword arguments kwargs of oriented_score.
  • data: The price or returns data, a Prices_RR, at the input level of pipe.

Validation

Returns

  • res::SearchCrossValidationResult: The field test_scores is $\mathbf{S}$, the raw matrix, with NaN at a failed path. The field train_scores is nothing when gscv.train_score is false. Otherwise it is a vector of $n_{p}$ matrices, one per path, and each matrix holds one row per fold of the path and one column per candidate. Each entry is $\mathcal{O}$ of the fitted result of the fold, through its prior result. The folds of one path train on different windows, so the train scores stay per fold. The field opt is candidate $i^{\star}$, and idx is $i^{\star}$.

Related

source