Grid search cross validation

PortfolioOptimisers.GridSearchCrossValidation — Type
struct GridSearchCrossValidation{__T_p, __T_cv, __T_r, __T_r_sca, __T_scorer, __T_s_sca, __T_n_best, __T_ex, __T_train_score, __T_rank_prediction, __T_kwargs} <: AbstractSearchCrossValidationEstimator

Performs grid search cross-validation for portfolio optimisation estimators. Iterates over parameter grids, applies cross-validation splits, and scores each configuration to select the optimal parameters.

Fields

  • p: Hyperparameter search grid.
  • cv: Cross-validation estimator.
  • r: Risk measure or vector of risk measures.
  • r_sca: Reduction of the oriented scores of a vector of risk measures in r to one score in each fold or path, a Sca_VecToScaM, or nothing for the sum that SumScalariser gives. EachValue keeps one score matrix per measure, and gives one ranking per measure. PerCandidate keeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert when r holds one measure.
  • scorer: Search scorer, a CrossValSearchScorer, or a vector of them. Given the orientation-normalised score matrix (rows = CV splits, columns = parameter sets), it gives one value per parameter set, bigger is better, or the column index of the best parameter set. The matrix is normalised so that higher is always better, whatever the risk measure, so a scorer selects the largest aggregate score (see CrossValidationSearchScorer).
  • s_sca: Reduction of the values of a vector of search scorers in scorer to one value per candidate, after the values of each scorer are multiplied by its scorer_scale, a Sca_VecToScaM, or nothing for the sum that SumScalariser gives. EachValue lets each scorer order the candidates alone, and gives one ranking per scorer. Inert when scorer holds one scorer.
  • n_best: Count of the best candidates that the search keeps in rank order. A count larger than the count of the candidates that finished every fold keeps those candidates alone. A scorer that gives the index of one winner keeps one candidate.
  • ex: Parallel execution strategy.
  • train_score: Whether to also compute the training set score.
  • rank_prediction: Whether the Result also keeps, in rank_predictions, the prediction that the search scored for each kept candidate. With false, the search stores no prediction.
  • kwargs: More keyword arguments.

Constructors

GridSearchCrossValidation(    p::MultiGSCVValType_VecMultiGSCVValType;    cv::CVE_Onl = KFold(),    r::BaseRM_VecBaseRM = ConditionalValueatRisk(),    r_sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction} = nothing,    scorer::Union{CrossValSearchScorer, AbstractVector} = HighestMeanScore(),    s_sca::Union{Option{<:Sca_VecToScaM}, EachValue} = nothing,    n_best::Integer = 1,    ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(),    train_score::Bool = false,    rank_prediction::Bool = false,    kwargs::NamedTuple = (;),) -> GridSearchCrossValidation

r_sca takes a PerCandidate, and s_sca refuses it with a TypeError.

Positional and keyword arguments correspond to fields above. The positional constructor takes rank_prediction after train_score, so a positional call without it raises a MethodError. The outer constructor takes a vector of parameter sets whose types differ. Such a literal infers the element type Vector, so the constructor narrows it through narrow_mixed_vector. A vector of search scorers of different types, such as [HighestMeanScore(), f], infers the element type Any, so the constructor narrows it the same way.

Multiplicity

r takes one risk measure or a vector of them. Each test score and each train score is oriented_score with r_sca, so each measure is oriented before r_sca reduces the vector, and a higher score is always better. A vector whose measures disagree on orientation is admitted. The default r_sca = nothing gives the sum, as SumScalariser() does. For one measure r_sca is inert.

scorer takes one search scorer or a vector of them. The search multiplies the values of each scorer by its scorer_scale, and s_sca reduces the values of each candidate to one, through candidate_values. The values are bigger-is-better, so MaxScalariser() gives the best scorer of each candidate. The default s_sca = nothing gives the sum. For one scorer s_sca and the scale are inert.

One ranking for each value

EachValue in r_sca or s_sca keeps each value apart, as the multi-metric search of sklearn does. The search gives one ranking for each value per candidate, in one SearchCrossValidationResult. r_sca = EachValue() keeps one score matrix for each of the K measures of r, and s_sca = EachValue() lets each of the L scorers of scorer order the candidates alone. Then opt, idx, ranked, rank_idx and rank_scores each hold a vector of K or L entries, or a K × L matrix under both, with row k for measure k and column l for scorer l. The types decide the shape, so a vector of one measure under r_sca = EachValue() gives a vector of one entry. For one measure, or one scorer, the rule is inert.

The search takes no refit, as the refit of sklearn does. To fit the winner of value v on the whole sample, call optimise(res.opt[v], rd).

The measures after the search scorer

r_sca = PerCandidate(sca) combines the measures once per candidate, after the search scorer reduces the folds or the paths of each measure. The search keeps one score matrix for each of the K measures of r, as under EachValue. The scorer reduces each matrix, and sca combines the K values of each candidate, so the search gives one ranking, or L rankings under s_sca = EachValue(). A candidate enters only when it finished every fold under every measure. For a sum and HighestMeanScore the two stages agree. For MaxScalariser they do not, see PerCandidate.

The predictions of the kept candidates

rank_prediction = true makes the Result keep, in rank_predictions[j], the prediction that the search scored for candidate ranked[j]: the object that fit_and_predict gave, with no copy and no refit. So expected_risk(r, p) of each fold prediction p of res.rank_predictions[j].pred, with the sign of the search, gives column rank_idx[j] of test_scores. The ranking is known only after the search scores every candidate, so the search keeps the prediction of each candidate until the end, and then drops the predictions of the candidates that it does not keep.

Warning

With rank_prediction = true, the peak memory of the search holds the predictions of every candidate of the grid, and each prediction holds one optimisation Result for each fold. A large grid over many folds can need much memory. With the default false, the search stores no prediction.

Validation

  • !isempty(p).
  • If p is a vector of parameter sets: each element must not be empty.
  • All keys in p must be of type GSCVKey (i.e. String, Symbol, or Integer).
  • n_best > 0 and finite, through assert_search_scorer.
  • If scorer is a vector: it is not empty, and each member is a CrossValSearchScorer.
  • kwargs must not contain sca, through assert_search_reduction. The reduction is the field r_sca.

When the search runs, it refuses a scorer that gives an index with n_best > 1, a member of a vector that gives an index, and a scorer that gives an index under r_sca = PerCandidate(sca) with a vector of measures. A function with n_best > 1 is refused there, not here, because only its return shows its form.

A search grid can turn a plain count into a fraction

A field that takes a CountOf or a FractionOf also takes a plain number, and count_or_fraction reads an Integer as a count and any other real number as a fraction. Julia promotes the elements of a vector literal to one type before the library reads them. So ["opt.ret.ucs.alg.method.active" => [3, 5, 8], "r.alpha" => [0.05, 0.1]] is a Vector{Pair{String, Vector{Float64}}}, and the counts arrive as 3.0, 5.0 and 8.0. The field then reads them as fractions, and the check of FractionOf refuses them. State the grid in the wrappers, such as [CountOf(3), CountOf(5), FractionOf(0.2)]. A wrapper keeps its type in every container.

Examples

julia> GridSearchCrossValidation(Dict("alpha" => [0.1, 0.2], "beta" => [1.0, 2.0]))GridSearchCrossValidation                p ┼ Dict{String, Vector{Float64}}: Dict("alpha" => [0.1, 0.2], "beta" => [1.0, 2.0])               cv ┼ KFold                  │                   n ┼ Int64: 5                  │         purged_size ┼ Int64: 0                  │        embargo_size ┼ Int64: 0                  │                  wd ┼ nothing                  │                  fa ┼ nothing                  │   store_weight_path ┼ Bool: false                  │              strict ┴ Bool: false                r ┼ ConditionalValueatRisk                  │   settings ┼ RiskMeasureSettings                  │            │   scale ┼ Float64: 1.0                  │            │      ub ┼ nothing                  │            │     rke ┴ Bool: true                  │      alpha ┼ Float64: 0.05                  │          w ┴ nothing            r_sca ┼ nothing           scorer ┼ HighestMeanScore                  │   scale ┴ Float64: 1.0            s_sca ┼ nothing           n_best ┼ Int64: 1               ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()      train_score ┼ Bool: false  rank_prediction ┼ Bool: false           kwargs ┴ @NamedTuple{}: NamedTuple()

Related

References

  • [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012). Section 2.
source
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
                        gscv::GridSearchCrossValidation, rd::ReturnsResult)

Score every candidate of the grid of gscv on every fold of gscv.cv, and return the winner.

Each candidate runs through fit_and_predict, the fold loop of every cross-validation entry point, so the candidate runs the scheme it declares. A walk-forward threads the weights of the previous fold through the pws of the scheme, and a TimeDependent schedule resolves per fold. Under an Online Scheme, every candidate warms up cold on the first training window and steps through the folds, and no state passes from one candidate to the next. The online search then gives the score matrix of the batch search over the expanding walk-forward. The two matrices agree to the tolerance of the moment estimators and of the solver, and the two searches select the same candidate.

Mathematical definition

\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]

Where:

  • $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
  • $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
  • $F$: Fold count, the count of the folds that split enumerates.
  • $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
  • $\Theta$: Search grid, the sequence of grid points that lens_val_grid builds from p. For a Pipeline, pipeline_lens_val_grid builds it.
  • $\mathcal{O}$: Oriented score under r, from oriented_score with the reduction r_sca and the keyword arguments kwargs. Each measure is oriented so that a higher score is always better, and r_sca reduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better.
  • $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores NaN, so a candidate that failed one is not in $\mathcal{C}$.
  • $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
  • $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
  • $\sigma$: Search scorer scorer, a map from a score matrix to the position of its best column. The default HighestMeanScore takes the column of greatest mean.
  • $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.

A candidate that failed one fold is never selected, whatever the scorer computes, because the scorer never reads its column.

Algorithm

  1. Refuse an estimator that carries a partial-fit state, through assert_search_entry.
  2. Build lens_grid and val_grid through lens_val_grid.
  3. Run the entry checks of cross_val_predict on every candidate through assert_search_candidates, before any candidate is scored.
  4. Split rd by gscv.cv once, giving cv. Every candidate scores the folds of cv, so a scheme whose split draws at random draws once.
  5. Read the rows of the folds through score_rows, giving rows.
  6. Allocate test_scores, of size M × N for M folds and N candidates, with the element type of rd.X, through allocate_search_scores. When gscv.train_score is true, it allocates train_scores of the same size.
  7. When gscv.train_score is true, view the training returns of each fold through fold_train_returns, giving train_X.
  8. For each candidate i, in parallel over gscv.ex, do steps 9 to 11.
  9. Build opti through search_candidate.
  10. Fit and predict opti over the folds of cv through fit_and_predict, with the folds in sequence, giving predictions.
  11. Write $\mathcal{O}$ of each fold into column i of test_scores and train_scores through write_candidate_scores!. Store predictions at entry i of the store of allocate_search_predictions through store_search_prediction!, which stores nothing when gscv.rank_prediction is false.
  12. Order the finite candidates, keep the best gscv.n_best, build each from opt, and return the SearchCrossValidationResult through search_result, with the predictions of the kept candidates. The path of each row is score_path_ids of cv.

Arguments

  • opt: The estimator to tune. The search reads it as configuration alone.
  • gscv: The grid search. It gives the grid p, the scheme cv, the risk measures r and their reduction r_sca, the scorer, the executor ex, the train_score flag and the keyword arguments kwargs of oriented_score.
  • rd: The returns result to use.

Validation

Returns

  • res::SearchCrossValidationResult: The field test_scores is $\mathbf{S}$. It is the raw matrix, with NaN at a failed fold, so its columns agree with the grid and a reader sees which fold failed. Row $f$ is fold $f$ in the order of split. Under a MultipleRandomised, score_rows puts the scores of each path back onto the rows of the split. The field train_scores holds $\mathcal{O}$ of each fitted result over its own training window, in a matrix of the same size, or nothing when gscv.train_score is false. The field opt is candidate $i^{\star}$, and idx is $i^{\star}$.

Related

source
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
                        gscv::GridSearchCrossValidation{<:Any, <:CombinatorialCrossValidation},
                        rd::ReturnsResult)

Tune opt over the grid of gscv under a CombinatorialCrossValidation, with one score per candidate and backtest path.

A combinatorial scheme recombines its disjoint test groups into full backtest paths, and a split scored alone mixes the groups of different paths. So this method scores each path. For each candidate, the scheme runs through fit_and_predict, sort_predictions recombines the groups into a PopulationPredictionResult, and oriented_score gives one score per path. The candidates run in sequence, and the fold loop of each candidate runs with the executor gscv.ex.

Mathematical definition

\[\begin{align} S_{pi} &= \mathcal{O}\left(\hat{P}_{p}(\theta_i)\right)\,, \quad p = 1, \ldots, n_{p}\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{pi} \text{ is finite for every } p \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]

Where:

  • $S_{pi}$: Path score, the test score of candidate $i$ on path $p$, and the entry of the score matrix $\mathbf{S}$, $n_{p} \times |\Theta|$.
  • $\hat{P}_{p}(\theta_i)$: Path prediction, the predictions of candidate $i$ for the test groups that path $p$ holds, pooled into one series. The weights that $\theta_i$ fits on the training window of the fold that tests a group predict that group.
  • $n_{p}$: Path count, the greatest entry of path_ids of the split.
  • $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
  • $\Theta$: Search grid, the sequence of grid points that lens_val_grid builds from p. For a Pipeline, pipeline_lens_val_grid builds it.
  • $\mathcal{O}$: Oriented score under r, from oriented_score with the reduction r_sca and the keyword arguments kwargs. Each measure is oriented so that a higher score is always better, and r_sca reduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better.
  • $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores NaN, so a candidate that failed one is not in $\mathcal{C}$.
  • $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
  • $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
  • $\sigma$: Search scorer scorer, a map from a score matrix to the position of its best column. The default HighestMeanScore takes the column of greatest mean.
  • $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.

A candidate that failed one path is never selected, whatever the scorer computes, because the scorer never reads its column.

Algorithm

  1. Refuse an estimator that carries a partial-fit state, through assert_search_entry.
  2. Build lens_grid and val_grid through lens_val_grid.
  3. Run the entry checks of cross_val_predict on every candidate through assert_search_candidates, before any candidate is scored.
  4. Split rd by gscv.cv, giving cv.
  5. Set M, the count of the paths, to the greatest entry of cv.path_ids.
  6. Allocate test_scores, of size M × N for N candidates, with the element type of rd.X, through allocate_search_scores. When gscv.train_score is true, it allocates train_scores, one matrix per path, with one row per fold of the path and one column per candidate.
  7. View the training returns of each fold of each path through fold_train_returns, giving path_X. The entries of a path in cv.path_ids are read in column order, and the column is the fold, which is the order of the folds in the path that sort_predictions gives.
  8. For each candidate i, in sequence, do steps 9 to 11.
  9. Build opti through search_candidate.
  10. Fit and predict opti over the folds of cv through fit_and_predict, with the executor gscv.ex, giving predictions, one path per member.
  11. Write column i through write_candidate_scores!, with row p for path p. The test score of each path is $\mathcal{O}$ of the path. When gscv.train_score is true, the train score of each fold of each path goes into column i of the matrix of that path, through candidate_train_score with the view of path_X. Store predictions at entry i of the store of allocate_search_predictions through store_search_prediction!, which stores nothing when gscv.rank_prediction is false.
  12. Order the finite candidates, keep the best gscv.n_best, build each from opt, and return the SearchCrossValidationResult through search_result, with the predictions of the kept candidates. Row p is path p, so the path of each row is 1:M.

Arguments

  • opt: The estimator to tune. The search reads it as configuration alone.
  • gscv: The grid search over a CombinatorialCrossValidation. It gives the grid p, the scheme cv, the risk measures r and their reduction r_sca, the scorer, the executor ex, the train_score flag and the keyword arguments kwargs of oriented_score.
  • rd: The returns result to use.

Validation

Returns

  • res::SearchCrossValidationResult: The field test_scores is $\mathbf{S}$, the raw matrix, with NaN at a failed path. The field train_scores is nothing when gscv.train_score is false. Otherwise it is a vector of $n_{p}$ matrices, one per path, and each matrix holds one row per fold of the path and one column per candidate. The method keeps the train scores per fold, because the folds of one path train on different windows. The test returns of one path pool into one series, so the test score is one per path. The field opt is candidate $i^{\star}$, and idx is $i^{\star}$.

Related

source

References

[226]
J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).