Grid search cross validation
PortfolioOptimisers.GridSearchCrossValidation — Type
struct GridSearchCrossValidation{__T_p, __T_cv, __T_r, __T_r_sca, __T_scorer, __T_s_sca, __T_n_best, __T_ex, __T_train_score, __T_rank_prediction, __T_kwargs} <: AbstractSearchCrossValidationEstimatorPerforms grid search cross-validation for portfolio optimisation estimators. Iterates over parameter grids, applies cross-validation splits, and scores each configuration to select the optimal parameters.
Fields
p: Hyperparameter search grid.
cv: Cross-validation estimator.
r: Risk measure or vector of risk measures.
r_sca: Reduction of the oriented scores of a vector of risk measures inrto one score in each fold or path, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuekeeps one score matrix per measure, and gives one ranking per measure.PerCandidatekeeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert whenrholds one measure.
scorer: Search scorer, aCrossValSearchScorer, or a vector of them. Given the orientation-normalised score matrix (rows = CV splits, columns = parameter sets), it gives one value per parameter set, bigger is better, or the column index of the best parameter set. The matrix is normalised so that higher is always better, whatever the risk measure, so a scorer selects the largest aggregate score (seeCrossValidationSearchScorer).
s_sca: Reduction of the values of a vector of search scorers inscorerto one value per candidate, after the values of each scorer are multiplied by itsscorer_scale, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuelets each scorer order the candidates alone, and gives one ranking per scorer. Inert whenscorerholds one scorer.
n_best: Count of the best candidates that the search keeps in rank order. A count larger than the count of the candidates that finished every fold keeps those candidates alone. A scorer that gives the index of one winner keeps one candidate.
ex: Parallel execution strategy.
train_score: Whether to also compute the training set score.
rank_prediction: Whether the Result also keeps, inrank_predictions, the prediction that the search scored for each kept candidate. Withfalse, the search stores no prediction.
kwargs: More keyword arguments.
Constructors
GridSearchCrossValidation( p::MultiGSCVValType_VecMultiGSCVValType; cv::CVE_Onl = KFold(), r::BaseRM_VecBaseRM = ConditionalValueatRisk(), r_sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction} = nothing, scorer::Union{CrossValSearchScorer, AbstractVector} = HighestMeanScore(), s_sca::Union{Option{<:Sca_VecToScaM}, EachValue} = nothing, n_best::Integer = 1, ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(), train_score::Bool = false, rank_prediction::Bool = false, kwargs::NamedTuple = (;),) -> GridSearchCrossValidationr_sca takes a PerCandidate, and s_sca refuses it with a TypeError.
Positional and keyword arguments correspond to fields above. The positional constructor takes rank_prediction after train_score, so a positional call without it raises a MethodError. The outer constructor takes a vector of parameter sets whose types differ. Such a literal infers the element type Vector, so the constructor narrows it through narrow_mixed_vector. A vector of search scorers of different types, such as [HighestMeanScore(), f], infers the element type Any, so the constructor narrows it the same way.
Multiplicity
r takes one risk measure or a vector of them. Each test score and each train score is oriented_score with r_sca, so each measure is oriented before r_sca reduces the vector, and a higher score is always better. A vector whose measures disagree on orientation is admitted. The default r_sca = nothing gives the sum, as SumScalariser() does. For one measure r_sca is inert.
scorer takes one search scorer or a vector of them. The search multiplies the values of each scorer by its scorer_scale, and s_sca reduces the values of each candidate to one, through candidate_values. The values are bigger-is-better, so MaxScalariser() gives the best scorer of each candidate. The default s_sca = nothing gives the sum. For one scorer s_sca and the scale are inert.
One ranking for each value
EachValue in r_sca or s_sca keeps each value apart, as the multi-metric search of sklearn does. The search gives one ranking for each value per candidate, in one SearchCrossValidationResult. r_sca = EachValue() keeps one score matrix for each of the K measures of r, and s_sca = EachValue() lets each of the L scorers of scorer order the candidates alone. Then opt, idx, ranked, rank_idx and rank_scores each hold a vector of K or L entries, or a K × L matrix under both, with row k for measure k and column l for scorer l. The types decide the shape, so a vector of one measure under r_sca = EachValue() gives a vector of one entry. For one measure, or one scorer, the rule is inert.
The search takes no refit, as the refit of sklearn does. To fit the winner of value v on the whole sample, call optimise(res.opt[v], rd).
The measures after the search scorer
r_sca = PerCandidate(sca) combines the measures once per candidate, after the search scorer reduces the folds or the paths of each measure. The search keeps one score matrix for each of the K measures of r, as under EachValue. The scorer reduces each matrix, and sca combines the K values of each candidate, so the search gives one ranking, or L rankings under s_sca = EachValue(). A candidate enters only when it finished every fold under every measure. For a sum and HighestMeanScore the two stages agree. For MaxScalariser they do not, see PerCandidate.
The predictions of the kept candidates
rank_prediction = true makes the Result keep, in rank_predictions[j], the prediction that the search scored for candidate ranked[j]: the object that fit_and_predict gave, with no copy and no refit. So expected_risk(r, p) of each fold prediction p of res.rank_predictions[j].pred, with the sign of the search, gives column rank_idx[j] of test_scores. The ranking is known only after the search scores every candidate, so the search keeps the prediction of each candidate until the end, and then drops the predictions of the candidates that it does not keep.
With rank_prediction = true, the peak memory of the search holds the predictions of every candidate of the grid, and each prediction holds one optimisation Result for each fold. A large grid over many folds can need much memory. With the default false, the search stores no prediction.
Validation
!isempty(p).- If
pis a vector of parameter sets: each element must not be empty. - All keys in
pmust be of typeGSCVKey(i.e.String,Symbol, orInteger). n_best > 0and finite, throughassert_search_scorer.- If
scoreris a vector: it is not empty, and each member is aCrossValSearchScorer. kwargsmust not containsca, throughassert_search_reduction. The reduction is the fieldr_sca.
When the search runs, it refuses a scorer that gives an index with n_best > 1, a member of a vector that gives an index, and a scorer that gives an index under r_sca = PerCandidate(sca) with a vector of measures. A function with n_best > 1 is refused there, not here, because only its return shows its form.
A field that takes a CountOf or a FractionOf also takes a plain number, and count_or_fraction reads an Integer as a count and any other real number as a fraction. Julia promotes the elements of a vector literal to one type before the library reads them. So ["opt.ret.ucs.alg.method.active" => [3, 5, 8], "r.alpha" => [0.05, 0.1]] is a Vector{Pair{String, Vector{Float64}}}, and the counts arrive as 3.0, 5.0 and 8.0. The field then reads them as fractions, and the check of FractionOf refuses them. State the grid in the wrappers, such as [CountOf(3), CountOf(5), FractionOf(0.2)]. A wrapper keeps its type in every container.
Examples
julia> GridSearchCrossValidation(Dict("alpha" => [0.1, 0.2], "beta" => [1.0, 2.0]))GridSearchCrossValidation p ┼ Dict{String, Vector{Float64}}: Dict("alpha" => [0.1, 0.2], "beta" => [1.0, 2.0]) cv ┼ KFold │ n ┼ Int64: 5 │ purged_size ┼ Int64: 0 │ embargo_size ┼ Int64: 0 │ wd ┼ nothing │ fa ┼ nothing │ store_weight_path ┼ Bool: false │ strict ┴ Bool: false r ┼ ConditionalValueatRisk │ settings ┼ RiskMeasureSettings │ │ scale ┼ Float64: 1.0 │ │ ub ┼ nothing │ │ rke ┴ Bool: true │ alpha ┼ Float64: 0.05 │ w ┴ nothing r_sca ┼ nothing scorer ┼ HighestMeanScore │ scale ┴ Float64: 1.0 s_sca ┼ nothing n_best ┼ Int64: 1 ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx() train_score ┼ Bool: false rank_prediction ┼ Bool: false kwargs ┴ @NamedTuple{}: NamedTuple()Related
MultiGSCVValType_VecMultiGSCVValTypeBaseRM_VecBaseRMSca_VecToScaMoriented_scoreCrossValSearchScorersearch_cross_validation
References
- [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012). Section 2.
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
gscv::GridSearchCrossValidation, rd::ReturnsResult)Score every candidate of the grid of gscv on every fold of gscv.cv, and return the winner.
Each candidate runs through fit_and_predict, the fold loop of every cross-validation entry point, so the candidate runs the scheme it declares. A walk-forward threads the weights of the previous fold through the pws of the scheme, and a TimeDependent schedule resolves per fold. Under an Online Scheme, every candidate warms up cold on the first training window and steps through the folds, and no state passes from one candidate to the next. The online search then gives the score matrix of the batch search over the expanding walk-forward. The two matrices agree to the tolerance of the moment estimators and of the solver, and the two searches select the same candidate.
Mathematical definition
\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]
Where:
- $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
- $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
- $F$: Fold count, the count of the folds that
splitenumerates. - $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
- $\Theta$: Search grid, the sequence of grid points that
lens_val_gridbuilds fromp. For aPipeline,pipeline_lens_val_gridbuilds it. - $\mathcal{O}$: Oriented score under
r, fromoriented_scorewith the reductionr_scaand the keyword argumentskwargs. Each measure is oriented so that a higher score is always better, andr_scareduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better. - $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores
NaN, so a candidate that failed one is not in $\mathcal{C}$. - $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
- $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
- $\sigma$: Search scorer
scorer, a map from a score matrix to the position of its best column. The defaultHighestMeanScoretakes the column of greatest mean. - $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.
A candidate that failed one fold is never selected, whatever the scorer computes, because the scorer never reads its column.
Algorithm
- Refuse an estimator that carries a partial-fit state, through
assert_search_entry. - Build
lens_gridandval_gridthroughlens_val_grid. - Run the entry checks of
cross_val_predicton every candidate throughassert_search_candidates, before any candidate is scored. - Split
rdbygscv.cvonce, givingcv. Every candidate scores the folds ofcv, so a scheme whosesplitdraws at random draws once. - Read the rows of the folds through
score_rows, givingrows. - Allocate
test_scores, of sizeM × NforMfolds andNcandidates, with the element type ofrd.X, throughallocate_search_scores. Whengscv.train_scoreistrue, it allocatestrain_scoresof the same size. - When
gscv.train_scoreistrue, view the training returns of each fold throughfold_train_returns, givingtrain_X. - For each candidate
i, in parallel overgscv.ex, do steps 9 to 11. - Build
optithroughsearch_candidate. - Fit and predict
optiover the folds ofcvthroughfit_and_predict, with the folds in sequence, givingpredictions. - Write $\mathcal{O}$ of each fold into column
ioftest_scoresandtrain_scoresthroughwrite_candidate_scores!. Storepredictionsat entryiof the store ofallocate_search_predictionsthroughstore_search_prediction!, which stores nothing whengscv.rank_predictionisfalse. - Order the finite candidates, keep the best
gscv.n_best, build each fromopt, and return theSearchCrossValidationResultthroughsearch_result, with the predictions of the kept candidates. The path of each row isscore_path_idsofcv.
Arguments
opt: The estimator to tune. The search reads it as configuration alone.gscv: The grid search. It gives the gridp, the schemecv, the risk measuresrand their reductionr_sca, thescorer, the executorex, thetrain_scoreflag and the keyword argumentskwargsoforiented_score.rd: The returns result to use.
Validation
optcarries no partial-fit state, throughassert_search_entry. Under an Online Scheme,optalso passes every check ofassert_online_entry.- Every candidate passes the entry checks of
cross_val_predict, throughassert_search_candidates. - The grid is not empty, and it holds at most
RESOURCE_LIMITS[].max_search_gridcandidates, throughlens_val_grid. - At least one candidate finishes every fold. Otherwise
finite_candidate_orderthrows anIsNonFiniteError.
Returns
res::SearchCrossValidationResult: The fieldtest_scoresis $\mathbf{S}$. It is the raw matrix, withNaNat a failed fold, so its columns agree with the grid and a reader sees which fold failed. Row $f$ is fold $f$ in the order ofsplit. Under aMultipleRandomised,score_rowsputs the scores of each path back onto the rows of the split. The fieldtrain_scoresholds $\mathcal{O}$ of each fitted result over its own training window, in a matrix of the same size, ornothingwhengscv.train_scoreisfalse. The fieldoptis candidate $i^{\star}$, andidxis $i^{\star}$.
Related
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
gscv::GridSearchCrossValidation{<:Any, <:CombinatorialCrossValidation},
rd::ReturnsResult)Tune opt over the grid of gscv under a CombinatorialCrossValidation, with one score per candidate and backtest path.
A combinatorial scheme recombines its disjoint test groups into full backtest paths, and a split scored alone mixes the groups of different paths. So this method scores each path. For each candidate, the scheme runs through fit_and_predict, sort_predictions recombines the groups into a PopulationPredictionResult, and oriented_score gives one score per path. The candidates run in sequence, and the fold loop of each candidate runs with the executor gscv.ex.
Mathematical definition
\[\begin{align} S_{pi} &= \mathcal{O}\left(\hat{P}_{p}(\theta_i)\right)\,, \quad p = 1, \ldots, n_{p}\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{pi} \text{ is finite for every } p \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]
Where:
- $S_{pi}$: Path score, the test score of candidate $i$ on path $p$, and the entry of the score matrix $\mathbf{S}$, $n_{p} \times |\Theta|$.
- $\hat{P}_{p}(\theta_i)$: Path prediction, the predictions of candidate $i$ for the test groups that path $p$ holds, pooled into one series. The weights that $\theta_i$ fits on the training window of the fold that tests a group predict that group.
- $n_{p}$: Path count, the greatest entry of
path_idsof the split. - $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
- $\Theta$: Search grid, the sequence of grid points that
lens_val_gridbuilds fromp. For aPipeline,pipeline_lens_val_gridbuilds it. - $\mathcal{O}$: Oriented score under
r, fromoriented_scorewith the reductionr_scaand the keyword argumentskwargs. Each measure is oriented so that a higher score is always better, andr_scareduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better. - $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores
NaN, so a candidate that failed one is not in $\mathcal{C}$. - $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
- $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
- $\sigma$: Search scorer
scorer, a map from a score matrix to the position of its best column. The defaultHighestMeanScoretakes the column of greatest mean. - $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.
A candidate that failed one path is never selected, whatever the scorer computes, because the scorer never reads its column.
Algorithm
- Refuse an estimator that carries a partial-fit state, through
assert_search_entry. - Build
lens_gridandval_gridthroughlens_val_grid. - Run the entry checks of
cross_val_predicton every candidate throughassert_search_candidates, before any candidate is scored. - Split
rdbygscv.cv, givingcv. - Set
M, the count of the paths, to the greatest entry ofcv.path_ids. - Allocate
test_scores, of sizeM × NforNcandidates, with the element type ofrd.X, throughallocate_search_scores. Whengscv.train_scoreistrue, it allocatestrain_scores, one matrix per path, with one row per fold of the path and one column per candidate. - View the training returns of each fold of each path through
fold_train_returns, givingpath_X. The entries of a path incv.path_idsare read in column order, and the column is the fold, which is the order of the folds in the path thatsort_predictionsgives. - For each candidate
i, in sequence, do steps 9 to 11. - Build
optithroughsearch_candidate. - Fit and predict
optiover the folds ofcvthroughfit_and_predict, with the executorgscv.ex, givingpredictions, one path per member. - Write column
ithroughwrite_candidate_scores!, with rowpfor pathp. The test score of each path is $\mathcal{O}$ of the path. Whengscv.train_scoreistrue, the train score of each fold of each path goes into columniof the matrix of that path, throughcandidate_train_scorewith the view ofpath_X. Storepredictionsat entryiof the store ofallocate_search_predictionsthroughstore_search_prediction!, which stores nothing whengscv.rank_predictionisfalse. - Order the finite candidates, keep the best
gscv.n_best, build each fromopt, and return theSearchCrossValidationResultthroughsearch_result, with the predictions of the kept candidates. Rowpis pathp, so the path of each row is1:M.
Arguments
opt: The estimator to tune. The search reads it as configuration alone.gscv: The grid search over aCombinatorialCrossValidation. It gives the gridp, the schemecv, the risk measuresrand their reductionr_sca, thescorer, the executorex, thetrain_scoreflag and the keyword argumentskwargsoforiented_score.rd: The returns result to use.
Validation
optcarries no partial-fit state, throughassert_search_entry.- Every candidate passes the entry checks of
cross_val_predict, throughassert_search_candidates. - The grid is not empty, and it holds at most
RESOURCE_LIMITS[].max_search_gridcandidates, throughlens_val_grid. - At least one candidate finishes every path. Otherwise
finite_candidate_orderthrows anIsNonFiniteError.
Returns
res::SearchCrossValidationResult: The fieldtest_scoresis $\mathbf{S}$, the raw matrix, withNaNat a failed path. The fieldtrain_scoresisnothingwhengscv.train_scoreisfalse. Otherwise it is a vector of $n_{p}$ matrices, one per path, and each matrix holds one row per fold of the path and one column per candidate. The method keeps the train scores per fold, because the folds of one path train on different windows. The test returns of one path pool into one series, so the test score is one per path. The fieldoptis candidate $i^{\star}$, andidxis $i^{\star}$.
Related
CombinatorialCrossValidationGridSearchCrossValidationRandomisedSearchCrossValidation: its search samples a grid and runs this method on it under a combinatorial scheme.fit_and_predictsort_predictionsexpected_riskfinite_candidate_ordercandidate_train_scorefold_train_returns
References
- [226]
- J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).