Randomised search cross validation
PortfolioOptimisers.RandomisedSearchCrossValidation — Type
struct RandomisedSearchCrossValidation{__T_p, __T_cv, __T_r, __T_r_sca, __T_scorer, __T_s_sca, __T_n_best, __T_ex, __T_n_iter, __T_rng, __T_seed, __T_train_score, __T_rank_prediction, __T_kwargs} <: AbstractSearchCrossValidationEstimatorRandomised search cross-validation estimator for portfolio optimisation. It draws n_iter trials from the search space, each a joint value of every key of one parameter set, through make_p_grid. Then it scores the trials with the search of GridSearchCrossValidation, and selects one with the scorer. The count of fits grows with n_iter, not with the count of keys.
Fields
p: Hyperparameter search grid.
cv: Cross-validation estimator.
r: Risk measure or vector of risk measures.
r_sca: Reduction of the oriented scores of a vector of risk measures inrto one score in each fold or path, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuekeeps one score matrix per measure, and gives one ranking per measure.PerCandidatekeeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert whenrholds one measure.
scorer: Search scorer, aCrossValSearchScorer, or a vector of them. Given the orientation-normalised score matrix (rows = CV splits, columns = parameter sets), it gives one value per parameter set, bigger is better, or the column index of the best parameter set. The matrix is normalised so that higher is always better, whatever the risk measure, so a scorer selects the largest aggregate score (seeCrossValidationSearchScorer).
s_sca: Reduction of the values of a vector of search scorers inscorerto one value per candidate, after the values of each scorer are multiplied by itsscorer_scale, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuelets each scorer order the candidates alone, and gives one ranking per scorer. Inert whenscorerholds one scorer.
n_best: Count of the best candidates that the search keeps in rank order. A count larger than the count of the candidates that finished every fold keeps those candidates alone. A scorer that gives the index of one winner keeps one candidate.
ex: Parallel execution strategy.
n_iter: Trial count, the count of the candidates that the randomised search draws. Each trial sets every key of one parameter set. When every value is a vector and the space holds fewer points, the search takes each point once.
rng: Random number generator.
seed: Optional seed. If set, the draws come from a copy ofrngseeded with it, sorngdoes not advance and every call draws the same values. Ifnothing, the draws come fromrngitself.
train_score: Whether to also compute the training set score.
rank_prediction: Whether the Result also keeps, inrank_predictions, the prediction that the search scored for each kept candidate. Withfalse, the search stores no prediction.
kwargs: More keyword arguments.
Constructors
RandomisedSearchCrossValidation( p::Union{AbstractVector{<:Pair{<:GSCVKey, <:RSCVVal}}, AbstractVector{<:AbstractVector{<:Pair{<:GSCVKey, <:RSCVVal}}}, AbstractDict{<:GSCVKey, <:RSCVVal}, AbstractVector{<:AbstractDict{<:GSCVKey, <:RSCVVal}}}; cv::CVE_Onl = KFold(), r::BaseRM_VecBaseRM = ConditionalValueatRisk(), r_sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction} = nothing, scorer::Union{CrossValSearchScorer, AbstractVector} = HighestMeanScore(), s_sca::Union{Option{<:Sca_VecToScaM}, EachValue} = nothing, n_best::Integer = 1, ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(), n_iter::Integer = 10, rng::Random.AbstractRNG = Random.default_rng(), seed::Option{<:Integer} = nothing, train_score::Bool = false, rank_prediction::Bool = false, kwargs::NamedTuple = (;),) -> RandomisedSearchCrossValidationPositional and keyword arguments correspond to the struct's fields. The positional constructor takes rank_prediction after train_score, so a positional call without it raises a MethodError. The outer constructor takes a vector of parameter sets whose types differ, such as a vector of pairs and a vector of pairs with other value types. Such a literal infers the element type Vector, so the constructor narrows it through narrow_mixed_vector.
The search draws its trials from rng, so the trials, and the order of the n_best kept candidates, are the same in two runs only when seed is set.
Multiplicity
r takes one risk measure or a vector of them, and r_sca reduces a vector, as in GridSearchCrossValidation. For one measure r_sca is inert. scorer takes one search scorer or a vector of them, and s_sca reduces a vector, as in GridSearchCrossValidation. For one scorer s_sca is inert. EachValue in r_sca or s_sca gives one ranking for each value per candidate, and PerCandidate in r_sca combines the measures after the search scorer, as in GridSearchCrossValidation.
rank_prediction = true keeps the prediction that the search scored for each kept candidate in rank_predictions, as in GridSearchCrossValidation.
With rank_prediction = true, the peak memory of the search holds the predictions of every trial, and each prediction holds one optimisation Result for each fold.
Validation
!isempty(p).- If
pis a vector of parameter sets: each element must not be empty. - All keys in
pmust be of typeGSCVKey(i.e.String,Symbol, orInteger). - All values in
pmust be of typeRSCVVal(i.e. anAbstractVectororDistributions.Distribution). n_iter > 0and finite.n_best > 0and finite, throughassert_search_scorer.- If
scoreris a vector: it is not empty, and each member is aCrossValSearchScorer. kwargsmust not containsca, throughassert_search_reduction. The reduction is the fieldr_sca.
When the search runs, it refuses a scorer that gives an index with n_best > 1, and a member of a vector that gives an index, as in GridSearchCrossValidation.
A field that takes a CountOf or a FractionOf also takes a plain number, and count_or_fraction reads an Integer as a count and any other real number as a fraction. Julia promotes the elements of a vector literal to one type before the library reads them. So ["opt.ret.ucs.alg.method.active" => [3, 5, 8], "r.alpha" => [0.05, 0.1]] is a Vector{Pair{String, Vector{Float64}}}, and the counts arrive as 3.0, 5.0 and 8.0. The field then reads them as fractions, and the check of FractionOf refuses them. State the grid in the wrappers, such as [CountOf(3), CountOf(5), FractionOf(0.2)]. A wrapper keeps its type in every container.
Examples
julia> RandomisedSearchCrossValidation(Dict("alpha" => [0.1, 0.2, 0.3], "beta" => Normal(1.0, 0.5)))RandomisedSearchCrossValidation p ┼ Dict{String, Any}: Dict{String, Any}("alpha" => [0.1, 0.2, 0.3], "beta" => Normal(1.0, 0.5)) cv ┼ KFold │ n ┼ Int64: 5 │ purged_size ┼ Int64: 0 │ embargo_size ┼ Int64: 0 │ wd ┼ nothing │ fa ┼ nothing │ store_weight_path ┼ Bool: false │ strict ┴ Bool: false r ┼ ConditionalValueatRisk │ settings ┼ RiskMeasureSettings │ │ scale ┼ Float64: 1.0 │ │ ub ┼ nothing │ │ rke ┴ Bool: true │ alpha ┼ Float64: 0.05 │ w ┴ nothing r_sca ┼ nothing scorer ┼ HighestMeanScore │ scale ┴ Float64: 1.0 s_sca ┼ nothing n_best ┼ Int64: 1 ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx() n_iter ┼ Int64: 10 rng ┼ Random.TaskLocalRNG: Random.TaskLocalRNG() seed ┼ nothing train_score ┼ Bool: false rank_prediction ┼ Bool: false kwargs ┴ @NamedTuple{}: NamedTuple()Related
AbstractSearchCrossValidationEstimatorGridSearchCrossValidationSearchCrossValidationResultCrossValSearchScorer
References
- [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
rscv::RandomisedSearchCrossValidation, rd::ReturnsResult)Score rscv.n_iter random candidates of the search space of rscv on every fold of rscv.cv, and return the winner.
The search draws the trials through make_p_grid, and runs the search of GridSearchCrossValidation on them. Each trial is one candidate. So the count of fits grows with n_iter, not with the count of keys. The scheme, the scorer and the treatment of a failed fold are those of the grid search. Under a CombinatorialCrossValidation, the grid search scores each backtest path.
Mathematical definition
\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]
Where:
- $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
- $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
- $F$: Fold count, the count of the folds that
splitenumerates. - $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
- $\Theta$: Trial sequence, the points of the search space that
make_p_griddraws. It is the grid that the grid search of the trials scores. - $\mathcal{O}$: Oriented score under
r, fromoriented_scorewith the reductionr_scaand the keyword argumentskwargs. Each measure is oriented so that a higher score is always better, andr_scareduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better. - $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores
NaN, so a candidate that failed one is not in $\mathcal{C}$. - $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
- $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
- $\sigma$: Search scorer
scorer, a map from a score matrix to the position of its best column. The defaultHighestMeanScoretakes the column of greatest mean. - $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.
Algorithm
- Resolve the random number generator from
rscv.rngandrscv.seedthroughresolve_rng, givingrng. - Draw the trials from
rscv.pthroughmake_p_grid, withrscv.n_iterandrng. - Build a
GridSearchCrossValidationwith the trials as its grid and the other fields ofrscv. - Run
search_cross_validationof that grid search onoptandrd, and return its result.
Arguments
opt: The estimator to tune. The search reads it as configuration alone.rscv: The randomised search. It gives the spacep, the trial countn_iter, the generatorrngandseed, and the fields of a grid search: the schemecv, the risk measuresrand their reductionr_sca, thescorer, the executorex, thetrain_scoreflag and the keyword argumentskwargsoforiented_score.rd: The returns result to use.
Validation
- The checks of
make_p_gridonrscv.pandrscv.n_iter. - The checks of the grid search on
optand on each candidate.
Returns
res::SearchCrossValidationResult: The result of the grid search of the trials. Itslens_gridandval_gridhold one entry per trial, in the order of $\Theta$, andidxis $i^{\star}$. Itsrankedholds therscv.n_bestbest trials, best first. The trials come fromrng, so two runs give the samerankedonly whenrscv.seedis set.
Related
RandomisedSearchCrossValidationGridSearchCrossValidationmake_p_gridresolve_rngNonFiniteAllocationOptimisationEstimatorReturnsResult
References
- [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012). Section 1, Equation 4.
References
- [226]
- J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).