Randomised search cross validation

PortfolioOptimisers.RandomisedSearchCrossValidation — Type
struct RandomisedSearchCrossValidation{__T_p, __T_cv, __T_r, __T_r_sca, __T_scorer, __T_s_sca, __T_n_best, __T_ex, __T_n_iter, __T_rng, __T_seed, __T_train_score, __T_rank_prediction, __T_kwargs} <: AbstractSearchCrossValidationEstimator

Randomised search cross-validation estimator for portfolio optimisation. It draws n_iter trials from the search space, each a joint value of every key of one parameter set, through make_p_grid. Then it scores the trials with the search of GridSearchCrossValidation, and selects one with the scorer. The count of fits grows with n_iter, not with the count of keys.

Fields

  • p: Hyperparameter search grid.
  • cv: Cross-validation estimator.
  • r: Risk measure or vector of risk measures.
  • r_sca: Reduction of the oriented scores of a vector of risk measures in r to one score in each fold or path, a Sca_VecToScaM, or nothing for the sum that SumScalariser gives. EachValue keeps one score matrix per measure, and gives one ranking per measure. PerCandidate keeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert when r holds one measure.
  • scorer: Search scorer, a CrossValSearchScorer, or a vector of them. Given the orientation-normalised score matrix (rows = CV splits, columns = parameter sets), it gives one value per parameter set, bigger is better, or the column index of the best parameter set. The matrix is normalised so that higher is always better, whatever the risk measure, so a scorer selects the largest aggregate score (see CrossValidationSearchScorer).
  • s_sca: Reduction of the values of a vector of search scorers in scorer to one value per candidate, after the values of each scorer are multiplied by its scorer_scale, a Sca_VecToScaM, or nothing for the sum that SumScalariser gives. EachValue lets each scorer order the candidates alone, and gives one ranking per scorer. Inert when scorer holds one scorer.
  • n_best: Count of the best candidates that the search keeps in rank order. A count larger than the count of the candidates that finished every fold keeps those candidates alone. A scorer that gives the index of one winner keeps one candidate.
  • ex: Parallel execution strategy.
  • n_iter: Trial count, the count of the candidates that the randomised search draws. Each trial sets every key of one parameter set. When every value is a vector and the space holds fewer points, the search takes each point once.
  • rng: Random number generator.
  • seed: Optional seed. If set, the draws come from a copy of rng seeded with it, so rng does not advance and every call draws the same values. If nothing, the draws come from rng itself.
  • train_score: Whether to also compute the training set score.
  • rank_prediction: Whether the Result also keeps, in rank_predictions, the prediction that the search scored for each kept candidate. With false, the search stores no prediction.
  • kwargs: More keyword arguments.

Constructors

RandomisedSearchCrossValidation(    p::Union{AbstractVector{<:Pair{<:GSCVKey, <:RSCVVal}},             AbstractVector{<:AbstractVector{<:Pair{<:GSCVKey,                                                    <:RSCVVal}}},             AbstractDict{<:GSCVKey, <:RSCVVal},             AbstractVector{<:AbstractDict{<:GSCVKey,                                           <:RSCVVal}}};    cv::CVE_Onl = KFold(),    r::BaseRM_VecBaseRM = ConditionalValueatRisk(),    r_sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction} = nothing,    scorer::Union{CrossValSearchScorer, AbstractVector} = HighestMeanScore(),    s_sca::Union{Option{<:Sca_VecToScaM}, EachValue} = nothing,    n_best::Integer = 1,    ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(),    n_iter::Integer = 10,    rng::Random.AbstractRNG = Random.default_rng(),    seed::Option{<:Integer} = nothing,    train_score::Bool = false,    rank_prediction::Bool = false,    kwargs::NamedTuple = (;),) -> RandomisedSearchCrossValidation

Positional and keyword arguments correspond to the struct's fields. The positional constructor takes rank_prediction after train_score, so a positional call without it raises a MethodError. The outer constructor takes a vector of parameter sets whose types differ, such as a vector of pairs and a vector of pairs with other value types. Such a literal infers the element type Vector, so the constructor narrows it through narrow_mixed_vector.

The search draws its trials from rng, so the trials, and the order of the n_best kept candidates, are the same in two runs only when seed is set.

Multiplicity

r takes one risk measure or a vector of them, and r_sca reduces a vector, as in GridSearchCrossValidation. For one measure r_sca is inert. scorer takes one search scorer or a vector of them, and s_sca reduces a vector, as in GridSearchCrossValidation. For one scorer s_sca is inert. EachValue in r_sca or s_sca gives one ranking for each value per candidate, and PerCandidate in r_sca combines the measures after the search scorer, as in GridSearchCrossValidation.

rank_prediction = true keeps the prediction that the search scored for each kept candidate in rank_predictions, as in GridSearchCrossValidation.

Warning

With rank_prediction = true, the peak memory of the search holds the predictions of every trial, and each prediction holds one optimisation Result for each fold.

Validation

  • !isempty(p).
  • If p is a vector of parameter sets: each element must not be empty.
  • All keys in p must be of type GSCVKey (i.e. String, Symbol, or Integer).
  • All values in p must be of type RSCVVal (i.e. an AbstractVector or Distributions.Distribution).
  • n_iter > 0 and finite.
  • n_best > 0 and finite, through assert_search_scorer.
  • If scorer is a vector: it is not empty, and each member is a CrossValSearchScorer.
  • kwargs must not contain sca, through assert_search_reduction. The reduction is the field r_sca.

When the search runs, it refuses a scorer that gives an index with n_best > 1, and a member of a vector that gives an index, as in GridSearchCrossValidation.

A search grid can turn a plain count into a fraction

A field that takes a CountOf or a FractionOf also takes a plain number, and count_or_fraction reads an Integer as a count and any other real number as a fraction. Julia promotes the elements of a vector literal to one type before the library reads them. So ["opt.ret.ucs.alg.method.active" => [3, 5, 8], "r.alpha" => [0.05, 0.1]] is a Vector{Pair{String, Vector{Float64}}}, and the counts arrive as 3.0, 5.0 and 8.0. The field then reads them as fractions, and the check of FractionOf refuses them. State the grid in the wrappers, such as [CountOf(3), CountOf(5), FractionOf(0.2)]. A wrapper keeps its type in every container.

Examples

julia> RandomisedSearchCrossValidation(Dict("alpha" => [0.1, 0.2, 0.3],                                            "beta" => Normal(1.0, 0.5)))RandomisedSearchCrossValidation                p ┼ Dict{String, Any}: Dict{String, Any}("alpha" => [0.1, 0.2, 0.3], "beta" => Normal(1.0, 0.5))               cv ┼ KFold                  │                   n ┼ Int64: 5                  │         purged_size ┼ Int64: 0                  │        embargo_size ┼ Int64: 0                  │                  wd ┼ nothing                  │                  fa ┼ nothing                  │   store_weight_path ┼ Bool: false                  │              strict ┴ Bool: false                r ┼ ConditionalValueatRisk                  │   settings ┼ RiskMeasureSettings                  │            │   scale ┼ Float64: 1.0                  │            │      ub ┼ nothing                  │            │     rke ┴ Bool: true                  │      alpha ┼ Float64: 0.05                  │          w ┴ nothing            r_sca ┼ nothing           scorer ┼ HighestMeanScore                  │   scale ┴ Float64: 1.0            s_sca ┼ nothing           n_best ┼ Int64: 1               ex ┼ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()           n_iter ┼ Int64: 10              rng ┼ Random.TaskLocalRNG: Random.TaskLocalRNG()             seed ┼ nothing      train_score ┼ Bool: false  rank_prediction ┼ Bool: false           kwargs ┴ @NamedTuple{}: NamedTuple()

Related

References

  • [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).
source
PortfolioOptimisers.search_cross_validation — Method
search_cross_validation(opt::NonFiniteAllocationOptimisationEstimator,
                        rscv::RandomisedSearchCrossValidation, rd::ReturnsResult)

Score rscv.n_iter random candidates of the search space of rscv on every fold of rscv.cv, and return the winner.

The search draws the trials through make_p_grid, and runs the search of GridSearchCrossValidation on them. Each trial is one candidate. So the count of fits grows with n_iter, not with the count of keys. The scheme, the scorer and the treatment of a failed fold are those of the grid search. Under a CombinatorialCrossValidation, the grid search scores each backtest path.

Mathematical definition

\[\begin{align} S_{fi} &= \mathcal{O}\left(\hat{P}_{f}(\theta_i)\right)\,, \quad f = 1, \ldots, F\,, \quad \theta_i \in \Theta\,, \\ \mathcal{C} &= \left\{ i : S_{fi} \text{ is finite for every } f \right\}\,, \\ i^{\star} &= c_{\sigma\left(\mathbf{S}_{:,\,\mathcal{C}}\right)}\,. \end{align}\]

Where:

  • $S_{fi}$: Fold score, the test score of candidate $i$ on fold $f$, and the entry of the score matrix $\mathbf{S}$, $F \times |\Theta|$.
  • $\hat{P}_{f}(\theta_i)$: Fold prediction, the returns over the test window of fold $f$ of the weights that $\theta_i$ fits on the training window of the fold.
  • $F$: Fold count, the count of the folds that split enumerates.
  • $\theta_i$: Candidate $i$, the estimator with each tuned parameter set to its value at grid point $i$. Column $i$ of every score matrix belongs to it.
  • $\Theta$: Trial sequence, the points of the search space that make_p_grid draws. It is the grid that the grid search of the trials scores.
  • $\mathcal{O}$: Oriented score under r, from oriented_score with the reduction r_sca and the keyword arguments kwargs. Each measure is oriented so that a higher score is always better, and r_sca reduces a vector of measures to one score. For one measure, $\mathcal{O}$ is its expected risk, negated when a smaller value is better.
  • $\mathcal{C}$: Finite candidates, the candidates whose every score is finite. A failed fold scores NaN, so a candidate that failed one is not in $\mathcal{C}$.
  • $\mathbf{S}_{:,\,\mathcal{C}}$: Finite columns, the columns of $\mathbf{S}$ at the finite candidates, in grid order.
  • $c_j$: Finite candidate $j$, the $j$-th smallest entry of $\mathcal{C}$.
  • $\sigma$: Search scorer scorer, a map from a score matrix to the position of its best column. The default HighestMeanScore takes the column of greatest mean.
  • $i^{\star}$: Selected candidate, the position in the grid of the candidate that the search returns.

Algorithm

  1. Resolve the random number generator from rscv.rng and rscv.seed through resolve_rng, giving rng.
  2. Draw the trials from rscv.p through make_p_grid, with rscv.n_iter and rng.
  3. Build a GridSearchCrossValidation with the trials as its grid and the other fields of rscv.
  4. Run search_cross_validation of that grid search on opt and rd, and return its result.

Arguments

  • opt: The estimator to tune. The search reads it as configuration alone.
  • rscv: The randomised search. It gives the space p, the trial count n_iter, the generator rng and seed, and the fields of a grid search: the scheme cv, the risk measures r and their reduction r_sca, the scorer, the executor ex, the train_score flag and the keyword arguments kwargs of oriented_score.
  • rd: The returns result to use.

Validation

  • The checks of make_p_grid on rscv.p and rscv.n_iter.
  • The checks of the grid search on opt and on each candidate.

Returns

  • res::SearchCrossValidationResult: The result of the grid search of the trials. Its lens_grid and val_grid hold one entry per trial, in the order of $\Theta$, and idx is $i^{\star}$. Its ranked holds the rscv.n_best best trials, best first. The trials come from rng, so two runs give the same ranked only when rscv.seed is set.

Related

References

  • [226] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012). Section 1, Equation 4.
source

References

[226]
J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).