Base search cross validation

PortfolioOptimisers.SearchCrossValidationResult — Type
struct SearchCrossValidationResult{__T_opt, __T_test_scores, __T_train_scores, __T_lens_grid, __T_val_grid, __T_idx, __T_ranked, __T_rank_idx, __T_rank_scores, __T_rank_predictions, __T_path_ids} <: AbstractSearchCrossValidationResult

Result type for search-based cross-validation routines. Stores the optimal estimator, score matrices, parameter grid, and selected index for hyperparameter search.

The search keeps the n_best best candidates in rank order, as configurations, in ranked. It keeps no fitted result. Fit a kept candidate outside the search, as you fit opt. A candidate that failed a fold never enters ranked, so ranked holds fewer than n_best candidates when fewer finished every fold.

A search with rank_prediction = true also keeps, in rank_predictions[j], the prediction that it scored for candidate ranked[j]. A caller can give it to expected_risk or performance_summary, and get the numbers that the search used. With the default rank_prediction = false, rank_predictions is nothing. The positional constructor takes rank_predictions after rank_scores.

The scores of the kept candidate j over the rows of the scheme are test_scores[:, rank_idx[j]]. The Result keeps no summary of their spread. Compute it from the matrix, for example std(res.test_scores[:, res.rank_idx]; dims = 1). Under a MultipleRandomised scheme, path_ids gives the path of each row, so test_scores[path_ids .== k, rank_idx[j]] are the scores of candidate j over path k.

One ranking for each value

Under EachValue in r_sca or s_sca of the search, opt, idx, ranked, rank_idx, rank_scores and rank_predictions hold one entry for each value per candidate. The types of r, r_sca, scorer and s_sca decide the container:

r_scas_scaContainer of each field
a reduction, PerCandidate, or one measurea reduction, or one scorerOne ranking, the fields above.
a reduction, PerCandidate, or one measureEachValue() with L scorersA vector of L entries, in the order of scorer.
EachValue() with K measuresa reduction, or one scorerA vector of K entries, in the order of r.
EachValue() with K measuresEachValue() with L scorersA K × L matrix, row k for measure k and column l for scorer l.

So res.opt[k, l] is the winner of measure k under scorer l. Under r_sca = EachValue() or r_sca = PerCandidate(sca) with a vector of measures, test_scores is a vector of K matrices in the order of r, and train_scores takes the same shape. Each ranking keeps at most n_best candidates, and a candidate enters the ranking of a measure only when it finished every fold under that measure. A measure under which no candidate finished every fold gives an empty entry: opt[v] and idx[v] are nothing, and ranked[v], rank_idx[v] and rank_predictions[v] are empty. A prediction belongs to a candidate, so two rankings that keep the same candidate hold the same prediction, with no copy. A scorer that gives the index of one winner keeps one candidate, and its entry of rank_scores is nothing.

Fields

  • opt: Optimal estimator found by cross-validation, ranked[1]. Under EachValue, the winner of each value, or nothing for an empty entry.
  • test_scores: Test set scores, one row per fold or path and one column per candidate, oriented so that a higher score is better. Under a rule of AbstractSearchReduction in r_sca, EachValue or PerCandidate, with a vector of measures, one such matrix per measure, in the order of r.
  • train_scores: Training set scores, in the shape of test_scores, or nothing when the search records none. Under a CombinatorialCrossValidation, one matrix per path, and under a rule of AbstractSearchReduction in r_sca with a vector of measures, one of these per measure, so that train_scores[k][p] is path p of measure k.
  • lens_grid: Lenses of each candidate, one vector of lenses per column of the score matrices, in grid order.
  • val_grid: Values of each candidate, one tuple per column of the score matrices, in the order of the lenses of that candidate.
  • idx: Index of the optimal parameter configuration, rank_idx[1]. Under EachValue, the index of the winner of each value, or nothing for an empty entry.
  • ranked: The kept candidates as configurations, best first. Each one is built from the estimator that the search was given, through the lenses of its own grid point. The search keeps no fitted result. Under EachValue, one such vector for each value.
  • rank_idx: Column of the score matrices of each kept candidate, in the order of ranked. Under EachValue, one such vector for each value.
  • rank_scores: Value of candidate_values of each kept candidate, in the order of ranked, signed as test_scores: the value of one scorer, or the reduction through s_sca of a vector of scorers. Under PerCandidate, the value of each scorer is the combination of its values over the measures. nothing when the scorer gives the index of one winner. Under EachValue, one entry for each value.

Related

source
PortfolioOptimisers.AbstractSearchCrossValidationResult — Type
abstract type AbstractSearchCrossValidationResult <: AbstractResult

Abstract result type for search-based cross-validation routines. Serves as the parent for all result types produced by search cross-validation algorithms, encapsulating optimal estimator, score matrices, parameter grid, and selected index.

Interfaces

  • Subtypes must store the optimal estimator opt, the score matrices test_scores and train_scores, the parameter grid lens_grid and val_grid, and the selected index idx.
  • Subtypes must store the kept candidates in rank order, ranked, with their columns rank_idx and their scorer values rank_scores, and the path of each row of the score matrix, path_ids. opt is ranked[1], and idx is rank_idx[1].
source