Base search cross validation
PortfolioOptimisers.SearchCrossValidationResult — Type
struct SearchCrossValidationResult{__T_opt, __T_test_scores, __T_train_scores, __T_lens_grid, __T_val_grid, __T_idx, __T_ranked, __T_rank_idx, __T_rank_scores, __T_rank_predictions, __T_path_ids} <: AbstractSearchCrossValidationResultResult type for search-based cross-validation routines. Stores the optimal estimator, score matrices, parameter grid, and selected index for hyperparameter search.
The search keeps the n_best best candidates in rank order, as configurations, in ranked. It keeps no fitted result. Fit a kept candidate outside the search, as you fit opt. A candidate that failed a fold never enters ranked, so ranked holds fewer than n_best candidates when fewer finished every fold.
A search with rank_prediction = true also keeps, in rank_predictions[j], the prediction that it scored for candidate ranked[j]. A caller can give it to expected_risk or performance_summary, and get the numbers that the search used. With the default rank_prediction = false, rank_predictions is nothing. The positional constructor takes rank_predictions after rank_scores.
The scores of the kept candidate j over the rows of the scheme are test_scores[:, rank_idx[j]]. The Result keeps no summary of their spread. Compute it from the matrix, for example std(res.test_scores[:, res.rank_idx]; dims = 1). Under a MultipleRandomised scheme, path_ids gives the path of each row, so test_scores[path_ids .== k, rank_idx[j]] are the scores of candidate j over path k.
One ranking for each value
Under EachValue in r_sca or s_sca of the search, opt, idx, ranked, rank_idx, rank_scores and rank_predictions hold one entry for each value per candidate. The types of r, r_sca, scorer and s_sca decide the container:
r_sca | s_sca | Container of each field |
|---|---|---|
a reduction, PerCandidate, or one measure | a reduction, or one scorer | One ranking, the fields above. |
a reduction, PerCandidate, or one measure | EachValue() with L scorers | A vector of L entries, in the order of scorer. |
EachValue() with K measures | a reduction, or one scorer | A vector of K entries, in the order of r. |
EachValue() with K measures | EachValue() with L scorers | A K × L matrix, row k for measure k and column l for scorer l. |
So res.opt[k, l] is the winner of measure k under scorer l. Under r_sca = EachValue() or r_sca = PerCandidate(sca) with a vector of measures, test_scores is a vector of K matrices in the order of r, and train_scores takes the same shape. Each ranking keeps at most n_best candidates, and a candidate enters the ranking of a measure only when it finished every fold under that measure. A measure under which no candidate finished every fold gives an empty entry: opt[v] and idx[v] are nothing, and ranked[v], rank_idx[v] and rank_predictions[v] are empty. A prediction belongs to a candidate, so two rankings that keep the same candidate hold the same prediction, with no copy. A scorer that gives the index of one winner keeps one candidate, and its entry of rank_scores is nothing.
Fields
opt: Optimal estimator found by cross-validation,ranked[1]. UnderEachValue, the winner of each value, ornothingfor an empty entry.
test_scores: Test set scores, one row per fold or path and one column per candidate, oriented so that a higher score is better. Under a rule ofAbstractSearchReductioninr_sca,EachValueorPerCandidate, with a vector of measures, one such matrix per measure, in the order ofr.
train_scores: Training set scores, in the shape oftest_scores, ornothingwhen the search records none. Under aCombinatorialCrossValidation, one matrix per path, and under a rule ofAbstractSearchReductioninr_scawith a vector of measures, one of these per measure, so thattrain_scores[k][p]is pathpof measurek.
lens_grid: Lenses of each candidate, one vector of lenses per column of the score matrices, in grid order.
val_grid: Values of each candidate, one tuple per column of the score matrices, in the order of the lenses of that candidate.
idx: Index of the optimal parameter configuration,rank_idx[1]. UnderEachValue, the index of the winner of each value, ornothingfor an empty entry.
ranked: The kept candidates as configurations, best first. Each one is built from the estimator that the search was given, through the lenses of its own grid point. The search keeps no fitted result. UnderEachValue, one such vector for each value.
rank_idx: Column of the score matrices of each kept candidate, in the order ofranked. UnderEachValue, one such vector for each value.
rank_scores: Value ofcandidate_valuesof each kept candidate, in the order ofranked, signed astest_scores: the value of one scorer, or the reduction throughs_scaof a vector of scorers. UnderPerCandidate, the value of each scorer is the combination of its values over the measures.nothingwhen the scorer gives the index of one winner. UnderEachValue, one entry for each value.
rank_predictions: The prediction that the search scored for each kept candidate, in the order ofranked, the object thatfit_and_predictgave, ornothingwhen the search hasrank_prediction = false. Its type follows the scheme: aMultiPeriodPredictionResult, or aPopulationPredictionResultunder aMultipleRandomisedor aCombinatorialCrossValidationscheme. UnderEachValue, one such vector for each value, and two rankings that keep the same candidate hold the same object.
path_ids: Path of each row oftest_scores: the path of each fold under aMultipleRandomisedscheme,1:Munder aCombinatorialCrossValidation, whose rows are paths, andnothingfor a scheme with one path.
Related
PortfolioOptimisers.AbstractSearchCrossValidationResult — Type
abstract type AbstractSearchCrossValidationResult <: AbstractResultAbstract result type for search-based cross-validation routines. Serves as the parent for all result types produced by search cross-validation algorithms, encapsulating optimal estimator, score matrices, parameter grid, and selected index.
Interfaces
- Subtypes must store the optimal estimator
opt, the score matricestest_scoresandtrain_scores, the parameter gridlens_gridandval_grid, and the selected indexidx. - Subtypes must store the kept candidates in rank order,
ranked, with their columnsrank_idxand their scorer valuesrank_scores, and the path of each row of the score matrix,path_ids.optisranked[1], andidxisrank_idx[1].