Base search cross validation: private API
PortfolioOptimisers.AbstractSearchCrossValidationEstimator — Type
abstract type AbstractSearchCrossValidationEstimator <: AbstractEstimatorAbstract supertype for all search-based cross-validation estimators.
Subtypes implement hyperparameter search strategies (e.g. grid search, randomised search) that use cross-validation to select the best estimator configuration.
Related
PortfolioOptimisers.assert_search_scorer — Function
assert_search_scorer(scorer, n_best::Integer)Refuse a scorer slot or a count of kept candidates that a search estimator cannot use, when the estimator is built.
The configuration alone decides these rules, so the constructors of GridSearchCrossValidation and RandomisedSearchCrossValidation call this check. The form that a scorer gives, a value per candidate or an index, is known only when the search runs, so finite_candidate_order and candidate_values refuse an index that the search cannot use. The constructors check the type of each member of a vector through narrow_mixed_vector.
Arguments
scorer: One search scorer, or a vector of them.n_best: The count of the candidates that the search keeps.
Validation
n_best > 0and finite.- If
scoreris a vector:!isempty(scorer).
Returns
nothing.
Related
PortfolioOptimisers.assert_search_reduction — Function
assert_search_reduction(kwargs::NamedTuple)Refuse a sca in the keyword arguments of a search estimator, when the estimator is built.
The constructors of GridSearchCrossValidation and RandomisedSearchCrossValidation call this check. The search forwards kwargs to oriented_score, and the reduction of a vector of risk measures is the field r_sca. A reader of a risk takes every keyword that it does not declare, so it would ignore a sca in kwargs without an error. The keys of a NamedTuple are part of its type, so the check costs nothing at run time.
Arguments
kwargs: The keyword arguments of the search.
Validation
kwargsmust not containsca. The error is aConflictingArgumentErrorthat namesr_sca.
Returns
nothing.
Related
assert_path_reduction: the same check for a reader of a population.oriented_score
PortfolioOptimisers.search_result — Function
search_result(base, gscv::GridSearchCrossValidation,
test_scores::Union{MatNum, AbstractVector{<:MatNum}}, train_scores,
lens_grid::AbstractVector, val_grid::AbstractVector,
path_ids::Option{<:AbstractVector{<:Integer}},
predictions::Option{<:AbstractVector{<:AbstractPredictionResult}})Build the Result of a search from its score matrices: order the candidates, keep the best gscv.n_best, and build each kept candidate from the estimator that the search was given.
The four methods of search_cross_validation on an optimiser and on a Pipeline, batch and combinatorial, call it after their score loop. Each kept candidate is built from base through the lenses of its own grid point. A grid that joins parameter sets with different keys therefore gives each kept candidate the values of its own set alone, and none of the values of opt.
Algorithm
- Order the finite candidates by the values of
gscv.scorer, reduced throughgscv.s_scafor a vector of scorers, and keep the firstgscv.n_bestthroughfinite_candidate_order, givingrank_idxandrank_scores. UnderEachValue, these hold one ranking for each value. UnderPerCandidate,gscv.r_scacombines the values of the measures first. - Build each kept candidate
kofrank_idxfrombasethroughranked_candidates, givingranked,optandidx. - Keep the prediction of each kept candidate through
ranked_predictions, givingrank_predictions, ornothingwhenpredictionsisnothing. - Return a
SearchCrossValidationResultwithopt = ranked[1]andidx = rank_idx[1], or these for each ranking.
Arguments
base: The estimator, or thePipeline, that the search tunes.gscv: The grid search. It givesr_sca,scorer,s_scaandn_best.test_scores: The raw test score matrix, or one matrix for each measure under a rule ofAbstractSearchReduction.train_scores: The train scores, ornothing.lens_grid: The lenses of each grid point.val_grid: The values of each grid point.path_ids: The path of each row oftest_scores, ornothingfor a scheme with one path.predictions: The prediction of each candidate, in grid order, fromallocate_search_predictions, ornothingwhengscv.rank_predictionisfalse.
Validation
- The checks of
finite_candidate_order.
Returns
res::SearchCrossValidationResult.
Related
PortfolioOptimisers.assert_search_ranking — Function
assert_search_ranking(opt::AbstractEstimator, idx::Integer, ranked::AbstractVector,
rank_idx::AbstractVector{<:Integer}, rank_scores)
assert_search_ranking(opt::AbstractArray, idx::AbstractArray, ranked::AbstractArray,
rank_idx::AbstractArray{<:AbstractVector{<:Integer}},
rank_scores::AbstractArray)Refuse the kept fields of a SearchCrossValidationResult when they do not agree.
The first form checks one ranking, the form of a search of one value. The second form checks one ranking for each value, the form of EachValue. An entry of the second form can be empty: a measure under which no candidate finished every fold has no winner. Its opt and idx are then nothing.
Arguments
opt,idx,ranked,rank_idx,rank_scores: The fields of the Result, seeSearchCrossValidationResult.
Validation
- Each ranking keeps one entry of
ranked, and one entry ofrank_scoreswhen they are notnothing, for each entry ofrank_idx. Otherwise aDimensionMismatch. - For each ranking that is not empty:
opt == ranked[1]andidx == rank_idx[1]. Otherwise aDimensionMismatch. - The first form: the ranking is not empty. Otherwise a
DimensionMismatch. - The second form:
opt,idx,rankedandrank_scoreshave the size ofrank_idx. Otherwise aDimensionMismatch. An empty entry hasoptandidxequal tonothing. Otherwise aDimensionMismatch. At least one entry is not empty. Otherwise anIsEmptyError.
Returns
nothing.
Related
PortfolioOptimisers.assert_search_entry — Function
assert_search_entry(est, cv)Refuse an estimator that is not the configuration alone at the entry to a search, once, before any candidate is built.
A search scores every candidate through the one fold loop, and a lens is applied before that loop's warm-up, so a cold estimator seeds one state per candidate and nothing is shared or reset. A warm one is refused here, with an error that names the field, rather than inside the candidate loop, where the workers of gscv.ex would raise it up to once per candidate. Under a plain scheme the walk is online_entry_state, because a search tunes the configuration alone whatever the scheme does with it; under an Online Scheme it is the whole assert_online_entry, so a schedule on a stateful field is refused at the entry too.
Arguments
est: The estimator the search tunes.cv: The scheme the search scores over.
Validation
- No
cachein the tree ofestholds a state. AnArgumentErrornaming the field is thrown otherwise. - Under an Online Scheme, everything
assert_online_entryrefuses.
Related
PortfolioOptimisers.search_candidate — Function
search_candidate(opt, lenses, vals)Build one candidate of a search: opt with each lens set to its value, in order.
Arguments
opt: The estimator the search tunes.lenses: The lenses of one grid point.vals: The value of each lens at that grid point.
Returns
- The candidate estimator.
Related
PortfolioOptimisers.assert_search_candidates — Method
assert_search_candidates(opt, lens_grid, val_grid) -> NothingRun the entry checks of cross_val_predict on every candidate of a search, before any candidate is scored.
A search scores each candidate through the fold loop directly, so the checks that the one-shot cross_val_predict runs at its entry must run here. They run on each candidate and not on opt alone, because a lens can write a value the checks refuse, such as a precomputed prior, and it can also replace one.
Arguments
opt: The estimator the search tunes.lens_grid: The lenses of each grid point.val_grid: The values of each grid point.
Validation
- Every candidate passes
assert_internal_optimiserandassert_external_optimiser.
Returns
nothing.
Related
PortfolioOptimisers.score_rows — Function
score_rows(cvr::CrossValidationResult)
score_rows(cvr::MultipleRandomisedResult)The rows of a search's score matrix that a candidate's predictions fill, in the order the fold loop returns them.
The score matrix is indexed by split: row j is the j-th fold the scheme enumerates, and a reader lines the rows up with split's enumeration. A contiguous scheme returns one MultiPeriodPredictionResult whose predictions are in that order, so its rows are every row in order. A MultipleRandomised returns one MultiPeriodPredictionResult per path, each sorted by the start of its test window through sort_predictions, so its rows are one vector per path: the split rows of that path, in the order the path's predictions come back. write_candidate_scores! zips a candidate's predictions against these rows, so the matrix keeps split's order under either shape.
Arguments
cvr: The split the search scores over.
Returns
rows: The row indices, one range for a contiguous scheme, or one vector of indices per path for a multiple-randomised one.
Related
PortfolioOptimisers.score_path_ids — Function
score_path_ids(cvr::CrossValidationResult)
score_path_ids(cvr::MultipleRandomisedResult)The path of each row of a search's score matrix, for the path_ids field of SearchCrossValidationResult.
Row j of the score matrix is fold j of split, see score_rows. A MultipleRandomised scheme runs one walk-forward per asset subset, and cvr.path_ids[j] is the path of fold j, so the method returns cvr.path_ids. Every other scheme of this method has one path, so the method returns nothing. The combinatorial methods of search_cross_validation score one row per path, and they give 1:M themselves.
Arguments
cvr: The split the search scores over.
Returns
path_ids::Option{<:AbstractVector{<:Integer}}: The path of each row, ornothingfor a scheme with one path.
Related
PortfolioOptimisers.fold_train_returns — Function
fold_train_returns(cv::CrossValidationResult, rd::ReturnsResult, k::Integer)
fold_train_returns(cv::MultipleRandomisedResult, rd::ReturnsResult, k::Integer)View out of rd the returns that fold k of cv was fitted on.
The rows are the fold's own train_idx. The columns are every asset, except under a MultipleRandomised, whose folds each run on a drawn subset of the universe and which records that subset on asset_idx. So the view is the returns the fold's own optimise call was handed, which is what an optimiser that keeps its prior result stores on the result's pr: measured over the eleven schemes the library ships, res.pr.X of a fold equals this view element for element.
candidate_train_score reads it, and only for a result that holds no prior result of its own.
Arguments
cv: The search's split.rd: The returns the search ran over.k: The fold's index intocv.train_idx.
Returns
X::SubArray: The fold's training returns.
Related
PortfolioOptimisers.candidate_train_score — Function
candidate_train_score(r_sca, r, res::OptimisationResult, X::Option{<:MatNum}, kwargs)Score a fold's fitted result over its own training sample, oriented so that a higher score is better.
The score is the form of oriented_score for a fitted result, so it is oriented and reduced as the test score is. This function selects the sample: the prior result of res, or X.
A result that holds a prior result is scored through it, so the figure resolves the measure exactly as the fit did: an unstated slot falls back to the prior result's own field, and a Deferred Quantity is fitted. This is the call the search has always made, and it is kept for every such result, because a bare matrix would opt out of that resolution and would refuse a measure like Variance whose sigma the prior result fills.
A result that carries none is scored over X, the fold's own training returns from fold_train_returns. Before this the fallback yielded nothing and the call raised a MethodError, so no such result could be scored at all — which is every optimiser whose fit reads no returns to keep, the online portfolio selection family first among them.
X is nothing where the search holds no trustworthy window, which today is the Pipeline's own search alone: its steps transform the data per fold, so the returns the optimiser was handed are not the returns the split names, and a selector narrows the universe by a rule the split does not record. That arm therefore scores through the prior result whatever the result holds, which is what every arm did before, and a result with no prior result under a Pipeline still meets the refusal.
A result that exposes no pr property at all keeps the refusal extract_pr already gave it.
Arguments
r_sca: Reduction of the oriented scores of a vector of risk measures inrto one score in each fold or path, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuekeeps one score matrix per measure, and gives one ranking per measure.PerCandidatekeeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert whenrholds one measure.r: The risk measure the search scores with, or a vector of them.res: The fold's fitted result.X: The fold's training returns, ornothingwhere the search holds none.kwargs: The keyword arguments forwarded tooriented_score.
Returns
score::Number: The oriented score of the result over its own training sample.
Related
PortfolioOptimisers.write_candidate_scores! — Function
write_candidate_scores!(test_scores, train_scores, i::Integer,
predictions::MultiPeriodPredictionResult,
rows::AbstractVector{<:Integer}, train_X, r_sca, r, kwargs)
write_candidate_scores!(test_scores, train_scores, i::Integer,
path::MultiPeriodPredictionResult, row::Integer, train_X, r_sca,
r, kwargs)
write_candidate_scores!(test_scores, train_scores, i::Integer,
predictions::PopulationPredictionResult, rows::AbstractVector,
train_X, r_sca, r, kwargs)Write the scores of candidate i into column i of a search's score matrices.
Every method of search_cross_validation writes through this function, and the type of each entry of rows selects the layout:
- A vector of rows, one per fold. The candidate's predictions are what
fit_and_predictreturned over the search's scheme, one per fold, androwsisscore_rowsof that scheme's split, so prediction and row are zipped and the column reads insplit's order. Each fold's test score isoriented_scoreof the fold's prediction. Its train score, written only whentrain_scoresis a matrix, iscandidate_train_scoreof the fold's fitted result. - One row for a whole path. A
CombinatorialCrossValidationpools the test groups of a path into one series, so the path's test score isoriented_scoreof the path, at its own row. The folds of one path train on different windows, so the train scores stay one per fold, in the matrix of the path,train_scores[row]. - A population. Each path is written against its own entry of
rows, by one of the two methods above. Under aMultipleRandomisedscheme an entry is a vector of rows, and under aCombinatorialCrossValidationit is one row.
Every score is oriented, so a higher score is always better whatever measures r holds, which is the orientation a CrossValidationSearchScorer relies on. r_sca reduces a vector of measures to one score. Under a rule of AbstractSearchReduction with a vector of measures, each measure keeps its own score, and store_search_score! writes score k into matrix k. A fold or a path that failed scores NaN, and finite_candidate_order keeps that column from the scorer.
Arguments
test_scores: The test score matrix, with one row per fold or per path, or one such matrix per measure under a rule ofAbstractSearchReduction.train_scores: The train score matrix of the same shape, a vector of one matrix per path under aCombinatorialCrossValidation, ornothingwhen the search records none. Under a rule ofAbstractSearchReduction, one of these per measure.i: The candidate's column.predictions,path: The candidate's predictions over the scheme, or one path of them.rows,row: The rows the predictions fill.train_X: The training views offold_train_returns, indexed by the same row the scores are written at, ornothingwhen the search holds no trustworthy window. Under aCombinatorialCrossValidation, the entry of a row holds one view per fold of the path.r_sca: Reduction of the oriented scores of a vector of risk measures inrto one score in each fold or path, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuekeeps one score matrix per measure, and gives one ranking per measure.PerCandidatekeeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert whenrholds one measure.r: The risk measure the search scores with, or a vector of them.kwargs: The keyword arguments forwarded tooriented_score.
Related
PortfolioOptimisers.allocate_search_scores — Function
allocate_search_scores(gscv::GridSearchCrossValidation, T::Type, test_rows::Integer,
train_rows::Union{Integer, AbstractVector{<:Integer}},
N::Integer)Allocate the test and the train scores of a search, with N columns, one per candidate.
The four methods of search_cross_validation on an optimiser and on a Pipeline, batch and combinatorial, call it. search_score_array allocates each array, and the types of gscv.r_sca and gscv.r select its method. Under a rule of AbstractSearchReduction in r_sca, EachValue or PerCandidate, with a vector of measures, each array holds one entry per measure.
Arguments
gscv: The grid search. It givesr_sca,randtrain_score.T: The element type of the scores.test_rows: The row count of the test scores: the fold count, or the path count under aCombinatorialCrossValidation.train_rows: The row count of the train scores, or one row count per path under aCombinatorialCrossValidation.N: The candidate count.
Returns
test_scores: The test scores.train_scores: The train scores, ornothingwhengscv.train_scoreisfalse.
Related
PortfolioOptimisers.search_score_array — Function
search_score_array(r_sca, r::BaseRM_VecBaseRM, T::Type, M::Integer, N::Integer)
search_score_array(::AbstractSearchReduction, rs::VecBaseRM, T::Type, M::Integer,
N::Integer)
search_score_array(r_sca, r::BaseRM_VecBaseRM, T::Type, Ms::AbstractVector{<:Integer},
N::Integer)
search_score_array(::AbstractSearchReduction, rs::VecBaseRM, T::Type,
Ms::AbstractVector{<:Integer}, N::Integer)Allocate one score array of a search, uninitialised.
A reduction to one score, or one measure, gives one M × N matrix, and a vector of row counts gives one matrix per path. A rule of AbstractSearchReduction, such as EachValue or PerCandidate, with a vector of K measures keeps one score for each measure, so it gives a vector of K such arrays, in the order of rs: scores[k] is a matrix, or scores[k][p] is the matrix of path p. The types of r_sca and r select the method, so the shape is fixed at compile time.
Arguments
r_sca: Reduction of the oriented scores of a vector of risk measures inrto one score in each fold or path, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.EachValuekeeps one score matrix per measure, and gives one ranking per measure.PerCandidatekeeps one score matrix per measure, and combines the measures once per candidate, after the search scorer. Inert whenrholds one measure.r: The risk measure the search scores with, or a vector of them.rs: A vector of risk measures.T: The element type of the scores.M: The row count.Ms: One row count per path.N: The candidate count.
Returns
scores::Matrix{T}: OneM × Nmatrix.scores::Vector{Matrix{T}}: OneMs[p] × Nmatrix per pathp, or oneM × Nmatrix per measure under a rule ofAbstractSearchReduction.scores::Vector{Vector{Matrix{T}}}: Under a rule ofAbstractSearchReduction, one matrix per measurekand pathp.
Related
PortfolioOptimisers.store_search_score! — Function
store_search_score!(scores::MatNum, j::Integer, i::Integer, v::Number)
store_search_score!(scores::AbstractVector{<:MatNum}, j::Integer, i::Integer, v::VecNum)Store the score of candidate i at row j of a score array of a search.
write_candidate_scores! calls it with the return of oriented_score. One score goes into one matrix. Under a rule of AbstractSearchReduction with a vector of measures, the score is a vector of one value per measure, and value k goes into matrix k.
Arguments
scores: One score matrix, or one matrix per measure.j: The row: a fold, a path, or a fold of a path.i: The column of the candidate.v: The score, or one score per measure.
Returns
nothing.
Related
PortfolioOptimisers.path_train_scores — Function
path_train_scores(train_scores::AbstractVector{<:MatNum}, p::Integer)
path_train_scores(train_scores::AbstractVector{<:AbstractVector{<:MatNum}}, p::Integer)Give the train scores of path p of a combinatorial search, one row per fold of the path.
The train scores hold one matrix per path. Under a rule of AbstractSearchReduction with a vector of measures they hold one such vector per measure, train_scores[k][p], so the scores of path p are one matrix per measure.
Arguments
train_scores: The train scores of a combinatorial search.p: The path.
Returns
scores: The matrix of pathp, or one matrix of pathpper measure.
Related
PortfolioOptimisers.parse_lens — Function
parse_lens(key::AbstractString)Parse a hyperparameter key string into an Accessors.jl lens.
Converts a dotted string path (e.g., "opt.pe.ce") into a composable lens for getting and setting nested fields of an estimator object.
Arguments
key: Dotted field path string.
Validation
- String keys longer than
EQUATION_LIMITS[].max_lengthare rejected beforeMeta.parse. - Keys deeper than
EQUATION_LIMITS[].max_depthare rejected before the lens-building walk. A string key is measured afterMeta.parse, so one depth bound holds for every shape ofkey.
Returns
- Composed Accessors.jl lens.
Related
PortfolioOptimisers.assert_search_grid_cap — Function
assert_search_grid_cap(total::Integer, detail::AbstractString)
assert_search_grid_cap(estval::Union{AbstractVector{<:Pair{<:Any, <:AbstractVector}},
AbstractDict{<:Any, <:AbstractVector}})Assert the total number of search-grid candidates does not exceed the active max_search_grid ceiling.
The grid is an Iterators.product materialised by collect, so k tuned parameters of N values each cost N^k candidates and N^k full cross-validated fits. A per-parameter check never sees that product — the same reasoning that gave the frontier sweep assert_frontier_sweep_cap — so the cap is asserted where the grid is formed, before the collect.
Concatenated parameter sets are a sum of products, not a product: each set is capped as it is built, and the concatenated total is capped again on the way out.
Arguments
total: Candidate count already computed (aBigInton the product path, sincekparameters at the ceiling overflow anInt64before the check reads it).detail: Phrase naming howtotalwas made, interpolated into the message.estval: The parameter grid itself,key => valuespairs or a dictionary from key to values. The count of each value vector is a factor of the product.
Validation
- Every factor holds at least one value. An
IsEmptyErrornames the first key whose value vector is empty, because the product of an empty set is an empty grid. - The count does not exceed
RESOURCE_LIMITS[].max_search_grid. ADomainErrornames the count, the factors that made it, and the knob that raises the ceiling.
Returns
nothing.
Related
PortfolioOptimisers._expr_to_lens — Function
_expr_to_lens(ex::Symbol)Convert a bare symbol into a PropertyLens for field access.
Base case for the lens-building recursion: a bare symbol maps directly to an Accessors.PropertyLens.
Arguments
ex::Symbol: A field name symbol.
Returns
Accessors.PropertyLensfor the symbol.
Related
PortfolioOptimisers._eval_index — Function
_eval_index(x)Evaluate a literal index node in the AST without runtime eval.
Converts integer, symbol, or vector expression AST nodes to concrete index values for use in Accessors.IndexLens.
Arguments
x::Integer: An integer index.x::Symbol: A symbolic index.ex::Expr: A vector expression (:vecthead).
Returns
- The evaluated index value.
Related
PortfolioOptimisers.expr_to_lens_chain — Function
expr_to_lens_chain(ex)Convert a Julia expression to a chain of lens accessors.
Internal helper for parsing hyperparameter key strings into composable Accessors.jl lenses.
Arguments
ex: Julia expression representing a field access chain.
Returns
- Composed lens.
Related