Search scorers

PortfolioOptimisers.HighestMeanScore — Type
struct HighestMeanScore{__T_scale} <: CrossValidationSearchScorer

A CrossValidationSearchScorer that selects the parameter set with the highest mean score across cross-validation splits.

When called with a score matrix (rows = CV splits, columns = parameter sets), it returns the column index with the largest mean score. Its candidate_scores are the column means, so a search orders its candidates by the mean score. A tie goes to the first column, in the index form and in the order.

The scale is the weight of the scorer in a vector of search scorers, the same word and mechanism as settings.scale of a risk measure. A negative scale is a negative weight. The scale is inert for one scorer, so neither form above reads it.

Fields

  • scale: Weight of this search scorer in a vector of search scorers, read through scorer_scale. A negative scale is a negative weight. It is inert for one scorer.

Constructors

HighestMeanScore(; scale::Number = 1.0) -> HighestMeanScore

Keyword arguments correspond to the fields above.

Validation

Examples

julia> scorer = PortfolioOptimisers.HighestMeanScore();julia> scores = [0.5 0.8; 0.6 0.7];julia> scorer(scores)2julia> PortfolioOptimisers.candidate_scores(scorer, scores)2-element Vector{Float64}: 0.55 0.75

Related

source
PortfolioOptimisers.CrossValidationSearchScorer — Type
abstract type CrossValidationSearchScorer <: AbstractCrossValidationScorer

Defines the interface for scoring strategies used in search cross-validation. Implementations select the optimal parameter set based on cross-validation scores.

It is the branch of AbstractCrossValidationScorer that selects a candidate of a search. Its sibling PredictionScorer selects a path of a cross-validation.

Interfaces

  • candidate_scores(::CrossValidationSearchScorer, scores::AbstractMatrix): Returns one value per column of scores, bigger is better. The search orders the candidates by this value, so it can keep the n_best best ones, and it can combine the values of a vector of scorers through s_sca. A subtype that defines no method gives nothing, and the search then calls the index form below and keeps one candidate.
  • (::CrossValidationSearchScorer)(scores::AbstractMatrix): Returns the index of the optimal parameter set, the column with the best (highest) aggregate score. It must equal argmax of candidate_scores when the subtype defines both.
  • scorer_scale(::CrossValidationSearchScorer): Optional. Returns the weight of the scorer in a vector of scorers. The fallback gives 1, so a subtype with a scale field defines this method to read it.

Arguments

  • scores: Matrix of scores, where each column corresponds to a parameter set and each row to a cross-validation split. The matrix is orientation-normalised: risk-measure scores are negated so that higher is always better, whatever measure r is. A scorer therefore returns the index of the column with the best (highest) aggregate score.

Returns

  • candidate_scores: VecNum, one value per column of the matrix the scorer received, in the order of the columns, and no value NaN. The search refuses a vector of another length with a DimensionMismatch, and a NaN with an IsNonFiniteError.
  • The index form: Int, the index of the optimal parameter set, as a position in the matrix the scorer received.

A scorer never sees a failed candidate

The search, through finite_candidate_order, hands the scorer the columns whose every entry is finite, and maps the order back to the parameter grid itself. A scorer therefore reads a matrix of finite numbers alone, and it may compute anything on it, such as a mean, a spread or a rank, and a candidate that failed a fold cannot win. The matrix a scorer receives is a view of the score matrix, and its column count can be smaller than the grid. So a scorer must return values or a position in that view, and must not index the grid itself.

Examples

julia> struct MyScore <: PortfolioOptimisers.CrossValidationSearchScorer endjulia> function PortfolioOptimisers.candidate_scores(::MyScore, X::AbstractMatrix)           return vec(minimum(X; dims = 1))       endjulia> (s::MyScore)(X::AbstractMatrix) = argmax(PortfolioOptimisers.candidate_scores(s, X))julia> scores = [0.5 0.6; 0.7 0.8];julia> scorer = MyScore()MyScore()julia> PortfolioOptimisers.candidate_scores(scorer, scores)2-element Vector{Float64}: 0.5 0.6julia> scorer(scores)2

Related

source
PortfolioOptimisers.candidate_scores — Function
candidate_scores(scorer::CrossValSearchScorer, X)
candidate_scores(scorer::HighestMeanScore, X::MatNum)

Give one value per candidate of a search, bigger is better, so that the search can order the candidates.

The column j of X holds the scores of candidate j over the rows of the scheme, signed so that a bigger score is better. The search sorts the candidates by these values, in descending order, and keeps the first n_best. The sort is stable, so a tie keeps the order of the grid.

The method on CrossValSearchScorer is the fallback. It gives nothing, which states that the scorer gives no value through this method. A function scorer, and a subtype of CrossValidationSearchScorer that defines no method, give nothing. The search then calls the scorer once, through candidate_values, and reads the form from the return. A function can give a vector of values or an index, and the index form of a subtype gives an index. The search keeps one candidate for an index, and it refuses n_best > 1 with an ArgumentError.

Arguments

  • scorer: The search scorer.
  • X: The rows × candidates score matrix that the scorer receives. Every entry is finite.

Returns

  • vals::Option{<:VecNum}: One value per column of X, or nothing when the scorer gives no value per candidate.

Examples

julia> PortfolioOptimisers.candidate_scores(HighestMeanScore(), [0.5 0.75 0.25; 1.0 0.5 0.25])3-element Vector{Float64}: 0.75 0.625 0.25

Related

source
PortfolioOptimisers.scorer_scale — Function
scorer_scale(scorer::CrossValSearchScorer)
scorer_scale(scorer::HighestMeanScore)

Give the weight of a search scorer in a vector of search scorers.

candidate_values multiplies the values of each member of a vector by this weight before s_sca reduces them. It is the same word and mechanism as settings.scale of a risk measure, which oriented_member_score reads. A negative scale is a negative weight. The scale is inert for one scorer.

The method on CrossValSearchScorer is the fallback, and it gives 1. So a function, and a subtype of CrossValidationSearchScorer with no scale field, have the weight one. A subtype with a scale field defines a method that reads it. A function writes its weight into its own values.

Arguments

  • scorer: The search scorer.

Returns

  • scale::Number: The weight of the scorer.

Examples

julia> PortfolioOptimisers.scorer_scale(HighestMeanScore(; scale = 2.0))2.0julia> PortfolioOptimisers.scorer_scale(X -> vec(sum(X; dims = 1)))1

Related

source