Scoring
PortfolioOptimisers.NearestQuantilePrediction — Type
struct NearestQuantilePrediction{__T_r, __T_sca, __T_q, __T_r_kwargs, __T_q_kwargs} <: PredictionScorerSelects the path of a population whose risk is nearest to a target quantile of the population's risks.
It is the scorer that NestedClustered and Stacking use under CombinatorialCrossValidation when their scorer is nothing. A path with a failed fold, or with a risk that is not finite, takes no part in the quantile, and the scorer never selects it.
Mathematical definition
\[\begin{align} \rho_i &= -\mathcal{O}(P_i)\,, \quad i \in \mathcal{S}\,, \\ i^{\star} &= \underset{i \in \mathcal{S}}{\arg\min} \left| \rho_i - Q_q\left(\{\rho_j\}_{j \in \mathcal{S}}\right) \right|\,. \end{align}\]
Where:
- $P_i$: Path $i$ of the population.
- $\mathcal{S}$: Set of the paths whose every fold solved and whose $\rho_i$ is finite.
- $\mathcal{O}(P_i)$: Oriented score of path $i$ under
r, fromoriented_score, so a higher score is better. The fieldscareduces a vector of risk measures to one value, and the defaultnothinggives the sum ofSumScalariser. - $\rho_i$: Risk of path $i$ on a scale where a larger value is worse.
- $Q_q$: $q$-th quantile of a set of values, as
Statistics.quantilecomputes it withq_kwargs. - $i^{\star}$: Selected path.
$q = 0$ selects the best path and $q = 1$ the worst path, whatever the quantile definition and whatever the orientation of r. For one risk measure $\rho_i$ is its expected risk, and for one measure of which a bigger value is better $\rho_i$ is the negated value.
Algorithm
- Keep the members of
ppredwhose every fold solved, throughsuccessful_members, givingpred. - Compute
rks, the vector of $-\mathcal{O}(P_i)$ overpred, throughoriented_scorewithscaandr_kwargs. - Drop every entry of
rksthat is not finite, and the member ofpredat its position. - Compute
rkq, theq-th quantile ofrks, throughStatistics.quantilewithq_kwargs. - Scan
rksin population order for the entry nearest torkq, givingidx. A tie goes to the first entry. - Return
pred[idx].
Fields
r: Risk measure or vector of risk measures.
sca: Reduction of the oriented scores of a vector of risk measures inrto one score for each path, aSca_VecToScaM, ornothing, which gives the sum ofSumScalariser. Inert whenrholds one measure.
q: Quantile level of the population's risks that the selected path is nearest to.
r_kwargs: Keyword arguments forwarded tooriented_score. It must not containsca, because the reduction of a vector of risk measures isscaitself.
q_kwargs: Keyword arguments forwarded toStatistics.quantile, such as itsalphaandbeta.
Constructors
NearestQuantilePrediction(; r::BaseRM_VecBaseRM = ConditionalValueatRisk(), sca::Option{<:Sca_VecToScaM} = nothing, q::Real = 0.5, r_kwargs::NamedTuple = (;), q_kwargs::NamedTuple = (;)) -> NearestQuantilePredictionMultiplicity
r takes one risk measure or a vector of them, and the field sca reduces a vector, as in sca = MinScalariser() or sca = MedianValue(). The default nothing gives the sum. The reduction reads oriented scores, so MinScalariser selects by the worst measure of each path and MaxScalariser by the best. For one measure sca is inert.
The scorer admits a vector whose members disagree on orientation, because oriented_score orients each member before the reduction.
Validation
0 <= q <= 1.r_kwargsmust not containsca, throughassert_path_reduction. The reduction is the fieldsca.
Functor
(s::NearestQuantilePrediction)(ppred::PopulationPredictionResult)Selects a path of ppred through quantile_by_measure and returns it.
When the population is empty, or every member has a failed fold or a risk that is not finite, Statistics.quantile receives no value and throws an ArgumentError.
Related
The slot r_sca of a search takes a rule that a scalariser cannot spell. Each rule keeps one score matrix for each risk measure.
PortfolioOptimisers.AbstractSearchReduction — Type
abstract type AbstractSearchReduction <: AbstractAlgorithmAbstract supertype of the rules of the slot r_sca of a search that a scalariser cannot spell.
A Sca_VecToScaM in r_sca combines the risk measures of r in each fold or path, so the search keeps one score matrix. A member of this family keeps one score matrix for each measure, so the search scores each fold under each measure apart. The members differ in what the search does with the K matrices:
EachValuecombines nothing, and gives one ranking for each measure.PerCandidatecombines the measures once per candidate, after the search scorer reduces the rows of each matrix, and gives one ranking.
So oriented_score gives the oriented score of each measure under every member, and the allocator of the scores gives one matrix for each measure.
expected_risk, NearestQuantilePrediction and FoldRiskScorer need one value for each sample, so their bounds refuse every member.
Interfaces
The family is closed. The selection of a search dispatches on EachValue and on PerCandidate, and holds the methods of each for its K score matrices. A new member needs those methods, so the library adds no member.
Related
A search keeps each value apart under EachValue, and gives one ranking for each value per candidate.
PortfolioOptimisers.EachValue — Type
struct EachValue <: AbstractSearchReductionThe per-value rule of a scalariser slot of a search: keep each value apart, and give one ranking for each value per candidate.
It is the multi-metric form of a search in sklearn. The search does not combine the values. It orders the candidates by each value apart, and it keeps one opt, one idx and one ranking for each value, in one SearchCrossValidationResult.
- In
r_scawith a vector ofKrisk measures, the search keeps one score matrix for each measure. Each matrix holds the oriented score of its measure, multiplied by the scale of the measure, seeoriented_score. - In
s_scawith a vector ofLsearch scorers, each scorer orders the candidates alone. Its values are multiplied by itsscorer_scale. - In both, the search keeps a
K × Lmatrix of rankings. Rowkis measurek, and columnlis scorerl.
For one measure in r, or one scorer in scorer, the rule is inert, as a reduction is. The types of r, r_sca, scorer and s_sca decide the shape of the Result, never the count at run time. So a vector of one measure under r_sca = EachValue() gives a vector of one matrix.
The search takes no refit. A caller who wants the winner of value v fitted on the whole sample calls optimise(res.opt[v], rd).
The rule is a rule of GridSearchCrossValidation and RandomisedSearchCrossValidation alone. It is not a Scalariser and not a member of Sca_VecToScaM. expected_risk, NearestQuantilePrediction and FoldRiskScorer need one value for each sample, so their bounds refuse it.
Constructors
EachValue() -> EachValueExamples
julia> GridSearchCrossValidation(["opt.l1" => [0.0, 0.01]]; r = [ConditionalValueatRisk(), MeanReturn()], r_sca = EachValue()).r_scaEachValue()Related
A search combines its risk measures once per candidate under PerCandidate, after the search scorer reduces the folds or the paths of each measure.
PortfolioOptimisers.PerCandidate — Type
struct PerCandidate{T} <: AbstractSearchReductionThe per-candidate rule of the slot r_sca of a search: combine the risk measures once per candidate, after the search scorer reduces the folds or the paths of each measure.
A Sca_VecToScaM in r_sca combines the K measures of r in each fold, or in each path of a combinatorial scheme, and then the search scorer reduces the rows. PerCandidate(sca) takes the stages in the other order. The search keeps one score matrix for each measure, as under EachValue. The search scorer reduces the rows of each matrix, and gives the value $v_{kj}$ of measure $k$ for candidate $j$. Then sca combines the K values of each candidate to one. The search gives one ranking, as for one measure.
For a sum, and a search scorer that is linear in the rows, such as HighestMeanScore, the two orders give the same value. For MaxScalariser, MinScalariser, LogSumExpScalariser, a VectorToScalarMeasure or a function they do not. The mean over the folds of the best measure is not the best measure of the means. For one candidate, two measures and three folds:
fold 1 fold 2 fold 3 mean
measure 1 3 0 0 1
measure 2 0 2 1 1
r_sca = MaxScalariser(): max per fold [3, 2, 1], then mean = 2
r_sca = PerCandidate(MaxScalariser()): mean per measure [1, 1], then max = 1The first asks how good the best measure is in each fold. The second asks how good the best measure is over all the folds.
Mathematical definition
\[\begin{align} v_{kj} &= c\left(\mathbf{S}^{(k)}_{:,\,\mathcal{C}}\right)_j\,, \\ u_j &= S\left(v_{1j}, \, \ldots, \, v_{Kj}\right)\,. \end{align}\]
Where:
- $\mathbf{S}^{(k)}$: Score matrix of measure $k$. Each entry is the oriented score of the measure, multiplied by its scale, see
oriented_score. So the scale of a measure is its weight insca. - $\mathcal{C}$: The candidates whose every entry is finite in every matrix. A value of fewer than $K$ measures does not compare with a value of $K$.
- $c$: The search scorer, through
candidate_values. - $v_{kj}$: Value of measure $k$ for candidate $j$.
- $S$: The reduction
sca. - $u_j$: Value of candidate $j$ for the scorer. A vector of search scorers multiplies $u_j$ of each scorer by its
scorer_scale, ands_scareduces them, as for one measure. - $K$: Count of the risk measures in
r.
Fields
sca: Reduction of the values of the risk measures of one candidate, each the value that the search scorer gives for the score matrix of the measure, aSca_VecToScaM, ornothingfor the sum thatSumScalarisergives.
Constructors
PerCandidate(; sca::Option{<:Sca_VecToScaM} = nothing) -> PerCandidatePerCandidate(sca) takes the reduction as the one positional argument.
Validation
- When the search runs, a search scorer that gives the index of one winner is refused with an
ArgumentError, because an index has no value to combine. For one measure inrit is admitted. - When the search runs, a reduction that gives
NaNis refused with anIsNonFiniteError.
For one measure in r the rule is inert, as a reduction is. The search scorer of a vector of one measure gives one value, and sca reduces that one value.
The rule is a rule of the slot r_sca of GridSearchCrossValidation and RandomisedSearchCrossValidation alone. The slot s_sca refuses it, because no stage follows the scorers. expected_risk, NearestQuantilePrediction and FoldRiskScorer have no scorer stage, so their bounds refuse it.
Examples
julia> PerCandidate(MaxScalariser())PerCandidate sca ┴ MaxScalariser()julia> GridSearchCrossValidation(["opt.l1" => [0.0, 0.01]]; r = [ConditionalValueatRisk(), MeanReturn()], r_sca = PerCandidate(MaxScalariser())).r_sca.scaMaxScalariser()Related