Scoring

PortfolioOptimisers.NearestQuantilePrediction — Type
struct NearestQuantilePrediction{__T_r, __T_sca, __T_q, __T_r_kwargs, __T_q_kwargs} <: PredictionScorer

Selects the path of a population whose risk is nearest to a target quantile of the population's risks.

It is the scorer that NestedClustered and Stacking use under CombinatorialCrossValidation when their scorer is nothing. A path with a failed fold, or with a risk that is not finite, takes no part in the quantile, and the scorer never selects it.

Mathematical definition

\[\begin{align} \rho_i &= -\mathcal{O}(P_i)\,, \quad i \in \mathcal{S}\,, \\ i^{\star} &= \underset{i \in \mathcal{S}}{\arg\min} \left| \rho_i - Q_q\left(\{\rho_j\}_{j \in \mathcal{S}}\right) \right|\,. \end{align}\]

Where:

  • $P_i$: Path $i$ of the population.
  • $\mathcal{S}$: Set of the paths whose every fold solved and whose $\rho_i$ is finite.
  • $\mathcal{O}(P_i)$: Oriented score of path $i$ under r, from oriented_score, so a higher score is better. The field sca reduces a vector of risk measures to one value, and the default nothing gives the sum of SumScalariser.
  • $\rho_i$: Risk of path $i$ on a scale where a larger value is worse.
  • $Q_q$: $q$-th quantile of a set of values, as Statistics.quantile computes it with q_kwargs.
  • $i^{\star}$: Selected path.

$q = 0$ selects the best path and $q = 1$ the worst path, whatever the quantile definition and whatever the orientation of r. For one risk measure $\rho_i$ is its expected risk, and for one measure of which a bigger value is better $\rho_i$ is the negated value.

Algorithm

  1. Keep the members of ppred whose every fold solved, through successful_members, giving pred.
  2. Compute rks, the vector of $-\mathcal{O}(P_i)$ over pred, through oriented_score with sca and r_kwargs.
  3. Drop every entry of rks that is not finite, and the member of pred at its position.
  4. Compute rkq, the q-th quantile of rks, through Statistics.quantile with q_kwargs.
  5. Scan rks in population order for the entry nearest to rkq, giving idx. A tie goes to the first entry.
  6. Return pred[idx].

Fields

  • r: Risk measure or vector of risk measures.
  • sca: Reduction of the oriented scores of a vector of risk measures in r to one score for each path, a Sca_VecToScaM, or nothing, which gives the sum of SumScalariser. Inert when r holds one measure.
  • q: Quantile level of the population's risks that the selected path is nearest to.
  • r_kwargs: Keyword arguments forwarded to oriented_score. It must not contain sca, because the reduction of a vector of risk measures is sca itself.
  • q_kwargs: Keyword arguments forwarded to Statistics.quantile, such as its alpha and beta.

Constructors

NearestQuantilePrediction(;    r::BaseRM_VecBaseRM = ConditionalValueatRisk(),    sca::Option{<:Sca_VecToScaM} = nothing,    q::Real = 0.5,    r_kwargs::NamedTuple = (;),    q_kwargs::NamedTuple = (;)) -> NearestQuantilePrediction

Multiplicity

r takes one risk measure or a vector of them, and the field sca reduces a vector, as in sca = MinScalariser() or sca = MedianValue(). The default nothing gives the sum. The reduction reads oriented scores, so MinScalariser selects by the worst measure of each path and MaxScalariser by the best. For one measure sca is inert.

The scorer admits a vector whose members disagree on orientation, because oriented_score orients each member before the reduction.

Validation

  • 0 <= q <= 1.
  • r_kwargs must not contain sca, through assert_path_reduction. The reduction is the field sca.

Functor

(s::NearestQuantilePrediction)(ppred::PopulationPredictionResult)

Selects a path of ppred through quantile_by_measure and returns it.

When the population is empty, or every member has a failed fold or a risk that is not finite, Statistics.quantile receives no value and throws an ArgumentError.

Related

source

The slot r_sca of a search takes a rule that a scalariser cannot spell. Each rule keeps one score matrix for each risk measure.

PortfolioOptimisers.AbstractSearchReduction — Type
abstract type AbstractSearchReduction <: AbstractAlgorithm

Abstract supertype of the rules of the slot r_sca of a search that a scalariser cannot spell.

A Sca_VecToScaM in r_sca combines the risk measures of r in each fold or path, so the search keeps one score matrix. A member of this family keeps one score matrix for each measure, so the search scores each fold under each measure apart. The members differ in what the search does with the K matrices:

  • EachValue combines nothing, and gives one ranking for each measure.
  • PerCandidate combines the measures once per candidate, after the search scorer reduces the rows of each matrix, and gives one ranking.

So oriented_score gives the oriented score of each measure under every member, and the allocator of the scores gives one matrix for each measure.

expected_risk, NearestQuantilePrediction and FoldRiskScorer need one value for each sample, so their bounds refuse every member.

Interfaces

The family is closed. The selection of a search dispatches on EachValue and on PerCandidate, and holds the methods of each for its K score matrices. A new member needs those methods, so the library adds no member.

Related

source

A search keeps each value apart under EachValue, and gives one ranking for each value per candidate.

PortfolioOptimisers.EachValue — Type
struct EachValue <: AbstractSearchReduction

The per-value rule of a scalariser slot of a search: keep each value apart, and give one ranking for each value per candidate.

It is the multi-metric form of a search in sklearn. The search does not combine the values. It orders the candidates by each value apart, and it keeps one opt, one idx and one ranking for each value, in one SearchCrossValidationResult.

  • In r_sca with a vector of K risk measures, the search keeps one score matrix for each measure. Each matrix holds the oriented score of its measure, multiplied by the scale of the measure, see oriented_score.
  • In s_sca with a vector of L search scorers, each scorer orders the candidates alone. Its values are multiplied by its scorer_scale.
  • In both, the search keeps a K × L matrix of rankings. Row k is measure k, and column l is scorer l.

For one measure in r, or one scorer in scorer, the rule is inert, as a reduction is. The types of r, r_sca, scorer and s_sca decide the shape of the Result, never the count at run time. So a vector of one measure under r_sca = EachValue() gives a vector of one matrix.

The search takes no refit. A caller who wants the winner of value v fitted on the whole sample calls optimise(res.opt[v], rd).

The rule is a rule of GridSearchCrossValidation and RandomisedSearchCrossValidation alone. It is not a Scalariser and not a member of Sca_VecToScaM. expected_risk, NearestQuantilePrediction and FoldRiskScorer need one value for each sample, so their bounds refuse it.

Constructors

EachValue() -> EachValue

Examples

julia> GridSearchCrossValidation(["opt.l1" => [0.0, 0.01]];                                 r = [ConditionalValueatRisk(), MeanReturn()], r_sca = EachValue()).r_scaEachValue()

Related

source

A search combines its risk measures once per candidate under PerCandidate, after the search scorer reduces the folds or the paths of each measure.

PortfolioOptimisers.PerCandidate — Type
struct PerCandidate{T} <: AbstractSearchReduction

The per-candidate rule of the slot r_sca of a search: combine the risk measures once per candidate, after the search scorer reduces the folds or the paths of each measure.

A Sca_VecToScaM in r_sca combines the K measures of r in each fold, or in each path of a combinatorial scheme, and then the search scorer reduces the rows. PerCandidate(sca) takes the stages in the other order. The search keeps one score matrix for each measure, as under EachValue. The search scorer reduces the rows of each matrix, and gives the value $v_{kj}$ of measure $k$ for candidate $j$. Then sca combines the K values of each candidate to one. The search gives one ranking, as for one measure.

For a sum, and a search scorer that is linear in the rows, such as HighestMeanScore, the two orders give the same value. For MaxScalariser, MinScalariser, LogSumExpScalariser, a VectorToScalarMeasure or a function they do not. The mean over the folds of the best measure is not the best measure of the means. For one candidate, two measures and three folds:

            fold 1  fold 2  fold 3   mean
measure 1      3       0       0      1
measure 2      0       2       1      1

r_sca = MaxScalariser():               max per fold [3, 2, 1], then mean = 2
r_sca = PerCandidate(MaxScalariser()): mean per measure [1, 1], then max = 1

The first asks how good the best measure is in each fold. The second asks how good the best measure is over all the folds.

Mathematical definition

\[\begin{align} v_{kj} &= c\left(\mathbf{S}^{(k)}_{:,\,\mathcal{C}}\right)_j\,, \\ u_j &= S\left(v_{1j}, \, \ldots, \, v_{Kj}\right)\,. \end{align}\]

Where:

  • $\mathbf{S}^{(k)}$: Score matrix of measure $k$. Each entry is the oriented score of the measure, multiplied by its scale, see oriented_score. So the scale of a measure is its weight in sca.
  • $\mathcal{C}$: The candidates whose every entry is finite in every matrix. A value of fewer than $K$ measures does not compare with a value of $K$.
  • $c$: The search scorer, through candidate_values.
  • $v_{kj}$: Value of measure $k$ for candidate $j$.
  • $S$: The reduction sca.
  • $u_j$: Value of candidate $j$ for the scorer. A vector of search scorers multiplies $u_j$ of each scorer by its scorer_scale, and s_sca reduces them, as for one measure.
  • $K$: Count of the risk measures in r.

Fields

  • sca: Reduction of the values of the risk measures of one candidate, each the value that the search scorer gives for the score matrix of the measure, a Sca_VecToScaM, or nothing for the sum that SumScalariser gives.

Constructors

PerCandidate(; sca::Option{<:Sca_VecToScaM} = nothing) -> PerCandidate

PerCandidate(sca) takes the reduction as the one positional argument.

Validation

  • When the search runs, a search scorer that gives the index of one winner is refused with an ArgumentError, because an index has no value to combine. For one measure in r it is admitted.
  • When the search runs, a reduction that gives NaN is refused with an IsNonFiniteError.

For one measure in r the rule is inert, as a reduction is. The search scorer of a vector of one measure gives one value, and sca reduces that one value.

The rule is a rule of the slot r_sca of GridSearchCrossValidation and RandomisedSearchCrossValidation alone. The slot s_sca refuses it, because no stage follows the scorers. expected_risk, NearestQuantilePrediction and FoldRiskScorer have no scorer stage, so their bounds refuse it.

Examples

julia> PerCandidate(MaxScalariser())PerCandidate  sca ┴ MaxScalariser()julia> GridSearchCrossValidation(["opt.l1" => [0.0, 0.01]];                                 r = [ConditionalValueatRisk(), MeanReturn()],                                 r_sca = PerCandidate(MaxScalariser())).r_sca.scaMaxScalariser()

Related

source