Scoring: private API

PortfolioOptimisers.AbstractCrossValidationScorer — Type
abstract type AbstractCrossValidationScorer <: AbstractEstimator

Abstract supertype of the cross-validation scorers.

A scorer reads the scores of a cross-validation and selects one of the items it scored. The family has two branches:

Related

source
PortfolioOptimisers.PredictionCrossValScorer — Type
const PredictionCrossValScorer = Union{PredictionScorer, Function}

Union of the values that the scorer field of NestedClustered and Stacking accepts: a PredictionScorer or a plain function.

The union exists so that a caller can select a path with a function and write no type. The function takes a PopulationPredictionResult and returns one of its members, as a PredictionScorer does.

Related

source

A scorer reads the oriented score of a path. The score orients each risk measure before it reduces a vector, so a higher score is always better.

PortfolioOptimisers.oriented_score — Function
oriented_score(sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction},
               r::BaseRM_VecBaseRM, p::PredRes_MultiPredRes; kwargs...)
oriented_score(sca::Option{<:Scalariser}, rs::VecBaseRM, X::VecNum; kwargs...)
oriented_score(sca::Union{VectorToScalarMeasure, Function}, rs::VecBaseRM, X::VecNum;
               kwargs...)
oriented_score(::AbstractSearchReduction, rs::VecBaseRM, X::VecNum; kwargs...)
oriented_score(sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction},
               r::AbstractBaseRiskMeasure, res::OptimisationResult,
               X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(sca::Option{<:Scalariser}, rs::VecBaseRM, res::OptimisationResult,
               X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(sca::Union{VectorToScalarMeasure, Function}, rs::VecBaseRM,
               res::OptimisationResult, X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(::AbstractSearchReduction, rs::VecBaseRM, res::OptimisationResult,
               X::Option{<:MatNum} = nothing; kwargs...)

Score a prediction, or a fitted result over its own training sample, under one risk measure or a vector of them, so that a higher score is always better.

A vector of measures also scores a return series X, one sample of a prediction. FoldRiskScorer scores the returns of one asset over one test window in this way. The form for a fitted result scores the result. With X = nothing it scores the result through its prior result, as expected_risk(r, res) does. With a matrix X it scores the result over X, as expected_risk(r, res, X) does. A search calls it through candidate_train_score for its train scores.

The function orients each measure before it reduces the vector. So a vector whose measures disagree on orientation gives a score with one orientation, and bigger_is_better of the vector is never read. Because the values are oriented, MaxScalariser and MaxValue give the score of the best measure, and MinScalariser and MinValue give the score of the worst. The function reads every reduction as "higher is better", so StdValue gives the highest score to the largest spread.

Mathematical definition

\[\begin{align} \mathcal{O}(P) &= o \, \mathcal{R}(P)\,, \\ \mathcal{O}(P) &= S\left(o_1 s_1 \mathcal{R}_1(P), \, \ldots, \, o_n s_n \mathcal{R}_n(P)\right)\,, \\ o_k &= \begin{cases} 1 & \mathcal{R}_k \ \mathrm{bigger\ is\ better} \\ -1 & \mathrm{otherwise} \end{cases}\,. \end{align}\]

Where:

  • $\mathcal{O}(P)$: Oriented score of the prediction or the fitted result $P$. The first line is the case of one measure, and the second line the case of a vector.
  • $\mathcal{R}(P)$, $\mathcal{R}_k(P)$: Expected risk of $P$ under the measure, or under the $k$-th measure of the vector. For a fitted result, it is the risk over its training sample.
  • $o$, $o_k$: Orientation of the measure, from bigger_is_better.
  • $s_k$: scale of the $k$-th risk measure of a vector, its weight in the combination. A negative scale is a negative weight, so that measure counts against the score.
  • $S$: The reduction sca.
  • $n$: Size of the vector of risk measures.

One measure ignores sca and its scale, as expected_risk does. A search calls the function with a rule of AbstractSearchReduction in sca, such as EachValue or PerCandidate. For a vector, such a rule does not reduce: it gives the vector of the values $o_k s_k \mathcal{R}_k(P)$, in the order of rs. For one measure it is inert.

Algorithm

  1. One measure: return $o \, \mathcal{R}(P)$, with $\mathcal{R}(P)$ from expected_risk with kwargs, through oriented_value.
  2. A vector of a prediction: read the return series of $P$ once with prediction_returns, and score the series as in step 3.
  3. A vector of a series or of a fitted result: compute $o_k s_k \mathcal{R}_k(P)$ for each measure on the sample, with oriented_member_score with kwargs. Reduce the values by the type of sca:
    1. A Scalariser or nothing: through scalarise. nothing gives the sum of SumScalariser.
    2. A VectorToScalarMeasure or a function: collect the values into a vector, and apply vec_to_real_measure.
    3. A rule of AbstractSearchReduction: collect the values into a vector, and return the vector. The search combines them later, or not at all.

The reduction of step 3 is written twice, once for a series and once for a fitted result, so that each sample is a method of this function and no helper holds a part of the job. The two copies agree: each reads the members through oriented_member_score, and each reduces by the type of sca in the same two ways. The tests of a search hold the copy for a fitted result, and the tests of a scorer of predictions and of FoldRiskScorer hold the copy for a series.

Arguments

  • r: One risk measure, or a vector of them.
  • rs: A vector of risk measures.
  • p: A prediction of one fold, a PredictionResult, or of one path, a MultiPeriodPredictionResult.
  • res: A fitted result.
  • X: In the form for a series, the return series. In the form for a fitted result, the training returns of res, or nothing to score res through its prior result.
  • sca: The reduction of a vector. A VectorToScalarMeasure or a function must take a vector of numbers. nothing gives the sum of SumScalariser. A rule of AbstractSearchReduction keeps each value.
  • kwargs...: Keyword arguments forwarded to expected_risk.

Returns

  • score::Number: The oriented score.
  • scores::VecNum: Under a rule of AbstractSearchReduction with a vector, the oriented score of each measure, multiplied by its scale.

Related

source
PortfolioOptimisers.oriented_member_score — Function
oriented_member_score(r::AbstractBaseRiskMeasure, X::VecNum; kwargs...)
oriented_member_score(r::AbstractBaseRiskMeasure, res::OptimisationResult,
                      X::Option{<:MatNum}; kwargs...)

Score one member of a vector of risk measures on a return series, or a fitted result over its training sample, oriented so that a higher value is better and multiplied by its scale.

It is the value $o_k s_k \mathcal{R}_k(P)$ of oriented_score, which reduces these values of a vector. The vector methods of oriented_score call this function once for each member, on one sample.

Arguments

  • r: One member of the vector.
  • X: The return series of the prediction. In the second form, the training returns of res, or nothing to score res through its prior result.
  • res: A fitted result.
  • kwargs...: Keyword arguments forwarded to expected_risk_from_returns, or to expected_risk for a fitted result.

Returns

  • score::Number: The value of r, multiplied by -1 when bigger_is_better does not hold for r, and by r.settings.scale.

Related

source
PortfolioOptimisers.prediction_returns — Function
prediction_returns(p::PredictionResult)
prediction_returns(p::MultiPeriodPredictionResult)

Give the realised return series of a prediction, the series that expected_risk reads.

A PredictionResult is one fold, and its series is p.rd.X. A MultiPeriodPredictionResult is one path, and its series is p.mrd.X, the folds of the path in one series.

Arguments

  • p: A prediction of one fold or of one path.

Returns

  • X::VecNum: The return series of p.

Related

source

A reader of a population takes its reduction as an argument. So it refuses a sca in the keywords that it forwards to the score.

PortfolioOptimisers.assert_path_reduction — Function
assert_path_reduction(kwargs, name::AbstractString)

Check the keyword arguments of a reader that gives one score for each path of a population.

NearestQuantilePrediction, sort_by_measure and quantile_by_measure call it before they score a path. The reduction is the argument sca, so a sca in the keyword arguments that the reader forwards to oriented_score is refused. A reader of a risk takes every keyword that it does not declare, so it would ignore that sca without an error. The keys of a NamedTuple are part of its type, so the check costs nothing at run time.

Arguments

  • kwargs: The keyword arguments that the reader forwards to oriented_score.
  • name: The name of kwargs in the call of the caller, for the error message.

Validation

Returns

  • nothing.

Related

source

A scorer ranks only the members of a population whose every fold solved.

PortfolioOptimisers.successful_members — Function
successful_members(ppred::PopulationPredictionResult)

Gives the members of ppred whose every fold solved.

sort_by_measure and quantile_by_measure rank only these members, so they never rank or return a path with a failed fold.

Arguments

  • ppred: Population prediction result.

Returns

  • pred::VecPredRes_MultiPredRes: The members for which member_solved holds, in population order.

Related

source

A PopulationPredictionResult gives each member without an id its position in the population as its id. So the id of the path that a scorer selects is its position in the population.

PortfolioOptimisers.with_position_id — Function
with_position_id(p::PredictionResult, i::Integer)
with_position_id(p::MultiPeriodPredictionResult, i::Integer)
with_position_id(p::MultiPeriodPredictionResult{<:Any, <:Any, Nothing}, i::Integer)

Gives the population member p the id i when it lacks one.

The type of the member's id selects the method, so the function makes no test at run time. The method for an id of nothing rebuilds the member with id = i, and keeps its folds and its estimator. The other methods return the member unchanged.

Arguments

Returns

  • p::PredRes_MultiPredRes: The member, with an id where it needs one.

Related

source
PortfolioOptimisers.population_ids — Function
population_ids(pred::VecPredRes_MultiPredRes)

Gives every member of a population an id, so that the id of a selected path is its position in the population.

The constructor of PopulationPredictionResult calls it. A member whose id is nothing takes its position in pred as its id. CombinatorialCrossValidation and MultipleRandomised number their paths by position too. When no member lacks_id, the function returns the same vector, so it does not copy a population that a scheme made.

Arguments

  • pred: The members of the population.

Returns

  • pred::VecPredRes_MultiPredRes: The members, each with an id where it needs one.

Related

source