Scoring: private API
PortfolioOptimisers.AbstractCrossValidationScorer — Type
abstract type AbstractCrossValidationScorer <: AbstractEstimatorAbstract supertype of the cross-validation scorers.
A scorer reads the scores of a cross-validation and selects one of the items it scored. The family has two branches:
PredictionScorerselects one path from the population of paths that a cross-validation scheme returns.CrossValidationSearchScorerselects one candidate of a search from the matrix of its cross-validation scores.
Related
PortfolioOptimisers.PredictionScorer — Type
abstract type PredictionScorer <: AbstractCrossValidationScorerAbstract supertype of the scorers that select one path from a population of predictions.
A subtype is callable. It takes a PopulationPredictionResult and returns one of its members.
Related
PortfolioOptimisers.PredictionCrossValScorer — Type
const PredictionCrossValScorer = Union{PredictionScorer, Function}Union of the values that the scorer field of NestedClustered and Stacking accepts: a PredictionScorer or a plain function.
The union exists so that a caller can select a path with a function and write no type. The function takes a PopulationPredictionResult and returns one of its members, as a PredictionScorer does.
Related
A scorer reads the oriented score of a path. The score orients each risk measure before it reduces a vector, so a higher score is always better.
PortfolioOptimisers.oriented_score — Function
oriented_score(sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction},
r::BaseRM_VecBaseRM, p::PredRes_MultiPredRes; kwargs...)
oriented_score(sca::Option{<:Scalariser}, rs::VecBaseRM, X::VecNum; kwargs...)
oriented_score(sca::Union{VectorToScalarMeasure, Function}, rs::VecBaseRM, X::VecNum;
kwargs...)
oriented_score(::AbstractSearchReduction, rs::VecBaseRM, X::VecNum; kwargs...)
oriented_score(sca::Union{Option{<:Sca_VecToScaM}, AbstractSearchReduction},
r::AbstractBaseRiskMeasure, res::OptimisationResult,
X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(sca::Option{<:Scalariser}, rs::VecBaseRM, res::OptimisationResult,
X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(sca::Union{VectorToScalarMeasure, Function}, rs::VecBaseRM,
res::OptimisationResult, X::Option{<:MatNum} = nothing; kwargs...)
oriented_score(::AbstractSearchReduction, rs::VecBaseRM, res::OptimisationResult,
X::Option{<:MatNum} = nothing; kwargs...)Score a prediction, or a fitted result over its own training sample, under one risk measure or a vector of them, so that a higher score is always better.
A vector of measures also scores a return series X, one sample of a prediction. FoldRiskScorer scores the returns of one asset over one test window in this way. The form for a fitted result scores the result. With X = nothing it scores the result through its prior result, as expected_risk(r, res) does. With a matrix X it scores the result over X, as expected_risk(r, res, X) does. A search calls it through candidate_train_score for its train scores.
The function orients each measure before it reduces the vector. So a vector whose measures disagree on orientation gives a score with one orientation, and bigger_is_better of the vector is never read. Because the values are oriented, MaxScalariser and MaxValue give the score of the best measure, and MinScalariser and MinValue give the score of the worst. The function reads every reduction as "higher is better", so StdValue gives the highest score to the largest spread.
Mathematical definition
\[\begin{align} \mathcal{O}(P) &= o \, \mathcal{R}(P)\,, \\ \mathcal{O}(P) &= S\left(o_1 s_1 \mathcal{R}_1(P), \, \ldots, \, o_n s_n \mathcal{R}_n(P)\right)\,, \\ o_k &= \begin{cases} 1 & \mathcal{R}_k \ \mathrm{bigger\ is\ better} \\ -1 & \mathrm{otherwise} \end{cases}\,. \end{align}\]
Where:
- $\mathcal{O}(P)$: Oriented score of the prediction or the fitted result $P$. The first line is the case of one measure, and the second line the case of a vector.
- $\mathcal{R}(P)$, $\mathcal{R}_k(P)$: Expected risk of $P$ under the measure, or under the $k$-th measure of the vector. For a fitted result, it is the risk over its training sample.
- $o$, $o_k$: Orientation of the measure, from
bigger_is_better. - $s_k$:
scaleof the $k$-th risk measure of a vector, its weight in the combination. A negative scale is a negative weight, so that measure counts against the score. - $S$: The reduction
sca. - $n$: Size of the vector of risk measures.
One measure ignores sca and its scale, as expected_risk does. A search calls the function with a rule of AbstractSearchReduction in sca, such as EachValue or PerCandidate. For a vector, such a rule does not reduce: it gives the vector of the values $o_k s_k \mathcal{R}_k(P)$, in the order of rs. For one measure it is inert.
Algorithm
- One measure: return $o \, \mathcal{R}(P)$, with $\mathcal{R}(P)$ from
expected_riskwithkwargs, throughoriented_value. - A vector of a prediction: read the return series of $P$ once with
prediction_returns, and score the series as in step 3. - A vector of a series or of a fitted result: compute $o_k s_k \mathcal{R}_k(P)$ for each measure on the sample, with
oriented_member_scorewithkwargs. Reduce the values by the type ofsca:- A
Scalariserornothing: throughscalarise.nothinggives the sum ofSumScalariser. - A
VectorToScalarMeasureor a function: collect the values into a vector, and applyvec_to_real_measure. - A rule of
AbstractSearchReduction: collect the values into a vector, and return the vector. The search combines them later, or not at all.
- A
The reduction of step 3 is written twice, once for a series and once for a fitted result, so that each sample is a method of this function and no helper holds a part of the job. The two copies agree: each reads the members through oriented_member_score, and each reduces by the type of sca in the same two ways. The tests of a search hold the copy for a fitted result, and the tests of a scorer of predictions and of FoldRiskScorer hold the copy for a series.
Arguments
r: One risk measure, or a vector of them.rs: A vector of risk measures.p: A prediction of one fold, aPredictionResult, or of one path, aMultiPeriodPredictionResult.res: A fitted result.X: In the form for a series, the return series. In the form for a fitted result, the training returns ofres, ornothingto scoreresthrough its prior result.sca: The reduction of a vector. AVectorToScalarMeasureor a function must take a vector of numbers.nothinggives the sum ofSumScalariser. A rule ofAbstractSearchReductionkeeps each value.kwargs...: Keyword arguments forwarded toexpected_risk.
Returns
score::Number: The oriented score.scores::VecNum: Under a rule ofAbstractSearchReductionwith a vector, the oriented score of each measure, multiplied by its scale.
Related
Sca_VecToScaMbigger_is_betterprediction_returnsoriented_member_scoresort_by_measure: sorts the paths of a population by this score.quantile_by_measure: selects a path by the quantile of the negated score.candidate_train_score: the train score of a search, through the form for a fitted result.
PortfolioOptimisers.oriented_value — Function
oriented_value(r::AbstractBaseRiskMeasure, x)Orient the value x of the risk measure r, so that a higher value is better.
Arguments
r: The risk measure ofx.x: A value ofr.
Returns
x:xwhenbigger_is_betterholds forr, and-xotherwise. The orientation is a trait of the type ofr, so the compiler drops the branch.
Related
PortfolioOptimisers.oriented_member_score — Function
oriented_member_score(r::AbstractBaseRiskMeasure, X::VecNum; kwargs...)
oriented_member_score(r::AbstractBaseRiskMeasure, res::OptimisationResult,
X::Option{<:MatNum}; kwargs...)Score one member of a vector of risk measures on a return series, or a fitted result over its training sample, oriented so that a higher value is better and multiplied by its scale.
It is the value $o_k s_k \mathcal{R}_k(P)$ of oriented_score, which reduces these values of a vector. The vector methods of oriented_score call this function once for each member, on one sample.
Arguments
r: One member of the vector.X: The return series of the prediction. In the second form, the training returns ofres, ornothingto scoreresthrough its prior result.res: A fitted result.kwargs...: Keyword arguments forwarded toexpected_risk_from_returns, or toexpected_riskfor a fitted result.
Returns
score::Number: The value ofr, multiplied by-1whenbigger_is_betterdoes not hold forr, and byr.settings.scale.
Related
PortfolioOptimisers.prediction_returns — Function
prediction_returns(p::PredictionResult)
prediction_returns(p::MultiPeriodPredictionResult)Give the realised return series of a prediction, the series that expected_risk reads.
A PredictionResult is one fold, and its series is p.rd.X. A MultiPeriodPredictionResult is one path, and its series is p.mrd.X, the folds of the path in one series.
Arguments
p: A prediction of one fold or of one path.
Returns
X::VecNum: The return series ofp.
Related
A reader of a population takes its reduction as an argument. So it refuses a sca in the keywords that it forwards to the score.
PortfolioOptimisers.assert_path_reduction — Function
assert_path_reduction(kwargs, name::AbstractString)Check the keyword arguments of a reader that gives one score for each path of a population.
NearestQuantilePrediction, sort_by_measure and quantile_by_measure call it before they score a path. The reduction is the argument sca, so a sca in the keyword arguments that the reader forwards to oriented_score is refused. A reader of a risk takes every keyword that it does not declare, so it would ignore that sca without an error. The keys of a NamedTuple are part of its type, so the check costs nothing at run time.
Arguments
kwargs: The keyword arguments that the reader forwards tooriented_score.name: The name ofkwargsin the call of the caller, for the error message.
Validation
kwargsmust not containsca. The error is aConflictingArgumentError.
Returns
nothing.
Related
A scorer ranks only the members of a population whose every fold solved.
PortfolioOptimisers.member_solved — Function
member_solved(p::PredictionResult)
member_solved(p::MultiPeriodPredictionResult)Says whether every fold of the population member p solved.
A PredictionResult is one fold, and it solved when its return code is an OptimisationSuccess. A MultiPeriodPredictionResult solved when each of its folds solved.
Arguments
p: A member of aPopulationPredictionResult.
Returns
flag::Bool:truewhen every fold ofpsolved.
Related
PortfolioOptimisers.successful_members — Function
successful_members(ppred::PopulationPredictionResult)Gives the members of ppred whose every fold solved.
sort_by_measure and quantile_by_measure rank only these members, so they never rank or return a path with a failed fold.
Arguments
ppred: Population prediction result.
Returns
pred::VecPredRes_MultiPredRes: The members for whichmember_solvedholds, in population order.
Related
A PopulationPredictionResult gives each member without an id its position in the population as its id. So the id of the path that a scorer selects is its position in the population.
PortfolioOptimisers.lacks_id — Function
lacks_id(p::PredictionResult)
lacks_id(p::MultiPeriodPredictionResult)Says whether the population member p needs an id.
A PredictionResult is one fold and has no id field, so it needs none. A MultiPeriodPredictionResult needs one when its id is nothing.
Arguments
p: A member of aPopulationPredictionResult.
Returns
flag::Bool:truewhenpneeds anid.
Related
PortfolioOptimisers.with_position_id — Function
with_position_id(p::PredictionResult, i::Integer)
with_position_id(p::MultiPeriodPredictionResult, i::Integer)
with_position_id(p::MultiPeriodPredictionResult{<:Any, <:Any, Nothing}, i::Integer)Gives the population member p the id i when it lacks one.
The type of the member's id selects the method, so the function makes no test at run time. The method for an id of nothing rebuilds the member with id = i, and keeps its folds and its estimator. The other methods return the member unchanged.
Arguments
p: A member of aPopulationPredictionResult.i: The position ofpin the population.
Returns
p::PredRes_MultiPredRes: The member, with anidwhere it needs one.
Related
PortfolioOptimisers.population_ids — Function
population_ids(pred::VecPredRes_MultiPredRes)Gives every member of a population an id, so that the id of a selected path is its position in the population.
The constructor of PopulationPredictionResult calls it. A member whose id is nothing takes its position in pred as its id. CombinatorialCrossValidation and MultipleRandomised number their paths by position too. When no member lacks_id, the function returns the same vector, so it does not copy a population that a scheme made.
Arguments
pred: The members of the population.
Returns
pred::VecPredRes_MultiPredRes: The members, each with anidwhere it needs one.
Related