Asset selection

An asset selector removes assets from the returns data. It can drop constant columns, keep the best or worst assets by a risk measure, or remove redundant assets. A selector is an ordinary preprocessing estimator of returns, and it does not depend on a pipeline. A Pipeline calls fit_preprocessing and apply_preprocessing on it, as on any other step.

A selector chooses its assets on the training window, and that choice is its fitted result. Applied to a later window, the fitted result keeps the same assets and does not choose again. The test window never changes the choice, so you can use a selector inside cross-validation.

Every selector subtypes PortfolioOptimisers.AbstractAssetSelector.

Scoring assets with a risk measure

A ScoreSelector computes a risk measure on the returns of each asset alone, and passes the scores to a rule. It accepts any risk measure whose supports_precomputed_returns is true, which includes the quantile and drawdown measures, the moment measures and MeanReturn. bigger_is_better tells the ordinal rules which end of the order is the best.

Variance and StandardDeviation do not work here. They are WeightsInput measures, which take portfolio weights and not a series of returns, so they cannot score one asset, and the constructor throws an error for them. Use SCM(), which computes the same quantity from a series of returns. ZeroVarianceFilter builds that selector for you.

PortfolioOptimisers.ScoreSelector — Type
struct ScoreSelector{__T_score, __T_rule} <: AbstractAssetSelector

Asset selector that scores every asset with a risk measure or an Asset Scorer, and keeps the assets that a rule admits.

score is an AbstractBaseRiskMeasure that can take the return series of one asset, or an AbstractAssetScorer. The score of asset i under a risk measure is score(X[:, i]). The risk measures of the library cover the in-sample scores. ConditionalValueatRisk and the drawdown measures score risk, SCM() scores variance, and MeanReturn scores the mean return. FoldRiskScorer scores a risk measure over the test windows of a cross-validation scheme, and WeightMassScorer scores the weight that an optimiser gives each asset. bigger_is_better tells the ordinal rules which end is the best.

rule sets what the selector does with the scores. A ThresholdRule keeps an absolute band. A RankRule takes a CountOf or a FractionOf from the tails.

A ScoreSelector is safe in cross-validation. It fits its universe on the training window, and every test window uses that universe.

Fields

Constructors

ScoreSelector(;    score::BaseRM_AsSc,    rule::AbstractSelectionRule,) -> ScoreSelector

Keywords correspond to the struct's fields.

Validation

Examples

julia> rd = ReturnsResult(; nx = ["A", "B", "C"], X = [0.1 0.0 -0.2; -0.1 0.0 0.3; 0.2 0.0 -0.1]);julia> sel = ScoreSelector(; score = SCM(), rule = ThresholdRule(; lo = 1e-12));julia> PortfolioOptimisers.fit_preprocessing(sel, rd).nx2-element Vector{String}: "A" "C"

Related

source
PortfolioOptimisers.CompleteAssetSelector — Type
struct CompleteAssetSelector <: AbstractAssetSelector

Asset selector that keeps the Coverage Universe of the training window, and drops every other asset column.

It does at the returns level what the column threshold of MissingDataFilter does at the price level. A pipeline that starts from returns data, and so runs no price stage, uses it. It cannot drop observations. A fitted selector cannot select the rows of a new window to drop, because the weights and the returns then no longer align.

The library fits every selector of the family on the Coverage Universe. This selector is then the identity on the window that it gets. It is the explicit step that asks for the reduction, and for nothing more. The reduction drops an asset whose return is non-finite at a row of the window. It also drops an asset that the AssetPanel marks inactive at a row of the window.

A ReturnsResult bounds the type of X to a matrix of numbers, so a missing never gets to this selector. The constructor of the ReturnsResult refuses a Matrix{Union{Missing, Float64}}. MissingDataFilter removes a missing from the price data at an earlier stage.

Constructors

CompleteAssetSelector() -> CompleteAssetSelector

Examples

julia> rd = ReturnsResult(; nx = ["A", "B"], X = [0.1 0.2; 0.3 NaN]);julia> PortfolioOptimisers.fit_preprocessing(CompleteAssetSelector(), rd).nx1-element Vector{String}: "A"

Related

source

Scoring assets with an asset scorer

The score of a selector can also be an asset scorer, which reads the whole returns data and not one asset alone. FoldRiskScorer scores a risk measure over the test windows of a cross-validation scheme, and WeightMassScorer scores the weight that an optimiser gives each asset. To write a scorer of your own, subtype PortfolioOptimisers.AbstractAssetScorer, and give it an asset_scores method and a bigger_is_better method.

PortfolioOptimisers.AbstractAssetScorer — Type
abstract type AbstractAssetScorer <: AbstractEstimator

Abstract supertype for the estimators that give one score to each asset from the whole returns data.

An Asset Scorer is the second kind of score that a ScoreSelector and a RedundancySelector take. A risk measure is the first kind, and it scores each asset on the whole training window, column by column. An Asset Scorer can read more than that. FoldRiskScorer scores each asset on each test window of a cross-validation scheme and aggregates the values. WeightMassScorer runs an optimiser through a cross-validation scheme and scores each asset by the weight that the optimiser gives it. The second score reads the whole cross-section, so no risk measure on one column can state it.

Every parameter of a scorer is a field. A selector is a step of a Pipeline, so a field has a parameter path that search_cross_validation can tune.

Interfaces

To add an Asset Scorer, subtype AbstractAssetScorer, put every parameter of the score in the struct, and add the two methods below. Both are mandatory. The library gives a scorer no default orientation, because an ordinal rule cuts the wrong end of the order when the orientation is a guess.

asset_scores

  • asset_scores(s::MyAssetScorer, rd::AbstractReturnsResult) -> VecNum: Give one score to each asset column of rd.X, in the order of the columns. Write NaN at an asset that the scorer cannot score, for example an asset that no fold offered. Every selection rule and every redundancy algorithm drops a NaN score.

Arguments

  • s: The concrete Asset Scorer.
  • rd: The returns result to use.

Returns

  • scores::VecNum: The score vector assets × 1, with one entry for each asset column of rd.X. An entry is NaN at an asset that has no score.

bigger_is_better

  • bigger_is_better(s::MyAssetScorer) -> Bool: State whether a bigger score is a better score.

Arguments

  • s: The concrete Asset Scorer.

Returns

  • bib::Bool: true when a bigger score is better, and false when a smaller score is better.

Related

source

Selection rules

A rule turns the scores into a mask of the assets to keep. ThresholdRule compares the raw scores with fixed bounds, and ignores which end is better, because a zero-variance filter must drop the assets with low variance. RankRule reads bigger_is_better, and takes a CountOf or a FractionOf of the assets from each end.

An ordinal rule drops every asset of a tie at its cut, so it can keep fewer assets than you ask for. If the 20th and 21st assets have the same score, RankRule(; best = CountOf(20)) keeps 19. The rule drops both tied assets, because it has no reason to keep one of them and not the other.

PortfolioOptimisers.AbstractSelectionRule — Type
abstract type AbstractSelectionRule <: AbstractAlgorithm

Abstract supertype for the rules that turn the scores of the assets into a keep-mask.

A selection rule is an AbstractAlgorithm. A ScoreSelector uses it, and it has no use on its own. There are two kinds of rule.

  • Literal. ThresholdRule compares a score with absolute bounds, and it ignores bigger_is_better. A threshold on a variance is the variance that it states. The rule does not read it as "keep the better assets". That change inverts the purpose of a ZeroVarianceFilter.
  • Ordinal. RankRule sorts the assets from the best to the worst, and takes a CountOf or a FractionOf from each end. bigger_is_better sets the order, so :best is the lowest risk for a risk measure and the highest value for a return measure.

Interfaces

To add a selection rule, subtype AbstractSelectionRule, put every parameter of the rule in the struct, and add this method:

  • rule_keep(rule::AbstractSelectionRule, scores::VecNum, bib::Bool) -> BitVector: Turn the scores of the assets into a keep-mask.

Arguments

  • rule: The concrete selection rule.
  • scores: The score vector assets × 1, with one score for each asset.
  • bib: The bigger_is_better flag of the risk measure that gave scores. A literal rule ignores it.

Returns

  • keep::BitVector: The mask assets × 1. It is true for every asset that the rule admits. It must be false at every non-finite score, under every parameter of the rule. A NaN score marks an asset that has no score, and that asset must not survive a selection.

Related

source
PortfolioOptimisers.ThresholdRule — Type
struct ThresholdRule{__T_lo, __T_hi} <: AbstractSelectionRule

Keep the assets whose score is strictly inside the band (lo, hi).

Both bounds are optional, and both are literal. The rule compares lo and hi with the raw score, and bigger_is_better does not change them. A bound that the caller omits sets no limit on that side.

Mathematical definition

\[\begin{align} \mathcal{K} &= \left\{ i : s_{i} \in \mathbb{R},\; l < s_{i} < u \right\}\,. \end{align}\]

Where:

  • $\mathcal{K}$: The set of the assets that a selector keeps.
  • $s_{i}$: The score of asset $i$. It is the value of the risk measure on the return series of asset $i$ alone.
  • $l$: lo, the lower bound. An omitted bound is $-\infty$.
  • $u$: hi, the upper bound. An omitted bound is $+\infty$.

Both comparisons are strict, so the band is open. The rule drops an asset whose score is equal to $l$ or to $u$. A score that is not in $\mathbb{R}$, a NaN or an infinite value, is outside every band. An omitted bound does not admit it.

Fields

  • lo: The exclusive lower bound on the score. nothing sets no lower bound.
  • hi: The exclusive upper bound on the score. nothing sets no upper bound.

Constructors

ThresholdRule(;    lo::Option{<:Number} = nothing,    hi::Option{<:Number} = nothing,) -> ThresholdRule

Keywords correspond to the struct's fields.

Validation

  • At least one of lo, hi is not nothing.
  • If both are given, lo < hi.

Examples

julia> ThresholdRule(; lo = 1e-12)   # drop (near-)constant assetsThresholdRule  lo ┼ Float64: 1.0e-12  hi ┴ nothing

Related

source
PortfolioOptimisers.RankRule — Type
struct RankRule{__T_best, __T_worst, __T_action} <: AbstractSelectionRule

Take a number of assets from each end of the order of the scores, then keep them or drop them.

best and worst are sizes of the part that the rule takes from each end, not positions. Each is a CountOf of assets or a FractionOf of the assets with a finite score. best = CountOf(20) means twenty assets, not the asset at rank twenty, and best = FractionOf(0.2) means a fifth of the scored assets. A plain number is read through count_or_fraction. bigger_is_better of the score sets which end is the best. For a risk measure, RankRule(; best = 20) keeps the twenty assets with the lowest risk. For MeanReturn, it keeps the twenty with the highest return. A rule with both tails takes both ends. action = :drop takes the complement of the selection. With it, a rule can drop the five worst assets with no knowledge of the size of the universe.

A count saturates at the number of assets. On a window of 30 assets, best = 50 keeps all 30 and does not throw. A hyperparameter search over best then does not fail at its largest value. A fraction becomes a count on the window that the rule fits, with round(Int, f * n, RoundNearestTiesUp) over the n assets with a finite score. An exact half rounds up. For example, best = FractionOf(0.625) on a window of 4 assets takes 3 assets, and not the 2 that the default RoundNearest of Julia gives. An asset with no score does not count in the size of the universe.

Warning

The rule excludes every asset of a tied block at the cut, so it can return fewer assets than the count. If the 20th and the 21st assets have equal scores, RankRule(; best = 20) keeps 19. The rule drops the tied block, and it does not split the block arbitrarily.

This is the tie policy of the library: if the library cannot tell two assets apart, it keeps neither of them. find_uncorrelated_indices has the same policy. It removes both assets of a correlated pair whose drop scores are equal. A window whose scores are all equal selects no asset, and fit_preprocessing throws.

Mathematical definition

\[\begin{align} \mathcal{F} &= \left\{ i : s_{i} \in \mathbb{R} \right\}\,, \\ \mathcal{T}(k) &= \left\{ i \in \mathcal{F} : a_{i} + e_{i} \leq k \right\}\,, \\ \mathcal{S} &= \mathcal{T}_{\mathrm{best}}(k_{b}) \cup \mathcal{T}_{\mathrm{worst}}(k_{w})\,, \\ \mathcal{K} &= \begin{cases} \mathcal{S} & \text{keep} \\ \mathcal{F} \setminus \mathcal{S} & \text{drop} \end{cases}\,. \end{align}\]

Where:

  • $\mathcal{K}$: The set of the assets that a selector keeps.
  • $s_{i}$: The score of asset $i$. It is the value of the risk measure on the return series of asset $i$ alone.
  • $k$: The number of assets that a rule takes from one end of the order of the scores.
  • $\mathcal{F}$: The set of the assets with a finite score. A NaN or an infinite score is outside it.
  • $N_{\mathcal{F}}$: The number of assets with a finite score. A NaN or an infinite score does not count. It is the size of $\mathcal{F}$.
  • $\mathcal{T}(k)$: The tail of size $k$, at the end that the subscript names.
  • $a_{i}$: The number of assets in $\mathcal{F}$ whose score is strictly better than $s_{i}$. At the best end, a larger score is better when bigger_is_better is true, and a smaller score is better when it is false. The worst end reverses this order.
  • $e_{i}$: The number of assets in $\mathcal{F}$ whose score is equal to $s_{i}$. The number includes asset $i$, so $e_{i} \geq 1$.
  • $k_{b}$, $k_{w}$: The counts of best and worst from rule_tail_count, each saturated to $[0,\, N_{\mathcal{F}}]$. A tail of nothing is $0$. A CountOf is its val. A FractionOf $f$ is $\left\lfloor f N_{\mathcal{F}} + \frac{1}{2} \right\rfloor$, the rounding mode RoundNearestTiesUp.
  • $\mathcal{S}$: The union of the two tails, before the rule applies action.

Three facts follow from this definition. A tail admits asset $i$ only when the whole tied block of $i$ fits in $k$. A tail excludes a block that crosses the cut, so $\left| \mathcal{T}(k) \right| \leq k$. If all the finite scores are equal, $a_{i} + e_{i} = N_{\mathcal{F}}$ for every asset of $\mathcal{F}$, so every tail of size $k < N_{\mathcal{F}}$ is empty. An asset outside $\mathcal{F}$ is in no tail and in neither keep-set, so an asset with no score never survives the rule.

Fields

  • best: The part to take from the best end, as a CountOf or a FractionOf. nothing takes no asset.
  • worst: The part to take from the worst end, as a CountOf or a FractionOf. nothing takes no asset.
  • action: :keep keeps the assets that the rule takes, and :drop keeps all the other assets.

Constructors

RankRule(;    best::Option{<:Real_CountFraction} = nothing,    worst::Option{<:Real_CountFraction} = nothing,    action::Symbol = :keep,) -> RankRule

Keywords correspond to the struct's fields. The constructor reads a plain best or worst through count_or_fraction, so each field holds a CountOf, a FractionOf or nothing.

Validation

  • At least one of best, worst is not nothing. Otherwise an IsNothingError.
  • Every plain tail passes count_or_fraction. A count is at least one, so nothing states a tail that takes no asset.
  • action in (:keep, :drop).
A search grid can turn a plain count into a fraction

A field that takes a CountOf or a FractionOf also takes a plain number, and count_or_fraction reads an Integer as a count and any other real number as a fraction. Julia promotes the elements of a vector literal to one type before the library reads them. So ["opt.ret.ucs.alg.method.active" => [3, 5, 8], "r.alpha" => [0.05, 0.1]] is a Vector{Pair{String, Vector{Float64}}}, and the counts arrive as 3.0, 5.0 and 8.0. The field then reads them as fractions, and the check of FractionOf refuses them. State the grid in the wrappers, such as [CountOf(3), CountOf(5), FractionOf(0.2)]. A wrapper keeps its type in every container.

Examples

julia> RankRule(; worst = CountOf(5), action = :drop)   # drop the five worstRankRule    best ┼ nothing   worst ┼ CountOf         │   val ┴ Int64: 5  action ┴ Symbol: :dropjulia> RankRule(; best = FractionOf(0.1)).best   # keep the best decileFractionOf  val ┴ Float64: 0.1

Related

source

Discarding redundant assets

A RedundancySelector removes assets whose returns carry the same information as the returns of other assets. Its alg field decides what counts as redundant, and its score field decides which asset of a redundant group stays.

PairwiseCorrelation removes one asset at a time until no remaining pair has a correlation above the threshold, and it does not follow chains. CorrelationComponents follows chains of the same correlations. If A is close to B and B is close to C, it puts A, B and C in one group and keeps one of them. It removes more assets than PairwiseCorrelation, and on the same data it can keep a different set. ClusterGroups groups the assets with clusterise and keeps one asset per cluster.

If you leave score as nothing, the two correlation algorithms keep the asset with the lowest summary correlation to the rest of the universe. ClusterGroups has no such default, and it needs a score.

ClusterGroups is also the only redundancy algorithm that takes a distance estimator, and so the only one that can group assets by a feature matrix in place of their returns. The other two take a StatsBase.CovarianceEstimator. Give its cle field a FeatureDistance, and the groups come from data outside the returns. For example, a sector classification, held as a categorical field of the asset panel through panel_input, leaves one asset per sector, not one per group of correlated assets. The selector reads the panel directly from the ReturnsResult, because asset selection runs before a prior exists. An asset panel estimator that reads a prior throws an error here, and PhylogenyPanel, which reads none, works.

PortfolioOptimisers.AbstractRedundancyAlgorithm — Type
abstract type AbstractRedundancyAlgorithm <: AbstractAlgorithm

Abstract supertype for the algorithms that select which assets a RedundancySelector drops as redundant.

Each algorithm takes the data, and a score for each asset when the selector has one. It returns a keep-mask of the columns that survive. The interface returns a keep-mask, not a partition into groups. PairwiseCorrelation drops one asset at a time, and it can keep two members of one correlated group. A partition with one survivor for each group cannot show that selection.

Two algorithms do make a partition, and both use groups_argbest:

Interfaces

To add a redundancy algorithm, subtype AbstractRedundancyAlgorithm and add these methods:

  • redundancy_keep(alg::MyAlgorithm, rd, scores, bib) -> BitVector.
  • requires_score(::MyAlgorithm) -> Bool, only when the algorithm can select a survivor with no score. The generic method returns true, so this method returns false.

Arguments

  • alg: The concrete redundancy algorithm.
  • rd: The returns result to use.
  • scores: The score vector assets × 1, with one score for each asset. It is nothing when the selector has no score.
  • bib: The bigger_is_better flag of the score. It is false when scores is nothing.

Returns

  • keep::BitVector: The mask assets × 1. It is true for every asset that survives. It must be false at every asset whose score in scores is not finite. A NaN score marks an asset that has no score, and that asset must not survive a selection.

Related

source
PortfolioOptimisers.RedundancySelector — Type
struct RedundancySelector{__T_alg, __T_score} <: AbstractAssetSelector

Asset selector that drops the assets whose information other assets already carry.

alg sets what "redundant" means, and it returns the keep-mask. PairwiseCorrelation prunes correlated pairs greedily. CorrelationComponents keeps one representative of each correlated group. ClusterGroups keeps one representative of each cluster.

score sets which asset of a redundancy group survives. It is a risk measure on the return series of each asset, or an AbstractAssetScorer, and bigger_is_better sets its order, as in ScoreSelector. When score is nothing, a correlation algorithm uses its own rule. The asset with the lower summary correlation with the rest of the universe survives. ClusterGroups has no such rule, and it needs a score.

Fields

Constructors

RedundancySelector(;    alg::AbstractRedundancyAlgorithm = PairwiseCorrelation(),    score::Option{<:BaseRM_AsSc} = nothing,) -> RedundancySelector

Keywords correspond to the struct's fields.

Validation

  • If score is a risk measure, supports_precomputed_returns(score).
  • If requires_score(alg), score is not nothing.

Examples

julia> rd = ReturnsResult(; nx = ["A", "B", "C"],                          X = [0.10 0.10 -0.05; -0.10 -0.10 0.07; 0.05 0.05 -0.02;                               0.02 0.02 0.09]);julia> sel = RedundancySelector(; alg = PairwiseCorrelation(; t = 0.99), score = SCM());julia> PortfolioOptimisers.fit_preprocessing(sel, rd).nx1-element Vector{String}: "C"

A and B are identical, so the tie policy drops both of them.

Related

source
PortfolioOptimisers.PairwiseCorrelation — Type
struct PairwiseCorrelation{__T_ce, __T_t, __T_absolute, __T_measure} <: AbstractRedundancyAlgorithm

Greedy removal of correlated assets. The algorithm drops assets until no pair of the kept assets has a correlation of t or more.

The algorithm reads the correlated pairs from the most to the least correlated, and it removes the worse asset of each pair. The worse asset has the higher drop score. The drop score comes from the score of the RedundancySelector when it has one. Otherwise it is the summary of the correlation of each asset with the rest of the universe. Then the asset that is redundant with most of the universe goes first.

This algorithm does not chain. Take t = 0.7, and a universe with ρ(A, B) = 0.80, ρ(B, C) = 0.81 and ρ(A, C) = 0.32. With the default drop score, the algorithm removes B and keeps both A and C, because their correlation is below t. CorrelationComponents reads the same three correlations as one chain, and keeps only A.

The two edges of a chain set a lower bound on the correlation of its end points. Two edges at ρ give the end points a correlation of ρ² - (1 - ρ²) or more. Thus a chain of two edges at 0.97 cannot have end points with a correlation below 0.88. A chain whose end points have a low correlation needs weak edges, and the two algorithms differ most when t is just below those edges.

The algorithm calls find_uncorrelated_indices.

Mathematical definition

\[\begin{align} \rho_{i,\,j} &< t \quad \forall\, i \neq j \in \mathcal{K}\,. \end{align}\]

Where:

  • $\rho_{i,\,j}$: Pairwise correlation coefficient between assets $i$ and $j$. The algorithm reads the absolute value $\left| \rho_{i,\,j} \right|$ when absolute is true.
  • $t$: The correlation at or above which two assets are redundant.
  • $\mathcal{K}$: The set of the assets that a selector keeps.

The condition is about the pairs of kept assets, and about nothing else. It does not say that $\mathcal{K}$ is the largest such set. It is also not transitive. A chain of two edges at or above the threshold, whose end points are below the threshold, meets the condition with both end points kept.

Fields

  • ce: The covariance estimator that gives the correlation matrix.
  • t: The correlation at or above which two assets are redundant.
  • absolute: If true, the algorithm compares the absolute value of the correlation with t.
  • measure: The measure that collapses the finite entries of each column of the correlation matrix into the fallback drop score. The correlation with a constant column is not defined, and the measure skips it. The algorithm ignores measure when the selector has a score.

Constructors

PairwiseCorrelation(;    ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),    t::Number = 0.95,    absolute::Bool = false,    measure::Num_VecToScaM = MeanValue(),) -> PairwiseCorrelation

Keywords correspond to the struct's fields.

Validation

Related

source
PortfolioOptimisers.CorrelationComponents — Type
struct CorrelationComponents{__T_ce, __T_t, __T_absolute, __T_measure} <: AbstractRedundancyAlgorithm

Group the assets by the connected components of a correlation graph, and keep the member of each group with the best score.

Two assets share an edge when their correlation is at or above t. When absolute is true, the algorithm compares the absolute value of the correlation. A component is transitive, so the algorithm reads a chain A ~ B ~ C as one redundant group, also when A and C are not correlated. It keeps one asset of the group. This claim is stronger than that of PairwiseCorrelation, and it removes more assets. Use it to keep one representative of each correlated group, and not to bound the correlation of the pairs that survive.

A component whose best score is tied keeps no asset. groups_argbest states the tie policy.

Mathematical definition

\[\begin{align} \mathcal{E} &= \left\{ \left\{ i,\, j \right\} : i \neq j\,,\; \rho_{i,\,j} \geq t \right\}\,. \end{align}\]

Where:

  • $\mathcal{E}$: The edge set of the redundancy graph, whose vertices are the assets.
  • $\rho_{i,\,j}$: Pairwise correlation coefficient between assets $i$ and $j$. The algorithm reads the absolute value $\left| \rho_{i,\,j} \right|$ when absolute is true.
  • $t$: The correlation at or above which two assets are redundant.

The groups are the connected components of that graph, and every asset is in exactly one group. An asset with no partner at or above the threshold is a group of one. A component is transitive, but the edge relation is not. This is the difference from PairwiseCorrelation. A chain $i \sim j \sim k$ is one component, whatever the value of $\rho_{i,\,k}$.

Fields

  • ce: The covariance estimator that gives the correlation matrix.
  • t: The correlation at or above which two assets are redundant.
  • absolute: If true, the algorithm compares the absolute value of the correlation with t.
  • measure: The measure that collapses the finite entries of each column of the correlation matrix into the fallback drop score. The correlation with a constant column is not defined, and the measure skips it. The algorithm ignores measure when the selector has a score. A lower value is better, so the representative that survives is the least redundant member of its component.

Constructors

CorrelationComponents(;    ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),    t::Number = 0.95,    absolute::Bool = false,    measure::Num_VecToScaM = MeanValue(),) -> CorrelationComponents

Keywords correspond to the struct's fields.

Validation

Related

source
PortfolioOptimisers.ClusterGroups — Type
struct ClusterGroups{__T_cle} <: AbstractRedundancyAlgorithm

Group the assets into clusters, and keep the member of each cluster with the best score.

clusterise makes the clusters, so every clustering estimator of the library can set what "redundant" means. This includes hierarchical linkage, DBHT, the non-hierarchical algorithms and the estimators of the optimal number of clusters. The correlation algorithms have a fallback rule for the survivor, but this algorithm has none. A RedundancySelector with ClusterGroups must have a score.

A cluster whose best score is tied keeps no asset. groups_argbest states the tie policy.

Clusters on a Feature Matrix

A FeatureDistance in the distance field of cle measures a Feature Matrix in place of the returns. It makes that matrix from the AssetPanel on rd, the returns data that the selector fits on.

Preselection runs before the prior, so it gives rd alone. fit_preprocessing fits a selector from the returns data alone, and the selector never gets a prior result. In a Pipeline, the selector writes :returns, and this invalidates a :prior that an earlier step computed. Thus a panel producer that needs a prior raises an error here, and the error names this site. Put an AssetPanel on the ReturnsResult, for example from asset_panel. Otherwise clusterise throws.

The cle reads rd.X, the returns as the selector gets them, because a selector fits before a prior or a benchmark frame exists. For this reason, the cle states no source, and the constructor refuses a cle whose x_src is not nothing.

The algorithm selects on the full universe, and the library slices the columns that survive after the selection. The library measures the Feature Matrix Z over every asset, before the selector drops one.

The Feature Matrix takes its column names from the same panel as its values. Thus a FeatureDistance with a sel selects the same feature columns here as at every other site. Without a sel, the Feature Matrix holds the value columns of every Panel Field, with one column for each level of a categorical field. A redundancy selector on all of these columns can drop assets on a distance that the caller did not intend.

Fields

Constructors

ClusterGroups(;    cle::AbstractClustersEstimator = ClustersEstimator(),) -> ClusterGroups

Keywords correspond to the struct's fields.

Related

source

Functions

PortfolioOptimisers.asset_scores — Function
asset_scores(score::AbstractBaseRiskMeasure, X::MatNum) -> VecNum
asset_scores(score::AbstractBaseRiskMeasure, rd::AbstractReturnsResult) -> VecNum
asset_scores(score::AbstractAssetScorer, rd::AbstractReturnsResult) -> VecNum

Evaluate score on every asset column of X.

The function gives each live column to score as a view. The functor of a risk measure reads its argument, and does not write to it. A column is live when it is in the Coverage Universe of the data, as coverage_mask states: every one of its returns is finite and, when rd carries an AssetPanel, the asset is active at every row. The matrix method has no panel, so it reads finiteness alone.

The method on rd with a risk measure reads rd.X and calls the method on the matrix. This is the in-sample score of a selector. The method on an AbstractAssetScorer is the interface of that family, and each concrete scorer adds its own.

The two routes give an asset that they cannot score the same score, NaN. A column outside the Coverage Universe, such as a late listing, a gap, a delisting or an asset that the active mask turns off, is not live. A risk measure does not score it, and its entry is NaN. An Asset Scorer writes NaN at an asset that it cannot score, for example an asset that fewer than min_folds folds offered. That entry is a gap in the data, and not a defect. Every selection rule and every redundancy algorithm drops it, as rule_keep states. A non-finite score on a live column is different. It is a defect of the measure, and the method throws.

Algorithm

  1. Check that a risk measure can take the return series of one asset, with assert_scoreable.
  2. Find the live columns, those in the Coverage Universe, with cv_live_assets.
  3. Evaluate score on a view of each live column, one column after the other, check that every score is finite, and write NaN at every other column, with live_asset_scores.
  4. Return scores.

Arguments

  • score: The risk measure to evaluate on each asset column, or the Asset Scorer.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • rd: The returns result to use.

Validation

  • With a risk measure, supports_precomputed_returns(score). Otherwise assert_scoreable throws an ArgumentError that names the measure and a scoreable measure that computes the same quantity. The answer of supports_precomputed_returns comes from the type of score alone, so the compiler removes the check for a scoreable measure.
  • No column is live. Otherwise coverage_mask throws an IsEmptyError, as it does for an Asset Scorer.
  • With a risk measure, every live column has a finite score. Otherwise the function throws a DomainError that names those columns. ThirdCentralMoment with a variance estimator in ve, on a constant series for example, divides by a zero standard deviation and gives NaN. A rule drops a NaN score, so the defect of the measure becomes silent if the function does not throw. A column that is not live is not scored and raises no error: its entry is NaN.

Returns

  • scores::VecNum: The score vector assets × 1, with one entry for each asset column of X. An entry is NaN at an asset that cannot be scored: a column outside the Coverage Universe under a risk measure, or an asset that the scorer cannot score under an Asset Scorer.

Related

source
PortfolioOptimisers.rule_keep — Function
rule_keep(
    rule::ThresholdRule,
    scores::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    _::Bool
) -> BitVector

Turn the scores of the assets into a keep-mask under a selection rule.

bib is the bigger_is_better flag of the score that gave scores. ThresholdRule ignores it.

There is one method for each rule. The ThresholdRule method compares each score with the open band, and returns the mask. The RankRule method converts each tail to a count with rule_tail_count, where n is the number of finite scores. Then it gives the two counts to tail_action_mask.

A non-finite score is outside every band and outside both tails, under each action. Each method returns false at a NaN and at an infinite score, so an asset with no score never survives the rule. A scorer can mark an asset that it cannot score with NaN, and the rule drops that asset with no exception. If no score is finite, the mask is empty, and fit_preprocessing throws.

Arguments

  • rule: The selection rule.
  • scores: The score vector assets × 1, with one score for each asset.
  • bib: The bigger_is_better flag of the score that gave scores.

Returns

  • keep::BitVector: The mask assets × 1. It is true for every asset that the rule admits, and false at every non-finite score.

Related

source
PortfolioOptimisers.redundancy_keep — Function
redundancy_keep(alg::AbstractRedundancyAlgorithm, rd, scores, bib) -> BitVector

Return the keep-mask that a redundancy algorithm admits.

scores is nothing when the RedundancySelector has no score. Otherwise scores holds one score for each asset, and bib is the bigger_is_better flag of the score.

This method is the fallback of the family. It runs only for an algorithm that has no method of its own, and it always throws.

Arguments

  • alg: The redundancy algorithm.
  • rd: The returns result to use.
  • scores: The score vector assets × 1, or nothing.
  • bib: The bigger_is_better flag of the score.

Validation

  • The concrete algorithm implements redundancy_keep. Otherwise this method throws an ArgumentError that names the two methods of the interface.

Returns

  • keep::BitVector: The mask assets × 1. It is true for every asset that survives.

Related

source
redundancy_keep(
    alg::PairwiseCorrelation,
    rd::AbstractReturnsResult,
    scores::Union{Nothing, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
    bib::Bool
) -> BitVector

Keep the assets that survive PairwiseCorrelation. No pair of them has a correlation of t or more.

The method calls find_uncorrelated_indices. A greedy search gives the set of kept assets, so it is not always the largest such set.

Algorithm

  1. Start from a keep-mask of size(rd.X, 2) falses.
  2. Turn scores into drop scores with drop_scores, so that after this step a lower number is better. When scores is nothing, pass nothing. find_uncorrelated_indices then collapses the finite entries of each column of the correlation matrix with measure into its own drop score.
  3. Call find_uncorrelated_indices on rd.X with the ce, t, absolute and measure of the algorithm. It returns idx, the indices of the assets that survive.
  4. Set the mask at idx, and return it.

Arguments

  • alg: The pairwise correlation algorithm.
  • rd: The returns result to use.
  • scores: The score vector assets × 1, or nothing.
  • bib: The bigger_is_better flag of the score.

Returns

  • keep::BitVector: The mask assets × 1. It is true for every asset that survives.

Related

source
redundancy_keep(
    alg::CorrelationComponents,
    rd::AbstractReturnsResult,
    scores::Union{Nothing, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
    bib::Bool
) -> BitVector

Keep one representative of each correlated group under CorrelationComponents.

Algorithm

  1. Compute the correlation matrix rho of rd.X with the ce of the algorithm.
  2. When absolute is true, replace each entry of rho with its absolute value. This step comes before the threshold, so a strongly negative correlation is an edge.
  3. When scores is nothing, collapse the finite entries of each column of rho with measure into the fallback score s, through finite_vec_to_real_measure, and set sbib to false. The summary skips the undefined correlation with a constant column, so that column does not make every score NaN. The column of a constant asset is NaN at every entry, its diagonal included, so its summary is NaN. Such an asset has no edge, and step 4 gives it a group of its own. Its score is set to zero, so it survives, because groups_argbest keeps no asset with a score that is not finite. A lower summary correlation is then better, so the representative that survives is the asset least correlated with the rest of the universe. Otherwise take s from scores and sbib from bib.
  4. Group the assets with correlation_components on rho and t.
  5. Return the mask that groups_argbest admits for those groups under s and sbib.

Arguments

  • alg: The correlation components algorithm.
  • rd: The returns result to use.
  • scores: The score vector assets × 1, or nothing.
  • bib: The bigger_is_better flag of the score.

Returns

  • keep::BitVector: The mask assets × 1. It is true for one asset of each component whose best score is not tied.

Related

source
redundancy_keep(
    alg::ClusterGroups,
    rd::AbstractReturnsResult,
    scores::Union{Nothing, AbstractVector{<:Union{Number, AbstractJuMPScalar}}},
    bib::Bool
) -> BitVector

Keep one representative of each cluster under ClusterGroups.

Algorithm

  1. Cluster the assets with clusterise on rd.X into the clustering result clr. Give rd too, so that a FeatureDistance reads its panel from rd.
  2. Read the cluster assignment of every asset into idx.
  3. Collect the asset indices of each of the clr.k clusters into groups.
  4. Return the mask that groups_argbest admits for those groups under scores and bib.

The method gives rd alone, because preselection runs before a prior exists. At this site, rd is the only source of a Feature Matrix. ClusterGroups states why the type has no source field.

Arguments

  • alg: The cluster groups algorithm.
  • rd: The returns result to use.
  • scores: The score vector assets × 1. The constructor of RedundancySelector refuses a nothing score for this algorithm, so scores is never nothing here.
  • bib: The bigger_is_better flag of the score.

Returns

  • keep::BitVector: The mask assets × 1. It is true for one asset of each cluster whose best score is not tied.

Related

source
PortfolioOptimisers.requires_score — Function
requires_score(_::AbstractRedundancyAlgorithm) -> Bool

Return whether a redundancy algorithm needs a score to select the survivor of a redundancy group.

The correlation algorithms return false. With no score, they use a summary of the correlation of each asset with the rest of the universe. ClusterGroups has no such fallback, and it returns true.

The generic method returns true, so a RedundancySelector asks a new algorithm for a score until the algorithm declares otherwise. This is the safe default. Without it, an algorithm with no fallback can get nothing in place of a score vector. It then has no data to select a survivor from.

Arguments

  • The method takes the algorithm by type alone. It reads no field of it.

Returns

  • req::Bool: true when the algorithm cannot select the survivor of a group without a score.

Related

source