Feature Distance
PortfolioOptimisers.AngularDist — Type
struct AngularDist <: Distances.MetricMeasures the angle between two feature vectors as a fraction of $\pi$.
The angular distance satisfies the triangle inequality, so it is a true metric, and a hierarchy built from it is well defined. Distances.CosineDist, which is $1 - \cos$, does not satisfy it. Lemma 3 of [111] states that the angle $\arccos$ of a cosine or of a correlation is a metric, and it shows that $1 - \cos$ is not one. Section 1 of [116] divides the angle by $\pi$. The metric maps $[-1,\,1] \to [1,\,0]$, so it is bounded, it is invariant to a positive scale of an asset's feature vector, and it accepts signed features. Its exact similarity counterpart is AngularSimilarity, which recovers the cosine from the distance alone.
A zero feature vector has no direction, so its cosine is undefined. The metric puts two zero vectors at distance 0, because they are identical. It puts a zero vector at distance 1 from every non-zero vector, the largest distance. This convention keeps $S = \cos(\pi D)$ true at every entry of the matching similarity matrix.
Mathematical definition
\[\begin{align} d_{i,\,j} &= \dfrac{1}{\pi}\arccos\left(\dfrac{\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j}}{\lVert\boldsymbol{z}_{i}\rVert \lVert\boldsymbol{z}_{j}\rVert}\right)\,, \end{align}\]
Where:
- $d_{i,\,j}$: Pairwise distance between assets $i$ and $j$.
- $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.
Algorithm
The metric has two paths, and both are part of its contract. The elementwise method measures one pair of feature vectors, and Distances._pairwise! measures a whole matrix.
The elementwise method, AngularDist()(a, b):
- Promote the element types of
aandb, and takeTas the type of $\arccos$ ofoneof that type. $\arccos$ changes an integer or a rational type to a float type, so anIntor aRational{Int}input givesFloat64. Every float type keeps its own type.Distances.result_typegives the sameT, so a matrix of the distances has the element typeTtoo. - Take the norm
naofaand the normnbofb. - Return
zero(T)when both norms are zero, andone(T)when exactly one of them is zero. This is the convention for a zero feature vector above. - Divide the dot product of
aandbbyna * nbto get the cosine. - Clamp the cosine to $[-1,\,1]$, take its $\arccos$, and divide by $\pi$.
The matrix method, Distances._pairwise!(::AngularDist, r, a), receives a with one column per asset, so a zero column of a is a zero feature vector:
- Pass the whole matrix to the
Distances.CosineDistkernel. The kernel writes $1 - \cos$ intorwith one BLASgemmcall. It divides by the norm, so a zero column ofaleavesNaNin its row and in its column ofr. - Mark the zero columns of
ain the maskz. - Rewrite every entry of
rin place. A diagonal entry and an entry of two zero columns becomezero(T). An entry of a zero column and a non-zero column becomesone(T). Every other entry becomes $\arccos(1 - r_{i,\,j}) / \pi$.
One matrix multiplication replaces $N^{2}$ scalar calls, and it is the faster path from three assets upward. At $N = 2$ it is slower, because the one distance that it saves costs less than the call. The loss is at $N = 2$ alone, so the metric has one matrix path and no setting that selects a path.
$\arccos(1 - r) / \pi$ is the algebraic identity of the elementwise method, not its floating-point result. Off the diagonal, the two paths agree to a few units in the last place. On the diagonal, they differ more. The cosine of a vector with itself rounds to a value near 1, not always to 1. $\arccos$ has an infinite derivative at 1, so it makes that small residual into a much larger error in the distance. The matrix path writes an exact zero instead.
Distances.pairwise writes an exact zero diagonal. FeatureDistance reaches the metric through the matrix entry points alone, so it never sees the residual. A direct call of the metric on two identical vectors returns it. The testset "AngularDist gemm path matches the elementwise method" compares the two paths, and for this reason it compares them with a tolerance.
Related
References
- [111] S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations. arXiv preprint arXiv:1208.3145 (2012). Section 4, Lemma 3.
- [116] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In: Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (2002); pp. 380–388. Section 1.
PortfolioOptimisers.MeanCollapse — Type
struct MeanCollapse <: AbstractCollapseAlgorithmAggregates along the observation axis with the possibly weighted arithmetic mean.
Of the two collapse algorithms, AggregateDistances accepts this one alone, because a convex combination of metrics is a metric. AggregateFeatures accepts it too. It is the default of both.
Mathematical definition
\[\begin{align} \bar{z}_{i,\,k} &= \dfrac{\sum\limits_{t=1}^{T} w_{t} z_{t,\,i,\,k}}{\sum\limits_{t=1}^{T} w_{t}}\,, \end{align}\]
Where:
- $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
- $z_{t,\,i,\,k}$: Value of feature $k$ of asset $i$ at observation $t$, one entry of the feature window.
- $w_{t}$: Observation weight of observation $t$.
- $T$: Number of observations.
An unweighted collapse sets every $w_{t}$ to $1$. The weights are non-negative and the denominator normalises them, so the aggregate is a convex combination of the window. For this reason, the aggregate of distance matrices is a metric.
Algorithm
- Reduce the first axis of
Z, the observation axis, withStatistics.mean, weighted bywwhenwis notnothing. - Drop the reduced axis to get an
assets × featuresmatrix.
Related
PortfolioOptimisers.MedianCollapse — Type
struct MedianCollapse <: AbstractCollapseAlgorithmAggregates along the observation axis with the median, weighted or not, so that one outlying observation moves it little.
AggregateFeatures accepts it, because it aggregates the features and applies the metric after, so the result is a metric. AggregateDistances refuses it at construction, because an entrywise median of distance matrices need not satisfy the triangle inequality.
A quantile interpolates, so the aggregate need not be an element of the window. Statistics.median(v, w) is the StatsBase $0.5$-quantile, not an order statistic. On the window [0, 1, 2, 3] under the weights [1, 2, 3, 4], it returns 11/6, which is strictly between the second and the third value. The interpolation does not always move the result off the window. The same window under the weights [4, 3, 2, 1] returns 1, an element. The unweighted median of an even window is the mean of the two central values for the same reason.
Mathematical definition
\[\begin{align} \bar{z}_{i,\,k} &= Q_{0.5}\left(\left\{z_{t,\,i,\,k}\right\}_{t=1}^{T},\, \left\{w_{t}\right\}_{t=1}^{T}\right)\,, \end{align}\]
Where:
- $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
- $z_{t,\,i,\,k}$: Value of feature $k$ of asset $i$ at observation $t$, one entry of the feature window.
- $w_{t}$: Observation weight of observation $t$.
- $T$: Number of observations.
- $Q_{0.5}$: The $0.5$-quantile of the window under those weights.
An unweighted collapse sets every $w_{t}$ to $1$.
Algorithm
- For each asset
jand each featurek, take the observation seriesview(Z, j, k, :). - Reduce that series with
Statistics.median, weighted bywwhenwis notnothing, to get the entry of the collapsed matrix.
Related
PortfolioOptimisers.LastObservation — Type
abstract type LastObservation <: AbstractFeatureCollapseAlgorithmAbstract supertype for the collapse algorithms that discard the window and measure one row of each asset's features.
The default member, LastRow, is the default collapse of FeatureDistance. The family is the cheapest of the collapse algorithms, and its result depends on no choice of aggregation. A time-varying Asset Panel lists and delists assets, so an asset can be inactive at the last row of the window. Each member states which row of the window the collapse reads for each asset. LastRow reads the last row of the window. LastActiveRow reads the last row at which each asset is readable, where the asset is active and each selected cell holds data, not a placeholder. The family is also the one that names its rows before the stack exists. readable_window_rows returns the rows that the member can read, so the kernel stacks those rows of an Asset Panel alone.
Algorithm
- Under
LastRow, or when every asset is readable at the last row, take the last slice of the observation axis,view(Z, :, :, size(Z, 3)). This is anassets × featuresmatrix. - Under
LastActiveRow, take for each asset the row of its last readable observation, to make anassets × featuresmatrix. - Apply the metric to that matrix once.
Interfaces
To add a rule, subtype LastObservation and implement the two methods below.
collapse_rows
collapse_rows(alg::MyRule, pnl::AssetPanel) -> Union{UnitRange{Int}, Colon}: The rows of the panel that the stack of the Feature Matrix holds for the rule.
Arguments
alg: The concrete subtype instance.pnl: The Asset Panel the Feature Matrix is stacked from.
Returns
- A range of observations, or
Colon()for every row.
feature_readable
feature_readable(alg::MyRule, A::AbstractMatrix{Bool}) -> BitVector: Whether the rule can read each asset of the window.
Arguments
alg: The concrete subtype instance.A: The mask of readable cells of the window,observations × assets. A cell is readable when the asset is active and every value column that the Feature Selector names holds data, observed or filled.
Returns
- One entry per asset,
truewhere the rule reads the asset.
Related
PortfolioOptimisers.LastRow — Type
struct LastRow <: LastObservationReads every asset at the last row of the window. It is the default collapse of FeatureDistance.
An asset that is inactive at the last row has no value to read. Inside a fit, the entry of the optimiser drops the asset as a non-investable asset. A direct call refuses the asset and names it.
Related
PortfolioOptimisers.LastActiveRow — Type
struct LastActiveRow <: LastObservationReads each asset at its last readable row of the window, the most recent value that an investor had. A row is readable where the asset is active and each selected cell holds data, so a placeholder moves the read to an earlier row.
The rule reads an asset that delisted before the last row at the last row at which the asset was active. An asset with no active row in the window has no value to read.
Related
PortfolioOptimisers.collapse_rows — Function
collapse_rows(alg::AbstractFeatureCollapseAlgorithm, pnl::AssetPanel)Name the observation rows that a collapse algorithm reads, so that the kernel stacks those rows alone.
The kernel stacks a time-varying Feature Matrix from an Asset Panel, and then collapses it along its observation axis. A collapse that reads one row does not need the other rows.
LastRowreads the last row, so it names that row, and the kernel stacks a window of one observation. The kernel then reads a lifted static Panel Field, whose values are aRepeatedObservations, once and not once per observation. Under the default collapse, the work of the kernel is theassets × featuresslice that it measures.LastActiveRownames the rows from the earliest last active row of an asset to the last row. These rows hold the last active row of each asset that has one.- Every other member returns
Colon(), which names every row. The two aggregates resolve their weights against the stacked window, ascollapse_weightsstates, so the rows that they could cut are not known before the stack exists.StackObservationsreads the whole stack by definition.
A static panel has no observation axis, so every member returns Colon() on it, and each member of LastObservation does too.
Algorithm
The method that Julia selects is the algorithm.
LastRowon a time-varying panel: the last observation,nobs:nobs, wherenobsis the observation countpanel_axesreads.LastActiveRowon a time-varying panel: the rows thatlast_active_rowsnames on the active mask.- Every other case:
Colon().
Arguments
alg: The collapse algorithm.pnl: The Asset Panel the Feature Matrix is stacked from.
Returns
rows: A range of observations, orColon(). It is the value of therowskeyword offeature_matrix.
Related
PortfolioOptimisers.AggregateFeatures — Type
struct AggregateFeatures{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithmCollapses the window to one assets × features matrix, then applies the metric once.
alg aggregates each feature along the observation axis. The metric runs after the aggregation, so the result is a metric under both MeanCollapse and MedianCollapse. No other member of the family accepts the median.
Algorithm
- Resolve
wagainstZwithcollapse_weightsto get a weight vector with one entry per observation, ornothing. - Collapse the observation axis of
Zwithalgto get oneassets × featuresmatrix. - Apply the metric to that matrix once, and apply the convention for a zero feature vector to the result.
Fields
w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.
alg: Collapse algorithm, the aggregator applied along the observation axis.
Constructors
AggregateFeatures(; w::Option{<:ObsWeights} = nothing, alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateFeaturesKeywords correspond to the struct's fields.
Validation
- If
wis notnothing,!isempty(w).
Propagated parameters
factory passes its arguments on to these @fprop fields:
w: Replaced with the incomingObsWeights.
Observation weight parameters
obs_weights_view indexes these fields to the selected observations:
w: Indexed to the selected observations throughobs_weights_view.
Examples
julia> AggregateFeatures()AggregateFeatures w ┼ nothing alg ┴ MeanCollapse()Related
PortfolioOptimisers.AbstractEmptyPairAlgorithm — Type
abstract type AbstractEmptyPairAlgorithm <: AbstractAlgorithmAbstract supertype for the rules that give the distance of a pair of assets with no shared active row.
AggregateDistances and StackObservations read each pair of assets at the rows at which both assets are active. Two assets whose active rows do not meet have no such row, so the collapse has nothing to read for the pair. A rule states what the collapse does then. It is the pair field of the two collapses.
Interfaces
To add a rule, subtype AbstractEmptyPairAlgorithm and implement the method below. A rule that drops an asset at the entry of a fit also adds a method of drop_empty_pairs!, whose fallback keeps every asset.
empty_pair_distance!
empty_pair_distance!(D::MatNum, pair::MyRule, E::AbstractVector, de::FeatureDistance, win::NamedTuple) -> MatNum: Writes the distance of each empty pair inEintoD, or refuses.
Arguments
D: The distance matrix,assets × assets, written in place at each pair ofE.pair: The concrete subtype instance.E: The empty pairs,(i, j)withi < j.de: The feature distance estimator.win: The window, aNamedTupleofZ,dims,Aandnx.
Returns
D, with a distance at each pair ofE.
Related
PortfolioOptimisers.RefusePair — Type
struct RefusePair <: AbstractEmptyPairAlgorithmRefuses a pair of assets with no shared active row, the default rule for an empty pair.
The refusal names the two assets and the two other rules.
Related
PortfolioOptimisers.DropFewerRows — Type
struct DropFewerRows <: AbstractEmptyPairAlgorithmDrops the asset of an empty pair that has fewer active rows, at the entry of a fit.
The entry of the optimiser finds the first pair of assets with no shared active row, drops the asset of the pair with fewer active rows, and repeats until no pair is empty. On a tie, it drops the asset that comes later in the universe. The fit treats a dropped asset as a non-investable asset. It announces the asset, gives it a zero weight, and lists it on the Non-Investable Axis.
A direct call of distance or clusterise has no entry of a fit, so it refuses an empty pair under this rule and names the remedy.
Related
PortfolioOptimisers.FeatureFallback — Type
struct FeatureFallback{__T_w, __T_alg} <: AbstractEmptyPairAlgorithmMeasures an empty pair by the features of the two assets, each aggregated over its own active rows.
For a pair of assets with no shared active row, alg collapses the features of each asset over the active rows of that asset, under the weights w cut to those rows. The distance of the pair is then the distance of the two assets as if each asset held its aggregated features at every row of the window. Every other pair keeps the rule of its collapse.
The rule gives one scale to the whole matrix. Under AggregateDistances, the distance of the pair is the metric of the two aggregated feature vectors, which is the value that each row of the window then gives. Under StackObservations, the aggregated feature vector of each asset fills each row of the stack, so a metric of the Minkowski family gives the same distance, times the rescale of the stack at one shared row.
Fields
w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.
alg: Collapse algorithm, the aggregator applied along the observation axis.
Constructors
FeatureFallback(; w::Option{<:ObsWeights} = nothing, alg::AbstractCollapseAlgorithm = MeanCollapse()) -> FeatureFallbackKeywords correspond to the struct's fields.
Validation
- If
wis notnothing,!isempty(w).
Propagated parameters
factory passes its arguments on to these @fprop fields:
w: Replaced with the incomingObsWeights.
Observation weight parameters
obs_weights_view indexes these fields to the selected observations:
w: Indexed to the selected observations throughobs_weights_view.
Examples
julia> FeatureFallback()FeatureFallback w ┼ nothing alg ┴ MeanCollapse()Related
PortfolioOptimisers.AggregateDistances — Type
struct AggregateDistances{__T_w, __T_alg, __T_pair} <: AbstractFeatureCollapseAlgorithmMeasures each observation, then aggregates the distance matrices of the observations.
It makes one distance matrix per observation and combines them into one assets × assets matrix. It applies the metric once per observation, where AggregateFeatures applies it once. It adds each matrix into one buffer, and it does not keep the matrices of the observations.
It accepts MeanCollapse and refuses MedianCollapse. A convex combination of metrics is a metric, but an entrywise median of metrics need not be one. The metric runs before the aggregation, so the convention for a zero feature vector applies at each observation. An asset whose feature vector is zero at some observations takes the convention at those observations alone.
A window of an Asset Panel can hold an inactive cell. The collapse then reads each pair of assets at the rows at which both assets are active, and it divides the weights of those rows by their sum. This is available-case estimation, the rule by which a Coverage Policy fits a covariance cell. A pair with no shared active row, or with a zero weight at each shared row, takes the rule in pair.
Mathematical definition
\[\begin{align} D_{i,\,j} &= \dfrac{\sum\limits_{t=1}^{T} w_{t} a_{t,\,i} a_{t,\,j} D_{t,\,i,\,j}}{\sum\limits_{t=1}^{T} w_{t} a_{t,\,i} a_{t,\,j}}\,, \end{align}\]
Where:
- $D_{i,\,j}$: The distance between assets $i$ and $j$.
- $D_{t,\,i,\,j}$: Distance of assets $i$ and $j$ at observation $t$.
- $a_{t,\,i}$: Window activity of asset $i$ at observation $t$,
1when the active mask of the feature window marks the asset active and0otherwise. - $w_{t}$: Observation weight of observation $t$.
- $T$: Number of observations.
Algorithm
- Resolve
wagainstZwithcollapse_weightsto get a weight vector with one entry per observation, ornothing. - Allocate the accumulator
Dand the bufferDtfor one observation, bothassets × assets, and set the weight totalswto zero. Their element typeTis the result type of the metric, promoted with the type of the weights and passed throughfloat_if_integer. A metric that counts, such asDistances.Hamming, has an integer result type on floating-point features. The weighted mean of counts is a fraction, so such a metric gives a matrix of floating point numbers. ARationalwindow gives an exact matrix. - For each observation
t, measure that slice ofZintoDt, and apply the convention for a zero feature vector toDt. Then addwt .* DttoD, and addwttosw. The weightwtof the observation isone(T)whenwisnothing. - Divide
Dbysw. The result is the convex combination of the distance matrices of the observations.
When the window holds an inactive cell, active_feature_distance keeps one weight total per pair in step 2, adds an entry in step 3 only at a row where both of its assets are active, divides each entry by its own total in step 4, and gives each empty pair to pair.
Fields
w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.
alg: Collapse algorithm, the aggregator applied along the observation axis.
pair: Rule for a pair of assets with no shared active row in the window.
Constructors
AggregateDistances(; w::Option{<:ObsWeights} = nothing, alg::AbstractCollapseAlgorithm = MeanCollapse(), pair::AbstractEmptyPairAlgorithm = RefusePair()) -> AggregateDistancesKeywords correspond to the struct's fields.
Validation
- If
wis notnothing,!isempty(w). algis not aMedianCollapse.
Propagated parameters
factory passes its arguments on to these @fprop fields:
w: Replaced with the incomingObsWeights.pair: Recursively updated throughfactory.
Observation weight parameters
obs_weights_view indexes these fields to the selected observations:
w: Indexed to the selected observations throughobs_weights_view.pair: Recursively indexed throughobs_weights_view.
Examples
julia> AggregateDistances()AggregateDistances w ┼ nothing alg ┼ MeanCollapse() pair ┴ RefusePair()julia> AggregateDistances(; alg = MedianCollapse())ERROR: ArgumentError: alg must not be a MedianCollapse: an entrywise median of distance matrices need not satisfy the triangle inequality, so the result would not be a metric. Use MeanCollapse, or aggregate the features instead with AggregateFeatures.[...]Related
PortfolioOptimisers.StackObservations — Type
struct StackObservations{__T_pair} <: AbstractFeatureCollapseAlgorithmConcatenates the window into one long feature vector per asset, so the collapse aggregates nothing.
It turns assets × features × observations into an assets × (observations · features) matrix, and applies the metric once. Two assets are near only when their whole trajectories agree. For the same reason, the observations with the largest magnitudes have the largest effect on the result. Standardise features of different scales before you use it.
A metric with one parameter per coordinate, such as Distances.WeightedEuclidean, measures the long vector. So it takes observations · features entries, not one entry per feature. The entries follow the columns of step 2 of the algorithm: the first observations entries are the observations of the first feature, and so on.
In general, its result is different from the result of each other member of the family. At observations == 1, all the members give the same result.
A window of an Asset Panel can hold an inactive cell. For each pair of assets, the collapse then stacks the $n$ rows at which both assets are active. stack_rescale rescales the distance of the pair from those rows to the $T$ rows of the window. A metric that sums over the coordinates, such as the Minkowski family, Distances.SqEuclidean or Distances.ChiSqDist, takes a factor of $T / n$ inside its power. A ratio metric, such as AngularDist or Distances.CosineDist, takes no factor. A metric with no rescale method refuses a pair that shares fewer rows than the window. A pair with no shared active row takes the rule in pair.
Mathematical definition
\[\begin{align} D_{i,\,j} &= \left(\dfrac{T}{n_{i,\,j}}\right)^{1/p} m\left(\boldsymbol{z}_{i,\,\mathcal{S}_{i,\,j}},\, \boldsymbol{z}_{j,\,\mathcal{S}_{i,\,j}}\right)\,, \end{align}\]
Where:
- $D_{i,\,j}$: The distance between assets $i$ and $j$.
- $\mathcal{S}_{i,\,j}$: Rows at which both assets $i$ and $j$ are active.
- $n_{i,\,j}$: Number of rows in $\mathcal{S}_{i,\,j}$.
- $\boldsymbol{z}_{i,\,\mathcal{S}_{i,\,j}}$: Stacked feature vector of asset $i$ over the rows of $\mathcal{S}_{i,\,j}$.
- $m$: Distance metric,
metric. - $p$: Power of the metric: $1$ for
Distances.Cityblock,Distances.SqEuclideanandDistances.ChiSqDist, $2$ forDistances.Euclidean, $p$ forDistances.Minkowski, and $\infty$ for a ratio metric, which takes no factor. - $T$: Number of observations.
This is the rule that R's stats::dist applies to a missing coordinate. A weighted metric of the Minkowski family takes the ratio of its weight sums in place of $T / n$.
Algorithm
- Permute
Zso that the asset axis is first:(2, 1, 3)atdims = 1, and(3, 1, 2)atdims = 2. - Reshape the permuted array to
assets × (observations · features), one long feature vector per asset. - Apply the metric to that matrix once, along its first axis.
When the window holds an inactive cell, active_feature_distance then measures each pair that shares fewer rows than the window again, over its shared rows, rescales it with stack_rescale, and gives each empty pair to pair.
Fields
pair: Rule for a pair of assets with no shared active row in the window.
Constructors
StackObservations(; pair::AbstractEmptyPairAlgorithm = RefusePair()) -> StackObservationsKeywords correspond to the struct's fields.
Propagated parameters
factory passes its arguments on to these @fprop fields:
pair: Recursively updated throughfactory.
Examples
julia> StackObservations()StackObservations pair ┴ RefusePair()Related
PortfolioOptimisers.FeatureDistance — Type
struct FeatureDistance{__T_metric, __T_alg, __T_sim, __T_ape, __T_sel, __T_strict} <: AbstractDistanceEstimatorApplies a metric to the rows of a feature matrix to make a distance matrix.
A feature matrix describes the assets by their exposures, memberships, loadings or adjacencies, not by their returns. This estimator is a sibling of Distance and DistanceDistance. These two read a correlation matrix. This estimator never reads one, and it works where the returns hold no useful data, or where no returns exist.
The estimator accepts each Distances.SemiMetric, and a user-defined one too. Every metric has a similarity, so no combination throws on this path. The remarks below are about the metric that a caller chooses, not about this type.
ComplementSimilarity is the default for each metric other than AngularDist. Under it, a distance above 1 gives a negative similarity, and a distance above 2 gives a similarity below -1. plot_clusters clips such a similarity with no warning. The first limit is 1, so a bounded metric can cross it too. Distances.CosineDist and Distances.CorrDist have a bound of 2, and they cross 1 at every negative cosine or correlation. Distances.Euclidean has no bound, and it crosses 2 on features of large magnitude.
This warning applies to this path alone. When this estimator is the de of a NetworkEstimator, DBHT or LoGo, its distance matrix goes to the PMFG path. There, the similarity field of that estimator applies, not sim. assert_similarity_domain refuses a distance above 1 under ComplementSimilarity, and a non-finite distance under MaximumDistanceSimilarity.
Every metric other than AngularDist and Distances.CorrDist changes when the feature vector of an asset changes scale. AngularDist is invariant to a positive scale of the feature vector of an asset, but it changes when one feature changes scale across the assets. Standardise features of different scales before use. Distances.CorrDist cannot measure a single feature. Each feature vector is then constant, and Distances.CorrDist leaves NaN against a constant feature vector that is not zero.
Distances.Jaccard is the general Ruzicka form over the non-negative reals, not the Jaccard index of two binary sets. On signed input it returns values up to 2, with no error. Distances.Jaccard, Distances.BrayCurtis and Distances.ChiSqDist need a non-negative feature matrix. The constructor does not know the feature matrix, so assert_metric_domain checks it in the kernel.
Selection of the columns
sel names the Panel Fields that the Feature Matrix stacks. nothing stacks the values of every Panel Field. This is correct while the panel holds features alone. When the panel holds other data too, nothing stacks that data into the Feature Matrix.
An entry of sel has one of five forms, and one vector can mix them:
"industry"is a Panel Field name. It selects the value columns of that Panel Field alone."industry" => ["Tech", "Energy"]keeps the levels or labels that it names, in that order."industry" => "Tech"keeps one level or label. A column label has this form."exposures" => LabelGroup("momentum")keeps each label of a tensor Panel Field whose group is"momentum", in the order of the labels. SeeLabelGroup."mcap" => :observedis the observed mask of the Panel Field, one0/1column.
An entry cannot be an integer. Every Panel Field, level and label has a name, so a position has nothing to index. To select a taxonomy, name the categorical Panel Field that holds it in the panel.
An entry can name a field, a level or a label that the panel does not hold. When strict is true, the entry throws. Otherwise, it warns, and the estimator drops the entry.
The order of sel is the order of the columns that the metric reads, so the caller sets it. feature_matrix stacks the matrix, and feature_labels names its columns, one selector entry per column.
Mathematical definition
\[\begin{align} D_{i,\,j} &= m\left(\boldsymbol{z}_{i},\, \boldsymbol{z}_{j}\right)\\ S_{i,\,j} &= \sigma\left(D_{i,\,j}\right)\,, \end{align}\]
Where:
- $D_{i,\,j}$: The distance between assets $i$ and $j$.
- $S_{i,\,j}$: The similarity between assets $i$ and $j$.
- $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.
- $m$: Distance metric,
metric. - $\sigma$: Similarity transformation,
sim.
Fields
metric: Distance metric applied to the rows of the feature matrix.
alg: Feature collapse algorithm, which reduces a window of time-varying features to one distance matrix. It has no effect on a 2-D feature matrix.
sim: Similarity matrix algorithm that computes the similarity matrix from the feature distance matrix.
ape: Asset Panel producer, ornothingto read the panel that the returns data holds. A producer is configuration. It builds a static panel at the point of use, from the prior result and the returns of the subproblem that runs it. So a view passes it through, and a fold refits it.
sel: Feature Selector that names the Panel Fields that the Feature Matrix stacks, ornothingto stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with aLabelGroup, or a field paired with:observed. The order of the vector is the order of the columns.
strict: Whether aselentry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. Whenstrictisfalse, the entry warns, and the estimator drops it.
Constructors
FeatureDistance(; metric::Distances.SemiMetric = AngularDist(), alg::AbstractFeatureCollapseAlgorithm = LastRow(), sim::AbstractSimilarityMatrixAlgorithm = default_similarity(metric), ape::Option{<:AbstractAssetPanelEstimator} = nothing, sel::Option{<:AbstractVector} = nothing, strict::Bool = false) -> FeatureDistanceKeywords correspond to the struct's fields. The constructor sets sim to default_similarity of metric, so the printed object shows the resolved similarity, and the distance kernel resolves nothing.
Validation
assert_feature_selectorcheckssel. It isnothing, or a non-empty vector of distinct entries, each in one of the five admitted forms.
Propagated parameters
factory passes its arguments on to these @fprop fields:
Examples
julia> FeatureDistance()FeatureDistance metric ┼ AngularDist: AngularDist() alg ┼ LastRow() sim ┼ AngularSimilarity() ape ┼ nothing sel ┼ nothing strict ┴ Bool: falsejulia> FeatureDistance(; metric = PortfolioOptimisers.Distances.CosineDist())FeatureDistance metric ┼ Distances.CosineDist: Distances.CosineDist() alg ┼ LastRow() sim ┼ ComplementSimilarity() ape ┼ nothing sel ┼ nothing strict ┴ Bool: falseRelated
AbstractDistanceEstimatorAngularDistAbstractFeatureCollapseAlgorithmAbstractSimilarityMatrixAlgorithmdefault_similarityDistancedistancecor_and_distassert_metric_domain: the check in the kernel that the feature matrix of the three restricted metrics is non-negative.assert_feature_selector: the check ofselat construction.select_fields: the one resolution ofselagainst anAssetPanel.feature_matrix: the function that stacks the Feature Matrix.feature_labels: one selector entry per column of the stacked matrix.DBHT: has asimfield of its own, with the same name, the same type and the same function. When both are set, the field ofDBHTapplies, becauseclusterisewrites over the similarity matrix aftercor_and_distreturns.factory
PortfolioOptimisers.feature_matrix — Function
feature_matrix(pnl::AssetPanel, sel = nothing; strict::Bool = false, rows = Colon(),
placeholder::Option{<:Real} = NaN) -> ArrayStack the Panel Fields a Feature Selector names into the Feature Matrix a distance measures.
No object keeps the result. The Asset Panel is the data and the Feature Matrix is one view of it, so the caller that measures a distance builds the matrix and drops it after.
A static panel gives an assets × features matrix, and a time-varying one an assets × features × observations array. A numeric Panel Field gives one column, a categorical Panel Field one 0/1 column per level, a tensor Panel Field one column per label, and an observed mask one 0/1 column. The order of sel is the column order.
A time-varying panel stacks every observation unless rows names the rows to stack. Then it stacks those rows alone, one row of the stack for each row that rows names. A consumer that reads one row stacks one row this way. A FeatureDistance under LastRow passes the last row through collapse_rows, so it reads a lifted static Panel Field, whose values are a RepeatedObservations, once and does not copy it once per observation. The stack keeps its observation axis whatever rows holds. A one-row stack is therefore a window of one observation, and every collapse algorithm agrees on it. A static panel has no observation axis, so it takes Colon() alone.
A cell of a value column holds data where the raw input carried a value or a fill policy wrote one, and the Feature Matrix holds that value. A cell that holds the placeholder of a blank that no fill reached holds no data. The Feature Matrix holds placeholder in its place, NaN by default, so the answer does not change with the placeholder. placeholder = nothing keeps the stored placeholder. An observed-mask column shows where the raw input held a blank, filled or not.
Mathematical definition
\[\begin{align} Z_{s,\,i,\,c} &= \begin{cases} u & c \text{ is a value column of a Panel Field that holds a placeholder, and } d^{(k,l)}_{r_{s},\,i} = 0 \\ v^{(k)}_{r_{s},\,i} & c \text{ is the value column of a numeric Panel Field} \\ \mathbf{1}\left[g^{(k)}_{r_{s},\,i} = l\right] & c \text{ is level } l \text{ of a categorical Panel Field} \\ V^{(k)}_{r_{s},\,i,\,l} & c \text{ is label } l \text{ of a tensor Panel Field} \\ m^{(k)}_{r_{s},\,i} & c \text{ is the observed mask of a numeric or a categorical Panel Field} \\ \prod_{j} m^{(k)}_{r_{s},\,i,\,j} & c \text{ is the observed mask of a tensor Panel Field} \end{cases}\,, \\ & s = 1, \ldots, R\,, \quad i = 1, \ldots, N\,, \quad c = 1, \ldots, C\,. \end{align}\]
A static panel has no observation index, so $s$ and $r_{s}$ drop out and $\mathbf{Z}$ is $N \times C$. A Panel Field that cannot blank has $m^{(k)} = 1$ in every cell. Each code lies on the levels of its Panel Field, so the level columns of one categorical Panel Field sum to one in every row.
Where:
- $Z_{s,\,i,\,c}$: Entry of the Feature Matrix $\mathbf{Z}$ at stacked row $s$, asset $i$ and column $c$.
- $r_{s}$: The observation that stacked row $s$ reads.
- $R$: Number of observations the stack reads.
- $k$: The Panel Field that column $c$ reads.
- $l$: Position of the level or the label that column $c$ reads.
- $v^{(k)}_{t,\,i}$: Value of a numeric Panel Field at observation $t$ and asset $i$.
- $g^{(k)}_{t,\,i}$: Level position of a categorical Panel Field at observation $t$ and asset $i$.
- $V^{(k)}_{t,\,i,\,l}$: Value of a tensor Panel Field at observation $t$, asset $i$ and label $l$.
- $m^{(k)}_{t,\,i}$, $m^{(k)}_{t,\,i,\,j}$: Observed mask of a Panel Field, $1$ where the cell was observed. A tensor Panel Field's mask carries the label index $j$.
- $d^{(k,l)}_{t,\,i}$: Data mask of the value column that $c$ reads, $0$ where the cell holds a placeholder, see
panel_field_column_mask. The first case applies first, and it does not apply to a Panel Field that holds no placeholder. - $u$: The value of
placeholder. Withplaceholder = nothing, the first case does not apply. - $\mathbf{1}[\cdot]$: Indicator function, $1$ when its condition holds and $0$ otherwise.
- $C$: Number of columns the Feature Selector resolves to.
- $N$: Number of assets.
Algorithm
- Resolve
selagainst the panel withselect_fields. - Derive the element type as the promotion over the Panel Fields whose value columns
selresolves to. An observed-mask column is a0/1column that every type holds, so it contributes nothing, and an indicator contributes nothing either. A selection of mask and indicator columns alone stacks in the panel's own type, the promotion over the values of every Panel Field. So the one-hot block of aFloat32panel isFloat32. A panel with no numeric or tensor Panel Field stacks inFloat64. Seepanel_value_eltype. - When a value column reads a Panel Field that holds a placeholder, widen the type to hold
placeholderthroughpanel_read_eltype. An integer panel with the defaultNaNstacks in a floating-point type. - Allocate the matrix as zeros, over the asset axis of the panel, the resolved column count and the observation rows that
rowsnames, in that order. Seestacked_axes. - Write each column, cut to
rows, withpanel_field_value_column!orpanel_field_observed_column!. Then writeplaceholderinto each cell of a value column whose mask frompanel_field_column_maskisfalse, throughpanel_read_write!.
Arguments
pnl: The Asset Panel.- Feature Selector that names the Panel Fields that the Feature Matrix stacks, or
nothingto stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with aLabelGroup, or a field paired with:observed. The order of the vector is the order of the columns. - Whether a
selentry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. Whenstrictisfalse, the entry warns, and the estimator drops it. - The observation rows a time-varying Asset Panel stacks, as positions or as a
Boolmask, andColon()for every row. A static panel has no observation axis and refuses any other value. placeholder: The value that the Feature Matrix holds at a value column's cell that holds the placeholder of a blank that no fill reached, ornothingto keep the stored placeholder. A cell that a fill policy wrote holds data, and the Feature Matrix keeps its value. The defaultNaNshows where no data exists, so no reader takes the placeholder as data.
Validation
- The panel holds a Panel Field, and
selresolves to at least one column. Seeselect_fields. placeholderconverts to the element type of the matrix. ARationalpanel that holds a placeholder refusesNaNwith anInexactError.rowsisColon()on a static panel, and indexes the observation axis of a time-varying one. Seestacked_axes.
Returns
Z::Array: The Feature Matrix, in the type derived over the Panel Fields it stacks.
Related
feature_matrix(de::FeatureDistance, pr, rd, X) -> AbstractArray{<:Number}Stack the Feature Matrix that a FeatureDistance measures, from the panel that its ape field resolves.
One call resolves the panel. Under a nothing producer, asset_panel(de.ape, pr, rd, X) returns the panel that the ReturnsResult holds. Otherwise, it builds one. This method returns the stack of feature_window, which is the stack that the kernel measures. The stack holds the columns that de.sel names, over the observation rows that de.alg reads, with a zero at each cell that holds a placeholder. feature_labels resolves the panel through the same call, so the labels that a caller rebuilds name the columns that the kernel measured.
The panel method of feature_matrix writes NaN at a placeholder by default. It gives the same matrix at placeholder = 0, over the same rows.
The collapse algorithm names the rows through readable_window_rows. Under LastRow, a time-varying panel stacks its last observation alone, 1 × assets × features, which is the slice that the collapse measures. With LastActiveRow, it stacks the rows from the earliest last readable row of an asset to the last row. Every other collapse stacks every observation.
Algorithm
- Stack the Feature Matrix and the mask of its readable cells with
feature_window. - Return the Feature Matrix, and drop the mask.
Arguments
de: Feature distance estimator.pr: Prior result or returns result. Both carry the asset returns matrixX, so either can supply it.rd: The returns result to use.X: Returns matrix of the subproblem,observations × assets. A producer reads it.
Validation
- The panel resolves.
asset_panelraises anIsNothingErrorthat names the site when it does not. - Under
de.strict = true, every entry ofde.selnames a Panel Field, a level, a label or a group that the panel holds. Raises anArgumentError. Underde.strict = false, such an entry warns, and the estimator drops it. - The panel holds a Panel Field, and
de.selresolves to at least one column. Seeselect_fields.
Returns
Z::Array: The Feature Matrix,assets × featuresorassets × features × observations, where the observation count is the one thatde.algreads.
Related
PortfolioOptimisers.feature_labels — Function
feature_labels(pnl::AssetPanel, sel = nothing; strict::Bool = false) -> VectorName the columns feature_matrix stacks, one Feature Selector entry per column.
A label is the entry that selects its own column and no other. The returned vector is therefore a Feature Selector, and a stack of the panel against it rebuilds the same matrix. So a caller can ask what a distance measured, and no object has to keep the matrix.
The distance kernel never calls this function. It reads the matrix alone, so a path that reads no label allocates none.
Algorithm
- Resolve
selagainst the panel withselect_fields. - Name each resolved column with
panel_column_label.
Arguments
pnl: The Asset Panel.- Feature Selector that names the Panel Fields that the Feature Matrix stacks, or
nothingto stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with aLabelGroup, or a field paired with:observed. The order of the vector is the order of the columns. - Whether a
selentry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. Whenstrictisfalse, the entry warns, and the estimator drops it.
Returns
labels::Vector: One Feature Selector entry per column offeature_matrix, in column order.
Related
feature_labels(de::FeatureDistance, pr, rd, X) -> VectorName each column of the Feature Matrix that a FeatureDistance measures.
It resolves the panel and the selector as feature_matrix does, so the two always agree. A label is the selector entry that selects exactly that column. So the vector of labels is a selector that rebuilds the columns of the matrix, and it tells a caller what the kernel measured.
The kernel never calls it. No clustering or phylogeny result records the labels, because the estimator, pr and rd give them without a distance computation. To get them, call feature_labels(de, res.pr, rd, rd.X) with the arguments that the optimiser received.
Algorithm
- Resolve the panel with
asset_panel. - Label it with
feature_labels, underde.selandde.strict.
Arguments
de: Feature distance estimator.pr: Prior result or returns result. Both carry the asset returns matrixX, so either can supply it.rd: The returns result to use.X: Returns matrix of the subproblem,observations × assets. A producer reads it.
Validation
- The panel resolves.
asset_panelraises anIsNothingErrorthat names the site when it does not. - Under
de.strict = true, every entry ofde.selnames a Panel Field, a level, a label or a group that the panel holds. Raises anArgumentError. Underde.strict = false, such an entry warns, and the estimator drops it. - The panel holds a Panel Field, and
de.selresolves to at least one column. Seeselect_fields.
Returns
labels::Vector: One Feature Selector entry per column of the Feature Matrix, in column order.
Related
PortfolioOptimisers.LabelGroup — Type
struct LabelGroup{__T_group} <: AbstractAlgorithmSelects the labels of a tensor Panel Field whose group is one name, as the right side of a Feature Selector entry.
The entry "exposures" => LabelGroup("momentum") selects every label of the tensor Panel Field "exposures" whose group is "momentum", one column per label, in the order of the labels. The groups of a factor exposure tensor are its Factor Families, so this entry selects one family. Two groups take two entries. A string on the right side of an entry already names one label, so a group needs its own type.
A group that the Panel Field does not hold, and a Panel Field that carries no groups, resolve against nothing. The entry is then dropped with a warning, or refused under strict = true, as an unknown label is.
Fields
group: The group that the entry selects, one of the entries of thegroupsof a tensor Panel Field.
Constructors
LabelGroup(group::AbstractString) -> LabelGroupLabelGroup(; group::AbstractString) -> LabelGroupValidation
!isempty(group). Raises anIsEmptyError.
Examples
julia> f = TensorPanelField(; name = "exposures", axis = "factor", labels = ["mom_12", "value", "mom_6"], groups = ["momentum", "value", "momentum"], vals = [1.0 2.0 3.0; 4.0 5.0 6.0]);julia> pnl = AssetPanel(; pf = [f]);julia> feature_matrix(pnl, ["exposures" => LabelGroup("momentum")])2×2 Matrix{Float64}: 1.0 3.0 4.0 6.0julia> feature_labels(pnl, ["exposures" => LabelGroup("momentum")])2-element Vector{Pair{String, String}}: "exposures" => "mom_12" "exposures" => "mom_6"Related
References
- [111]
- S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations, arXiv preprint arXiv:1208.3145 (2012).
- [116]
- M. S. Charikar. Similarity estimation techniques from rounding algorithms. In: Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (2002); pp. 380–388.