GitHub

Feature Distance

PortfolioOptimisers.AngularDist — Type
struct AngularDist <: Distances.Metric

Normalised angular distance metric.

Mathematical definition

\[\begin{align} d_{i,\,j} &= \dfrac{1}{\pi}\arccos\left(\dfrac{\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j}}{\lVert\boldsymbol{z}_{i}\rVert \lVert\boldsymbol{z}_{j}\rVert}\right)\,, \end{align}\]

Where:

  • $d_{i,\,j}$: Pairwise distance between assets $i$ and $j$.
  • $\boldsymbol{z}_{i}$: Feature vector of asset $i$.

Unlike Distances.CosineDist ($1 - \cos$), the angular distance satisfies the triangle inequality, so it is a true metric and the hierarchies built from it are well defined. It maps $[-1,\,1] \to [1,\,0]$, so it is bounded, scale-invariant per asset, and admits signed features. Its exact similarity counterpart is AngularSimilarity, which recovers the cosine from the distance alone.

A zero feature vector has no direction, so the cosine is undefined. By convention two zero vectors are at distance 0 from each other (they are identical) and at distance 1 from every non-zero vector (maximally dissimilar), which keeps $S = \cos(\pi D)$ true on every entry of the matching similarity matrix.

A self-pair is not exactly zero elementwise

AngularDist()(a, a) returns up to 6.707879276254074e-9 rather than 0. The cosine of a vector with itself rounds to 0.9999999999999999, and $\arccos$ has an infinite derivative at 1, so a 1e-16 error there becomes a 1e-8 error in the distance.

Distances.pairwise writes an exact zero diagonal, so the matrix entry points — which are the only route FeatureDistance takes — never see it. Call the metric directly on a pair of identical vectors and the residual is there.

Related

References

  • [46] S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations. arXiv preprint arXiv:1208.3145 (2012).
source
PortfolioOptimisers.MedianCollapse — Type
struct MedianCollapse <: AbstractCollapseAlgorithm

Aggregates along the observation axis with the possibly weighted median, which resists an outlying observation.

Only AggregateFeatures accepts it: it aggregates the features and applies the metric afterwards, so the result is a metric. AggregateDistances rejects it at construction, because an entrywise median of distance matrices need not satisfy the triangle inequality.

Related

source
PortfolioOptimisers.AbstractFeatureCollapseAlgorithm — Type
abstract type AbstractFeatureCollapseAlgorithm <: AbstractAlgorithm

Abstract supertype for all feature collapse algorithms.

A feature collapse algorithm reduces a window of time-varying features, observations × assets × features, to a single assets × assets distance matrix. It is the FeatureDistance alg field, and is inert when the feature matrix is 2-D — a static feature matrix has no observation axis to collapse. At observations == 1 every algorithm in the family agrees exactly.

Related

source
PortfolioOptimisers.AggregateFeatures — Type
struct AggregateFeatures{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithm

Collapses the window to one assets × features matrix, then applies the metric once.

Each feature is aggregated along the observation axis. The metric runs after the aggregation, so the result is a metric for both MeanCollapse and MedianCollapse, and this is the only consumer that takes the median.

Fields

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

AggregateFeatures(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateFeatures

Keywords correspond to the struct's fields.

Validation

  • If w is not nothing, !isempty(w).

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

Examples

julia> AggregateFeatures()AggregateFeatures    w ┼ nothing  alg ┴ MeanCollapse()

Related

source
PortfolioOptimisers.AggregateDistances — Type
struct AggregateDistances{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithm

Measures every observation, then aggregates the resulting distance matrices.

Produces one distance matrix per observation and combines them into a single assets × assets matrix. Costs observations metric evaluations against AggregateFeatures's one, and accumulates into a single buffer rather than materialising the whole stack.

Only MeanCollapse is accepted: a convex combination of metrics is a metric, an entrywise median of them is not. Because the metric is applied before the aggregation, the zero-feature convention is applied per observation — an asset that is zero at some observations but not others is treated as zero only in the observations where it is.

Fields

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

AggregateDistances(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateDistances

Keywords correspond to the struct's fields.

Validation

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

Examples

julia> AggregateDistances()AggregateDistances    w ┼ nothing  alg ┴ MeanCollapse()julia> AggregateDistances(; alg = MedianCollapse())ERROR: ArgumentError: alg must not be a MedianCollapse: an entrywise median of distance matrices need not satisfy the triangle inequality, so the result would not be a metric. Use MeanCollapse, or aggregate the features instead with AggregateFeatures.[...]

Related

source
PortfolioOptimisers.StackObservations — Type
struct StackObservations <: AbstractFeatureCollapseAlgorithm

Concatenates the window into one long feature vector per asset, so nothing is averaged away.

Turns observations × assets × features into an assets × (observations · features) matrix along the feature axis, and applies the metric once. Two assets are close only when their whole trajectories agree — which is also why the result is dominated by whichever observations carry the largest magnitudes, and why heterogeneous features should be standardised before it is used.

Equals none of the other members of the family in general, but agrees with all of them when observations == 1.

Related

source
PortfolioOptimisers.FeatureDistance — Type
struct FeatureDistance{__T_metric, __T_alg, __T_sim} <: AbstractDistanceEstimator

Turns a feature matrix into a distance matrix, by applying a metric to the rows of that matrix.

A feature matrix describes assets by their exposures, memberships, loadings or adjacencies rather than by their returns. This estimator is a peer of Distance and DistanceDistance: unlike them, it never consults a correlation matrix, so it is usable where returns are uninformative or unavailable.

Mathematical definition

\[\begin{align} D_{i,\,j} &= m\left(\boldsymbol{z}_{i},\, \boldsymbol{z}_{j}\right)\\ S_{i,\,j} &= \sigma\left(D_{i,\,j}\right)\,, \end{align}\]

Where:

  • $D_{i,\,j}$: Distance between assets $i$ and $j$.
  • $S_{i,\,j}$: Similarity between assets $i$ and $j$.
  • $\boldsymbol{z}_{i}$: Feature vector of asset $i$.
  • $m$: Distance metric, metric.
  • $\sigma$: Similarity transformation, sim.

Fields

  • metric: Distance metric applied to the rows of the feature matrix.
  • alg: Feature collapse algorithm, used to reduce a window of time-varying features to a single distance matrix. Inert for a 2-D feature matrix.
  • sim: Similarity matrix algorithm used to derive the similarity counterpart of the feature distance matrix.

Constructors

FeatureDistance(;    metric::Distances.SemiMetric = AngularDist(),    alg::AbstractFeatureCollapseAlgorithm = LastObservation(),    sim::AbstractSimilarityMatrixAlgorithm = default_similarity(metric)) -> FeatureDistance

Keywords correspond to the struct's fields.

Validation

  • sim is defaulted from metric via default_similarity, so the resolved value is visible on the printed object rather than hidden inside the distance kernel.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

  • alg: Recursively updated via factory.

Details

  • Any Distances.SemiMetric is accepted, including user-defined ones. Every metric yields a similarity, so no combination throws on this path; a metric returning a distance above 1 gives similarities outside $[-1,\,1]$ under the default ComplementSimilarity, which plot_clusters silently clips. The threshold is 1, not "the metric is unbounded" — Distances.CosineDist and Distances.CorrDist are bounded by 2 and cross it routinely.
  • The claim above is scoped to this path. Handing this estimator to a NetworkEstimator, DBHT or LoGo as their de puts the resulting distance matrix on the PMFG path, where their own similarity field applies rather than sim, and where assert_similarity_domain refuses a distance above 1 under ComplementSimilarity and a non-finite one under MaximumDistanceSimilarity.
  • Every metric other than AngularDist and Distances.CorrDist is scale-sensitive, and even AngularDist is invariant to scaling an asset's feature vector but not to scaling a feature across assets. Heterogeneous features should be standardised before use.
  • Distances.Jaccard is the general non-negative-real (Ruzicka) form, not the binary-set Jaccard, and returns values up to 2 on signed input without erroring. It, Distances.BrayCurtis and Distances.ChiSqDist therefore require a non-negative feature matrix, which is checked in the kernel rather than at construction because the feature matrix is not known here.
  • Distances.CorrDist is NaN against any constant feature vector, hence unusable with a single feature.
  • The field name sim is shared with DBHT's, deliberately: same type, same job. When both are set DBHT's wins, because clusterise overwrites the similarity matrix immediately after cor_and_dist returns.

Examples

julia> FeatureDistance()FeatureDistance  metric ┼ AngularDist: AngularDist()     alg ┼ LastObservation()     sim ┴ AngularSimilarity()julia> FeatureDistance(; metric = PortfolioOptimisers.Distances.CosineDist())FeatureDistance  metric ┼ Distances.CosineDist: Distances.CosineDist()     alg ┼ LastObservation()     sim ┴ ComplementSimilarity()

Related

source
PortfolioOptimisers.assert_metric_domain — Function
assert_metric_domain(metric::Distances.SemiMetric, Z::ArrNum, sym::Symbol = :Z)

Assert that Z lies in metric's domain. The fallback is a no-op: most metrics accept any finite real input, and a blanket non-negativity check would reject signed factor loadings and the FeatureDistance default metric alike.

Distances.Jaccard (the Ruzicka form), Distances.BrayCurtis and Distances.ChiSqDist are the exceptions, all defined only on non-negative reals. The check matters most for Distances.Jaccard, which fails silently: it returns values up to 2 on signed input, with no error, straight into a clustering routine.

Related

source
PortfolioOptimisers.assert_feature_matrix — Function
assert_feature_matrix(de::FeatureDistance, Z::ArrNum, dims::Integer)

Validate a feature matrix at the distance/cor_and_dist entry point: dims selects a valid axis, Z is non-empty, every entry is finite, and Z lies in the metric's domain.

Non-finite entries are rejected because no metric produces a usable distance from them — the Minkowski family gives Inf and the cosine family gives NaN — and neither can be clustered. Structurally degenerate inputs that a metric can handle are admitted: zero feature vectors are given a documented convention (see AngularDist), and duplicate or constant features are legitimate.

Related

source
PortfolioOptimisers.patch_zero_feature_vectors! — Function
patch_zero_feature_vectors!(D::MatNum, Z::MatNum, dims::Integer)

Apply the zero-feature-vector convention to D in place: two zero vectors are at distance 0, a zero vector and a non-zero one at distance 1.

Only entries the metric left as NaN are rewritten. A zero feature vector is structurally valid input, so construction-time validation cannot catch it, but it is undefined for the metrics normalised by a norm — the cosine family gives NaN against anything, and Distances.Jaccard/Distances.BrayCurtis give NaN between two zero vectors. It is perfectly well defined for the Minkowski family, which places it at the origin; restricting the patch to NaN entries fixes the former without corrupting the latter.

The convention is the one that keeps $S = \cos(\pi D)$ true on every entry, so AngularSimilarity yields +1 between two zero vectors and -1 against a non-zero one, with a unit diagonal. Distances.pairwise always writes an exact zero diagonal, so self-distance needs no patching.

Related

source
PortfolioOptimisers.feature_distance — Function
feature_distance(metric::Distances.SemiMetric, Z::MatNum, dims::Integer)

Turn a 2-D feature matrix into a distance matrix. This is the shared kernel behind every FeatureDistance entry point: the collapse algorithms differ only in the matrix they hand it, except for AggregateDistances, which calls it once per observation and aggregates the results.

Related

source
feature_distance(de::FeatureDistance, Z::Arr3Num, dims::Integer)

Turn a window of time-varying features into a distance matrix, by the collapse algorithm in de.alg.

Related

source
PortfolioOptimisers.collapse_weights — Function
collapse_weights(w::Option{<:ObsWeights}, Z::Arr3Num)

Resolve the observation weights of a collapse algorithm against a window of time-varying features.

Z is matricised to observations × (assets · features) first, because get_observation_weights's documented interface is VecNum/MatNum and a raw 3-D array matches neither — a user's correct MatNum method would otherwise never fire. There is no caller-side nothing guard: get_observation_weights raises ObservationWeightsError itself when a DynamicAbstractWeights cannot resolve, so nothing here means only that no weights were requested (ADR 0043).

Validation

  • length(w) == size(Z, 1) once resolved.

Details

  • Cross-fold weighting requires a DynamicAbstractWeights. It resolves against the Z it is handed, so it is fold-local and correct automatically. A static AbstractWeights is fixed at construction and outlives the fold: a longer one used to be read positionally by AggregateDistances, giving the oldest weights to the newest observations with no bounds error, and a shorter one gave a bare BoundsError. The length check makes both loud.

Related

source
PortfolioOptimisers.assert_feature_matrix_supplied — Function
assert_feature_matrix_supplied(Z::Option{<:ArrNum}, z_src::Symbol)

Assert that a feature matrix reached FeatureDistance's three-argument entry point, and name the reason when none did.

Every way of failing to supply Z arrives here identically, as Z === nothing. z_src is the diagnostic that tells them apart — usually resolved by feature_matrix_picker, and riding the wire beside Z purely so this message can be specific:

  • :none: nothing supplied Z at all. The estimator was driven from a raw returns matrix, which carries no feature matrix — the two-argument distance(de, Z; dims) entry point, a ReturnsResult or a prior result is needed.
  • :neither: a carrier was available but neither it nor the returns result holds a feature matrix. The feature matrix has not been supplied or produced.
  • :data / :prior: z_src selected a carrier that holds no feature matrix, while the other one does. This is the typo/wrong-selector case, and the message says which value to use instead.
  • :data_only: the call runs before any prior exists, so the data carrier is the only one that could have supplied a feature matrix and it holds none. It is named for the situation rather than for the caller, so any pre-prior site inherits it; ClusterGroups is the one that exists today. Sending the user to a FeaturePrior — :neither's remedy — would be actively wrong here, because a prior is structurally unreachable from a selector.

Any unrecognised symbol falls through to :neither's text.

Related

source

References

[46]
S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations, arXiv preprint arXiv:1208.3145 (2012).