Feature Distance

struct AngularDist <: Distances.Metric

Normalised angular distance metric.

Unlike Distances.CosineDist ($1 - \cos$), the angular distance satisfies the triangle inequality, so it is a true metric and the hierarchies built from it are well defined. It maps $[-1,\,1] \to [1,\,0]$, so it is bounded, scale-invariant per asset, and admits signed features. Its exact similarity counterpart is AngularSimilarity, which recovers the cosine from the distance alone.

A zero feature vector has no direction, so the cosine is undefined. By convention two zero vectors are at distance 0 from each other (they are identical) and at distance 1 from every non-zero vector (maximally dissimilar), which keeps $S = \cos(\pi D)$ true on every entry of the matching similarity matrix.

Mathematical definition

\[\begin{align} d_{i,\,j} &= \dfrac{1}{\pi}\arccos\left(\dfrac{\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j}}{\lVert\boldsymbol{z}_{i}\rVert \lVert\boldsymbol{z}_{j}\rVert}\right)\,, \end{align}\]

Where:

• $d_{i,\,j}$: Pairwise distance between assets $i$ and $j$.
• $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.

Algorithm

The metric carries two paths, and both are its contract. The elementwise method answers one pair of feature vectors, and Distances._pairwise! answers a whole matrix.

The elementwise method, AngularDist()(a, b):

1. Promote the element types of a and b with Float64, giving T.
2. Take the norms of a and b, giving na and nb.
3. Return zero(T) when both norms are zero, and one(T) when exactly one of them is. This is the zero-feature-vector convention above.
4. Divide the dot product of a and b by na * nb, giving the cosine.
5. Clamp the cosine to $[-1,\,1]$, take its $\arccos$, and divide by $\pi$.

The matrix method, Distances._pairwise!(::AngularDist, r, a). It receives a already permuted to columns-as-observations, so a zero column of a is a zero feature vector:

1. Delegate the whole matrix to the Distances.CosineDist kernel, which writes $1 - \cos$ into r with one BLAS gemm call. That kernel divides by the norm, so a zero column of a leaves NaN in its row and its column of r.
2. Mark the zero columns of a, giving z.
3. Rewrite every entry of r in place: the diagonal to zero(T); a pair of zero columns to zero(T); a zero column against a non-zero one to one(T); every other entry to $\arccos(1 - r_{i,\,j}) / \pi$.

One matrix multiplication replaces $N^{2}$ scalar calls, and it is the faster path from three assets upward. It loses only at $N = 2$, where the single distance it saves does not pay for the call. So there is one matrix path and nothing to tune.

Related

• AngularSimilarity
• FeatureDistance
• default_similarity
• Distances.jl

References

• [44] S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations. arXiv preprint arXiv:1208.3145 (2012).

struct MeanCollapse <: AbstractCollapseAlgorithm

Aggregates along the observation axis with the possibly weighted arithmetic mean.

This is the only collapse algorithm AggregateDistances accepts, because a convex combination of metrics is itself a metric. AggregateFeatures accepts it too, so it is the one member both consumers share, and the default of both.

Mathematical definition

\[\begin{align} \bar{z}_{i,\,k} &= \dfrac{\sum\limits_{t=1}^{T} w_{t} z_{t,\,i,\,k}}{\sum\limits_{t=1}^{T} w_{t}}\,, \end{align}\]

Where:

• $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
• $z_{t,\,i,\,k}$: Feature window entry: feature $k$ of asset $i$ at observation $t$.
• $w_{t}$: Observation weight of observation $t$.
• $T$: Number of observations.

An unweighted collapse sets every $w_{t}$ to $1$. The weights are non-negative and the denominator normalises them, so the aggregate is a convex combination of the window. That is what makes it a metric when it is applied to distance matrices.

Algorithm

1. Reduce the leading observation axis of Z with Statistics.mean, weighted by w when w is not nothing.
2. Drop the reduced axis, giving an assets × features matrix.

Related

• AbstractCollapseAlgorithm
• MedianCollapse
• AggregateFeatures
• AggregateDistances

struct MedianCollapse <: AbstractCollapseAlgorithm

Aggregates along the observation axis with the possibly weighted median, which resists an outlying observation.

Only AggregateFeatures accepts it: it aggregates the features and applies the metric afterwards, so the result is a metric. AggregateDistances rejects it at construction, because an entrywise median of distance matrices need not satisfy the triangle inequality.

A quantile interpolates, so the aggregate need not be an element of the window. Statistics.median(v, w) is the StatsBase $0.5$-quantile rather than an order statistic: on the window [0, 1, 2, 3] under the weights [1, 2, 3, 4] it returns 11/6, which lies strictly between the second and the third value. Interpolation is what the quantile may do rather than what it always does — the same window under the weights [4, 3, 2, 1] returns 1, an element. The unweighted median of an even window averages the two central values for the same reason.

Mathematical definition

\[\begin{align} \bar{z}_{i,\,k} &= Q_{0.5}\left(\left\{z_{t,\,i,\,k}\right\}_{t=1}^{T},\, \left\{w_{t}\right\}_{t=1}^{T}\right)\,, \end{align}\]

Where:

• $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
• $z_{t,\,i,\,k}$: Feature window entry: feature $k$ of asset $i$ at observation $t$.
• $w_{t}$: Observation weight of observation $t$.
• $T$: Number of observations.
• $Q_{0.5}$: The $0.5$-quantile of the window under those weights.

An unweighted collapse sets every $w_{t}$ to $1$.

Algorithm

1. For each asset j and each feature k, take the observation series view(Z, :, j, k).
2. Reduce that series with Statistics.median, weighted by w when w is not nothing, giving the entry of the collapsed matrix.

Related

• AbstractCollapseAlgorithm
• MeanCollapse
• AggregateFeatures

struct LastObservation <: AbstractFeatureCollapseAlgorithm

Discards the window and measures the last observation's feature matrix alone.

The cheapest member of the family and its default, because it is the only one whose result depends on no aggregation choice. It is also the one member that names its rows before the stack exists: collapse_rows answers the last row, so the kernel stacks a window of one observation from an Asset Panel rather than every observation the collapse then discards.

Algorithm

1. Take the last slice of the observation axis, view(Z, size(Z, 1), :, :), giving an assets × features matrix.
2. Apply the metric to that matrix once.

Related

• AbstractFeatureCollapseAlgorithm
• AggregateFeatures
• AggregateDistances
• StackObservations
• collapse_rows

struct AggregateFeatures{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithm

Collapses the window to one assets × features matrix, then applies the metric once.

Each feature is aggregated along the observation axis. The metric runs after the aggregation, so the result is a metric for both MeanCollapse and MedianCollapse, and this is the only consumer that takes the median.

Algorithm

1. Resolve w against Z with collapse_weights, giving a weight vector of one entry per observation, or nothing.
2. Collapse the observation axis of Z with alg, giving one assets × features matrix.
3. Apply the metric to that matrix once, and apply the zero-feature-vector convention to the result.

Fields

• w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.

• alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

AggregateFeatures(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateFeatures

Keywords correspond to the struct's fields.

Validation

• If w is not nothing, !isempty(w).

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• w: Replaced with the incoming ObsWeights.

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

• w: Indexed to the selected observations via obs_weights_view.

Examples

julia> AggregateFeatures()AggregateFeatures    w ┼ nothing  alg ┴ MeanCollapse()

Related

• AbstractFeatureCollapseAlgorithm
• AbstractCollapseAlgorithm
• AggregateDistances
• FeatureDistance
• factory
• obs_weights_view

struct AggregateDistances{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithm

Measures every observation, then aggregates the resulting distance matrices.

Produces one distance matrix per observation and combines them into a single assets × assets matrix. Costs observations metric evaluations against AggregateFeatures's one, and accumulates into a single buffer rather than materialising the whole stack.

Only MeanCollapse is accepted: a convex combination of metrics is a metric, an entrywise median of them is not. Because the metric is applied before the aggregation, the zero-feature convention is applied per observation — an asset that is zero at some observations but not others is treated as zero only in the observations where it is.

Algorithm

1. Resolve w against Z with collapse_weights, giving a weight vector of one entry per observation, or nothing.
2. Allocate the accumulator D and the single per-observation buffer Dt, both assets × assets, and set the weight total sw to zero.
3. For each observation t: measure that slice of Z into Dt; apply the zero-feature-vector convention to Dt; read the observation's weight wt, which is one(T) when w is nothing; add wt .* Dt to D; and add wt to sw.
4. Divide D by sw, giving the convex combination of the per-observation distance matrices.

Fields

• w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.

• alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

AggregateDistances(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateDistances

Keywords correspond to the struct's fields.

Validation

• If w is not nothing, !isempty(w).
• alg is not a MedianCollapse.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• w: Replaced with the incoming ObsWeights.

Observation weight parameters

When obs_weights_view is called on this type, the following fields are automatically indexed to the selected observations:

• w: Indexed to the selected observations via obs_weights_view.

Examples

julia> AggregateDistances()AggregateDistances    w ┼ nothing  alg ┴ MeanCollapse()julia> AggregateDistances(; alg = MedianCollapse())ERROR: ArgumentError: alg must not be a MedianCollapse: an entrywise median of distance matrices need not satisfy the triangle inequality, so the result would not be a metric. Use MeanCollapse, or aggregate the features instead with AggregateFeatures.[...]

Related

• AbstractFeatureCollapseAlgorithm
• AbstractCollapseAlgorithm
• AggregateFeatures
• FeatureDistance
• factory
• obs_weights_view

struct StackObservations <: AbstractFeatureCollapseAlgorithm

Concatenates the window into one long feature vector per asset, so nothing is averaged away.

Turns observations × assets × features into an assets × (observations · features) matrix along the feature axis, and applies the metric once. Two assets are close only when their whole trajectories agree — which is also why the result is dominated by whichever observations carry the largest magnitudes, and why heterogeneous features should be standardised before it is used.

Equals none of the other members of the family in general, but agrees with all of them when observations == 1.

Algorithm

1. Permute Z so the asset axis leads: (2, 1, 3) at dims = 1, and (3, 1, 2) at dims = 2.
2. Reshape the permuted array to assets × (observations · features), giving one long feature vector per asset.
3. Apply the metric to that matrix once, along its first axis.

Related

• AbstractFeatureCollapseAlgorithm
• LastObservation
• AggregateFeatures
• AggregateDistances

struct FeatureDistance{__T_metric, __T_alg, __T_sim, __T_ape, __T_sel, __T_strict} <: AbstractDistanceEstimator

Turns a feature matrix into a distance matrix, by applying a metric to the rows of that matrix.

A feature matrix describes assets by their exposures, memberships, loadings or adjacencies rather than by their returns. This estimator is a peer of Distance and DistanceDistance: unlike them, it never consults a correlation matrix, so it is usable where returns are uninformative or unavailable.

Any Distances.SemiMetric is accepted, including a user-defined one, and every metric yields a similarity, so no combination throws on this path. The remarks below are about the metric a caller chooses, not about this type.

Every metric other than AngularDist and Distances.CorrDist is scale-sensitive, and even AngularDist is invariant to scaling an asset's feature vector but not to scaling a feature across assets. Heterogeneous features should be standardised before use. Distances.CorrDist is NaN against any constant feature vector, hence unusable with a single feature.

Distances.Jaccard is the general non-negative-real (Ruzicka) form, not the binary-set Jaccard, and returns values up to 2 on signed input without erroring. It, Distances.BrayCurtis and Distances.ChiSqDist therefore require a non-negative feature matrix, which assert_metric_domain checks in the kernel rather than at construction, because the feature matrix is not known here.

Choosing the columns

sel names the Panel Fields the Feature Matrix stacks, and nothing stacks every Panel Field's values. Without it this estimator swallows the whole panel, which is harmless while a panel holds only features and wrong the moment it holds anything else.

An entry of sel takes one of four forms, and they mix freely in one vector:

• "industry" is a Panel Field name, and stands for that Panel Field's value columns alone.
• "industry" => ["Tech", "Energy"] keeps the levels or labels it names, in that order.
• "industry" => "Tech" keeps one level or label. This is the form a column label takes.
• "mcap" => :observed is the Panel Field's observed mask, one 0/1 column.

There is no integer entry: every Panel Field, level and label carries a name, so a position has nothing to index. A taxonomy is selected by the name of the categorical Panel Field it entered the panel as.

strict decides what an entry naming a field, a level or a label the panel does not hold does: it throws when strict is true, and warns and drops the entry otherwise.

The order of sel is the column order the metric reads, so a caller decides it. feature_matrix stacks the matrix and feature_labels names its columns, one selector entry per column.

Mathematical definition

\[\begin{align} D_{i,\,j} &= m\left(\boldsymbol{z}_{i},\, \boldsymbol{z}_{j}\right)\\ S_{i,\,j} &= \sigma\left(D_{i,\,j}\right)\,, \end{align}\]

Where:

• $D_{i,\,j}$: Distance between assets $i$ and $j$.
• $S_{i,\,j}$: Similarity between assets $i$ and $j$.
• $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.
• $m$: Distance metric, metric.
• $\sigma$: Similarity transformation, sim.

Fields

• metric: Distance metric applied to the rows of the feature matrix.

• alg: Feature collapse algorithm, used to reduce a window of time-varying features to a single distance matrix. Inert for a 2-D feature matrix.

• sim: Similarity matrix algorithm used to derive the similarity counterpart of the feature distance matrix.

• ape: Asset Panel producer, or nothing to read the panel the data carrier holds. A producer is configuration: it builds a static panel at the point of use, from the prior result and the returns of the subproblem that runs it, so a view passes it through and a fold refits it.

• sel: Feature Selector naming the Panel Fields the Feature Matrix stacks, or nothing to stack every field's values. An entry is a field name, a field paired with the levels or labels it keeps, a field paired with one level or label, or a field paired with :observed. The vector order is the column order.

• strict: Whether a sel entry naming a field, a level or a label the Asset Panel does not hold throws instead of warning and being dropped.

Constructors

FeatureDistance(;    metric::Distances.SemiMetric = AngularDist(),    alg::AbstractFeatureCollapseAlgorithm = LastObservation(),    sim::AbstractSimilarityMatrixAlgorithm = default_similarity(metric),    ape::Option{<:AbstractAssetPanelEstimator} = nothing,    sel::Option{<:AbstractVector} = nothing,    strict::Bool = false) -> FeatureDistance

Keywords correspond to the struct's fields.

Validation

• sim is defaulted from metric via default_similarity, so the resolved value is visible on the printed object rather than hidden inside the distance kernel.
• sel is checked by assert_feature_selector: nothing, or a non-empty vector of distinct entries, each of the four admitted forms.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• alg: Recursively updated via factory.
• ape: Recursively updated via factory.

Examples

julia> FeatureDistance()FeatureDistance  metric ┼ AngularDist: AngularDist()     alg ┼ LastObservation()     sim ┼ AngularSimilarity()     ape ┼ nothing     sel ┼ nothing  strict ┴ Bool: falsejulia> FeatureDistance(; metric = PortfolioOptimisers.Distances.CosineDist())FeatureDistance  metric ┼ Distances.CosineDist: Distances.CosineDist()     alg ┼ LastObservation()     sim ┼ ComplementSimilarity()     ape ┼ nothing     sel ┼ nothing  strict ┴ Bool: false

Related

• AbstractDistanceEstimator
• AngularDist
• AbstractFeatureCollapseAlgorithm
• AbstractSimilarityMatrixAlgorithm
• default_similarity
• Distance
• distance
• cor_and_dist
• assert_metric_domain: the non-negativity check that the three restricted metrics take in the kernel.
• assert_feature_selector: the construction check on sel.
• select_fields: the one resolution of sel against an AssetPanel.
• feature_matrix: the stacking itself.
• feature_labels: one selector entry per column of the stacked matrix.
• DBHT: carries a sim field of its own, deliberately named alike — same type, same job. When both are set DBHT's wins, because clusterise overwrites the similarity matrix immediately after cor_and_dist returns.
• factory

feature_matrix(pnl::AssetPanel, sel = nothing; strict::Bool = false, rows = Colon()) -> Array

Stack the Panel Fields a Feature Selector names into the Feature Matrix a distance measures.

Nothing stores the result. The Asset Panel is the data, and the Feature Matrix is one view of it, so it is built where it is measured and thrown away after.

A static panel gives an assets × features matrix, and a time-varying one an observations × assets × features array. A numeric Panel Field gives one column, a categorical Panel Field one 0/1 column per level, a tensor Panel Field one column per label, and an observed mask one 0/1 column. The order of sel is the column order.

A time-varying panel stacks every observation unless rows names the rows to stack, and then it stacks those alone, length(rows) × assets × features. That is how a consumer that reads one row stacks one row: a FeatureDistance under LastObservation passes the last row through collapse_rows, so a lifted static Panel Field, whose values are a RepeatedLeading, is read once rather than copied once per observation. The stack keeps its observation axis whatever rows holds, so a one-row stack is a window of one observation, on which every collapse algorithm agrees. A static panel has no observation axis, so it takes Colon() alone.

Algorithm

1. Resolve sel against the panel with select_fields.
2. Derive the element type, as the promotion over the Panel Fields whose value columns were resolved. An observed-mask column is a 0/1 column that every type carries, so it contributes nothing, and neither does an indicator. A selection of mask and indicator columns alone stacks in the panel's own type, the promotion over every Panel Field's values, so a Float32 panel's one-hot block is Float32; a panel with no numeric or tensor Panel Field at all stacks in Float64. See panel_value_eltype.
3. Allocate the matrix as zeros, over the observation rows rows names, the panel's asset axis and the resolved column count. See stacked_axes.
4. Write each column, cut to rows, with panel_field_value_column! or panel_field_observed_column!.

Arguments

• pnl: The Asset Panel.
• Feature Selector naming the Panel Fields the Feature Matrix stacks, or nothing to stack every field's values. An entry is a field name, a field paired with the levels or labels it keeps, a field paired with one level or label, or a field paired with :observed. The vector order is the column order.
• Whether a sel entry naming a field, a level or a label the Asset Panel does not hold throws instead of warning and being dropped.
• The observation rows a time-varying Asset Panel stacks, Colon() for every row. A static panel has no observation axis and refuses any other value.

Validation

• The panel holds a Panel Field, and sel resolves to at least one column. See select_fields.
• rows is Colon() on a static panel, and indexes the observation axis of a time-varying one. See stacked_axes.

Returns

• Z::Array: The Feature Matrix, in the type derived over the Panel Fields it stacks.

Related

• AssetPanel
• feature_labels
• select_fields
• stacked_axes
• panel_value_eltype
• FeatureDistance
• collapse_rows

feature_matrix(de::FeatureDistance, pr, rd, X) -> AbstractArray{<:Number}

Stack the Feature Matrix a FeatureDistance measures, from the panel its ape slot resolves.

The resolution has one site. asset_panel(de.ape, pr, rd, X) answers the carrier's panel under a nothing producer and builds one otherwise, and feature_matrix's panel method then stacks the columns de.sel names, over the observation rows de.alg reads. The kernel calls this, and a caller who asks what a clustering measured calls feature_labels with the arguments the optimiser received, so the caller's rebuild is the kernel's measurement by construction.

The rows are the collapse algorithm's, read through collapse_rows: under LastObservation a time-varying panel stacks its last observation alone, 1 × assets × features, which is the slice that collapse measures and the whole of what a caller asking what was measured is answered with. Every other collapse stacks every observation.

Algorithm

1. Resolve the panel with asset_panel.
2. Stack it with feature_matrix, reading de.sel and de.strict, over the rows collapse_rows names for de.alg.

Arguments

• de: Feature distance estimator.
• pr: Prior result or returns result. Both carry the asset returns matrix X and the feature matrix Z, so either can supply them.
• rd: The returns result to use.
• X: Returns matrix of the subproblem, observations × assets. A producer reads it.

Returns

• The Feature Matrix, assets × features or observations × assets × features, where the observation count is the one de.alg reads.

Related

• FeatureDistance
• feature_labels
• collapse_rows
• asset_panel
• AssetPanel
• cor_and_dist

feature_labels(pnl::AssetPanel, sel = nothing; strict::Bool = false) -> Vector

Name the columns feature_matrix stacks, one Feature Selector entry per column.

A label is the entry that selects exactly its column, so the returned vector is itself a Feature Selector and stacking the panel against it rebuilds the same matrix. That is what lets a caller ask what a distance measured without the matrix being stored anywhere.

The kernel never calls this: it reads the matrix alone, so no label is allocated on a path that does not read one.

Algorithm

1. Resolve sel against the panel with select_fields.
2. Name each resolved column with panel_column_label.

Arguments

• pnl: The Asset Panel.
• Feature Selector naming the Panel Fields the Feature Matrix stacks, or nothing to stack every field's values. An entry is a field name, a field paired with the levels or labels it keeps, a field paired with one level or label, or a field paired with :observed. The vector order is the column order.
• Whether a sel entry naming a field, a level or a label the Asset Panel does not hold throws instead of warning and being dropped.

Returns

• labels::Vector: One Feature Selector entry per column of feature_matrix, in column order.

Related

• AssetPanel
• feature_matrix
• select_fields
• panel_column_label

feature_labels(de::FeatureDistance, pr, rd, X) -> Vector

Name each column of the Feature Matrix a FeatureDistance measures.

The sibling of feature_matrix, and it resolves the panel and the selector the same way, so the two agree by construction. A label is the selector entry that selects exactly that column, so the label vector is itself a selector that rebuilds the matrix — which is what a caller who asks what was measured needs.

The kernel never calls it: no clustering or phylogeny result records the labels, because the estimator and the carriers derive them with no distance computed. A caller who wants them calls feature_labels(de, res.pr, rd, rd.X) with the arguments the optimiser received.

Algorithm

1. Resolve the panel with asset_panel.
2. Label it with feature_labels, reading de.sel and de.strict.

Arguments

• de: Feature distance estimator.
• pr: Prior result or returns result. Both carry the asset returns matrix X and the feature matrix Z, so either can supply them.
• rd: The returns result to use.
• X: Returns matrix of the subproblem, observations × assets. A producer reads it.

Returns

• One label per column of the Feature Matrix.

Related

• FeatureDistance
• feature_matrix
• asset_panel
• AssetPanel