Feature Distance

PortfolioOptimisers.AngularDist — Type
struct AngularDist <: Distances.Metric

Measures the angle between two feature vectors as a fraction of $\pi$.

The angular distance satisfies the triangle inequality, so it is a true metric, and a hierarchy built from it is well defined. Distances.CosineDist, which is $1 - \cos$, does not satisfy it. Lemma 3 of [111] states that the angle $\arccos$ of a cosine or of a correlation is a metric, and it shows that $1 - \cos$ is not one. Section 1 of [116] divides the angle by $\pi$. The metric maps $[-1,\,1] \to [1,\,0]$, so it is bounded, it is invariant to a positive scale of an asset's feature vector, and it accepts signed features. Its exact similarity counterpart is AngularSimilarity, which recovers the cosine from the distance alone.

A zero feature vector has no direction, so its cosine is undefined. The metric puts two zero vectors at distance 0, because they are identical. It puts a zero vector at distance 1 from every non-zero vector, the largest distance. This convention keeps $S = \cos(\pi D)$ true at every entry of the matching similarity matrix.

Mathematical definition

\[\begin{align} d_{i,\,j} &= \dfrac{1}{\pi}\arccos\left(\dfrac{\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j}}{\lVert\boldsymbol{z}_{i}\rVert \lVert\boldsymbol{z}_{j}\rVert}\right)\,, \end{align}\]

Where:

  • $d_{i,\,j}$: Pairwise distance between assets $i$ and $j$.
  • $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.

Algorithm

The metric has two paths, and both are part of its contract. The elementwise method measures one pair of feature vectors, and Distances._pairwise! measures a whole matrix.

The elementwise method, AngularDist()(a, b):

  1. Promote the element types of a and b, and take T as the type of $\arccos$ of one of that type. $\arccos$ changes an integer or a rational type to a float type, so an Int or a Rational{Int} input gives Float64. Every float type keeps its own type. Distances.result_type gives the same T, so a matrix of the distances has the element type T too.
  2. Take the norm na of a and the norm nb of b.
  3. Return zero(T) when both norms are zero, and one(T) when exactly one of them is zero. This is the convention for a zero feature vector above.
  4. Divide the dot product of a and b by na * nb to get the cosine.
  5. Clamp the cosine to $[-1,\,1]$, take its $\arccos$, and divide by $\pi$.

The matrix method, Distances._pairwise!(::AngularDist, r, a), receives a with one column per asset, so a zero column of a is a zero feature vector:

  1. Pass the whole matrix to the Distances.CosineDist kernel. The kernel writes $1 - \cos$ into r with one BLAS gemm call. It divides by the norm, so a zero column of a leaves NaN in its row and in its column of r.
  2. Mark the zero columns of a in the mask z.
  3. Rewrite every entry of r in place. A diagonal entry and an entry of two zero columns become zero(T). An entry of a zero column and a non-zero column becomes one(T). Every other entry becomes $\arccos(1 - r_{i,\,j}) / \pi$.

One matrix multiplication replaces $N^{2}$ scalar calls, and it is the faster path from three assets upward. At $N = 2$ it is slower, because the one distance that it saves costs less than the call. The loss is at $N = 2$ alone, so the metric has one matrix path and no setting that selects a path.

The two paths differ on the diagonal, and the matrix path is the correct one

$\arccos(1 - r) / \pi$ is the algebraic identity of the elementwise method, not its floating-point result. Off the diagonal, the two paths agree to a few units in the last place. On the diagonal, they differ more. The cosine of a vector with itself rounds to a value near 1, not always to 1. $\arccos$ has an infinite derivative at 1, so it makes that small residual into a much larger error in the distance. The matrix path writes an exact zero instead.

Distances.pairwise writes an exact zero diagonal. FeatureDistance reaches the metric through the matrix entry points alone, so it never sees the residual. A direct call of the metric on two identical vectors returns it. The testset "AngularDist gemm path matches the elementwise method" compares the two paths, and for this reason it compares them with a tolerance.

Related

References

  • [111] S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations. arXiv preprint arXiv:1208.3145 (2012). Section 4, Lemma 3.
  • [116] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In: Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (2002); pp. 380–388. Section 1.
source
PortfolioOptimisers.MeanCollapse — Type
struct MeanCollapse <: AbstractCollapseAlgorithm

Aggregates along the observation axis with the possibly weighted arithmetic mean.

Of the two collapse algorithms, AggregateDistances accepts this one alone, because a convex combination of metrics is a metric. AggregateFeatures accepts it too. It is the default of both.

Mathematical definition

\[\begin{align} \bar{z}_{i,\,k} &= \dfrac{\sum\limits_{t=1}^{T} w_{t} z_{t,\,i,\,k}}{\sum\limits_{t=1}^{T} w_{t}}\,, \end{align}\]

Where:

  • $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
  • $z_{t,\,i,\,k}$: Value of feature $k$ of asset $i$ at observation $t$, one entry of the feature window.
  • $w_{t}$: Observation weight of observation $t$.
  • $T$: Number of observations.

An unweighted collapse sets every $w_{t}$ to $1$. The weights are non-negative and the denominator normalises them, so the aggregate is a convex combination of the window. For this reason, the aggregate of distance matrices is a metric.

Algorithm

  1. Reduce the first axis of Z, the observation axis, with Statistics.mean, weighted by w when w is not nothing.
  2. Drop the reduced axis to get an assets × features matrix.

Related

source
PortfolioOptimisers.MedianCollapse — Type
struct MedianCollapse <: AbstractCollapseAlgorithm

Aggregates along the observation axis with the median, weighted or not, so that one outlying observation moves it little.

AggregateFeatures accepts it, because it aggregates the features and applies the metric after, so the result is a metric. AggregateDistances refuses it at construction, because an entrywise median of distance matrices need not satisfy the triangle inequality.

A quantile interpolates, so the aggregate need not be an element of the window. Statistics.median(v, w) is the StatsBase $0.5$-quantile, not an order statistic. On the window [0, 1, 2, 3] under the weights [1, 2, 3, 4], it returns 11/6, which is strictly between the second and the third value. The interpolation does not always move the result off the window. The same window under the weights [4, 3, 2, 1] returns 1, an element. The unweighted median of an even window is the mean of the two central values for the same reason.

Mathematical definition

\[\begin{align} \bar{z}_{i,\,k} &= Q_{0.5}\left(\left\{z_{t,\,i,\,k}\right\}_{t=1}^{T},\, \left\{w_{t}\right\}_{t=1}^{T}\right)\,, \end{align}\]

Where:

  • $\bar{z}_{i,\,k}$: Collapsed feature $k$ of asset $i$, the aggregate of $z_{t,\,i,\,k}$ over the observation axis.
  • $z_{t,\,i,\,k}$: Value of feature $k$ of asset $i$ at observation $t$, one entry of the feature window.
  • $w_{t}$: Observation weight of observation $t$.
  • $T$: Number of observations.
  • $Q_{0.5}$: The $0.5$-quantile of the window under those weights.

An unweighted collapse sets every $w_{t}$ to $1$.

Algorithm

  1. For each asset j and each feature k, take the observation series view(Z, j, k, :).
  2. Reduce that series with Statistics.median, weighted by w when w is not nothing, to get the entry of the collapsed matrix.

Related

source
PortfolioOptimisers.LastObservation — Type
abstract type LastObservation <: AbstractFeatureCollapseAlgorithm

Abstract supertype for the collapse algorithms that discard the window and measure one row of each asset's features.

The default member, LastRow, is the default collapse of FeatureDistance. The family is the cheapest of the collapse algorithms, and its result depends on no choice of aggregation. A time-varying Asset Panel lists and delists assets, so an asset can be inactive at the last row of the window. Each member states which row of the window the collapse reads for each asset. LastRow reads the last row of the window. LastActiveRow reads the last row at which each asset is readable, where the asset is active and each selected cell holds data, not a placeholder. The family is also the one that names its rows before the stack exists. readable_window_rows returns the rows that the member can read, so the kernel stacks those rows of an Asset Panel alone.

Algorithm

  1. Under LastRow, or when every asset is readable at the last row, take the last slice of the observation axis, view(Z, :, :, size(Z, 3)). This is an assets × features matrix.
  2. Under LastActiveRow, take for each asset the row of its last readable observation, to make an assets × features matrix.
  3. Apply the metric to that matrix once.

Interfaces

To add a rule, subtype LastObservation and implement the two methods below.

collapse_rows

  • collapse_rows(alg::MyRule, pnl::AssetPanel) -> Union{UnitRange{Int}, Colon}: The rows of the panel that the stack of the Feature Matrix holds for the rule.

Arguments

  • alg: The concrete subtype instance.
  • pnl: The Asset Panel the Feature Matrix is stacked from.

Returns

  • A range of observations, or Colon() for every row.

feature_readable

  • feature_readable(alg::MyRule, A::AbstractMatrix{Bool}) -> BitVector: Whether the rule can read each asset of the window.

Arguments

  • alg: The concrete subtype instance.
  • A: The mask of readable cells of the window, observations × assets. A cell is readable when the asset is active and every value column that the Feature Selector names holds data, observed or filled.

Returns

  • One entry per asset, true where the rule reads the asset.

Related

source
PortfolioOptimisers.LastRow — Type
struct LastRow <: LastObservation

Reads every asset at the last row of the window. It is the default collapse of FeatureDistance.

An asset that is inactive at the last row has no value to read. Inside a fit, the entry of the optimiser drops the asset as a non-investable asset. A direct call refuses the asset and names it.

Related

source
PortfolioOptimisers.LastActiveRow — Type
struct LastActiveRow <: LastObservation

Reads each asset at its last readable row of the window, the most recent value that an investor had. A row is readable where the asset is active and each selected cell holds data, so a placeholder moves the read to an earlier row.

The rule reads an asset that delisted before the last row at the last row at which the asset was active. An asset with no active row in the window has no value to read.

Related

source
PortfolioOptimisers.collapse_rows — Function
collapse_rows(alg::AbstractFeatureCollapseAlgorithm, pnl::AssetPanel)

Name the observation rows that a collapse algorithm reads, so that the kernel stacks those rows alone.

The kernel stacks a time-varying Feature Matrix from an Asset Panel, and then collapses it along its observation axis. A collapse that reads one row does not need the other rows.

  • LastRow reads the last row, so it names that row, and the kernel stacks a window of one observation. The kernel then reads a lifted static Panel Field, whose values are a RepeatedObservations, once and not once per observation. Under the default collapse, the work of the kernel is the assets × features slice that it measures.
  • LastActiveRow names the rows from the earliest last active row of an asset to the last row. These rows hold the last active row of each asset that has one.
  • Every other member returns Colon(), which names every row. The two aggregates resolve their weights against the stacked window, as collapse_weights states, so the rows that they could cut are not known before the stack exists. StackObservations reads the whole stack by definition.

A static panel has no observation axis, so every member returns Colon() on it, and each member of LastObservation does too.

Algorithm

The method that Julia selects is the algorithm.

  1. LastRow on a time-varying panel: the last observation, nobs:nobs, where nobs is the observation count panel_axes reads.
  2. LastActiveRow on a time-varying panel: the rows that last_active_rows names on the active mask.
  3. Every other case: Colon().

Arguments

  • alg: The collapse algorithm.
  • pnl: The Asset Panel the Feature Matrix is stacked from.

Returns

  • rows: A range of observations, or Colon(). It is the value of the rows keyword of feature_matrix.

Related

source
PortfolioOptimisers.AggregateFeatures — Type
struct AggregateFeatures{__T_w, __T_alg} <: AbstractFeatureCollapseAlgorithm

Collapses the window to one assets × features matrix, then applies the metric once.

alg aggregates each feature along the observation axis. The metric runs after the aggregation, so the result is a metric under both MeanCollapse and MedianCollapse. No other member of the family accepts the median.

Algorithm

  1. Resolve w against Z with collapse_weights to get a weight vector with one entry per observation, or nothing.
  2. Collapse the observation axis of Z with alg to get one assets × features matrix.
  3. Apply the metric to that matrix once, and apply the convention for a zero feature vector to the result.

Fields

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

AggregateFeatures(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> AggregateFeatures

Keywords correspond to the struct's fields.

Validation

  • If w is not nothing, !isempty(w).

Propagated parameters

factory passes its arguments on to these @fprop fields:

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Examples

julia> AggregateFeatures()AggregateFeatures    w ┼ nothing  alg ┴ MeanCollapse()

Related

source
PortfolioOptimisers.AbstractEmptyPairAlgorithm — Type
abstract type AbstractEmptyPairAlgorithm <: AbstractAlgorithm

Abstract supertype for the rules that give the distance of a pair of assets with no shared active row.

AggregateDistances and StackObservations read each pair of assets at the rows at which both assets are active. Two assets whose active rows do not meet have no such row, so the collapse has nothing to read for the pair. A rule states what the collapse does then. It is the pair field of the two collapses.

Interfaces

To add a rule, subtype AbstractEmptyPairAlgorithm and implement the method below. A rule that drops an asset at the entry of a fit also adds a method of drop_empty_pairs!, whose fallback keeps every asset.

empty_pair_distance!

  • empty_pair_distance!(D::MatNum, pair::MyRule, E::AbstractVector, de::FeatureDistance, win::NamedTuple) -> MatNum: Writes the distance of each empty pair in E into D, or refuses.

Arguments

  • D: The distance matrix, assets × assets, written in place at each pair of E.
  • pair: The concrete subtype instance.
  • E: The empty pairs, (i, j) with i < j.
  • de: The feature distance estimator.
  • win: The window, a NamedTuple of Z, dims, A and nx.

Returns

  • D, with a distance at each pair of E.

Related

source
PortfolioOptimisers.DropFewerRows — Type
struct DropFewerRows <: AbstractEmptyPairAlgorithm

Drops the asset of an empty pair that has fewer active rows, at the entry of a fit.

The entry of the optimiser finds the first pair of assets with no shared active row, drops the asset of the pair with fewer active rows, and repeats until no pair is empty. On a tie, it drops the asset that comes later in the universe. The fit treats a dropped asset as a non-investable asset. It announces the asset, gives it a zero weight, and lists it on the Non-Investable Axis.

A direct call of distance or clusterise has no entry of a fit, so it refuses an empty pair under this rule and names the remedy.

Related

source
PortfolioOptimisers.FeatureFallback — Type
struct FeatureFallback{__T_w, __T_alg} <: AbstractEmptyPairAlgorithm

Measures an empty pair by the features of the two assets, each aggregated over its own active rows.

For a pair of assets with no shared active row, alg collapses the features of each asset over the active rows of that asset, under the weights w cut to those rows. The distance of the pair is then the distance of the two assets as if each asset held its aggregated features at every row of the window. Every other pair keeps the rule of its collapse.

The rule gives one scale to the whole matrix. Under AggregateDistances, the distance of the pair is the metric of the two aggregated feature vectors, which is the value that each row of the window then gives. Under StackObservations, the aggregated feature vector of each asset fills each row of the stack, so a metric of the Minkowski family gives the same distance, times the rescale of the stack at one shared row.

Fields

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • alg: Collapse algorithm, the aggregator applied along the observation axis.

Constructors

FeatureFallback(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse()) -> FeatureFallback

Keywords correspond to the struct's fields.

Validation

  • If w is not nothing, !isempty(w).

Propagated parameters

factory passes its arguments on to these @fprop fields:

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Examples

julia> FeatureFallback()FeatureFallback    w ┼ nothing  alg ┴ MeanCollapse()

Related

source
PortfolioOptimisers.AggregateDistances — Type
struct AggregateDistances{__T_w, __T_alg, __T_pair} <: AbstractFeatureCollapseAlgorithm

Measures each observation, then aggregates the distance matrices of the observations.

It makes one distance matrix per observation and combines them into one assets × assets matrix. It applies the metric once per observation, where AggregateFeatures applies it once. It adds each matrix into one buffer, and it does not keep the matrices of the observations.

It accepts MeanCollapse and refuses MedianCollapse. A convex combination of metrics is a metric, but an entrywise median of metrics need not be one. The metric runs before the aggregation, so the convention for a zero feature vector applies at each observation. An asset whose feature vector is zero at some observations takes the convention at those observations alone.

A window of an Asset Panel can hold an inactive cell. The collapse then reads each pair of assets at the rows at which both assets are active, and it divides the weights of those rows by their sum. This is available-case estimation, the rule by which a Coverage Policy fits a covariance cell. A pair with no shared active row, or with a zero weight at each shared row, takes the rule in pair.

Mathematical definition

\[\begin{align} D_{i,\,j} &= \dfrac{\sum\limits_{t=1}^{T} w_{t} a_{t,\,i} a_{t,\,j} D_{t,\,i,\,j}}{\sum\limits_{t=1}^{T} w_{t} a_{t,\,i} a_{t,\,j}}\,, \end{align}\]

Where:

  • $D_{i,\,j}$: The distance between assets $i$ and $j$.
  • $D_{t,\,i,\,j}$: Distance of assets $i$ and $j$ at observation $t$.
  • $a_{t,\,i}$: Window activity of asset $i$ at observation $t$, 1 when the active mask of the feature window marks the asset active and 0 otherwise.
  • $w_{t}$: Observation weight of observation $t$.
  • $T$: Number of observations.

Algorithm

  1. Resolve w against Z with collapse_weights to get a weight vector with one entry per observation, or nothing.
  2. Allocate the accumulator D and the buffer Dt for one observation, both assets × assets, and set the weight total sw to zero. Their element type T is the result type of the metric, promoted with the type of the weights and passed through float_if_integer. A metric that counts, such as Distances.Hamming, has an integer result type on floating-point features. The weighted mean of counts is a fraction, so such a metric gives a matrix of floating point numbers. A Rational window gives an exact matrix.
  3. For each observation t, measure that slice of Z into Dt, and apply the convention for a zero feature vector to Dt. Then add wt .* Dt to D, and add wt to sw. The weight wt of the observation is one(T) when w is nothing.
  4. Divide D by sw. The result is the convex combination of the distance matrices of the observations.

When the window holds an inactive cell, active_feature_distance keeps one weight total per pair in step 2, adds an entry in step 3 only at a row where both of its assets are active, divides each entry by its own total in step 4, and gives each empty pair to pair.

Fields

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • alg: Collapse algorithm, the aggregator applied along the observation axis.
  • pair: Rule for a pair of assets with no shared active row in the window.

Constructors

AggregateDistances(;    w::Option{<:ObsWeights} = nothing,    alg::AbstractCollapseAlgorithm = MeanCollapse(),    pair::AbstractEmptyPairAlgorithm = RefusePair()) -> AggregateDistances

Keywords correspond to the struct's fields.

Validation

Propagated parameters

factory passes its arguments on to these @fprop fields:

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Examples

julia> AggregateDistances()AggregateDistances     w ┼ nothing   alg ┼ MeanCollapse()  pair ┴ RefusePair()julia> AggregateDistances(; alg = MedianCollapse())ERROR: ArgumentError: alg must not be a MedianCollapse: an entrywise median of distance matrices need not satisfy the triangle inequality, so the result would not be a metric. Use MeanCollapse, or aggregate the features instead with AggregateFeatures.[...]

Related

source
PortfolioOptimisers.StackObservations — Type
struct StackObservations{__T_pair} <: AbstractFeatureCollapseAlgorithm

Concatenates the window into one long feature vector per asset, so the collapse aggregates nothing.

It turns assets × features × observations into an assets × (observations · features) matrix, and applies the metric once. Two assets are near only when their whole trajectories agree. For the same reason, the observations with the largest magnitudes have the largest effect on the result. Standardise features of different scales before you use it.

A metric with one parameter per coordinate, such as Distances.WeightedEuclidean, measures the long vector. So it takes observations · features entries, not one entry per feature. The entries follow the columns of step 2 of the algorithm: the first observations entries are the observations of the first feature, and so on.

In general, its result is different from the result of each other member of the family. At observations == 1, all the members give the same result.

A window of an Asset Panel can hold an inactive cell. For each pair of assets, the collapse then stacks the $n$ rows at which both assets are active. stack_rescale rescales the distance of the pair from those rows to the $T$ rows of the window. A metric that sums over the coordinates, such as the Minkowski family, Distances.SqEuclidean or Distances.ChiSqDist, takes a factor of $T / n$ inside its power. A ratio metric, such as AngularDist or Distances.CosineDist, takes no factor. A metric with no rescale method refuses a pair that shares fewer rows than the window. A pair with no shared active row takes the rule in pair.

Mathematical definition

\[\begin{align} D_{i,\,j} &= \left(\dfrac{T}{n_{i,\,j}}\right)^{1/p} m\left(\boldsymbol{z}_{i,\,\mathcal{S}_{i,\,j}},\, \boldsymbol{z}_{j,\,\mathcal{S}_{i,\,j}}\right)\,, \end{align}\]

Where:

  • $D_{i,\,j}$: The distance between assets $i$ and $j$.
  • $\mathcal{S}_{i,\,j}$: Rows at which both assets $i$ and $j$ are active.
  • $n_{i,\,j}$: Number of rows in $\mathcal{S}_{i,\,j}$.
  • $\boldsymbol{z}_{i,\,\mathcal{S}_{i,\,j}}$: Stacked feature vector of asset $i$ over the rows of $\mathcal{S}_{i,\,j}$.
  • $m$: Distance metric, metric.
  • $p$: Power of the metric: $1$ for Distances.Cityblock, Distances.SqEuclidean and Distances.ChiSqDist, $2$ for Distances.Euclidean, $p$ for Distances.Minkowski, and $\infty$ for a ratio metric, which takes no factor.
  • $T$: Number of observations.

This is the rule that R's stats::dist applies to a missing coordinate. A weighted metric of the Minkowski family takes the ratio of its weight sums in place of $T / n$.

Algorithm

  1. Permute Z so that the asset axis is first: (2, 1, 3) at dims = 1, and (3, 1, 2) at dims = 2.
  2. Reshape the permuted array to assets × (observations · features), one long feature vector per asset.
  3. Apply the metric to that matrix once, along its first axis.

When the window holds an inactive cell, active_feature_distance then measures each pair that shares fewer rows than the window again, over its shared rows, rescales it with stack_rescale, and gives each empty pair to pair.

Fields

  • pair: Rule for a pair of assets with no shared active row in the window.

Constructors

StackObservations(;    pair::AbstractEmptyPairAlgorithm = RefusePair()) -> StackObservations

Keywords correspond to the struct's fields.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • pair: Recursively updated through factory.

Examples

julia> StackObservations()StackObservations  pair ┴ RefusePair()

Related

source
PortfolioOptimisers.FeatureDistance — Type
struct FeatureDistance{__T_metric, __T_alg, __T_sim, __T_ape, __T_sel, __T_strict} <: AbstractDistanceEstimator

Applies a metric to the rows of a feature matrix to make a distance matrix.

A feature matrix describes the assets by their exposures, memberships, loadings or adjacencies, not by their returns. This estimator is a sibling of Distance and DistanceDistance. These two read a correlation matrix. This estimator never reads one, and it works where the returns hold no useful data, or where no returns exist.

The estimator accepts each Distances.SemiMetric, and a user-defined one too. Every metric has a similarity, so no combination throws on this path. The remarks below are about the metric that a caller chooses, not about this type.

A metric is not automatically in the domain of the similarity

ComplementSimilarity is the default for each metric other than AngularDist. Under it, a distance above 1 gives a negative similarity, and a distance above 2 gives a similarity below -1. plot_clusters clips such a similarity with no warning. The first limit is 1, so a bounded metric can cross it too. Distances.CosineDist and Distances.CorrDist have a bound of 2, and they cross 1 at every negative cosine or correlation. Distances.Euclidean has no bound, and it crosses 2 on features of large magnitude.

This warning applies to this path alone. When this estimator is the de of a NetworkEstimator, DBHT or LoGo, its distance matrix goes to the PMFG path. There, the similarity field of that estimator applies, not sim. assert_similarity_domain refuses a distance above 1 under ComplementSimilarity, and a non-finite distance under MaximumDistanceSimilarity.

Every metric other than AngularDist and Distances.CorrDist changes when the feature vector of an asset changes scale. AngularDist is invariant to a positive scale of the feature vector of an asset, but it changes when one feature changes scale across the assets. Standardise features of different scales before use. Distances.CorrDist cannot measure a single feature. Each feature vector is then constant, and Distances.CorrDist leaves NaN against a constant feature vector that is not zero.

Distances.Jaccard is the general Ruzicka form over the non-negative reals, not the Jaccard index of two binary sets. On signed input it returns values up to 2, with no error. Distances.Jaccard, Distances.BrayCurtis and Distances.ChiSqDist need a non-negative feature matrix. The constructor does not know the feature matrix, so assert_metric_domain checks it in the kernel.

Selection of the columns

sel names the Panel Fields that the Feature Matrix stacks. nothing stacks the values of every Panel Field. This is correct while the panel holds features alone. When the panel holds other data too, nothing stacks that data into the Feature Matrix.

An entry of sel has one of five forms, and one vector can mix them:

  • "industry" is a Panel Field name. It selects the value columns of that Panel Field alone.
  • "industry" => ["Tech", "Energy"] keeps the levels or labels that it names, in that order.
  • "industry" => "Tech" keeps one level or label. A column label has this form.
  • "exposures" => LabelGroup("momentum") keeps each label of a tensor Panel Field whose group is "momentum", in the order of the labels. See LabelGroup.
  • "mcap" => :observed is the observed mask of the Panel Field, one 0/1 column.

An entry cannot be an integer. Every Panel Field, level and label has a name, so a position has nothing to index. To select a taxonomy, name the categorical Panel Field that holds it in the panel.

An entry can name a field, a level or a label that the panel does not hold. When strict is true, the entry throws. Otherwise, it warns, and the estimator drops the entry.

The order of sel is the order of the columns that the metric reads, so the caller sets it. feature_matrix stacks the matrix, and feature_labels names its columns, one selector entry per column.

Mathematical definition

\[\begin{align} D_{i,\,j} &= m\left(\boldsymbol{z}_{i},\, \boldsymbol{z}_{j}\right)\\ S_{i,\,j} &= \sigma\left(D_{i,\,j}\right)\,, \end{align}\]

Where:

  • $D_{i,\,j}$: The distance between assets $i$ and $j$.
  • $S_{i,\,j}$: The similarity between assets $i$ and $j$.
  • $\boldsymbol{z}_{i}$: Feature vector of asset $i$, its row of the feature matrix.
  • $m$: Distance metric, metric.
  • $\sigma$: Similarity transformation, sim.

Fields

  • metric: Distance metric applied to the rows of the feature matrix.
  • alg: Feature collapse algorithm, which reduces a window of time-varying features to one distance matrix. It has no effect on a 2-D feature matrix.
  • sim: Similarity matrix algorithm that computes the similarity matrix from the feature distance matrix.
  • ape: Asset Panel producer, or nothing to read the panel that the returns data holds. A producer is configuration. It builds a static panel at the point of use, from the prior result and the returns of the subproblem that runs it. So a view passes it through, and a fold refits it.
  • sel: Feature Selector that names the Panel Fields that the Feature Matrix stacks, or nothing to stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with a LabelGroup, or a field paired with :observed. The order of the vector is the order of the columns.
  • strict: Whether a sel entry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. When strict is false, the entry warns, and the estimator drops it.

Constructors

FeatureDistance(;    metric::Distances.SemiMetric = AngularDist(),    alg::AbstractFeatureCollapseAlgorithm = LastRow(),    sim::AbstractSimilarityMatrixAlgorithm = default_similarity(metric),    ape::Option{<:AbstractAssetPanelEstimator} = nothing,    sel::Option{<:AbstractVector} = nothing,    strict::Bool = false) -> FeatureDistance

Keywords correspond to the struct's fields. The constructor sets sim to default_similarity of metric, so the printed object shows the resolved similarity, and the distance kernel resolves nothing.

Validation

  • assert_feature_selector checks sel. It is nothing, or a non-empty vector of distinct entries, each in one of the five admitted forms.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • alg: Recursively updated through factory.
  • ape: Recursively updated through factory.

Examples

julia> FeatureDistance()FeatureDistance  metric ┼ AngularDist: AngularDist()     alg ┼ LastRow()     sim ┼ AngularSimilarity()     ape ┼ nothing     sel ┼ nothing  strict ┴ Bool: falsejulia> FeatureDistance(; metric = PortfolioOptimisers.Distances.CosineDist())FeatureDistance  metric ┼ Distances.CosineDist: Distances.CosineDist()     alg ┼ LastRow()     sim ┼ ComplementSimilarity()     ape ┼ nothing     sel ┼ nothing  strict ┴ Bool: false

Related

source
PortfolioOptimisers.feature_matrix — Function
feature_matrix(pnl::AssetPanel, sel = nothing; strict::Bool = false, rows = Colon(),
               placeholder::Option{<:Real} = NaN) -> Array

Stack the Panel Fields a Feature Selector names into the Feature Matrix a distance measures.

No object keeps the result. The Asset Panel is the data and the Feature Matrix is one view of it, so the caller that measures a distance builds the matrix and drops it after.

A static panel gives an assets × features matrix, and a time-varying one an assets × features × observations array. A numeric Panel Field gives one column, a categorical Panel Field one 0/1 column per level, a tensor Panel Field one column per label, and an observed mask one 0/1 column. The order of sel is the column order.

A time-varying panel stacks every observation unless rows names the rows to stack. Then it stacks those rows alone, one row of the stack for each row that rows names. A consumer that reads one row stacks one row this way. A FeatureDistance under LastRow passes the last row through collapse_rows, so it reads a lifted static Panel Field, whose values are a RepeatedObservations, once and does not copy it once per observation. The stack keeps its observation axis whatever rows holds. A one-row stack is therefore a window of one observation, and every collapse algorithm agrees on it. A static panel has no observation axis, so it takes Colon() alone.

A cell of a value column holds data where the raw input carried a value or a fill policy wrote one, and the Feature Matrix holds that value. A cell that holds the placeholder of a blank that no fill reached holds no data. The Feature Matrix holds placeholder in its place, NaN by default, so the answer does not change with the placeholder. placeholder = nothing keeps the stored placeholder. An observed-mask column shows where the raw input held a blank, filled or not.

Mathematical definition

\[\begin{align} Z_{s,\,i,\,c} &= \begin{cases} u & c \text{ is a value column of a Panel Field that holds a placeholder, and } d^{(k,l)}_{r_{s},\,i} = 0 \\ v^{(k)}_{r_{s},\,i} & c \text{ is the value column of a numeric Panel Field} \\ \mathbf{1}\left[g^{(k)}_{r_{s},\,i} = l\right] & c \text{ is level } l \text{ of a categorical Panel Field} \\ V^{(k)}_{r_{s},\,i,\,l} & c \text{ is label } l \text{ of a tensor Panel Field} \\ m^{(k)}_{r_{s},\,i} & c \text{ is the observed mask of a numeric or a categorical Panel Field} \\ \prod_{j} m^{(k)}_{r_{s},\,i,\,j} & c \text{ is the observed mask of a tensor Panel Field} \end{cases}\,, \\ & s = 1, \ldots, R\,, \quad i = 1, \ldots, N\,, \quad c = 1, \ldots, C\,. \end{align}\]

A static panel has no observation index, so $s$ and $r_{s}$ drop out and $\mathbf{Z}$ is $N \times C$. A Panel Field that cannot blank has $m^{(k)} = 1$ in every cell. Each code lies on the levels of its Panel Field, so the level columns of one categorical Panel Field sum to one in every row.

Where:

  • $Z_{s,\,i,\,c}$: Entry of the Feature Matrix $\mathbf{Z}$ at stacked row $s$, asset $i$ and column $c$.
  • $r_{s}$: The observation that stacked row $s$ reads.
  • $R$: Number of observations the stack reads.
  • $k$: The Panel Field that column $c$ reads.
  • $l$: Position of the level or the label that column $c$ reads.
  • $v^{(k)}_{t,\,i}$: Value of a numeric Panel Field at observation $t$ and asset $i$.
  • $g^{(k)}_{t,\,i}$: Level position of a categorical Panel Field at observation $t$ and asset $i$.
  • $V^{(k)}_{t,\,i,\,l}$: Value of a tensor Panel Field at observation $t$, asset $i$ and label $l$.
  • $m^{(k)}_{t,\,i}$, $m^{(k)}_{t,\,i,\,j}$: Observed mask of a Panel Field, $1$ where the cell was observed. A tensor Panel Field's mask carries the label index $j$.
  • $d^{(k,l)}_{t,\,i}$: Data mask of the value column that $c$ reads, $0$ where the cell holds a placeholder, see panel_field_column_mask. The first case applies first, and it does not apply to a Panel Field that holds no placeholder.
  • $u$: The value of placeholder. With placeholder = nothing, the first case does not apply.
  • $\mathbf{1}[\cdot]$: Indicator function, $1$ when its condition holds and $0$ otherwise.
  • $C$: Number of columns the Feature Selector resolves to.
  • $N$: Number of assets.

Algorithm

  1. Resolve sel against the panel with select_fields.
  2. Derive the element type as the promotion over the Panel Fields whose value columns sel resolves to. An observed-mask column is a 0/1 column that every type holds, so it contributes nothing, and an indicator contributes nothing either. A selection of mask and indicator columns alone stacks in the panel's own type, the promotion over the values of every Panel Field. So the one-hot block of a Float32 panel is Float32. A panel with no numeric or tensor Panel Field stacks in Float64. See panel_value_eltype.
  3. When a value column reads a Panel Field that holds a placeholder, widen the type to hold placeholder through panel_read_eltype. An integer panel with the default NaN stacks in a floating-point type.
  4. Allocate the matrix as zeros, over the asset axis of the panel, the resolved column count and the observation rows that rows names, in that order. See stacked_axes.
  5. Write each column, cut to rows, with panel_field_value_column! or panel_field_observed_column!. Then write placeholder into each cell of a value column whose mask from panel_field_column_mask is false, through panel_read_write!.

Arguments

  • pnl: The Asset Panel.
  • Feature Selector that names the Panel Fields that the Feature Matrix stacks, or nothing to stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with a LabelGroup, or a field paired with :observed. The order of the vector is the order of the columns.
  • Whether a sel entry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. When strict is false, the entry warns, and the estimator drops it.
  • The observation rows a time-varying Asset Panel stacks, as positions or as a Bool mask, and Colon() for every row. A static panel has no observation axis and refuses any other value.
  • placeholder: The value that the Feature Matrix holds at a value column's cell that holds the placeholder of a blank that no fill reached, or nothing to keep the stored placeholder. A cell that a fill policy wrote holds data, and the Feature Matrix keeps its value. The default NaN shows where no data exists, so no reader takes the placeholder as data.

Validation

  • The panel holds a Panel Field, and sel resolves to at least one column. See select_fields.
  • placeholder converts to the element type of the matrix. A Rational panel that holds a placeholder refuses NaN with an InexactError.
  • rows is Colon() on a static panel, and indexes the observation axis of a time-varying one. See stacked_axes.

Returns

  • Z::Array: The Feature Matrix, in the type derived over the Panel Fields it stacks.

Related

source
feature_matrix(de::FeatureDistance, pr, rd, X) -> AbstractArray{<:Number}

Stack the Feature Matrix that a FeatureDistance measures, from the panel that its ape field resolves.

One call resolves the panel. Under a nothing producer, asset_panel(de.ape, pr, rd, X) returns the panel that the ReturnsResult holds. Otherwise, it builds one. This method returns the stack of feature_window, which is the stack that the kernel measures. The stack holds the columns that de.sel names, over the observation rows that de.alg reads, with a zero at each cell that holds a placeholder. feature_labels resolves the panel through the same call, so the labels that a caller rebuilds name the columns that the kernel measured.

The panel method of feature_matrix writes NaN at a placeholder by default. It gives the same matrix at placeholder = 0, over the same rows.

The collapse algorithm names the rows through readable_window_rows. Under LastRow, a time-varying panel stacks its last observation alone, 1 × assets × features, which is the slice that the collapse measures. With LastActiveRow, it stacks the rows from the earliest last readable row of an asset to the last row. Every other collapse stacks every observation.

Algorithm

  1. Stack the Feature Matrix and the mask of its readable cells with feature_window.
  2. Return the Feature Matrix, and drop the mask.

Arguments

  • de: Feature distance estimator.
  • pr: Prior result or returns result. Both carry the asset returns matrix X, so either can supply it.
  • rd: The returns result to use.
  • X: Returns matrix of the subproblem, observations × assets. A producer reads it.

Validation

  • The panel resolves. asset_panel raises an IsNothingError that names the site when it does not.
  • Under de.strict = true, every entry of de.sel names a Panel Field, a level, a label or a group that the panel holds. Raises an ArgumentError. Under de.strict = false, such an entry warns, and the estimator drops it.
  • The panel holds a Panel Field, and de.sel resolves to at least one column. See select_fields.

Returns

  • Z::Array: The Feature Matrix, assets × features or assets × features × observations, where the observation count is the one that de.alg reads.

Related

source
PortfolioOptimisers.feature_labels — Function
feature_labels(pnl::AssetPanel, sel = nothing; strict::Bool = false) -> Vector

Name the columns feature_matrix stacks, one Feature Selector entry per column.

A label is the entry that selects its own column and no other. The returned vector is therefore a Feature Selector, and a stack of the panel against it rebuilds the same matrix. So a caller can ask what a distance measured, and no object has to keep the matrix.

The distance kernel never calls this function. It reads the matrix alone, so a path that reads no label allocates none.

Algorithm

  1. Resolve sel against the panel with select_fields.
  2. Name each resolved column with panel_column_label.

Arguments

  • pnl: The Asset Panel.
  • Feature Selector that names the Panel Fields that the Feature Matrix stacks, or nothing to stack the values of every field. An entry is a field name, a field paired with the levels or labels that it keeps, a field paired with one level or label, a tensor field paired with a LabelGroup, or a field paired with :observed. The order of the vector is the order of the columns.
  • Whether a sel entry throws when it names a field, a level, a label or a group that the Asset Panel does not hold. When strict is false, the entry warns, and the estimator drops it.

Returns

  • labels::Vector: One Feature Selector entry per column of feature_matrix, in column order.

Related

source
feature_labels(de::FeatureDistance, pr, rd, X) -> Vector

Name each column of the Feature Matrix that a FeatureDistance measures.

It resolves the panel and the selector as feature_matrix does, so the two always agree. A label is the selector entry that selects exactly that column. So the vector of labels is a selector that rebuilds the columns of the matrix, and it tells a caller what the kernel measured.

The kernel never calls it. No clustering or phylogeny result records the labels, because the estimator, pr and rd give them without a distance computation. To get them, call feature_labels(de, res.pr, rd, rd.X) with the arguments that the optimiser received.

Algorithm

  1. Resolve the panel with asset_panel.
  2. Label it with feature_labels, under de.sel and de.strict.

Arguments

  • de: Feature distance estimator.
  • pr: Prior result or returns result. Both carry the asset returns matrix X, so either can supply it.
  • rd: The returns result to use.
  • X: Returns matrix of the subproblem, observations × assets. A producer reads it.

Validation

  • The panel resolves. asset_panel raises an IsNothingError that names the site when it does not.
  • Under de.strict = true, every entry of de.sel names a Panel Field, a level, a label or a group that the panel holds. Raises an ArgumentError. Under de.strict = false, such an entry warns, and the estimator drops it.
  • The panel holds a Panel Field, and de.sel resolves to at least one column. See select_fields.

Returns

  • labels::Vector: One Feature Selector entry per column of the Feature Matrix, in column order.

Related

source
PortfolioOptimisers.LabelGroup — Type
struct LabelGroup{__T_group} <: AbstractAlgorithm

Selects the labels of a tensor Panel Field whose group is one name, as the right side of a Feature Selector entry.

The entry "exposures" => LabelGroup("momentum") selects every label of the tensor Panel Field "exposures" whose group is "momentum", one column per label, in the order of the labels. The groups of a factor exposure tensor are its Factor Families, so this entry selects one family. Two groups take two entries. A string on the right side of an entry already names one label, so a group needs its own type.

A group that the Panel Field does not hold, and a Panel Field that carries no groups, resolve against nothing. The entry is then dropped with a warning, or refused under strict = true, as an unknown label is.

Fields

  • group: The group that the entry selects, one of the entries of the groups of a tensor Panel Field.

Constructors

LabelGroup(group::AbstractString) -> LabelGroupLabelGroup(; group::AbstractString) -> LabelGroup

Validation

Examples

julia> f = TensorPanelField(; name = "exposures", axis = "factor",                            labels = ["mom_12", "value", "mom_6"],                            groups = ["momentum", "value", "momentum"],                            vals = [1.0 2.0 3.0; 4.0 5.0 6.0]);julia> pnl = AssetPanel(; pf = [f]);julia> feature_matrix(pnl, ["exposures" => LabelGroup("momentum")])2×2 Matrix{Float64}: 1.0  3.0 4.0  6.0julia> feature_labels(pnl, ["exposures" => LabelGroup("momentum")])2-element Vector{Pair{String, String}}: "exposures" => "mom_12" "exposures" => "mom_6"

Related

source

References

[111]
S. Van Dongen and A. J. Enright. Metric distances derived from cosine similarity and Pearson and Spearman correlations, arXiv preprint arXiv:1208.3145 (2012).
[116]
M. S. Charikar. Similarity estimation techniques from rounding algorithms. In: Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (2002); pp. 380–388.