Feature Distance: private API

PortfolioOptimisers.AbstractFeatureCollapseAlgorithm — Type
abstract type AbstractFeatureCollapseAlgorithm <: AbstractAlgorithm

Abstract supertype for all feature collapse algorithms.

A feature collapse algorithm reduces a window of time-varying features, assets × features × observations, to one assets × assets distance matrix. It is the alg field of FeatureDistance. It has no effect on a 2-D feature matrix, because a static feature matrix has no observation axis to collapse. At observations == 1, every algorithm of the family gives the same result.

Related

source
PortfolioOptimisers.assert_metric_domain — Function
assert_metric_domain(metric::Distances.SemiMetric, Z::ArrNum, sym::Symbol = :Z)

Assert that Z lies in the domain of metric. The fallback does nothing. Most metrics accept each finite real input, and a non-negativity check on every metric would refuse signed factor loadings and the default metric of FeatureDistance.

Distances.Jaccard, in its Ruzicka form, Distances.BrayCurtis and Distances.ChiSqDist are the exceptions. Their domain is the non-negative reals. The check is most important for Distances.Jaccard, because it gives no error on signed input. It returns values up to 2, and a clustering routine reads them as distances.

Algorithm

  1. Select the method by the type of metric. The three metrics above share one method. Every other metric reaches the Distances.SemiMetric method, which does nothing and returns.
  2. On the method of the three metrics, check that Z is non-negative with assert_nonneg. It raises a DomainError that names sym when an entry is negative.

Arguments

  • metric: Distance metric whose domain Z must lie in.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • sym::Symbol = :Z: Name that the error message gives to Z.

Validation

  • Under Distances.Jaccard, Distances.BrayCurtis and Distances.ChiSqDist: all(x -> x >= 0, Z).

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_feature_matrix — Function
assert_feature_matrix(de::FeatureDistance, Z::ArrNum, dims::Integer)

Validate a feature matrix at the entry point of distance and cor_and_dist. dims selects a valid axis, Z is not empty, every entry is finite, and Z lies in the domain of the metric.

The checks read every cell of Z. On a window that holds an unreadable cell, distance gives it the copy from blank_unreadable, which holds a zero at each such cell. So the checks find a fault only in the readable cells, which are the cells that the collapse reads.

The check refuses a non-finite entry, because no metric makes a usable distance from it. The Minkowski family gives Inf, the cosine family gives NaN, and a clustering cannot use either. The check accepts a degenerate input that a metric can measure. A zero feature vector has a documented convention, which AngularDist states, and a duplicate or constant feature is valid input.

Algorithm

  1. Check dims with assert_dims.
  2. Check that Z is non-empty with assert_nonempty.
  3. Check that every entry of Z is finite with assert_all_finite.
  4. Check that Z lies in the domain of de.metric with assert_metric_domain.

Arguments

  • de: Feature distance estimator, read for its metric.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Validation

  • dims in (1, 2).
  • !isempty(Z).
  • all(isfinite, Z).
  • Z lies in the domain of de.metric. See assert_metric_domain.

Returns

  • nothing.

Related

source
PortfolioOptimisers.zero_feature_vectors — Function
zero_feature_vectors(Z::MatNum, dims::Integer)

Mark the assets whose feature vector is all zero, in the layout that dims states.

Algorithm

  1. Select the asset axis from dims: the rows of Z at dims = 1, and its columns at dims = 2.
  2. Test the feature vector of each asset with all(iszero, ...), to get one entry of the mask per asset.

Arguments

  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Returns

  • z::Vector{Bool}: Mask with one entry per asset, true where the feature vector of that asset is all zero.

Related

source
PortfolioOptimisers.patch_zero_feature_vectors! — Function
patch_zero_feature_vectors!(D::MatNum, Z::MatNum, dims::Integer)

Apply the convention for a zero feature vector to D in place. Two zero vectors are at distance 0, and a zero vector and a non-zero vector are at distance 1.

The function rewrites only the entries that the metric left as NaN. A zero feature vector is valid input, so no check at construction can find it. A metric that divides by a norm cannot measure it. The cosine family gives NaN against every vector, and Distances.Jaccard and Distances.BrayCurtis give NaN between two zero vectors. The Minkowski family puts a zero vector at the origin and measures it. Because the function rewrites only NaN entries, it corrects the first metrics and leaves the Minkowski family unchanged.

This convention keeps $S = \cos(\pi D)$ true at every entry. With it, AngularSimilarity gives +1 between two zero vectors and -1 between a zero vector and a non-zero vector, with a diagonal of ones. Distances.pairwise always writes an exact zero diagonal, so the function does not change the diagonal.

Algorithm

  1. Make the zero mask z of Z with zero_feature_vectors.
  2. Return D unchanged when the mask marks no asset. Most inputs take this step.
  3. Otherwise, read each off-diagonal entry of D. Rewrite it only when the metric left it as NaN and the mask marks at least one of its two assets. Write zero(T) when the mask marks both, and one(T) when it marks exactly one.

Arguments

  • D: Distance matrix assets × assets, rewritten in place.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Returns

  • D::MatNum: Distance matrix assets x assets, in the units the distance algorithm defines.

Related

source
PortfolioOptimisers.feature_distance — Function
feature_distance(metric::Distances.SemiMetric, Z::MatNum, dims::Integer)

Turn a 2-D feature matrix into a distance matrix. Each entry point of FeatureDistance uses this kernel, and three of the collapse algorithms differ only in the matrix that they give it. AggregateDistances does not call it. It does the same two steps once per observation, into one buffer that it uses again, and aggregates the results.

Algorithm

  1. Apply metric to every pair of assets of Z with Distances.pairwise, along the axis that dims names, to get D.
  2. Apply the convention for a zero feature vector to D in place with patch_zero_feature_vectors!.

Arguments

  • metric: Distance metric applied to the assets of Z.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Returns

  • D::MatNum: Distance matrix assets x assets, in the units the distance algorithm defines.

Related

source
feature_distance(de::FeatureDistance, Z::Arr3Num, dims::Integer)

Turn a window of time-varying features into a distance matrix, by the collapse algorithm in de.alg.

Algorithm

The type of de.alg selects one of four methods. The docstring of each type states its method in full. In short:

  1. LastObservation: pass the last slice of the observation axis to the 2-D kernel.
  2. StackObservations: pass the stacked matrix from stack_observations to the 2-D kernel, along its first axis.
  3. AggregateFeatures: resolve the weights with collapse_weights, collapse the window with collapse_features, and pass the collapsed matrix to the 2-D kernel.
  4. AggregateDistances: resolve the weights with collapse_weights, then accumulate one weighted distance matrix per observation and divide by the weight total.

Arguments

  • de: Feature distance estimator, read for its metric and its alg.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Returns

  • D::MatNum: Distance matrix assets x assets, in the units the distance algorithm defines.

Related

source
PortfolioOptimisers.collapse_features — Function
collapse_features(alg::AbstractCollapseAlgorithm, Z::Arr3Num, w::Option{<:VecNum})

Aggregate a window of time-varying features along its last axis, the observation axis. The result is a matrix with the two first axes of Z. AggregateFeatures calls it.

The element type of the result comes from the aggregate, not from the window. The mean and the median of a window of integers can be fractions. A window with an even count of observations can have a fractional median even when each value in it is a whole number. For this reason, both routes make their result from the values that they compute.

Algorithm

The MeanCollapse route:

  1. Reduce the last axis of Z with Statistics.mean, weighted by w when w is not nothing.
  2. Drop the reduced axis.

The MedianCollapse route:

  1. For each entry (j, k) of the two first axes, take the observation series view(Z, j, k, :).
  2. Reduce that series with Statistics.median, weighted by w when w is not nothing, to get one entry of the result.

Arguments

  • alg: Collapse algorithm, which aggregates along the observation axis.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • w: Resolved observation weights, one entry per observation, or nothing for an unweighted collapse.

Returns

  • Zc::Matrix{<:Number}: Collapsed feature matrix, with the two first axes of Z.

Related

source
PortfolioOptimisers.stack_observations — Function
stack_observations(Z::Arr3Num, dims::Integer)

Reshape a window of time-varying features into an assets × (observations · features) matrix. Its rows are the assets, on whichever of the two first axes dims puts them.

Algorithm

  1. Permute Z so that the asset axis is first: (1, 3, 2) at dims = 1, and (2, 3, 1) at dims = 2. Both permutations put the observation axis second and the feature axis third.
  2. Reshape the permuted array to assets × (observations · features), one long feature vector per asset.

Arguments

  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.

Returns

  • Za::Matrix{<:Number}: Stacked feature matrix, assets × (observations · features).

Related

source
PortfolioOptimisers.collapse_weights — Function
collapse_weights(w::Option{<:ObsWeights}, Z::Arr3Num)

Resolve the observation weights of a collapse algorithm against a window of time-varying features.

The function first reshapes Z to an observations × (assets · features) matrix. The documented interface of get_observation_weights is VecNum and MatNum, and a 3-D array matches neither. Without the reshape, a correct MatNum method that a user adds never runs. The caller does not check for nothing. get_observation_weights raises ObservationWeightsError when a DynamicAbstractWeights cannot resolve, so nothing here means only that the caller asked for no weights.

Observation weights for a cross-validation need a DynamicAbstractWeights. It resolves against the Z that it receives, so its length is the length of each fold. A static AbstractWeights has the length that it had at construction, in each fold. Before the length check, AggregateDistances read a longer one by position. It gave the oldest weights to the newest observations, with no bounds error. A shorter one raised a BoundsError with no explanation. The length check now raises a clear error for both.

Algorithm

  1. Permute and reshape Z to observations × (assets · features), so that the observation axis is the first axis of a matrix, as the returns hold it.
  2. Resolve w against that matrix with get_observation_weights, along its first axis.
  3. Check the resolved length against the count of observations of Z, unless the resolution gave nothing.

Arguments

  • w: Optional observation weights vector observations × 1, or a concrete subtype of DynamicAbstractWeights. If nothing, the computation is unweighted.
  • Z: Feature matrix, assets × features when dims = 1 and features × assets when dims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis: assets × features × observations when dims = 1, and features × assets × observations when dims = 2.

Validation

  • length(w) == size(Z, 3) after the resolution. Raises a DimensionMismatch that names both lengths.

Returns

  • w::Option{<:VecNum}: Resolved observation weights, one entry per observation, or nothing when the caller asked for no weights.

Related

source
PortfolioOptimisers.feature_window — Function
feature_window(de::FeatureDistance, pr, rd, X) -> (Z, A)

Stack the Feature Matrix that a FeatureDistance measures, with the mask of the cells that it can read in the stacked rows.

It returns the same stack as feature_matrix, and the mask of that stack. A cell is readable where the asset is active and every value column of de.sel holds data. Data is a value that the raw input held, or a value that a fill policy wrote. The function cuts the mask to the rows that the collapse reads, so the mask has one row per observation of the stack. A static panel has one row, the cells of the assets. Its mask is nothing when every cell holds data, and a 1 × assets matrix otherwise. A time-varying panel whose every cell is readable gives the active mask, and the kernel drops that mask when it is all true.

The stack holds a zero at a cell that holds a placeholder, not the placeholder that the panel stores. The mask leaves that cell out, so no step reads the zero, and the stack does not change with the placeholder.

Algorithm

  1. Resolve the panel with asset_panel, and resolve de.sel against it with select_fields.
  2. Name the rows that the collapse reads, and cut the mask of readable cells to them, with feature_window_mask.
  3. Stack the panel over those rows with feature_stack, with a zero at each cell that holds a placeholder.

Arguments

  • de: Feature distance estimator.
  • pr: Prior result or returns result. Both carry the asset returns matrix X, so either can supply it.
  • rd: The returns result to use.
  • X: Returns matrix of the subproblem, observations × assets. A producer reads it.

Returns

  • Z::Array: The Feature Matrix, as feature_matrix returns it.
  • A::Option{<:AbstractMatrix{Bool}}: The mask of readable cells of the stacked rows, observations × assets, or 1 × assets for a static panel with a placeholder, or nothing.

Related

source
PortfolioOptimisers.window_activity — Function
window_activity(amsk, rows)

Cut a mask of the cells of an Asset Panel to the rows that a Feature Matrix stacks. A static panel has no observation axis. Its mask, one entry per asset, becomes a window of one row, 1 × assets, or nothing when every entry is true. A nothing mask gives nothing.

Arguments

  • amsk: The mask, observations × assets or assets, or nothing.
  • rows: The observation rows a time-varying Asset Panel stacks, as positions or as a Bool mask, and Colon() for every row. A static panel has no observation axis and refuses any other value.

Returns

  • A::Option{<:AbstractMatrix{Bool}}: amsk[rows, :], the one-row window of a static mask, or nothing.

Related

source