Feature Distance: private API
PortfolioOptimisers.AbstractCollapseAlgorithm — Type
abstract type AbstractCollapseAlgorithm <: AbstractAlgorithmAbstract supertype for all collapse algorithms.
A collapse algorithm states how to aggregate along the observation axis of a window of time-varying features. AggregateFeatures and AggregateDistances read it. These two state what to aggregate, the features or the distances, so the two families are independent.
Related
PortfolioOptimisers.AbstractFeatureCollapseAlgorithm — Type
abstract type AbstractFeatureCollapseAlgorithm <: AbstractAlgorithmAbstract supertype for all feature collapse algorithms.
A feature collapse algorithm reduces a window of time-varying features, assets × features × observations, to one assets × assets distance matrix. It is the alg field of FeatureDistance. It has no effect on a 2-D feature matrix, because a static feature matrix has no observation axis to collapse. At observations == 1, every algorithm of the family gives the same result.
Related
PortfolioOptimisers.assert_metric_domain — Function
assert_metric_domain(metric::Distances.SemiMetric, Z::ArrNum, sym::Symbol = :Z)Assert that Z lies in the domain of metric. The fallback does nothing. Most metrics accept each finite real input, and a non-negativity check on every metric would refuse signed factor loadings and the default metric of FeatureDistance.
Distances.Jaccard, in its Ruzicka form, Distances.BrayCurtis and Distances.ChiSqDist are the exceptions. Their domain is the non-negative reals. The check is most important for Distances.Jaccard, because it gives no error on signed input. It returns values up to 2, and a clustering routine reads them as distances.
Algorithm
- Select the method by the type of
metric. The three metrics above share one method. Every other metric reaches theDistances.SemiMetricmethod, which does nothing and returns. - On the method of the three metrics, check that
Zis non-negative withassert_nonneg. It raises aDomainErrorthat namessymwhen an entry is negative.
Arguments
metric: Distance metric whose domainZmust lie in.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.sym::Symbol = :Z: Name that the error message gives toZ.
Validation
- Under
Distances.Jaccard,Distances.BrayCurtisandDistances.ChiSqDist:all(x -> x >= 0, Z).
Returns
nothing.
Related
PortfolioOptimisers.assert_feature_matrix — Function
assert_feature_matrix(de::FeatureDistance, Z::ArrNum, dims::Integer)Validate a feature matrix at the entry point of distance and cor_and_dist. dims selects a valid axis, Z is not empty, every entry is finite, and Z lies in the domain of the metric.
The checks read every cell of Z. On a window that holds an unreadable cell, distance gives it the copy from blank_unreadable, which holds a zero at each such cell. So the checks find a fault only in the readable cells, which are the cells that the collapse reads.
The check refuses a non-finite entry, because no metric makes a usable distance from it. The Minkowski family gives Inf, the cosine family gives NaN, and a clustering cannot use either. The check accepts a degenerate input that a metric can measure. A zero feature vector has a documented convention, which AngularDist states, and a duplicate or constant feature is valid input.
Algorithm
- Check
dimswithassert_dims. - Check that
Zis non-empty withassert_nonempty. - Check that every entry of
Zis finite withassert_all_finite. - Check that
Zlies in the domain ofde.metricwithassert_metric_domain.
Arguments
de: Feature distance estimator, read for itsmetric.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Validation
dims in (1, 2).!isempty(Z).all(isfinite, Z).Zlies in the domain ofde.metric. Seeassert_metric_domain.
Returns
nothing.
Related
PortfolioOptimisers.zero_feature_vectors — Function
zero_feature_vectors(Z::MatNum, dims::Integer)Mark the assets whose feature vector is all zero, in the layout that dims states.
Algorithm
- Select the asset axis from
dims: the rows ofZatdims = 1, and its columns atdims = 2. - Test the feature vector of each asset with
all(iszero, ...), to get one entry of the mask per asset.
Arguments
Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Returns
z::Vector{Bool}: Mask with one entry per asset,truewhere the feature vector of that asset is all zero.
Related
PortfolioOptimisers.patch_zero_feature_vectors! — Function
patch_zero_feature_vectors!(D::MatNum, Z::MatNum, dims::Integer)Apply the convention for a zero feature vector to D in place. Two zero vectors are at distance 0, and a zero vector and a non-zero vector are at distance 1.
The function rewrites only the entries that the metric left as NaN. A zero feature vector is valid input, so no check at construction can find it. A metric that divides by a norm cannot measure it. The cosine family gives NaN against every vector, and Distances.Jaccard and Distances.BrayCurtis give NaN between two zero vectors. The Minkowski family puts a zero vector at the origin and measures it. Because the function rewrites only NaN entries, it corrects the first metrics and leaves the Minkowski family unchanged.
This convention keeps $S = \cos(\pi D)$ true at every entry. With it, AngularSimilarity gives +1 between two zero vectors and -1 between a zero vector and a non-zero vector, with a diagonal of ones. Distances.pairwise always writes an exact zero diagonal, so the function does not change the diagonal.
Algorithm
- Make the zero mask
zofZwithzero_feature_vectors. - Return
Dunchanged when the mask marks no asset. Most inputs take this step. - Otherwise, read each off-diagonal entry of
D. Rewrite it only when the metric left it asNaNand the mask marks at least one of its two assets. Writezero(T)when the mask marks both, andone(T)when it marks exactly one.
Arguments
D: Distance matrixassets × assets, rewritten in place.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Returns
D::MatNum: Distance matrixassets x assets, in the units the distance algorithm defines.
Related
PortfolioOptimisers.feature_distance — Function
feature_distance(metric::Distances.SemiMetric, Z::MatNum, dims::Integer)Turn a 2-D feature matrix into a distance matrix. Each entry point of FeatureDistance uses this kernel, and three of the collapse algorithms differ only in the matrix that they give it. AggregateDistances does not call it. It does the same two steps once per observation, into one buffer that it uses again, and aggregates the results.
Algorithm
- Apply
metricto every pair of assets ofZwithDistances.pairwise, along the axis thatdimsnames, to getD. - Apply the convention for a zero feature vector to
Din place withpatch_zero_feature_vectors!.
Arguments
metric: Distance metric applied to the assets ofZ.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Returns
D::MatNum: Distance matrixassets x assets, in the units the distance algorithm defines.
Related
feature_distance(de::FeatureDistance, Z::Arr3Num, dims::Integer)Turn a window of time-varying features into a distance matrix, by the collapse algorithm in de.alg.
Algorithm
The type of de.alg selects one of four methods. The docstring of each type states its method in full. In short:
LastObservation: pass the last slice of the observation axis to the 2-D kernel.StackObservations: pass the stacked matrix fromstack_observationsto the 2-D kernel, along its first axis.AggregateFeatures: resolve the weights withcollapse_weights, collapse the window withcollapse_features, and pass the collapsed matrix to the 2-D kernel.AggregateDistances: resolve the weights withcollapse_weights, then accumulate one weighted distance matrix per observation and divide by the weight total.
Arguments
de: Feature distance estimator, read for itsmetricand itsalg.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Returns
D::MatNum: Distance matrixassets x assets, in the units the distance algorithm defines.
Related
PortfolioOptimisers.collapse_features — Function
collapse_features(alg::AbstractCollapseAlgorithm, Z::Arr3Num, w::Option{<:VecNum})Aggregate a window of time-varying features along its last axis, the observation axis. The result is a matrix with the two first axes of Z. AggregateFeatures calls it.
The element type of the result comes from the aggregate, not from the window. The mean and the median of a window of integers can be fractions. A window with an even count of observations can have a fractional median even when each value in it is a whole number. For this reason, both routes make their result from the values that they compute.
Algorithm
The MeanCollapse route:
- Reduce the last axis of
ZwithStatistics.mean, weighted bywwhenwis notnothing. - Drop the reduced axis.
The MedianCollapse route:
- For each entry
(j, k)of the two first axes, take the observation seriesview(Z, j, k, :). - Reduce that series with
Statistics.median, weighted bywwhenwis notnothing, to get one entry of the result.
Arguments
alg: Collapse algorithm, which aggregates along the observation axis.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.w: Resolved observation weights, one entry per observation, ornothingfor an unweighted collapse.
Returns
Zc::Matrix{<:Number}: Collapsed feature matrix, with the two first axes ofZ.
Related
PortfolioOptimisers.stack_observations — Function
stack_observations(Z::Arr3Num, dims::Integer)Reshape a window of time-varying features into an assets × (observations · features) matrix. Its rows are the assets, on whichever of the two first axes dims puts them.
Algorithm
- Permute
Zso that the asset axis is first:(1, 3, 2)atdims = 1, and(2, 3, 1)atdims = 2. Both permutations put the observation axis second and the feature axis third. - Reshape the permuted array to
assets × (observations · features), one long feature vector per asset.
Arguments
Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.dims: Dimension along which the computation runs.
Returns
Za::Matrix{<:Number}: Stacked feature matrix,assets × (observations · features).
Related
PortfolioOptimisers.collapse_weights — Function
collapse_weights(w::Option{<:ObsWeights}, Z::Arr3Num)Resolve the observation weights of a collapse algorithm against a window of time-varying features.
The function first reshapes Z to an observations × (assets · features) matrix. The documented interface of get_observation_weights is VecNum and MatNum, and a 3-D array matches neither. Without the reshape, a correct MatNum method that a user adds never runs. The caller does not check for nothing. get_observation_weights raises ObservationWeightsError when a DynamicAbstractWeights cannot resolve, so nothing here means only that the caller asked for no weights.
Observation weights for a cross-validation need a DynamicAbstractWeights. It resolves against the Z that it receives, so its length is the length of each fold. A static AbstractWeights has the length that it had at construction, in each fold. Before the length check, AggregateDistances read a longer one by position. It gave the oldest weights to the newest observations, with no bounds error. A shorter one raised a BoundsError with no explanation. The length check now raises a clear error for both.
Algorithm
- Permute and reshape
Ztoobservations × (assets · features), so that the observation axis is the first axis of a matrix, as the returns hold it. - Resolve
wagainst that matrix withget_observation_weights, along its first axis. - Check the resolved length against the count of observations of
Z, unless the resolution gavenothing.
Arguments
w: Optional observation weights vectorobservations × 1, or a concrete subtype ofDynamicAbstractWeights. Ifnothing, the computation is unweighted.Z: Feature matrix,assets × featureswhendims = 1andfeatures × assetswhendims = 2. It can also be a 3-D array of time-varying features, whose last axis is always the observation axis:assets × features × observationswhendims = 1, andfeatures × assets × observationswhendims = 2.
Validation
length(w) == size(Z, 3)after the resolution. Raises aDimensionMismatchthat names both lengths.
Returns
w::Option{<:VecNum}: Resolved observation weights, one entry per observation, ornothingwhen the caller asked for no weights.
Related
PortfolioOptimisers.feature_window — Function
feature_window(de::FeatureDistance, pr, rd, X) -> (Z, A)Stack the Feature Matrix that a FeatureDistance measures, with the mask of the cells that it can read in the stacked rows.
It returns the same stack as feature_matrix, and the mask of that stack. A cell is readable where the asset is active and every value column of de.sel holds data. Data is a value that the raw input held, or a value that a fill policy wrote. The function cuts the mask to the rows that the collapse reads, so the mask has one row per observation of the stack. A static panel has one row, the cells of the assets. Its mask is nothing when every cell holds data, and a 1 × assets matrix otherwise. A time-varying panel whose every cell is readable gives the active mask, and the kernel drops that mask when it is all true.
The stack holds a zero at a cell that holds a placeholder, not the placeholder that the panel stores. The mask leaves that cell out, so no step reads the zero, and the stack does not change with the placeholder.
Algorithm
- Resolve the panel with
asset_panel, and resolvede.selagainst it withselect_fields. - Name the rows that the collapse reads, and cut the mask of readable cells to them, with
feature_window_mask. - Stack the panel over those rows with
feature_stack, with a zero at each cell that holds a placeholder.
Arguments
de: Feature distance estimator.pr: Prior result or returns result. Both carry the asset returns matrixX, so either can supply it.rd: The returns result to use.X: Returns matrix of the subproblem,observations × assets. A producer reads it.
Returns
Z::Array: The Feature Matrix, asfeature_matrixreturns it.A::Option{<:AbstractMatrix{Bool}}: The mask of readable cells of the stacked rows,observations × assets, or1 × assetsfor a static panel with a placeholder, ornothing.
Related
PortfolioOptimisers.window_activity — Function
window_activity(amsk, rows)Cut a mask of the cells of an Asset Panel to the rows that a Feature Matrix stacks. A static panel has no observation axis. Its mask, one entry per asset, becomes a window of one row, 1 × assets, or nothing when every entry is true. A nothing mask gives nothing.
Arguments
amsk: The mask,observations × assetsorassets, ornothing.rows: The observation rows a time-varying Asset Panel stacks, as positions or as aBoolmask, andColon()for every row. A static panel has no observation axis and refuses any other value.
Returns
A::Option{<:AbstractMatrix{Bool}}:amsk[rows, :], the one-row window of a static mask, ornothing.
Related
PortfolioOptimisers.feature_asset_names — Function
feature_asset_names(pr, rd)Give the asset names that a refusal of a FeatureDistance quotes. These are the names of the returns data rd, or the names of pr when pr is a ReturnsResult and rd is nothing. When rd is nothing and pr is not a ReturnsResult, the function returns nothing, and the refusal quotes positions. Any other rd is a MethodError.
Related