Similarity: private API
PortfolioOptimisers.AbstractSimilarityMatrixAlgorithm — Type
abstract type AbstractSimilarityMatrixAlgorithm <: AbstractAlgorithmAbstract supertype for all similarity matrix algorithms.
Every subtype defines a pure transformation of a distance matrix into a similarity matrix, and distance_to_similarity applies it. Two places read the family. The first is the construction of the Planar Maximally Filtered Graph (PMFG), which the similarity branch of NetworkEstimator, DBHT and LoGo reach. The second is the similarity that cor_and_dist returns.
The family is open. An extension defines a member and a distance_to_similarity method for it.
The PMFG path needs a similarity that is not negative, so the three fields that give it a similarity take the narrower AbstractNonNegativeSimilarityMatrixAlgorithm, not this type. These fields are alg of NetworkEstimator, sim of DBHT and sim of LoGo. The field sim of FeatureDistance takes all five members. Its similarity never reaches a PMFG, because each PMFG entry point computes the similarity again from its own field and discards the similarity that cor_and_dist returned.
Related
PortfolioOptimisers.AbstractNonNegativeSimilarityMatrixAlgorithm — Type
abstract type AbstractNonNegativeSimilarityMatrixAlgorithm <: AbstractSimilarityMatrixAlgorithmAbstract supertype for the similarity matrix algorithms that the PMFG path admits.
A member of this family promises that distance_to_similarity returns no negative entry for a finite distance matrix in the domain of the member. This type bounds Tree_SimMat, sim of DBHT and sim of LoGo, so a similarity that can go negative fails at construction, not inside PMFG_T2s.
Where the requirement comes from
The requirement does not come from PMFG_T2s, which runs on a signed input. Its argmax of the gain compares sums of exactly three weights, and a constant shift of the weights does not change its seed mask.
The requirement comes from DBHT, which runs after the PMFG. DirectHb compares the weight sums of bubbles of different sizes without a normalisation, and writes the winner into Hc. In Hc, a row sum of exactly zero marks a separating bubble, so weights of opposite sign can cancel and make a false one. BubbleMember divides by a signed total that can be zero, Inf or negative, and then argmax selects the worst bubble. Neither failure raises an error, so the caller gets wrong clusters without a message.
The type bound covers more consumers than DBHT. calc_adjacency makes the PMFG binary, clusterise takes matrix powers of it, and logo! reads only its structure. None of them reaches the bubble code of DBHT. The bound holds on them too, because it is the one guard that makes them safe now, and its removal needs a reason of its own.
Non-negative, not positive
exp(-Inf) is exactly 0, and real data reaches it, because LogDistance maps a correlation of exactly zero to an infinite distance. The type admits a similarity of zero. It does not admit a negative similarity.
The guarantee is per member, and it carries a domain
The type selects the algorithm. It cannot select the data. A member whose guarantee needs a precondition on D states the precondition in a method of assert_similarity_domain. The PMFG entry points call that function before the transformation.
| Member | Holds when |
|---|---|
ExponentialSimilarity | always |
GeneralExponentialSimilarity | always |
MaximumDistanceSimilarity | D is finite |
ComplementSimilarity | all(D .<= 1) |
The family never admits AngularSimilarity, and no precondition can change that. It is negative at each negative correlation, and a negative correlation is ordinary data.
Open by declaration, not by proof
The family is open. A subtype claims its membership, and the library does not check the claim. A probe cannot check it, because the contract holds over every admissible distance matrix. A probe that passes ComplementSimilarity at D = 0.5 does not find its failure at D = 7. PMFG_T2s keeps its own check of non-negativity, to stop an extension that claims membership and breaks the promise.
Related
PortfolioOptimisers.distance_to_similarity — Function
distance_to_similarity(se::AbstractSimilarityMatrixAlgorithm; D::MatNum, kwargs...)Compute a similarity matrix from a distance matrix with a similarity matrix algorithm.
Algorithm
- Select the method by the type of
se. Each member ofAbstractSimilarityMatrixAlgorithmhas one method, and the docstring of the member states the closed form and the steps of its branch. - Apply that transformation to each entry of
D. The result is a new matrix of the same size. The function does not write toD, and no method reads a keyword.
The function is a pure transformation with no domain of its own. It does not call assert_similarity_domain. The caller checks the precondition of a member on D, at the five PMFG entry points.
Arguments
se: The similarity matrix algorithm.se::MaximumDistanceSimilarity: Uses the maximum distance transformation.se::ExponentialSimilarity: Uses the exponential transformation.se::GeneralExponentialSimilarity: Uses a generalised exponential transformation.se::ComplementSimilarity: Uses the linear complement of the distance.se::AngularSimilarity: Inverts a normalised angular distance.
D: The distance matrix.kwargs...: More keyword arguments. No method reads them.
Returns
S::Matrix{<:Number}: The similarity matrix, with the size ofD.
Examples
julia> D = [0.0 0.25 0.5; 0.25 0.0 1.0; 0.5 1.0 0.0];julia> PortfolioOptimisers.distance_to_similarity(ComplementSimilarity(); D = D)3×3 Matrix{Float64}: 1.0 0.75 0.5 0.75 1.0 0.0 0.5 0.0 1.0julia> D = [0.0 1/3 2/3; 1/3 0.0 1.0; 2/3 1.0 0.0];julia> PortfolioOptimisers.distance_to_similarity(AngularSimilarity(); D = D)3×3 Matrix{Float64}: 1.0 0.5 -0.5 0.5 1.0 -1.0 -0.5 -1.0 1.0Related
PortfolioOptimisers.assert_similarity_domain — Function
assert_similarity_domain(sim::AbstractSimilarityMatrixAlgorithm,
de::AbstractDistanceEstimator, D::MatNum)
assert_similarity_domain(sim::ComplementSimilarity, de::AbstractDistanceEstimator,
D::MatNum)
assert_similarity_domain(sim::MaximumDistanceSimilarity,
de::AbstractDistanceEstimator, D::MatNum)Assert that D is in the domain of sim, so that the non-negativity guarantee of sim holds on it.
It is the second half of the enforcement of AbstractNonNegativeSimilarityMatrixAlgorithm. The type bound selects the algorithms that can reach the PMFG path, but it cannot select the data. Two of the four admitted members are non-negative only over a part of the distances that a caller can give.
| Member | Precondition |
|---|---|
ComplementSimilarity | all(D .<= 1) |
MaximumDistanceSimilarity | all(isfinite, D) |
| every other member | none, the fallback method does no operation |
Scoped to the interface, not to the member
The five call sites are the PMFG entry points, and there is no other call site. They are calc_weighted_adjacency_graph, calc_distance_weighted_graph, the two clusterise methods that build a PMFG, and logo!. distance_to_similarity does not call this function, so it stays a pure transformation with no domain of its own.
The scope follows the failure, and the failure depends on the path, not on the member. On the FeatureDistance path, ComplementSimilarity against a distance without an upper bound is documented and tested behaviour. There, each Distances.SemiMetric gives a similarity, and no call raises an error. The same value is unusable only on the PMFG path.
What it costs a caller
It costs nothing that works today. Each pair that this function refuses already raises an error one transformation later, at the check of PMFG_T2s. That message names W, and not the configuration that made it. This function gives a better message, because it names the two halves of the pair. These are the distance estimator de that made the bad value, and the similarity that refused it. This is the reason that the caller gives de.
Algorithm
- Select the method by the type of
sim. Each of the two members of the table above has a method. Each other member reaches the method ofAbstractSimilarityMatrixAlgorithm, which does no operation and returns at once. - Test the precondition of that member over all of
D. - Raise a
DomainErrorwhen the test fails. The message holdsmaximum(D), the name of the similarity that refused it, the closed form of that similarity, and the concrete type ofde. Returnnothingwhen the precondition holds.
Arguments
sim: The similarity matrix algorithm.de: The distance estimator that madeD. The function reads it only to name it in the error.D: The distance matrix.
Validation
- Under
ComplementSimilarity:all(x -> x <= 1, D). - Under
MaximumDistanceSimilarity:all(isfinite, D).
Returns
nothing.
Related
PortfolioOptimisers.default_similarity — Function
default_similarity(metric::Distances.SemiMetric)
default_similarity(metric::AngularDist)Select the similarity matrix algorithm that is the inverse of a distance metric.
A distance algorithm calls it to get the default of its similarity field from its metric. The printed object then shows the selected similarity, and the distance kernel does not select it out of view. The fallback is ComplementSimilarity, the named inverse of each metric that is one minus a similarity. A metric whose inverse is not linear adds its own method.
Algorithm
Select the method by the type of
metric. Dispatch does the selection, so a metric whose inverse is not linear adds a method of its own, and this function needs no branch for it.Return the similarity that the method names. There are two methods now.
metric::Distances.SemiMetricis the fallback, and returnsComplementSimilarity. Each metric that is one minus a similarity reaches this method, and the complement recovers that named similarity.metric::AngularDistreturnsAngularSimilarity, the exact algebraic inverse of the angular distance.src/06_Distance/05_FeatureDistance.jldeclares that method, next toAngularDist, and this docstring documents it.
The pair that this function makes is correct, because it selects the inverse of the metric that it gets. A similarity that the caller selects has no such guarantee. ComplementSimilarity and AngularSimilarity state what a wrong pair returns.
Arguments
metric: The distance metric.
Returns
sim::AbstractSimilarityMatrixAlgorithm: The similarity matrix algorithm.
Examples
julia> PortfolioOptimisers.default_similarity(PortfolioOptimisers.Distances.CosineDist())ComplementSimilarity()julia> PortfolioOptimisers.default_similarity(PortfolioOptimisers.AngularDist())AngularSimilarity()Related