Similarity: private API

PortfolioOptimisers.AbstractSimilarityMatrixAlgorithm — Type
abstract type AbstractSimilarityMatrixAlgorithm <: AbstractAlgorithm

Abstract supertype for all similarity matrix algorithms.

Every subtype defines a pure transformation of a distance matrix into a similarity matrix, and distance_to_similarity applies it. Two places read the family. The first is the construction of the Planar Maximally Filtered Graph (PMFG), which the similarity branch of NetworkEstimator, DBHT and LoGo reach. The second is the similarity that cor_and_dist returns.

The family is open. An extension defines a member and a distance_to_similarity method for it.

The PMFG path needs a similarity that is not negative, so the three fields that give it a similarity take the narrower AbstractNonNegativeSimilarityMatrixAlgorithm, not this type. These fields are alg of NetworkEstimator, sim of DBHT and sim of LoGo. The field sim of FeatureDistance takes all five members. Its similarity never reaches a PMFG, because each PMFG entry point computes the similarity again from its own field and discards the similarity that cor_and_dist returned.

Related

source
PortfolioOptimisers.AbstractNonNegativeSimilarityMatrixAlgorithm — Type
abstract type AbstractNonNegativeSimilarityMatrixAlgorithm <: AbstractSimilarityMatrixAlgorithm

Abstract supertype for the similarity matrix algorithms that the PMFG path admits.

A member of this family promises that distance_to_similarity returns no negative entry for a finite distance matrix in the domain of the member. This type bounds Tree_SimMat, sim of DBHT and sim of LoGo, so a similarity that can go negative fails at construction, not inside PMFG_T2s.

Where the requirement comes from

The requirement does not come from PMFG_T2s, which runs on a signed input. Its argmax of the gain compares sums of exactly three weights, and a constant shift of the weights does not change its seed mask.

The requirement comes from DBHT, which runs after the PMFG. DirectHb compares the weight sums of bubbles of different sizes without a normalisation, and writes the winner into Hc. In Hc, a row sum of exactly zero marks a separating bubble, so weights of opposite sign can cancel and make a false one. BubbleMember divides by a signed total that can be zero, Inf or negative, and then argmax selects the worst bubble. Neither failure raises an error, so the caller gets wrong clusters without a message.

The type bound covers more consumers than DBHT. calc_adjacency makes the PMFG binary, clusterise takes matrix powers of it, and logo! reads only its structure. None of them reaches the bubble code of DBHT. The bound holds on them too, because it is the one guard that makes them safe now, and its removal needs a reason of its own.

Non-negative, not positive

exp(-Inf) is exactly 0, and real data reaches it, because LogDistance maps a correlation of exactly zero to an infinite distance. The type admits a similarity of zero. It does not admit a negative similarity.

The guarantee is per member, and it carries a domain

The type selects the algorithm. It cannot select the data. A member whose guarantee needs a precondition on D states the precondition in a method of assert_similarity_domain. The PMFG entry points call that function before the transformation.

MemberHolds when
ExponentialSimilarityalways
GeneralExponentialSimilarityalways
MaximumDistanceSimilarityD is finite
ComplementSimilarityall(D .<= 1)

The family never admits AngularSimilarity, and no precondition can change that. It is negative at each negative correlation, and a negative correlation is ordinary data.

Open by declaration, not by proof

The family is open. A subtype claims its membership, and the library does not check the claim. A probe cannot check it, because the contract holds over every admissible distance matrix. A probe that passes ComplementSimilarity at D = 0.5 does not find its failure at D = 7. PMFG_T2s keeps its own check of non-negativity, to stop an extension that claims membership and breaks the promise.

Related

source
PortfolioOptimisers.distance_to_similarity — Function
distance_to_similarity(se::AbstractSimilarityMatrixAlgorithm; D::MatNum, kwargs...)

Compute a similarity matrix from a distance matrix with a similarity matrix algorithm.

Algorithm

  1. Select the method by the type of se. Each member of AbstractSimilarityMatrixAlgorithm has one method, and the docstring of the member states the closed form and the steps of its branch.
  2. Apply that transformation to each entry of D. The result is a new matrix of the same size. The function does not write to D, and no method reads a keyword.

The function is a pure transformation with no domain of its own. It does not call assert_similarity_domain. The caller checks the precondition of a member on D, at the five PMFG entry points.

Arguments

  • se: The similarity matrix algorithm.

    • se::MaximumDistanceSimilarity: Uses the maximum distance transformation.
    • se::ExponentialSimilarity: Uses the exponential transformation.
    • se::GeneralExponentialSimilarity: Uses a generalised exponential transformation.
    • se::ComplementSimilarity: Uses the linear complement of the distance.
    • se::AngularSimilarity: Inverts a normalised angular distance.
  • D: The distance matrix.

  • kwargs...: More keyword arguments. No method reads them.

Returns

  • S::Matrix{<:Number}: The similarity matrix, with the size of D.

Examples

julia> D = [0.0 0.25 0.5; 0.25 0.0 1.0; 0.5 1.0 0.0];julia> PortfolioOptimisers.distance_to_similarity(ComplementSimilarity(); D = D)3×3 Matrix{Float64}: 1.0   0.75  0.5 0.75  1.0   0.0 0.5   0.0   1.0julia> D = [0.0 1/3 2/3; 1/3 0.0 1.0; 2/3 1.0 0.0];julia> PortfolioOptimisers.distance_to_similarity(AngularSimilarity(); D = D)3×3 Matrix{Float64}:  1.0   0.5  -0.5  0.5   1.0  -1.0 -0.5  -1.0   1.0

Related

source
PortfolioOptimisers.assert_similarity_domain — Function
assert_similarity_domain(sim::AbstractSimilarityMatrixAlgorithm,
                         de::AbstractDistanceEstimator, D::MatNum)
assert_similarity_domain(sim::ComplementSimilarity, de::AbstractDistanceEstimator,
                         D::MatNum)
assert_similarity_domain(sim::MaximumDistanceSimilarity,
                         de::AbstractDistanceEstimator, D::MatNum)

Assert that D is in the domain of sim, so that the non-negativity guarantee of sim holds on it.

It is the second half of the enforcement of AbstractNonNegativeSimilarityMatrixAlgorithm. The type bound selects the algorithms that can reach the PMFG path, but it cannot select the data. Two of the four admitted members are non-negative only over a part of the distances that a caller can give.

MemberPrecondition
ComplementSimilarityall(D .<= 1)
MaximumDistanceSimilarityall(isfinite, D)
every other membernone, the fallback method does no operation

Scoped to the interface, not to the member

The five call sites are the PMFG entry points, and there is no other call site. They are calc_weighted_adjacency_graph, calc_distance_weighted_graph, the two clusterise methods that build a PMFG, and logo!. distance_to_similarity does not call this function, so it stays a pure transformation with no domain of its own.

The scope follows the failure, and the failure depends on the path, not on the member. On the FeatureDistance path, ComplementSimilarity against a distance without an upper bound is documented and tested behaviour. There, each Distances.SemiMetric gives a similarity, and no call raises an error. The same value is unusable only on the PMFG path.

What it costs a caller

It costs nothing that works today. Each pair that this function refuses already raises an error one transformation later, at the check of PMFG_T2s. That message names W, and not the configuration that made it. This function gives a better message, because it names the two halves of the pair. These are the distance estimator de that made the bad value, and the similarity that refused it. This is the reason that the caller gives de.

Algorithm

  1. Select the method by the type of sim. Each of the two members of the table above has a method. Each other member reaches the method of AbstractSimilarityMatrixAlgorithm, which does no operation and returns at once.
  2. Test the precondition of that member over all of D.
  3. Raise a DomainError when the test fails. The message holds maximum(D), the name of the similarity that refused it, the closed form of that similarity, and the concrete type of de. Return nothing when the precondition holds.

Arguments

  • sim: The similarity matrix algorithm.
  • de: The distance estimator that made D. The function reads it only to name it in the error.
  • D: The distance matrix.

Validation

Returns

  • nothing.

Related

source
PortfolioOptimisers.default_similarity — Function
default_similarity(metric::Distances.SemiMetric)
default_similarity(metric::AngularDist)

Select the similarity matrix algorithm that is the inverse of a distance metric.

A distance algorithm calls it to get the default of its similarity field from its metric. The printed object then shows the selected similarity, and the distance kernel does not select it out of view. The fallback is ComplementSimilarity, the named inverse of each metric that is one minus a similarity. A metric whose inverse is not linear adds its own method.

Algorithm

  1. Select the method by the type of metric. Dispatch does the selection, so a metric whose inverse is not linear adds a method of its own, and this function needs no branch for it.

  2. Return the similarity that the method names. There are two methods now.

    • metric::Distances.SemiMetric is the fallback, and returns ComplementSimilarity. Each metric that is one minus a similarity reaches this method, and the complement recovers that named similarity.
    • metric::AngularDist returns AngularSimilarity, the exact algebraic inverse of the angular distance. src/06_Distance/05_FeatureDistance.jl declares that method, next to AngularDist, and this docstring documents it.

The pair that this function makes is correct, because it selects the inverse of the metric that it gets. A similarity that the caller selects has no such guarantee. ComplementSimilarity and AngularSimilarity state what a wrong pair returns.

Arguments

  • metric: The distance metric.

Returns

  • sim::AbstractSimilarityMatrixAlgorithm: The similarity matrix algorithm.

Examples

julia> PortfolioOptimisers.default_similarity(PortfolioOptimisers.Distances.CosineDist())ComplementSimilarity()julia> PortfolioOptimisers.default_similarity(PortfolioOptimisers.AngularDist())AngularSimilarity()

Related

source