Base Phylogeny: private API
PortfolioOptimisers.AbstractPhylogenyEstimator — Type
abstract type AbstractPhylogenyEstimator <: AbstractEstimatorAbstract supertype for all phylogeny estimator types.
A subtype holds the configuration that finds the relations between assets from the data. A network estimator and a clusters estimator are the two families.
Related
PortfolioOptimisers.AbstractPhylogenyAlgorithm — Type
abstract type AbstractPhylogenyAlgorithm <: AbstractAlgorithmAbstract supertype for all phylogeny algorithm types.
A subtype selects one method inside a phylogeny estimator. A clustering algorithm, a tree type and a centrality algorithm are the three families.
Related
PortfolioOptimisers.AbstractPhylogenyResult — Type
abstract type AbstractPhylogenyResult <: AbstractResultAbstract supertype for all phylogeny result types.
A subtype holds the output of a phylogeny estimator, such as a clustering or a phylogeny matrix.
Related
PortfolioOptimisers.PlE_Pl — Type
const PlE_Pl = Union{AbstractPhylogenyEstimator, AbstractPhylogenyResult}Alias for a phylogeny estimator or result.
Matches an AbstractPhylogenyEstimator or an AbstractPhylogenyResult. The library dispatches on it where a function accepts a phylogeny estimator or a precomputed result.
Related
PortfolioOptimisers.AbstractSeparationAlgorithm — Type
abstract type AbstractSeparationAlgorithm <: AbstractAlgorithmAbstract supertype for all separation algorithms.
A separation algorithm states how far apart two assets are in a network, and how far apart two related assets can be. One object answers the two questions through three functions:
separation_graphbuilds the structure that the member measures.separation_matrixreads the denseassets × assetsseparations from that structure.separation_budgetresolves the budget. A pair further apart than the budget is unrelated.
The family is open. A new member is a struct and one method of each function.
The two questions share a unit. A hop count has its budget in hops, and a weighted path length has its budget in the units of the distance estimator. A budget apart from the rule that measures it is a number with no unit. So the budget is a field of the member, not of NetworkEstimator.
The structure and the measurement are two functions
The function that measures takes the structure, not the estimator that builds it. separation_matrix(sep, g) is the interface, and separation_matrix(sep, nte, X) is a wrapper that calls separation_graph first. calc_weighted_adjacency_graph has the same two entry points for the same reason. The structure is expensive, and a caller often has one already.
Under VariationInfoDistance, the structure takes 98% of the runtime of clusterise. A consumer that resolves a budget rule and also measures the separations must build the structure once and pass the graph to both functions.
A test or an extension also calls separation_matrix(sep, g). Every structure that a shipped estimator builds is connected, because it is a spanning tree or a PMFG. Only a graph that a caller gives can be disconnected and carry the unreachable sentinel.
Two more functions
resolve_separation changes a member whose budget is a rule into a member whose budget is a value. Every consumer calls it before the other two functions. It asks nothing new about the network. The fallback on this type returns a member with a numeric budget unchanged, so an extension inherits it and writes no method.
is_related applies the budget to one entry of the separation matrix, after the sentinel test of is_reachable. Each is one generic method on this type, because the rule is the same in every unit. A related pair is not the sentinel, and it is not further apart than the budget.
A member whose routine reports a different sentinel overrides is_reachable alone. Every consumer that applies a budget calls these two functions and writes no comparison of its own. So the rule of "The unreachable sentinel" of separation_matrix stays inside one interface.
The two shipped members accept a rule in the budget field, through HopCountValue and PathLengthValue. A caller who cannot state the budget in advance gives the rule that calculates it. The library resolves the rule where the data is available, because a rule needs the data. For this reason separation_budget refuses an unresolved member and does not return a function.
Separation is not decay
AbstractSeparationDecayAlgorithm changes a separation into a score. This family calculates the separation and its budget. sep decides which pairs are related, and every consumer of a network needs that. decay decides how strongly a pair is related, and only the feature producer needs that number. So sep is a field of NetworkEstimator and decay is a field of Proximity.
The name does not say graph
The name of the family says nothing about graphs. A taxonomy depth is also a separation. The name leaves room for a member that measures one, and AbstractGraphSeparationAlgorithm does not.
Related
PortfolioOptimisers.HopCountAlgorithm — Type
abstract type HopCountAlgorithm <: AbstractAlgorithmAbstract supertype for all rules that compute a hop count from the network and the data.
The caller usually states a HopCount budget as a CountOf of hops, or as a FractionOf of the reachable pairs to relate, but the budget can also be a rule. A subtype of HopCountAlgorithm is a callable struct in the n field. resolve_separation calls it with the network estimator and the data matrix.
A rule lets the budget follow a universe whose size the caller does not know in advance. Examples are a cross-validation fold and a subproblem of a meta optimiser such as NestedClustered. A FractionOf is the shipped form of such a budget, so the family ships no member, and it stays the extension point for a rule of another kind.
The extension contract
A subtype defines one method, the functor:
(rule::MySubtype)(nte::AbstractNetworkEstimator, X::MatNum, g::Graphs.AbstractGraph; dims::Int = 1, kwargs...) -> Integerg is the structure that separation_graph built for the separation that holds the rule. The rule reads what it needs from g through separation_matrix, which takes g directly, so the rule builds no structure. It still runs the traversal of all pairs, as resolve_separation states.
nte holds the separation, and X is the data that g comes from. An extension reads them to get what the graph does not hold, for example the distance estimator, the number of observations or a covariance.
The return value must be an Integer. The return type of a functor is not part of its signature, so the type system cannot state this requirement. resolve_separation checks the value instead. Three readers use 0:n as a count of matrix powers, and 0:1.5 drops a power with no error.
The same field also accepts a bare Function, which has the same requirement. The constructor does not check it. Subtype this type instead when the rule has parameters, because the struct holds them, prints them and compares equal by them.
Related
PortfolioOptimisers.PathLengthAlgorithm — Type
abstract type PathLengthAlgorithm <: AbstractAlgorithmAbstract supertype for all rules that compute a path-length budget from the network and the data.
This is the PathLength counterpart of HopCountAlgorithm. A subtype is a callable struct in the dmax field. resolve_separation calls it with the network estimator, the data matrix and the structure that it built from them.
The two families have different return types, so they are two types. The split lets resolve_separation check each return type. A hop count must be an Integer. A path-length budget is in the units of the distance estimator, so it can be any Number.
The extension contract
A subtype defines one method, the functor:
(rule::MySubtype)(nte::AbstractNetworkEstimator, X::MatNum, g::Graphs.AbstractGraph; dims::Int = 1, kwargs...) -> Numberg is the structure of separation_graph. Under a PathLength its edges carry distances on both branches. The same field also accepts a bare Function, which has the same requirement. The constructor does not check it.
A rule must return a Number, and nothing is not a Number. nothing in the dmax field means the observed diameter. The caller states it in place of a rule. A rule cannot ask for the diameter with nothing, because the field states that value already. If a rule returns nothing in error, resolve_separation throws an ArgumentError and does not give the largest ball. So PathLengthValue holds the rules and the numbers, and the field is an Option of it.
Related
PortfolioOptimisers.HopCountRule — Type
const HopCountRule = Union{HopCountAlgorithm, Function}Alias for the forms of a hop count that are rules.
Matches the two forms that resolve_separation calls and does not read: a HopCountAlgorithm and a bare Function. The library dispatches on it. HopCount{<:HopCountRule} and HopCount{<:FractionOf} are unresolved separations, and HopCount{<:CountOf} is a resolved one. A FractionOf is not a rule of this alias, because the library does not call it. resolve_separation reads it.
Related
PortfolioOptimisers.HopCountValue — Type
const HopCountValue = Union{Real_CountFraction, HopCountAlgorithm, Function}Alias for every value that the n field of HopCount accepts.
The field accepts a CountOf of hops, a FractionOf of the reachable pairs, a plain number that count_or_fraction reads, and the rules of HopCountRule. A CountOf is the resolved form, and every reader reads its val. resolve_separation resolves a fraction or a rule before a reader gets it.
Related
PortfolioOptimisers.PathLengthRule — Type
const PathLengthRule = Union{PathLengthAlgorithm, Function}Alias for the forms of a path-length budget that are rules.
This is the PathLength counterpart of HopCountRule.
Related
PortfolioOptimisers.PathLengthValue — Type
const PathLengthValue = Union{Number, FractionOf, PathLengthAlgorithm, Function}Alias for every value that the dmax field of PathLength accepts, except nothing.
This is the PathLength counterpart of HopCountValue, but the field is Option{PathLengthValue}. nothing in that field means the observed diameter. It is a stated budget, and a rule cannot return it. The alias leaves nothing out, so the check of resolve_separation is isa(dmax, Number) alone.
A plain number is a radius in the units of the distance estimator, and not a fraction, so the field takes no plain shorthand of count_or_fraction. A FractionOf states the radius that relates that fraction of the reachable pairs. The alias holds no CountOf, because a count of members is no radius.
Related
PortfolioOptimisers.AbstractSeparationDecayAlgorithm — Type
abstract type AbstractSeparationDecayAlgorithm <: AbstractAlgorithmAbstract supertype for all separation decay algorithms.
A separation decay changes a separation d >= 0 into a score. The separation is how far apart two assets are in the structure that the caller reads. separation_decay applies the decay. The family is open. For a different fall-off, define a member and a separation_decay method for it. AbstractSimilarityMatrixAlgorithm and distance_to_similarity have the same pattern.
d is a real separation and not an integer hop count. So one family serves an unweighted graph, where a hop count is a real with an integer value, and each structure whose separation is continuous.
The contract
fhas a value for everyd >= 0.f(0) > 0, andf(0)is the largest value. A decay must put an asset at the top of its own scale. If not, the result is a structural equivalence matrix, not a proximity matrix. See "Why the diagonal includes self" ofPhylogenyPanel.fis monotone non-increasing ind.- The contract does not ask
fto reach zero. The budget truncates, and the decay does not. The consumer applies the budget, which isseparation_budgetof theAbstractSeparationAlgorithmin scope. The decay shapes the fall-off inside the budget. An exponential never reaches zero, so one parameter cannot set the budget and the fall-off. f(d) >= 0for0 <= d <= dmax. The score of an unreachable pair is0. A negative score inside the budget puts a reachable pair below an unreachable pair on the scale of the producer.
The last clause does not say that a signed score is always wrong. The feature matrix accepts signed values by decision, and assert_metric_domain checks non-negativity for each metric at the consumer. The clause applies to the producer alone. It asks for non-negativity and not strict positivity, because a decay that falls to zero says that the pair is unrelated. An unreachable pair says the same.
The clause applies inside the budget alone, for two reasons. First, the sign outside the budget has no effect, because the consumer tests is_related first and does not evaluate the decay there. Second, the default of the family breaks the wider clause. LinearDecay is zero at d = dmax + 1 and negative above it. A clause on all d >= 0 needs a max(0, ⋅) floor, which is a second truncation inside the decay.
So a zero in the feature matrix means that the pair is unrelated. The pair is unreachable, it is outside the budget, or the decay fell to zero. A consumer of the feature matrix cannot tell these cases apart. No shipped member gives zero inside the budget. For the shipped members, a zero means unreachable or outside the budget.
The budget is an argument, not a field
separation_decay takes the budget in scope as its third argument, dmax. A member can ignore it, and only LinearDecay reads it, to set f(0). The member does not hold the budget, so the budget and the decay cannot disagree. The AbstractSeparationAlgorithm is the one place that holds the budget. A decay algorithm cannot see the budget when the caller constructs it. ExponentialDecay sets the contrast between an asset and its neighbour with its own rate, and adds no second truncation inside the decay.
Enforcement
The fallback assert_separation_decay on this type probes a member and enforces the contract. The shipped members satisfy the contract by construction, and their method of that function does nothing. So an extension that defines no method of assert_separation_decay gets the probe.
Related
PortfolioOptimisers.is_reachable — Function
is_reachable(sep::AbstractSeparationAlgorithm, d::Number)Is d a separation at all, or the sentinel an unreachable pair carries?
separation_matrix does not repair the sentinel of the routine that it calls. This test tells a measured separation from a missing one. It is one generic method on AbstractSeparationAlgorithm, because one expression covers the two shipped sentinels. A member whose routine reports a different sentinel overrides this method, and it inherits is_related unchanged.
The test is not isfinite alone, and not typemax alone
Each clause rejects a sentinel that the other clause accepts.
isfiniteistruefor everyInteger. Alone, it accepts thetypemax(Int)ofHopCount, andReciprocalDecayoverflows at that value.typemaxof aFloat64isInf, so the comparison also rejects the sentinel ofPathLength.isfiniterejects aNaN. No shipped path gives aNaN, and aNaNcomparesfalsewith every budget.
Algorithm
- Test
isfinite(d). This rejects aNaNand theInfofPathLength. - Compare
dwithtypemax(typeof(d)). This rejects thetypemax(Int)ofHopCount, which step 1 accepts. - Return
reachable, the conjunction of the two tests.&&short-circuits, so step 2 runs only on a finited.
Arguments
sep: Separation algorithm. The shipped members do not read it. An extension whose routine reports a different sentinel dispatches on it.d: One entry of a separation matrix fromseparation_matrix.
Returns
reachable::Bool:truewhendis a measured separation.
Related
PortfolioOptimisers.is_related — Function
is_related(sep::AbstractSeparationAlgorithm, d::Number, dmax::Number)Does a separation of d count as related under a budget of dmax?
This is the one function that applies the budget to an entry of a separation matrix. A related pair is reachable, and it is not further apart than dmax. Each consumer that compares a separation with a budget calls this function and writes no comparison of its own. The PathLength branch of phylogeny_matrix and the Proximity method of phylogeny_features call it.
The reachability test comes first
d <= dmax alone is not sufficient. It rejects the two shipped sentinels only because of the two shipped budgets. separation_budget clamps a PathLength budget to the observed finite diameter, and a HopCount budget has a cap far below typemax(Int). is_reachable makes the rejection part of the predicate. A budget at the largest value of its unit then still excludes an unreachable pair.
The caller must still short-circuit
A consumer that scores the separation must put the score inside a branch that short-circuits. Write is_related(...) ? separation_decay(...) : zero(...), and do not use ifelse. An ifelse evaluates the two branches. ReciprocalDecay overflows 1 + d at typemax(Int), and a fractional power then gives a DomainError. The predicate holds the rule, and the call site must not evaluate the score of an unrelated pair.
Algorithm
- Call
is_reachableonsepandd, which gives the reachability test. This test runs first, so the function rejects a sentinel at each value ofdmax. - Compare
dwithdmax, which gives the budget test. - Return
related, the conjunction of the two tests.&&short-circuits, so step 2 runs only on a reachabled.
Arguments
sep: Separation algorithm, which the function passes tois_reachable.d: One entry of a separation matrix fromseparation_matrix.dmax: Separation budget in scope, fromseparation_budget. It has the units thatsepmeasures in, so the two arguments come together.
Returns
related::Bool:truewhen the pair is reachable and inside the budget.
Related
PortfolioOptimisers.assert_separation_decay — Function
assert_separation_decay(dk::AbstractSeparationDecayAlgorithm, ds, dmax::Number)
assert_separation_decay(dk::Union{LinearDecay, ExponentialDecay, ReciprocalDecay}, ds,
dmax::Number)
assert_separation_decay(::Nothing, ds, dmax::Number)Check that a separation decay satisfies its contract over the separations that the caller will score.
The fallback probes the decay. It evaluates separation_decay over ds and checks four properties:
- Each value is finite.
f(0)is strictly positive, and it is the largest value.- The values are monotone non-increasing.
- No value is negative.
The three shipped members and nothing satisfy the contract by construction, and their method does nothing. So the probe costs nothing for a shipped member, and it checks each extension.
The probe is cheap where the library uses it, because ds is small and the loop after the probe is large. Proximity passes 0:dmax. Under HopCount, this range holds every separation that the assets × assets loop can score, in dmax + 1 evaluations. Under a budget that is not an integer, the same range is a sample with a step of one. A continuum permits only a sample, and the checks below need no more.
The non-negativity check has one more evaluation at d = dmax, also when ds does not contain dmax. It is the counterpart of the evaluation of f(0) before the loop. If the decay is monotone, f(dmax) >= 0 gives f(d) >= 0 for every d in [0, dmax]. So the check holds over the continuum, also when ds is a sample, as it is for weighted path lengths. The endpoint does not help the monotonicity check, which stays a sample.
Algorithm
These are the steps of the probe in the fallback. The method on the three shipped members and on nothing runs none of them.
- Score the separation
zero(dmax)withseparation_decay, which givesf0, the top of the scale. - Check that
f0is finite and strictly positive. - If
dsis not sorted, sort it, so that step 5 reads the separations in increasing order. - Set
fp, the score of the previous separation, tof0. - For each
dofds, score it withseparation_decay, which givesf. Check thatfis finite and not more thanf0. Check thatfis not more thanfp. Check thatfis non-negative. Setfptof. - Score
dmax, which givesfmax, and check thatfmaxis non-negative. This is the endpoint that the paragraph above states. Step 5 does not always reach it. - Return
nothing.
Arguments
dk: Separation decay algorithm.ds: Separations to probe, in any order. Each separation must be in[0, dmax], because the loop after the probe scores no separation outside the budget.dmax: Separation budget in scope. The function passes it toseparation_decay, and also probes it as an endpoint.
Validation
- Every probed value is finite.
f(0) > 0.f(0) >= f(d)for every probedd.- The probed values are monotone non-increasing in
d. f(d) >= 0for every probedd, and atd = dmaxwhether or not it was probed.
Returns
nothing.
Related
PortfolioOptimisers.assert_distance_reader_rows — Function
assert_distance_reader_rows(D::MatNum) -> NothingRefuse the assets whose correlation a distance reader could not estimate, before a graph verb builds its tree or its graph.
A distance reader gives a NaN row and column, its diagonal entry included, for an asset whose correlation it cannot estimate, see DistanceReaderRefusalError. Every distance of a correlation keeps the NaN, so the distance matrix has a NaN diagonal entry for that asset and a finite one for every other asset. A dendrogram and a weighted graph have no place for such an asset. The function reads only the diagonal of D, so it costs one pass over N entries, against the N^2 T of the estimate. A NaN off the diagonal between two assets with a finite diagonal has another cause, and the verb that reads the distance refuses it with its own error, see assert_no_nan_distance.
Algorithm
- Find the columns
idxwhose diagonal entry ofDisNaN. - Return
nothingwhenidxis empty. Otherwise throw aDistanceReaderRefusalErrorthat carriesidx.
Arguments
D: The distance matrix of the reader,N × N.
Validation
- No diagonal entry of
DisNaN. Raises aDistanceReaderRefusalErrorthat names the columns. A fit catches it and drops the assets, seedistance_reader_refusal_mask.
Returns
nothing.
Related
PortfolioOptimisers.assert_no_nan_distance — Function
assert_no_nan_distance(D::MatNum, what::AbstractString) -> NothingRefuse a NaN distance between two assets, before a clustering reads D.
Call it after assert_distance_reader_rows, so every diagonal entry of D is finite and each NaN is a pair. Two assets share too few observations for a correlation of the pair, while the reader estimates each of them. A peel of the CoveragePolicy of the reader removes one asset of such a pair. An asset with zero variance does not give such a pair, because its diagonal entry is NaN too, see cov2cor_nan!, and assert_distance_reader_rows refuses it first.
Clustering.hclust refuses a NaN as a matrix that is not symmetric, because NaN is not equal to itself, and Clustering.kmeans refuses it as a weight. Neither error names the cell or the cause. Inf stays, because LogDistance gives it for two uncorrelated assets.
Algorithm
- Find the first cell
cofDthat isNaN, in column order. - Return
nothingwhencisnothing. Otherwise throw aDomainErrorthat namesc.
Arguments
D: The distance matrix of the reader,N × N.what: The structure that the caller builds, for the message, such as"a dendrogram".
Validation
Dholds noNaN. Raises aDomainErrorthat names the firstNaNcell and its cause.
Returns
nothing.
Related