GitHub

Base Phylogeny: private API

PortfolioOptimisers.AbstractSeparationAlgorithm — Type
abstract type AbstractSeparationAlgorithm <: AbstractAlgorithm

Abstract supertype for all separation algorithms.

A separation algorithm is the rule saying how far apart two assets sit in a network, and how far is too far. It answers two questions with one object, through three kernels: separation_graph builds the structure the member measures over, separation_matrix reads the dense assets × assets separations off that structure, and separation_budget resolves the budget beyond which a pair counts as unrelated. The family is open: a new member is a struct and one method of each.

The two questions travel together because they share a unit. A hop count is budgeted in hops and a weighted path length in the distance estimator's units, so a budget stated apart from the rule that measures it would be a number nobody could interpret — which is why the budget lives on the member rather than on NetworkEstimator.

Building the structure is a separate kernel from measuring it

The measuring kernel takes the structure, not the estimator that produces one: separation_matrix(sep, g) is the interface, and separation_matrix(sep, nte, X) is a wrapper that calls separation_graph first. The split is calc_weighted_adjacency_graph's two-entry-point shape, for the same reason — the structure is expensive and a caller often holds one already. Under VariationInfoDistance building it is 98% of clusterise's runtime, so a consumer that resolves a budget rule and measures the separations must build once and pass the graph, not call two estimator-taking kernels.

It is also the seam a test or an extension enters through. Every structure a shipped estimator can build is connected — a spanning tree or a PMFG — so a disconnected graph, and with it the unreachable sentinel below, is reachable only by handing one in.

Two more kernels, and why neither is a third question

resolve_separation turns a member whose budget is a rule into one whose budget is a value, and is called by every consumer before the other two kernels. It is not a third question about the network: a member whose budget is already a number is returned unchanged by the fallback on this type, so an extension inherits the kernel and never writes one.

is_related applies the budget to one entry of the separation matrix, over is_reachable's sentinel test. Both are single generic methods on this type rather than per-member ones, because the rule — not the sentinel, and no further than the budget — is the same rule whatever the unit; a member whose underlying routine reports an exotic sentinel overrides is_reachable alone. Every consumer applying a budget calls them instead of open-coding the comparison, which is what keeps the ordering obligation of separation_matrix's "the unreachable sentinel" inside an interface.

The two shipped members widen their budget field to admit a rule — HopCountValue and PathLengthValue — so a caller who cannot state the budget in advance states what would produce it instead. The resolution happens where the data is in hand, which is the only place a rule can be answered, and it is why separation_budget refuses an unresolved member rather than returning a function.

Separation is not decay

AbstractSeparationDecayAlgorithm turns a separation into a score; this family produces the separation and says where it runs out. The seam is that sep decides which pairs are related — every consumer of a network needs that — while decay decides how strongly, as a number, which only the feature producer wants. That is why sep sits on NetworkEstimator and decay sits on Proximity.

The family is unqualified on purpose

The name says nothing about graphs. A taxonomy depth is a separation too, so the room is left for a member that measures one, rather than being closed off by an AbstractGraphSeparationAlgorithm.

Related

source
PortfolioOptimisers.HopCountAlgorithm — Type
abstract type HopCountAlgorithm <: AbstractAlgorithm

Abstract supertype for all rules computing a hop count from the network and the data.

A HopCount budget is usually a number the caller states. It does not have to be. A subtype of HopCountAlgorithm is a callable struct standing in the n field, and resolve_separation calls it with the network estimator and the data matrix in hand. That is what lets a budget follow a universe whose size the caller cannot know in advance — a cross-validation fold, or a subproblem of a meta optimiser such as NestedClustered.

The extension contract

A subtype defines one method, the functor:

(rule::MySubtype)(nte::AbstractNetworkEstimator, X::MatNum, g::Graphs.AbstractGraph;                  dims::Int = 1, kwargs...) -> Integer

g is the structure separation_graph built for the separation the rule stands in, so a rule reads what it needs off a graph it did not pay to build — through separation_matrix, which takes g directly. It still pays for the all-pairs traversal; see resolve_separation.

nte owns the separation and X is the data g was built from. Both are inert for the shipped rule, and are the channel through which an extension reaches what the graph does not carry — the distance estimator, the observation count, a covariance.

The return value must be an Integer, and this is checked rather than bounded. A functor's return type is not part of its signature, so the family cannot state the requirement in the type system. resolve_separation checks it instead, and the check is not a formality: three readers use 0:n as a matrix-power count, where 0:1.5 silently drops a power rather than failing.

A bare Function is admitted in the same field and carries the same obligation, unchecked at construction. Subtype this instead when the rule has parameters — the struct holds them, prints them, and is comparable.

Related

source
PortfolioOptimisers.PathLengthAlgorithm — Type
abstract type PathLengthAlgorithm <: AbstractAlgorithm

Abstract supertype for all rules computing a path-length budget from the network and the data.

The PathLength counterpart of HopCountAlgorithm: a callable struct standing in the dmax field, called by resolve_separation with the network estimator, the data matrix, and the structure already built from them in hand.

The two families are separate because their return obligations differ, and the split is what lets one of them be checked. A hop count must be an Integer; a path-length budget is stated in the distance estimator's units, so it is any Number — or nothing, which resolves to the observed diameter exactly as a stated nothing does.

The extension contract

A subtype defines one method, the functor:

(rule::MySubtype)(nte::AbstractNetworkEstimator, X::MatNum, g::Graphs.AbstractGraph;                  dims::Int = 1, kwargs...) -> Number

g is separation_graph's structure, weighted by distance on both branches under a PathLength. A bare Function is admitted in the same field and carries the same obligation, unchecked at construction.

A rule must return a Number, and nothing is not one. nothing in the dmax field means the observed diameter, which is a statement the caller makes instead of stating a rule. A rule that meant to ask for the diameter is asking for something the field already spells, and a rule that returned nothing by accident would silently get the maximal ball. So PathLengthValue covers the rules and the numbers, and the field is an Option of it.

Related

source
PortfolioOptimisers.PathLengthValue — Type
const PathLengthValue = Union{<:Number, <:PathLengthAlgorithm, <:Function}

Alias for everything PathLength's dmax field accepts, apart from nothing.

The PathLength counterpart of HopCountValue, and the field is Option{PathLengthValue} rather than this alias alone. The asymmetry is deliberate: nothing in that field means the observed diameter, which is one of the stated budgets and not something a rule may answer with. Keeping it outside the alias is what makes resolve_separation's check a plain isa(dmax, Number).

Related

source
PortfolioOptimisers.AbstractSeparationDecayAlgorithm — Type
abstract type AbstractSeparationDecayAlgorithm <: AbstractAlgorithm

Abstract supertype for all separation decay algorithms.

A separation decay turns a separation d >= 0 — how far apart two assets sit in whatever structure the caller is reading — into a score, and is applied by separation_decay. The family is open: a caller wanting a different fall-off defines a member and a separation_decay method for it, exactly as AbstractSimilarityMatrixAlgorithm is extended through distance_to_similarity.

d is a real separation rather than an integer hop count, so one family serves an unweighted graph — where hop counts enter as integer-valued reals — and any structure whose separation is continuous.

The contract

  • Defined for every d >= 0.
  • f(0) > 0 and maximal. Self-inclusion is load-bearing rather than cosmetic: a decay that does not put an asset at the top of its own scale silently produces a structural equivalence matrix instead of a proximity one — see PhylogenyPanel's "Why the diagonal includes self".
  • Monotone non-increasing in d.
  • Never assumed to reach zero. Truncation is a separate knob: the consumer applies its own budget — separation_budget of the AbstractSeparationAlgorithm in scope — and the decay only shapes the fall-off inside it. An exponential never reaches zero, so budget and fall-off cannot be the same dial.
  • f(d) >= 0 for 0 <= d <= dmax. 0 is the unreachable sentinel, so a negative score inside the budget would place a reachable pair strictly below an unreachable one — an ordering inversion within the producer's own scale. It is not a claim that a signed score is wrong in general: the feature matrix is signed-tolerant by decision, and assert_metric_domain checks non-negativity per metric at the consumer rather than blanket. This clause is producer-local, and it is non-negativity rather than strict positivity because a decay that bottoms out at zero says no relatedness, which is the same claim an unreachable pair makes.

The clause is scoped to the budget because the sign outside it is unobservable — the consumer's h[u] <= n test short-circuits before the decay is ever evaluated there — and because the family's own default violates the wider statement: LinearDecay crosses zero at d = dmax + 1 and is negative above it. Binding the clause on all d >= 0 would need the max(0, ⋅) floor the budget knob exists to avoid, a second truncation biting before n does.

A zero in the resulting feature matrix therefore means functionally unreachable: either the graph is disconnected there, or the decay has fallen to nothing — the same claim about the pair, and nothing downstream can act on the difference. No shipped member emits zero anywhere inside the budget, so for what ships a zero is disconnection and nothing else.

The budget is an argument, not a field

separation_decay takes the budget in scope as its third argument, dmax, and members may ignore it — only LinearDecay reads it, to set f(0). Keeping it off the member is what makes the two knobs impossible to desync: the AbstractSeparationAlgorithm stays the single source of truth for the budget, rather than mirroring it on an algorithm that cannot see it at construction. ExponentialDecay provides the self-versus-neighbour contrast a free top-of-scale would have bought, without the hazard of a second truncation hiding inside the decay.

Enforcement

The contract is enforced rather than merely documented, by a probing assert_separation_decay fallback on this type. The shipped members satisfy it by construction and override that fallback to a no-op, so the check is opt-out: an extension that says nothing about itself gets probed.

Related

source
PortfolioOptimisers.is_reachable — Function
is_reachable(sep::AbstractSeparationAlgorithm, d::Number)

Is d a separation at all, or the sentinel an unreachable pair carries?

separation_matrix passes the underlying routine's sentinel through unrepaired, so this is the test that tells a measured separation from a missing one. It is one generic method on AbstractSeparationAlgorithm rather than one per member, because the two shipped sentinels are both covered by the same expression; a member whose routine reports something else overrides this method, and inherits is_related unchanged.

The test is not isfinite alone, and not typemax alone

Both clauses carry a sentinel of their own.

  • isfinite is true for every Integer, so on its own it admits HopCount's typemax(Int) — which ReciprocalDecay then overflows.
  • typemax of a Float64 is Inf, so the comparison covers PathLength's sentinel as well; isfinite stays to reject a NaN, which no shipped path produces and which would compare false against every budget anyway.

Algorithm

  1. Test isfinite(d), rejecting a NaN and rejecting PathLength's Inf.
  2. Compare d against typemax(typeof(d)), rejecting HopCount's typemax(Int), which step 1 admits.
  3. Return reachable, the conjunction of the two tests. && short-circuits, so step 2 runs only on a finite d.

Arguments

  • sep: Separation algorithm. Inert for the shipped members, and the dispatch channel for an extension whose routine reports a different sentinel.
  • d: One entry of a separation matrix from separation_matrix.

Returns

  • reachable::Bool: true when d is a measured separation.

Related

source
PortfolioOptimisers.is_related — Function
is_related(sep::AbstractSeparationAlgorithm, d::Number, dmax::Number)

Does a separation of d count as related under a budget of dmax?

The one place the budget is applied to an entry of a separation matrix: reachable, and no further than dmax. Every consumer that selects on a budget calls this instead of writing the comparison out, so the rule has one spelling — phylogeny_matrix and phylogeny_features had two, and one of them was a budget test with no sentinel test behind it.

The reachability test comes first

d <= dmax is not sufficient on its own. It happens to reject both shipped sentinels, because separation_budget clamps a PathLength budget to the observed finite diameter and a HopCount budget is capped far below typemax(Int) — but that is a property of the two shipped budgets, not of the comparison. is_reachable makes the rejection the predicate's own, so a budget that reached its unit's ceiling would still exclude an unreachable pair.

It does not remove the caller's obligation to short-circuit

A consumer that scores the separation must still keep the evaluation of the score inside a short-circuiting branch — is_related(...) ? separation_decay(...) : zero(...), never ifelse — because an ifelse evaluates both arms and ReciprocalDecay overflows 1 + d at typemax(Int), which a fractional power turns into a DomainError. The predicate owns the rule; the call site owns the laziness.

Algorithm

  1. Call is_reachable on sep and d, giving the reachability test. This runs first, so a sentinel is rejected whatever dmax is.
  2. Compare d against dmax, giving the budget test.
  3. Return related, the conjunction of the two tests. && short-circuits, so step 2 runs only on a reachable d.

Arguments

  • sep: Separation algorithm, forwarded to is_reachable.
  • d: One entry of a separation matrix from separation_matrix.
  • dmax: Separation budget in scope, from separation_budget. In the units sep measures in, which is why the two arrive together.

Returns

  • related::Bool: true when the pair is reachable and inside the budget.

Related

source
PortfolioOptimisers.assert_separation_decay — Function
assert_separation_decay(dk::AbstractSeparationDecayAlgorithm, ds, dmax::Number)
assert_separation_decay(dk::Union{<:LinearDecay, <:ExponentialDecay, <:ReciprocalDecay,
                                  <:NoDecay}, ds, dmax::Number)

Check that a separation decay honours its contract over the separations it will be asked about.

The fallback probes: it evaluates separation_decay over ds and checks the result is finite, that f(0) is strictly positive and maximal, that the values are monotone non-increasing, and that none of them is negative. The four shipped members satisfy the contract by construction and override this to a no-op, so the probe costs nothing for what ships and is fail-safe for extensions.

Probing is cheap where it is used because ds is small and the loop it guards is not: Proximity passes 0:dmax, which under HopCount is exhaustive — every separation the assets × assets loop can ever ask about, in dmax + 1 evaluations. Under a separation whose budget is not an integer the same range is a unit-spaced sample, which is all a continuum admits and all the clauses below need.

Non-negativity gets one extra evaluation at d = dmax, whether or not dmax appears in ds, mirroring the out-of-loop evaluation of f(0). That endpoint is what closes the clause over a continuum: monotonicity is already promised, so f(dmax) >= 0 implies f(d) >= 0 for every d in [0, dmax], and a ds that can only ever be a sample — as it must be once separations are weighted path lengths — costs this clause nothing. Monotonicity itself gains nothing from the endpoint and remains genuinely sampled.

Algorithm

These are the steps of the probing fallback. The method on the four shipped members runs none of them.

  1. Score the separation zero(dmax) with separation_decay, giving f0, the top of the scale.
  2. Check that f0 is finite and strictly positive.
  3. Sort ds when it is not sorted already, so that step 5 reads the separations in increasing order.
  4. Set fp, the score of the previous separation, to f0.
  5. For each d of ds, score it with separation_decay, giving f. Check that f is finite and does not exceed f0. Check that f does not exceed fp. Check that f is non-negative. Set fp to f.
  6. Score dmax itself, giving fmax, and check that fmax is non-negative. This is the endpoint the paragraph above states, and step 5 need not have reached it.
  7. Return nothing.

Arguments

  • dk: Separation decay algorithm.
  • ds: Separations to probe. Need not be sorted. Precondition: ds ⊆ [0, dmax] — ds is what the guarded loop will ask about, and the loop never asks outside the budget.
  • dmax: Separation budget in scope, forwarded to separation_decay and probed as an endpoint in its own right.

Validation

  • Every probed value is finite.
  • f(0) > 0.
  • f(0) >= f(d) for every probed d.
  • The probed values are monotone non-increasing in d.
  • f(d) >= 0 for every probed d, and at d = dmax whether or not it was probed.

Returns

  • nothing.

Related

source