Clusters

struct Clusters{__T_res, __T_S, __T_D, __T_P, __T_k} <: AbstractClusteringResult

Carries a clustering of the asset universe together with the matrices it was computed from.

Holds the result of either family, because res takes both shapes clusterise produces: a Clustering.Hclust from a hierarchical algorithm and a Clustering.ClusteringResult from a non-hierarchical one. k is the number of clusters selected by an AbstractOptimalNumberClustersEstimator, and assignments is what reads a cluster label per asset off the pair — cutting the tree at k on the hierarchical branch, reading the field on the other.

D is what the clustering saw, unless P is present

S and D are the similarity and distance matrices of the universe, and an ordinary ClustersEstimator clusters D itself and leaves P as nothing. A NetworkClustersEstimator instead accumulates a pseudo-distance matrix out of the network structure and clusters that, so it fills P and res is the clustering of P. Both matrices are kept because a consumer wants the network's clustering and the universe's own distances.

Fields

• res: Clustering result.

• S: Similarity matrix.

• D: Distance matrix.

• P: Pseudo-distance matrix the clustering was run on, nothing when the clustering ran on D itself. A NetworkClustersEstimator builds it by accumulating the network structure out of the distance or similarity matrix; see clusterise.

• k: Optimal number of clusters.

Constructors

Clusters(;    res::ClTypes,    S::MatNum,    D::MatNum,    P::Option{<:MatNum} = nothing,    k::Integer) -> Clusters

Keywords correspond to the struct's fields.

Validation

• !isempty(S).
• !isempty(D).
• size(S) == size(D).
• If P is not nothing, !isempty(P) and size(S) == size(P).
• k >= 1.

Related

• AbstractClusteringResult
• ClustersEstimator
• NetworkClustersEstimator
• clusterise
• assignments

struct SecondOrderDifference{__T_alg} <: AbstractOptimalNumberClustersAlgorithm

Picks the number of clusters at which the within-cluster dispersion curve bends most sharply.

The two-difference gap statistic. It reads the second-order difference of the within-cluster dispersion across cluster counts, so it finds the elbow of that curve without the Monte Carlo simulation the original gap statistic needs.

Mathematical definition

\[\begin{align} c^{\star} &= \underset{c}{\arg\max} \left(W_{c+2} + W_{c} - 2 W_{c+1}\right)\,,\\ &\mathrm{s.t.} \quad 1 \leq c \leq \sqrt{N}\,,\\ W_{c} &= \sum_{j=1}^{c} g\left(\left\{d_{uv} : u, v \in \mathcal{C}_{j},\, u < v\right\}\right)\,. \end{align}\]

Where:

• $c^{\star}$: Selected number of clusters.
• $W_{c}$: Within-cluster dispersion of a cut into $c$ clusters.
• $\mathcal{C}_{j}$: Set of assets in the $j$-th cluster of that cut.
• $d_{uv}$: Entry of the distance matrix the clustering ran on.
• $g$: Vector-to-scalar measure alg, applied to one cluster's pairwise distances. A cluster of one asset has no pairwise distance and contributes $0$.
• $N$: Number of assets.

The selected $c$ is the left end of the triple, not its centre. That is the source's own statement of the problem, and the code maximises it as written.

The measure decides what "dispersion" means, and the default is not the source's

The source takes $W_{c}$ as the mean of a cluster's pairwise distances, which is alg = MeanValue(). The default alg = StandardisedValue() divides that mean by the corrected standard deviation of the same distances, which is a different statistic and selects a different $c$.

Algorithm

optimal_number_clusters runs these steps, and both branches run the same ones. The bodies live in 03_Hierarchical.jl and in 04_NonHierarchicalClustering.jl.

1. Take the ceiling min(floor(Int, sqrt(N)), max_k) from the enclosing OptimalNumberClusters, add $2$ to it, and cap the sum at $N$, giving c1, the number of candidate counts to build. The two extra counts exist because the difference at $c$ reads the dispersions of $c + 1$ and $c + 2$ as well.
2. Cluster the universe at every count from $1$ to c1, giving cluster_lvls.
3. Set W_list[1] to typemin, so that the difference at $c = 1$ cannot win.
4. Reduce every cut from $2$ to c1 to one dispersion with alg, filling the rest of W_list. On the hierarchical branch alg runs once per cluster over that cluster's pairwise distances and the results are summed; on the non-hierarchical branch it runs once over the k-means per-point costs.
5. Take the second-order difference of W_list, giving gaps, one entry per candidate count up to the ceiling. A universe so small that c1 does not exceed $2$ carries no such difference, and both branches return c1 itself.
6. Select the count. The hierarchical branch hands gaps to valid_k_clusters, which walks down from the largest entry until the dendrogram admits the count; the non-hierarchical branch has no dendrogram to reject a count, so it takes the argmax as it stands. Both branches read the length of the score array in place of the argmax when no entry of it is finite.

Fields

• alg: The measure used to evaluate clustering quality.

Constructors

SecondOrderDifference(;    alg::Num_VecToScaM = StandardisedValue()) -> SecondOrderDifference

Keywords correspond to the struct's fields.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• alg: Recursively updated via factory.

Examples

julia> SecondOrderDifference()SecondOrderDifference  alg ┼ StandardisedValue      │   mv ┼ MeanValue      │      │   w ┴ nothing      │   sv ┼ StdValue      │      │           w ┼ nothing      │      │   corrected ┴ Bool: true

Related

• AbstractOptimalNumberClustersAlgorithm
• OptimalNumberClusters
• SilhouetteScore
• VectorToScalarMeasure
• MeanValue
• StandardisedValue
• optimal_number_clusters
• factory

References

• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.2, Equation 12.15.
• [45] S. Yue, X. Wang and M. Wei. Application of two-order difference to gap statistic. Transactions of Tianjin University 14, 217–221 (2008).
• [46] R. Tibshirani, G. Walther and T. Hastie. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411–423 (2001).

struct SilhouetteScore{__T_alg} <: AbstractOptimalNumberClustersAlgorithm

Picks the number of clusters whose assets sit best inside their own cluster.

Each asset gets a silhouette, which compares how far it sits from its own cluster against how far it sits from the nearest other one. alg reduces the whole vector of them to one number per cluster count, and the count with the largest number wins.

Mathematical definition

\[\begin{align} s_{i} &= \frac{b_{i} - a_{i}}{\max\left(a_{i},\, b_{i}\right)}\,,\\ a_{i} &= \sum_{j \in \mathcal{C}_{I},\, j \neq i} \frac{d_{ij}}{\left|\mathcal{C}_{I}\right| - 1}\,,\\ b_{i} &= \underset{J \neq I}{\min} \sum_{j \in \mathcal{C}_{J}} \frac{d_{ij}}{\left|\mathcal{C}_{J}\right|}\,,\\ c^{\star} &= \underset{c}{\arg\max} \; g\left(\boldsymbol{s}\right)\,, \quad 1 \leq c \leq \sqrt{N}\,. \end{align}\]

Where:

• $s_{i}$: Silhouette of asset $i$, in $\left[-1,\, 1\right]$. It is $1$ when the asset is clustered well and $-1$ when it is clustered wrongly.
• $a_{i}$: Mean distance from asset $i$ to the other members of its own cluster $\mathcal{C}_{I}$.
• $b_{i}$: Smallest mean distance from asset $i$ to the members of any other cluster.
• $d_{ij}$: Entry of the distance matrix the clustering ran on.
• $\boldsymbol{s}$: Vector of silhouettes over all assets, one per asset.
• $g$: Vector-to-scalar measure alg.
• $c^{\star}$: Selected number of clusters.
• $N$: Number of assets.

The default reduction is the source's standardised score

alg = StandardisedValue() divides the mean of $\boldsymbol{s}$ by its corrected standard deviation, which is the source's quality measure term for term. The standardisation is what makes scores comparable across cluster counts, so it is the reduction the selection wants; the field is a knob only because a caller may want the plain mean instead. The two disagree in general, and the standardised form is the one this default computes.

Unlike SecondOrderDifference, the reduction here runs over one vector of length $N$, never over one cluster at a time, so no cluster size makes it undefined.

Algorithm

optimal_number_clusters runs these steps, and both branches run the same ones. The bodies live in 03_Hierarchical.jl and in 04_NonHierarchicalClustering.jl.

1. Take the ceiling min(floor(Int, sqrt(N)), max_k) from the enclosing OptimalNumberClusters, giving c1, the largest candidate count. No count beyond the ceiling is built here, because the score at $c$ reads that cut alone.
2. Cluster the universe at every count from $1$ to c1, giving cluster_lvls.
3. Set W_list[1] to typemin, so that a single cluster cannot win.
4. Compute the per-asset silhouettes of every cut from $2$ to c1 with Clustering.silhouettes, giving sl, one entry per asset.
5. Reduce each sl to one score with alg, filling the rest of W_list.
6. Select the count. The hierarchical branch hands W_list to valid_k_clusters, which walks down from the largest entry until the dendrogram admits the count; the non-hierarchical branch has no dendrogram to reject a count, so it takes the argmax as it stands. Both branches read the length of the score array in place of the argmax when no entry of it is finite.

Fields

• alg: The measure used to evaluate clustering quality.

Constructors

SilhouetteScore(;    alg::Num_VecToScaM = StandardisedValue()) -> SilhouetteScore

Keywords correspond to the struct's fields.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• alg: Recursively updated via factory.

Examples

julia> SilhouetteScore()SilhouetteScore  alg ┼ StandardisedValue      │   mv ┼ MeanValue      │      │   w ┴ nothing      │   sv ┼ StdValue      │      │           w ┼ nothing      │      │   corrected ┴ Bool: true

Related

• AbstractOptimalNumberClustersAlgorithm
• OptimalNumberClusters
• SecondOrderDifference
• VectorToScalarMeasure
• StandardisedValue
• optimal_number_clusters
• factory

References

• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.1, Equations 12.13-12.14.
• [47] P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65 (1987).
• [48] M. López de Prado and M. J. Lewis. Detection of false investment strategies using unsupervised learning methods. Quantitative Finance 19, 1555–1565 (2019).

struct OptimalNumberClusters{__T_max_k, __T_alg} <: AbstractOptimalNumberClustersEstimator

Decides how many clusters to cut a dendrogram or a partition into.

Pairs a selection rule with a ceiling on the answer. alg is either an AbstractOptimalNumberClustersAlgorithm, which chooses the count from the data, or an Integer, which states it outright.

A stated k is a request, not a guarantee

max_k caps every branch, and the cap is itself capped: the ceiling in force is min(floor(Int, sqrt(N)), max_k), so a max_k above $\sqrt{N}$ buys nothing. A stated k above that ceiling is lowered to it.

A stated k is also checked against the tree on the hierarchical branch. A cut into k clusters that no node of the dendrogram supports is not honoured; optimal_number_clusters searches upward and downward for the nearest k that is, and takes the nearer of the two. The data-driven branches apply the same test through valid_k_clusters, which walks the scoring array from its largest entry down until one passes.

Fields

• max_k: Maximum number of clusters to consider. If nothing, computed as the floor(Int, sqrt(assets)).

• alg: Algorithm for selecting the optimal number of clusters. If an integer, defines the number of clusters directly.

Constructors

OptimalNumberClusters(;    max_k::Option{<:Integer} = nothing,    alg::Int_ONC = SecondOrderDifference()) -> OptimalNumberClusters

Keywords correspond to the struct's fields.

Validation

• If max_k is not nothing, max_k >= 1.
• If alg is an Integer, alg >= 1.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• alg: Recursively updated via factory.

Examples

julia> OptimalNumberClusters(; max_k = 10)OptimalNumberClusters  max_k ┼ Int64: 10    alg ┼ SecondOrderDifference        │   alg ┼ StandardisedValue        │       │   mv ┼ MeanValue        │       │      │   w ┴ nothing        │       │   sv ┼ StdValue        │       │      │           w ┼ nothing        │       │      │   corrected ┴ Bool: true

Related

• AbstractOptimalNumberClustersEstimator
• AbstractOptimalNumberClustersAlgorithm
• SecondOrderDifference
• SilhouetteScore
• optimal_number_clusters
• valid_k_clusters
• factory

References

• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.

struct HClustAlgorithm{__T_linkage} <: AbstractHierarchicalClusteringAlgorithm

Builds a dendrogram by merging the two nearest clusters until one remains.

linkage is the criterion that says how near two clusters are once they hold more than one asset. :single takes the smallest distance between their members and :complete the largest; :average takes the mean over all cross pairs; :ward merges the pair that raises the total within-cluster variance least. The symbol is passed straight to Clustering.hclust, so every criterion that package accepts is available.

Fields

• linkage: Linkage method for hierarchical clustering from Clustering.jl. The constructor accepts any Symbol and checks none of them, because the set of criteria belongs to Clustering.hclust and not to this library. A symbol that function does not accept raises its own ArgumentError at the point of use inside clusterise, not at construction: HClustAlgorithm(; linkage = :nonsense) builds, and clustering with it raises ArgumentError: Unsupported cluster linkage nonsense.

Constructors

HClustAlgorithm(;    linkage::Symbol = :ward) -> HClustAlgorithm

Keywords correspond to the struct's fields.

Examples

julia> HClustAlgorithm(; linkage = :average)HClustAlgorithm  linkage ┴ Symbol: :average

Related

• AbstractHierarchicalClusteringAlgorithm
• ClustersEstimator
• DBHT
• clusterise

References

• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.1.1, Equations 12.6-12.12.
• [49] D. Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 (2011).

struct ClustersEstimator{__T_ce, __T_de, __T_alg, __T_onc} <: AbstractClustersEstimator

Turns a return matrix into a clustering of the asset universe.

Holds the four steps in the order they run: ce estimates the covariance, de turns it into a distance matrix, alg clusters that matrix, and onc decides how many clusters to keep. clusterise runs them and returns a Clusters.

The universe is clustered, not the observations

Every step reads an assets x assets matrix, so the number of observations leaves the picture at ce. That is why alg and onc carry no @fprop tag: an ObsWeights weights observations, and there is no observation left for it to weight once de has run. ce and de are tagged, and they are where a weight belongs.

Fields

• ce: Covariance estimator.

• de: Distance matrix estimator.

• alg: Clustering algorithm.

• onc: Optimal number of clusters estimator.

Constructors

ClustersEstimator(;    ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),    de::AbstractDistanceEstimator = Distance(; alg = CanonicalDistance()),    alg::AbstractClustersAlgorithm = HClustAlgorithm(),    onc::AbstractOptimalNumberClustersEstimator = OptimalNumberClusters()) -> ClustersEstimator

Keywords correspond to the struct's fields.

Propagated parameters

When factory is called on this type, the following @fprop-tagged fields are automatically propagated:

• ce: Recursively updated via factory.
• de: Recursively updated via factory.

Examples

julia> ClustersEstimator()ClustersEstimator   ce ┼ PortfolioOptimisersCovariance      │   ce ┼ Covariance      │      │    me ┼ SimpleExpectedReturns      │      │       │   w ┴ nothing      │      │    ce ┼ GeneralCovariance      │      │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)      │      │       │    w ┴ nothing      │      │   alg ┼ FullMoment()      │      │     w ┴ nothing      │   mp ┼ MatrixProcessing      │      │     pdm ┼ Posdef      │      │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton      │      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()      │      │      dn ┼ nothing      │      │      dt ┼ nothing      │      │     alg ┼ nothing      │      │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)   de ┼ Distance      │   power ┼ nothing      │     alg ┴ CanonicalDistance()  alg ┼ HClustAlgorithm      │   linkage ┴ Symbol: :ward  onc ┼ OptimalNumberClusters      │   max_k ┼ nothing      │     alg ┼ SecondOrderDifference      │         │   alg ┼ StandardisedValue      │         │       │   mv ┼ MeanValue      │         │       │      │   w ┴ nothing      │         │       │   sv ┼ StdValue      │         │       │      │           w ┼ nothing      │         │       │      │   corrected ┴ Bool: true

Related

• AbstractClustersEstimator
• AbstractHierarchicalClusteringAlgorithm
• AbstractNonHierarchicalClusteringAlgorithm
• AbstractOptimalNumberClustersEstimator
• Clusters
• clusterise
• factory

References

• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.1.1.

clusterise(cle::AbstractClusteringResult, args...; kwargs...)

Return the clustering result cle unchanged.

Identity pass-through, so that every consumer takes an estimator or a precomputed result through one call. A ClustersEstimator reaching this function has already been run; a result reaching it is not run again.

Algorithm

1. Return cle itself. No clustering is run, and args and kwargs are discarded.

Arguments

• cle: Clustering result.
• args...: Additional positional arguments, ignored.
• kwargs...: Additional keyword arguments, ignored.

Returns

• cle::AbstractClusteringResult: The input, unchanged.

Related

• AbstractClusteringResult
• Clusters
• ClustersEstimator

factory(
    alg::AbstractClustersAlgorithm,
    args...;
    kwargs...
) -> AbstractClustersAlgorithm

Return the clustering algorithm alg unchanged.

Identity pass-through used when a clustering algorithm is provided in a context that calls factory. A clustering algorithm carries no prior-dependent field to rebuild, so every field is passed through and none is replaced.

Algorithm

1. Return alg itself. No field of it is rebuilt, and args and kwargs are discarded.

Arguments

• alg: Clustering algorithm.
• args...: Optional arguments (ignored).
• kwargs...: Optional keyword arguments (ignored).

Returns

• alg::AbstractClustersAlgorithm: The original clustering algorithm.

Related

• AbstractClustersAlgorithm
• factory