Clusters

PortfolioOptimisers.Clusters — Type
struct Clusters{__T_res, __T_S, __T_D, __T_P, __T_k} <: AbstractClusteringResult

Holds a clustering of the asset universe and the matrices that the clustering used.

res holds either of the two outputs of clusterise. A hierarchical algorithm makes a Clustering.Hclust, and a non-hierarchical algorithm makes a Clustering.ClusteringResult. k is the number of clusters that an AbstractOptimalNumberClustersEstimator selected. assignments reads one cluster label per asset from res and k. On the hierarchical branch it cuts the tree at k, and on the other branch it reads the labels that res holds.

The matrix that the clustering used

S is the similarity matrix of the universe, and D is its distance matrix. A ClustersEstimator clusters D and sets P to nothing. A NetworkClustersEstimator makes a pseudo-distance matrix from the structure of the network and clusters that matrix. It stores the matrix in P, so res is the clustering of P. The result also keeps D, so a caller can read the distances of the universe next to the clustering of the network.

Fields

  • res: Clustering result.
  • S: Similarity matrix.
  • D: Distance matrix.
  • P: Pseudo-distance matrix that the clustering used, or nothing when the clustering used D. A NetworkClustersEstimator makes it from the structure of the network, see clusterise.
  • k: Selected number of clusters.

Constructors

Clusters(;    res::ClTypes,    S::MatNum,    D::MatNum,    P::Option{<:MatNum} = nothing,    k::Integer) -> Clusters

Keywords correspond to the struct's fields.

Validation

  • !isempty(S).
  • !isempty(D).
  • size(S) == size(D).
  • If P is not nothing, !isempty(P) and size(S) == size(P).
  • k >= 1.

Related

source
PortfolioOptimisers.SecondOrderDifference — Type
struct SecondOrderDifference{__T_alg} <: AbstractOptimalNumberClustersAlgorithm

Selects the number of clusters at which the within-cluster dispersion has its largest second-order difference.

This score is the two-difference gap statistic. The second-order difference finds the elbow of the curve of the dispersion over the cluster counts. The gap statistic of [118] needs a Monte Carlo simulation for this, and the second-order difference does not.

Mathematical definition

\[\begin{align} c^{\star} &= \underset{c}{\arg\max} \left(W_{c+2} + W_{c} - 2 W_{c+1}\right)\,,\\ &\mathrm{s.t.} \quad 1 \leq c \leq \sqrt{N}\,,\\ W_{c} &= \sum_{j=1}^{c} g\left(\left\{d_{uv} : u, v \in \mathcal{C}_{j},\, u < v\right\}\right)\,. \end{align}\]

Where:

  • $c^{\star}$: Selected number of clusters.
  • $W_{c}$: Within-cluster dispersion of a cut into $c$ clusters.
  • $\mathcal{C}_{j}$: Set of assets in the $j$-th cluster of that cut.
  • $d_{uv}$: Entry of the distance matrix that the clustering used.
  • $g$: Vector-to-scalar measure alg. It reduces the pairwise distances of one cluster to one number. A cluster of one asset has no pairwise distance and contributes $0$.
  • $N$: Number of assets.

The selected $c$ is the left end of the triple, not its centre. Equation 12.15 of [12] prints it so, and the code maximises the equation as printed. The equation admits $c = 1$, and step 3 of the algorithm below excludes it.

The measure defines the dispersion

Equation 12.15 of [12] calls $W_{c}$ the within-cluster dispersion and gives no other definition. The book takes the equation from [119]. No copy of that work is available, so its definition of $W_{c}$ is not known.

The gap statistic of [118], Equation 2, takes $W_{k} = \sum_{r} D_{r} / (2 n_{r})$. $D_{r}$ is the sum of the pairwise distances of cluster $r$ over the ordered pairs. That is $(n_{r} - 1) / 2$ times the mean of the distances, so a large cluster has more weight. alg = MeanValue() takes the mean of the pairwise distances of a cluster. The default alg = StandardisedValue() divides that mean by the corrected standard deviation of the same distances.

The three definitions can select three different counts. On the sample randn(StableRNG(987654321), 400, 40), MeanValue() selects 6 and StandardisedValue() selects 4.

Note

A cluster of two assets has one pairwise distance, and one value has no corrected standard deviation. StandardisedValue divides by $1$ in that case, so such a cluster contributes its mean pairwise distance to $W_{c}$. The gap series stays finite, and the selected $c$ is a real maximiser. The two measures still differ on every larger cluster.

Algorithm

optimal_number_clusters does these steps on both branches. Its methods are in 03_Hierarchical.jl and in 04_NonHierarchicalClustering.jl.

  1. Take the ceiling min(floor(Int, sqrt(N)), max_k) from the OptimalNumberClusters that holds the score. Add $2$ to it and cap the sum at $N$. The result is c1, the number of candidate counts. The difference at $c$ also reads the dispersions at $c + 1$ and $c + 2$, so the method builds two more counts.
  2. Cluster the universe at every count from $1$ to c1. The cuts are cluster_lvls.
  3. Set W_list[1] to typemin, so that the difference at $c = 1$ cannot win.
  4. Reduce every cut from $2$ to c1 to one dispersion with alg, and write it into W_list. On the hierarchical branch, alg runs once for each cluster over the pairwise distances of that cluster, and the method sums the results. On the non-hierarchical branch, alg runs once over the per-point costs of the k-means run.
  5. Take the second-order difference of W_list. The result is gaps, one entry for each candidate count up to the ceiling. If c1 is $2$ or less, there is no such difference. This happens only when $N \leq 2$, where the ceiling is $1$, and both branches return the ceiling.
  6. Select the count. The hierarchical branch gives gaps to valid_k_clusters. That function moves down from the largest entry until the dendrogram accepts the count. The non-hierarchical branch has no dendrogram to reject a count, so it takes the argmax of gaps. If no entry of the scores is finite, both branches take the length of the score array in place of the argmax.

Fields

  • alg: Vector-to-scalar measure that the score applies at each candidate count.

Constructors

SecondOrderDifference(;    alg::Num_VecToScaM = StandardisedValue()) -> SecondOrderDifference

Keywords correspond to the struct's fields.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • alg: Recursively updated through factory.

Examples

julia> SecondOrderDifference()SecondOrderDifference  alg ┼ StandardisedValue      │   mv ┼ MeanValue      │      │   w ┴ nothing      │   sv ┼ StdValue      │      │           w ┼ nothing      │      │   corrected ┴ Bool: true

Related

References

  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.2, Equation 12.15. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
  • [119] S. Yue, X. Wang and M. Wei. Application of two-order difference to gap statistic. Transactions of Tianjin University 14, 217–221 (2008).
  • [118] R. Tibshirani, G. Walther and T. Hastie. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411–423 (2001). Equations 1-3, for the gap statistic that needs the simulation.
source
PortfolioOptimisers.SilhouetteScore — Type
struct SilhouetteScore{__T_alg} <: AbstractOptimalNumberClustersAlgorithm

Selects the number of clusters from the silhouettes of the assets.

The silhouette of an asset compares its mean distance to its own cluster with its mean distance to the nearest other cluster. For each candidate count, alg reduces the vector of silhouettes to one number. The count with the largest number wins.

Mathematical definition

\[\begin{align} s_{i} &= \frac{b_{i} - a_{i}}{\max\left(a_{i},\, b_{i}\right)}\,,\\ a_{i} &= \sum_{j \in \mathcal{C}_{I},\, j \neq i} \frac{d_{ij}}{\left|\mathcal{C}_{I}\right| - 1}\,,\\ b_{i} &= \underset{J \neq I}{\min} \sum_{j \in \mathcal{C}_{J}} \frac{d_{ij}}{\left|\mathcal{C}_{J}\right|}\,,\\ c^{\star} &= \underset{c}{\arg\max} \; g\left(\boldsymbol{s}\right)\,, \quad 1 \leq c \leq \sqrt{N}\,. \end{align}\]

Where:

  • $s_{i}$: Silhouette of asset $i$, in $\left[-1,\, 1\right]$. It is $1$ when $a_{i} = 0$ and $-1$ when $b_{i} = 0$.
  • $a_{i}$: Mean distance from asset $i$ to the other members of its own cluster $\mathcal{C}_{I}$.
  • $b_{i}$: Smallest mean distance from asset $i$ to the members of any other cluster.
  • $d_{ij}$: Entry of the distance matrix that the clustering used.
  • $\boldsymbol{s}$: Vector of silhouettes over all assets, one per asset.
  • $g$: Vector-to-scalar measure alg.
  • $c^{\star}$: Selected number of clusters.
  • $N$: Number of assets.

The default reduction is the source's standardised score

alg = StandardisedValue() divides the mean of $\boldsymbol{s}$ by its corrected standard deviation. This is the quality measure of [120], term for term. [120] writes $q = \mathrm{E}[\{s_{i}\}] / \sqrt{\mathrm{V}[\{s_{i}\}]}$ and does not say which variance it takes. Equation 12.14 of [12] takes the corrected variance. At a fixed $N$ the two forms differ by a constant factor, so they select the same $c$.

alg = MeanValue() gives the rule of [121], which selects the count with the largest mean silhouette. The mean and the standardised score can select different counts. On the sample randn(StableRNG(987654321), 400, 40), the mean selects 6 and the standardised score selects 4.

Neither [121] nor [120] bounds $c$ by $\sqrt{N}$, and the second tries every count from $2$ to $N - 1$. The bound is the ceiling of OptimalNumberClusters, which applies to both scores.

The reduction runs over one vector of length $N$, not over one cluster at a time as in SecondOrderDifference. So a cluster of one or two assets needs no special case here.

Algorithm

optimal_number_clusters does these steps on both branches. Its methods are in 03_Hierarchical.jl and in 04_NonHierarchicalClustering.jl.

  1. Take the ceiling min(floor(Int, sqrt(N)), max_k) from the OptimalNumberClusters that holds the score. The result is c1, the largest candidate count. The score at $c$ reads only the cut at $c$, so the method builds no count above the ceiling.
  2. Cluster the universe at every count from $1$ to c1. The cuts are cluster_lvls.
  3. Set W_list[1] to typemin, so that a single cluster cannot win.
  4. Calculate the silhouettes of the assets for every cut from $2$ to c1 with Clustering.silhouettes. The result for one cut is sl, with one entry for each asset.
  5. Reduce each sl to one score with alg, and write it into W_list.
  6. Select the count. The hierarchical branch gives W_list to valid_k_clusters. That function moves down from the largest entry until the dendrogram accepts the count. The non-hierarchical branch has no dendrogram to reject a count, so it takes the argmax of W_list. If no entry of the scores is finite, both branches take the length of the score array in place of the argmax.

Fields

  • alg: Vector-to-scalar measure that the score applies at each candidate count.

Constructors

SilhouetteScore(;    alg::Num_VecToScaM = StandardisedValue()) -> SilhouetteScore

Keywords correspond to the struct's fields.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • alg: Recursively updated through factory.

Examples

julia> SilhouetteScore()SilhouetteScore  alg ┼ StandardisedValue      │   mv ┼ MeanValue      │      │   w ┴ nothing      │   sv ┼ StdValue      │      │           w ┼ nothing      │      │   corrected ┴ Bool: true

Related

References

  • [121] P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65 (1987). Section 2, pages 55-56, for $s_{i}$, and page 59 for the mean silhouette width.
  • [120] M. López de Prado and M. J. Lewis. Detection of false investment strategies using unsupervised learning methods. Quantitative Finance 19, 1555–1565 (2019). Section 3, for the quality measure $q$.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.1, Equation 12.14, for the corrected variance in $q$.
source
PortfolioOptimisers.OptimalNumberClusters — Type
struct OptimalNumberClusters{__T_max_k, __T_alg} <: AbstractOptimalNumberClustersEstimator

Selects the number of clusters of a dendrogram or a flat partition, under a ceiling.

alg is an AbstractOptimalNumberClustersAlgorithm, which selects the count from the data, or an Integer, which states the count.

The ceiling and a stated k

The ceiling is min(floor(Int, sqrt(N)), max_k) on every branch. A max_k above $\sqrt{N}$ has no effect, and max_k = nothing sets the ceiling to floor(Int, sqrt(N)). The selection lowers a stated k above the ceiling to the ceiling.

On the hierarchical branch, validate_k_value also tests a stated k against the tree. If no node of the dendrogram supports a cut into k clusters, optimal_number_clusters searches up and down for the nearest valid count and takes the nearer one. On the same branch, the two scores apply the same test through valid_k_clusters. That function moves down the score array from the largest entry until a count passes.

The ceiling $\sqrt{N}$ is the bound of Equation 12.15 of [12]. There, it applies only to the two-difference gap statistic. The book takes that equation from a work of which no copy is available, see SecondOrderDifference. The cap max_k and the search of the dendrogram for a valid k are additions of this library.

Fields

  • max_k: Largest number of clusters that the selection considers. The ceiling is min(floor(Int, sqrt(N)), max_k), where N is the number of assets. nothing sets the ceiling to floor(Int, sqrt(N)).
  • alg: Algorithm that selects the number of clusters, or an Integer that states it.

Constructors

OptimalNumberClusters(;    max_k::Option{<:Integer} = nothing,    alg::Int_ONC = SecondOrderDifference()) -> OptimalNumberClusters

Keywords correspond to the struct's fields.

Validation

  • If max_k is not nothing, max_k >= 1.
  • If alg is an Integer, alg >= 1.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • alg: Recursively updated through factory.

Examples

julia> OptimalNumberClusters(; max_k = 10)OptimalNumberClusters  max_k ┼ Int64: 10    alg ┼ SecondOrderDifference        │   alg ┼ StandardisedValue        │       │   mv ┼ MeanValue        │       │      │   w ┴ nothing        │       │   sv ┼ StdValue        │       │      │           w ┼ nothing        │       │      │   corrected ┴ Bool: true

Related

References

  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 12.2.1.2, Equation 12.15. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.HClustAlgorithm — Type
struct HClustAlgorithm{__T_linkage} <: AbstractHierarchicalClusteringAlgorithm

Builds a dendrogram that merges the two nearest clusters at each step, until one cluster remains.

linkage is the criterion for the distance between two clusters that have more than one asset.

  • :single takes the smallest distance between the members of the two clusters.
  • :complete takes the largest distance between the members of the two clusters.
  • :average takes the mean distance over all pairs with one member in each cluster.
  • :ward merges the pair of clusters that increases the total within-cluster variance least.
  • :ward_presquared is :ward for a matrix that holds squared distances.

clusterise gives the symbol to Clustering.hclust unchanged, so every criterion of that function is available.

Figure 2 of [122] states each criterion in two forms. One form updates the distance after a merge, and the other is a closed form over the members. The formulas of :ward hold for Euclidean distances between points. Clustering.hclust squares the distances for :ward, applies the update to the squares, and takes the square root of the merge heights. This is the form of that figure. Equation 12.12 of [12] prints the same update without the squares.

[122] is a survey. No copy is available of the works that first state the four criteria, or of the work that first states the update that they share.

Fields

  • linkage: Linkage criterion of Clustering.hclust, see Clustering.jl. The constructor accepts any Symbol and checks none, because Clustering.hclust owns the set of criteria. HClustAlgorithm(; linkage = :nonsense) builds. clusterise then throws ArgumentError: Unsupported cluster linkage nonsense from Clustering.hclust.

Constructors

HClustAlgorithm(;    linkage::Symbol = :ward) -> HClustAlgorithm

Keywords correspond to the struct's fields.

Examples

julia> HClustAlgorithm(; linkage = :average)HClustAlgorithm  linkage ┴ Symbol: :average

Related

References

  • [122] D. Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 (2011). Figure 2. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.ClustersEstimator — Type
struct ClustersEstimator{__T_ce, __T_de, __T_alg, __T_onc, __T_x_src} <: AbstractClustersEstimator

Clusters the assets of a universe from a return matrix.

Four of the fields are steps, in the order that they run.

  • ce estimates the correlation matrix S.
  • de makes the distance matrix.
  • alg clusters the distance matrix.
  • onc selects the number of clusters.

clusterise runs the steps and returns a Clusters. On the non-hierarchical branch, onc runs alg once for each candidate count.

The fields that take observation weights

ce and de read the observations of the return matrix. A VariationInfoDistance in de reads the return matrix itself, not the correlation matrix. So ce and de carry the @fprop tag, and the weights of an ObsWeights reach them. alg and onc read only the assets × assets distance matrix. They have no observation to weight, so they carry no @fprop tag.

Fields

  • ce: Covariance estimator.
  • de: Distance matrix estimator.
  • alg: Clustering algorithm.
  • onc: Estimator of the optimal number of clusters.
  • x_src: The source of the returns that the clustering, phylogeny or centrality estimator reads, nothing or an AbstractReturnsSource. nothing and DataSource read the asset returns rd.X, and nothing reads the X of the Prior Result when the call gives no returns data. ExcessSource reads the returns in excess of the benchmark, rd.X - rd.B, the Excess Frame. PriorSource reads the X of the Prior Result, in the frame that the prior fitted in. Under a factor prior that X is a reconstruction of the rank of the factors, and a scenario window cuts its rows. A stated source that the call cannot give, DataSource() or ExcessSource() with no returns data, or PriorSource() on returns data with no Prior Result, gives a warning, and the estimator reads the matrix that the call gives. A call on a bare matrix reads that matrix under every source. An estimator inside another graph estimator keeps nothing, and the outer estimator states the source.

Constructors

ClustersEstimator(;    ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),    de::AbstractDistanceEstimator = Distance(; alg = CanonicalDistance()),    alg::AbstractClustersAlgorithm = HClustAlgorithm(),    onc::AbstractOptimalNumberClustersEstimator = OptimalNumberClusters(),    x_src::Option{<:AbstractReturnsSource} = nothing) -> ClustersEstimator

Keywords correspond to the struct's fields.

Validation

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • ce: Recursively updated through factory.
  • de: Recursively updated through factory.

Examples

julia> ClustersEstimator()ClustersEstimator     ce ┼ PortfolioOptimisersCovariance        │   ce ┼ Covariance        │      │    me ┼ SimpleExpectedReturns        │      │       │   w ┴ nothing        │      │    ce ┼ GeneralCovariance        │      │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)        │      │       │    w ┴ nothing        │      │   alg ┼ FullMoment()        │      │     w ┴ nothing        │   mp ┼ MatrixProcessing        │      │     pdm ┼ Posdef        │      │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton        │      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()        │      │      dn ┼ nothing        │      │      dt ┼ nothing        │      │     alg ┼ nothing        │      │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)     de ┼ Distance        │   power ┼ nothing        │     alg ┴ CanonicalDistance()    alg ┼ HClustAlgorithm        │   linkage ┴ Symbol: :ward    onc ┼ OptimalNumberClusters        │   max_k ┼ nothing        │     alg ┼ SecondOrderDifference        │         │   alg ┼ StandardisedValue        │         │       │   mv ┼ MeanValue        │         │       │      │   w ┴ nothing        │         │       │   sv ┼ StdValue        │         │       │      │           w ┼ nothing        │         │       │      │   corrected ┴ Bool: true  x_src ┴ nothing

Related

source
PortfolioOptimisers.clusterise — Method
clusterise(cle::AbstractClusteringResult, args...; kwargs...)

Return the clustering result cle unchanged.

A consumer calls clusterise on a clustering estimator or on a precomputed result. For a result, this method runs no clustering, so the library never clusters a result again.

Algorithm

  1. Return cle itself. The method runs no clustering, and it discards args and kwargs.

Arguments

  • cle: Clustering result.
  • args...: Positional arguments. The method does not read them.
  • kwargs...: Keyword arguments. The method does not read them.

Returns

  • cle::AbstractClusteringResult: The input, unchanged.

Related

source
PortfolioOptimisers.factory — Method
factory(
    alg::AbstractClustersAlgorithm,
    args...;
    kwargs...
) -> AbstractClustersAlgorithm

Return the clustering algorithm alg unchanged.

A caller that rebuilds an estimator with factory reaches this method for a clustering algorithm. A clustering algorithm has no field that depends on a prior, so the method keeps every field and replaces none.

Algorithm

  1. Return alg itself. The method rebuilds no field of it, and it discards args and kwargs.

Arguments

  • alg: Clustering algorithm.
  • args...: Optional arguments. The method does not read them.
  • kwargs...: Optional keyword arguments. The method does not read them.

Returns

  • alg::AbstractClustersAlgorithm: The original clustering algorithm.

Related

source

References

[12]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[118]
R. Tibshirani, G. Walther and T. Hastie. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411–423 (2001).
[119]
S. Yue, X. Wang and M. Wei. Application of two-order difference to gap statistic. Transactions of Tianjin University 14, 217–221 (2008).
[120]
M. López de Prado and M. J. Lewis. Detection of false investment strategies using unsupervised learning methods. Quantitative Finance 19, 1555–1565 (2019).
[121]
P. J. Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65 (1987).
[122]
D. Müllner. Modern hierarchical, agglomerative clustering algorithms, arXiv preprint arXiv:1109.2378 (2011).