The source files can be found in examples/.
Clustering optimisers
The optimisers we have met so far, MeanRisk, RiskBudgeting and NearOptimalCentering, solve one problem over every asset at once. A clustering optimiser works in two steps. It first groups the assets into a tree, the dendrogram, built from how they move together. It then splits the budget inside each group and between the groups. It never inverts the full covariance matrix and it reads no expected returns, so an error in those estimates moves the weights less, and variance-based risk needs no solver.
PortfolioOptimisers has three members of this family.
HierarchicalRiskParity, or HRP, orders the assets by the dendrogram, splits that ordered list in half again and again, and splits the budget between the two halves in inverse proportion to their risk.HierarchicalEqualRiskContribution, or HERC, cuts the dendrogram into clusters, splits the budget between the clusters down the tree's branches in inverse proportion to their risk, and splits each cluster's share between its assets in inverse proportion to their own risk. Despite the name, the clusters do not contribute equal risk. Each of the two levels takes its own risk measure and its own scalariser.SchurComplementHierarchicalRiskParity, or SCHRP, corrects the covariance of each sub-cluster with a Schur complement. Atgamma = 0it gives HRP, and asgammarises toward 1 it moves toward the minimum-variance portfolio.
Reach for a clustering optimiser when you want the correlation structure of the assets to drive the allocation rather than a return forecast. It spreads the money over groups that move differently, it holds up when the covariance matrix is noisy or near-singular, and it needs no solver. Use HRP for the plain robust case, HERC when you want to set the split inside a cluster apart from the split between clusters, and SCHRP when you want some of the efficiency of mean-variance without losing the stability of the hierarchy. If you want a stated trade-off between return and risk instead, use MeanRisk.
using PortfolioOptimisers, PrettyTablestsfmt = (v, i, j) -> begin if j == 1 return Date(v) else return v endend;resfmt = (v, i, j) -> begin if j == 1 return v else return isa(v, Number) ? "$(round(v*100, digits=3)) %" : v endend;1. Data
We use one year of daily prices for twenty S&P 500 stocks.
using CSV, TimeSeries, DataFramesX = TimeArray(CSV.File(joinpath(@__DIR__, "..", "SP500.csv.gz")); timestamp = :Date)[(end - 252):end]pretty_table(X[(end - 5):end]; formatters = [tsfmt])rd = prices_to_returns(X)ReturnsResult
nx ┼ 20-element Vector{String}
X ┼ 252×20 Matrix{Float64}
nf ┼ nothing
F ┼ nothing
nb ┼ nothing
B ┼ nothing
ne ┼ nothing
E ┼ nothing
ts ┼ 252-element Vector{Date}
iv ┼ nothing
ivpa ┼ nothing
pnl ┼ AssetPanel
│ pf ┼ Vector{AbstractPanelField}: AbstractPanelField[]
│ amsk ┼ 252×20 AllTrueMask
│ emsk ┴ 252×20 AllTrueMask
2. Prior and clustering
A clustering optimiser needs two things you can compute once, a prior for the covariance and a clustering of the assets. We compute both here and hand them to every optimiser below, so the only difference between the results is the allocation rule.
We cluster with the Direct Bubble Hierarchical Tree algorithm, DBHT, which builds the tree from the distance matrix the correlations give.
pr = prior(EmpiricalPrior(), rd)clr = clusterise(ClustersEstimator(; alg = DBHT()), pr.X)Clusters
res ┼ Clustering.Hclust{Float64}([-1 -13; -7 -4; … ; 12 17; 10 18], [0.1, 0.1111111111111111, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 2.0], [17, 20, 5, 3, 9, 6, 2, 1, 13, 7, 4, 19, 14, 10, 16, 18, 11, 8, 12, 15], :DBHT)
S ┼ 20×20 Matrix{Float64}
D ┼ 20×20 Matrix{Float64}
P ┼ nothing
k ┴ Int64: 4
The two plots below show the structure every optimiser on this page acts on. The first is the dendrogram.
using StatsPlots, GraphRecipesplot_dendrogram(clr, rd.nx)
The second is the correlation heatmap, reordered by the tree, with a box around each cluster.
plot_clusters(clr, rd.nx)
3. Hierarchical risk parity (HRP)
HRP orders the assets by the dendrogram, splits that ordered list in half again and again, and splits the budget between the two halves of each split in inverse proportion to their risk. We pass the shared prior and clustering through a HierarchicalOptimiser. Variance needs no solver, so we pass none.
opt = HierarchicalOptimiser(; pe = pr, cle = clr)res_hrp = optimise(HierarchicalRiskParity(; opt = opt, r = Variance()))HierarchicalRiskParityResult
hr ┼ HierarchicalResult
│ pr ┼ LowOrderPrior
│ │ X ┼ 252×20 Matrix{Float64}
│ │ o_X ┼ nothing
│ │ mu ┼ 20-element Vector{Float64}
│ │ sigma ┼ 20×20 Matrix{Float64}
│ │ chol ┼ nothing
│ │ w ┼ nothing
│ │ ens ┼ nothing
│ │ kld ┼ nothing
│ │ ow ┼ nothing
│ │ rr ┼ nothing
│ │ fpr ┼ nothing
│ │ x_src ┼ nothing
│ │ ss_gaps ┼ nothing
│ │ pc ┴ nothing
│ clr ┼ Clusters
│ │ res ┼ Clustering.Hclust{Float64}([-1 -13; -7 -4; … ; 12 17; 10 18], [0.1, 0.1111111111111111, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 2.0], [17, 20, 5, 3, 9, 6, 2, 1, 13, 7, 4, 19, 14, 10, 16, 18, 11, 8, 12, 15], :DBHT)
│ │ S ┼ 20×20 Matrix{Float64}
│ │ D ┼ 20×20 Matrix{Float64}
│ │ P ┼ nothing
│ │ k ┴ Int64: 4
│ wb ┼ WeightBounds
│ │ lb ┼ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
│ │ ub ┴ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
│ fees ┼ nothing
│ retcode ┼ OptimisationSuccess
│ │ res ┴ nothing
│ w ┼ 20-element Vector{Float64}
│ imsk ┴ nothing
r ┼ Variance
│ settings ┼ RiskMeasureSettings
│ │ scale ┼ Float64: 1.0
│ │ ub ┼ nothing
│ │ rke ┴ Bool: true
│ sigma ┼ 20×20 Matrix{Float64}
│ chol ┼ nothing
│ rc ┼ nothing
│ alg ┼ SquaredSOCRiskExpr()
│ mtx_sqrt ┼ EigenFallbackSquareRoot()
│ ratio_alg ┼ SquareRootRatio()
│ corner ┴ nothing
sca ┼ SumScalariser()
fb ┴ nothing
4. Hierarchical equal risk contribution (HERC)
HERC splits the budget between the clusters down the tree's branches in inverse proportion to their risk, which is the outer level, and splits each cluster's share between its assets in inverse proportion to their own risk, which is the inner level. It takes one risk measure per level, ri and ro, and one scalariser per level, scai and scao. We use Variance at both levels.
res_herc = optimise(HierarchicalEqualRiskContribution(; opt = opt, ri = Variance(), ro = Variance()))HierarchicalEqualRiskContributionResult
hr ┼ HierarchicalResult
│ pr ┼ LowOrderPrior
│ │ X ┼ 252×20 Matrix{Float64}
│ │ o_X ┼ nothing
│ │ mu ┼ 20-element Vector{Float64}
│ │ sigma ┼ 20×20 Matrix{Float64}
│ │ chol ┼ nothing
│ │ w ┼ nothing
│ │ ens ┼ nothing
│ │ kld ┼ nothing
│ │ ow ┼ nothing
│ │ rr ┼ nothing
│ │ fpr ┼ nothing
│ │ x_src ┼ nothing
│ │ ss_gaps ┼ nothing
│ │ pc ┴ nothing
│ clr ┼ Clusters
│ │ res ┼ Clustering.Hclust{Float64}([-1 -13; -7 -4; … ; 12 17; 10 18], [0.1, 0.1111111111111111, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 2.0], [17, 20, 5, 3, 9, 6, 2, 1, 13, 7, 4, 19, 14, 10, 16, 18, 11, 8, 12, 15], :DBHT)
│ │ S ┼ 20×20 Matrix{Float64}
│ │ D ┼ 20×20 Matrix{Float64}
│ │ P ┼ nothing
│ │ k ┴ Int64: 4
│ wb ┼ WeightBounds
│ │ lb ┼ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
│ │ ub ┴ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
│ fees ┼ nothing
│ retcode ┼ OptimisationSuccess
│ │ res ┴ nothing
│ w ┼ 20-element Vector{Float64}
│ imsk ┴ nothing
ri ┼ Variance
│ settings ┼ RiskMeasureSettings
│ │ scale ┼ Float64: 1.0
│ │ ub ┼ nothing
│ │ rke ┴ Bool: true
│ sigma ┼ 20×20 Matrix{Float64}
│ chol ┼ nothing
│ rc ┼ nothing
│ alg ┼ SquaredSOCRiskExpr()
│ mtx_sqrt ┼ EigenFallbackSquareRoot()
│ ratio_alg ┼ SquareRootRatio()
│ corner ┴ nothing
ro ┼ Variance
│ settings ┼ RiskMeasureSettings
│ │ scale ┼ Float64: 1.0
│ │ ub ┼ nothing
│ │ rke ┴ Bool: true
│ sigma ┼ 20×20 Matrix{Float64}
│ chol ┼ nothing
│ rc ┼ nothing
│ alg ┼ SquaredSOCRiskExpr()
│ mtx_sqrt ┼ EigenFallbackSquareRoot()
│ ratio_alg ┼ SquareRootRatio()
│ corner ┴ nothing
scai ┼ SumScalariser()
scao ┼ SumScalariser()
fb ┴ nothing
5. Schur-complement HRP (SCHRP)
SCHRP corrects the covariance of each half of a split with a Schur complement. The correction subtracts a term that the covariances between the two halves build, and gamma scales it. At gamma = 0 the result is HRP. As gamma rises, more of the covariance between the halves enters, and the allocation moves toward the minimum-variance portfolio. Under the default MonotonicSchurComplement, gamma is the upper end of a search. The optimiser uses the largest value from 0 to gamma up to which the variance of the portfolio keeps falling. We run three values so you can compare them.
res_schur0 = optimise(SchurComplementHierarchicalRiskParity(; opt = opt, params = SchurComplementParams(; r = Variance(), gamma = 0.0)))res_schur5 = optimise(SchurComplementHierarchicalRiskParity(; opt = opt, params = SchurComplementParams(; r = Variance(), gamma = 0.5)))res_schur9 = optimise(SchurComplementHierarchicalRiskParity(; opt = opt, params = SchurComplementParams(; r = Variance(), gamma = 0.9)))SchurComplementHierarchicalRiskParityResult
pr ┼ LowOrderPrior
│ X ┼ 252×20 Matrix{Float64}
│ o_X ┼ nothing
│ mu ┼ 20-element Vector{Float64}
│ sigma ┼ 20×20 Matrix{Float64}
│ chol ┼ nothing
│ w ┼ nothing
│ ens ┼ nothing
│ kld ┼ nothing
│ ow ┼ nothing
│ rr ┼ nothing
│ fpr ┼ nothing
│ x_src ┼ nothing
│ ss_gaps ┼ nothing
│ pc ┴ nothing
wb ┼ WeightBounds
│ lb ┼ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
│ ub ┴ 20-element StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}
clr ┼ Clusters
│ res ┼ Clustering.Hclust{Float64}([-1 -13; -7 -4; … ; 12 17; 10 18], [0.1, 0.1111111111111111, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 0.125, 0.14285714285714285, 0.16666666666666666, 0.2, 0.25, 0.3333333333333333, 0.5, 1.0, 2.0], [17, 20, 5, 3, 9, 6, 2, 1, 13, 7, 4, 19, 14, 10, 16, 18, 11, 8, 12, 15], :DBHT)
│ S ┼ 20×20 Matrix{Float64}
│ D ┼ 20×20 Matrix{Float64}
│ P ┼ nothing
│ k ┴ Int64: 4
fees ┼ nothing
r ┼ Variance
│ settings ┼ RiskMeasureSettings
│ │ scale ┼ Float64: 1.0
│ │ ub ┼ nothing
│ │ rke ┴ Bool: true
│ sigma ┼ 20×20 Matrix{Float64}
│ chol ┼ nothing
│ rc ┼ nothing
│ alg ┼ SquaredSOCRiskExpr()
│ mtx_sqrt ┼ EigenFallbackSquareRoot()
│ ratio_alg ┼ SquareRootRatio()
│ corner ┴ nothing
gamma ┼ Float64: 0.042285156250000004
retcode ┼ OptimisationSuccess
│ res ┴ nothing
w ┼ 20-element Vector{Float64}
imsk ┼ nothing
fb ┴ nothing
6. Comparing the allocations
Read the SCHRP column at gamma = 0 against the HRP column, then read the two columns at the higher values of gamma. Under the search, two values of gamma can give almost the same weights, as 0.5 and 0.9 do here.
pretty_table(DataFrame(; :assets => rd.nx, :HRP => res_hrp.w, :HERC => res_herc.w, Symbol("SCHRP γ=0") => res_schur0.w, Symbol("SCHRP γ=0.5") => res_schur5.w, Symbol("SCHRP γ=0.9") => res_schur9.w); formatters = [resfmt])┌────────┬──────────┬──────────┬───────────┬─────────────┬─────────────┐
│ assets │ HRP │ HERC │ SCHRP γ=0 │ SCHRP γ=0.5 │ SCHRP γ=0.9 │
│ String │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │
├────────┼──────────┼──────────┼───────────┼─────────────┼─────────────┤
│ AAPL │ 3.26 % │ 1.847 % │ 3.26 % │ 3.269 % │ 3.269 % │
│ AMD │ 1.263 % │ 0.627 % │ 1.263 % │ 1.27 % │ 1.27 % │
│ BAC │ 2.742 % │ 2.221 % │ 2.742 % │ 2.674 % │ 2.674 % │
│ BBY │ 2.871 % │ 1.138 % │ 2.871 % │ 2.905 % │ 2.905 % │
│ CVX │ 5.049 % │ 4.525 % │ 5.049 % │ 5.008 % │ 5.008 % │
│ GE │ 3.879 % │ 1.924 % │ 3.879 % │ 3.839 % │ 3.839 % │
│ HD │ 2.605 % │ 2.386 % │ 2.605 % │ 2.605 % │ 2.605 % │
│ JNJ │ 13.136 % │ 10.746 % │ 13.136 % │ 13.111 % │ 13.111 % │
│ JPM │ 3.24 % │ 2.623 % │ 3.24 % │ 3.162 % │ 3.162 % │
│ KO │ 6.299 % │ 13.86 % │ 6.299 % │ 6.311 % │ 6.311 % │
│ LLY │ 5.995 % │ 4.388 % │ 5.995 % │ 6.091 % │ 6.09 % │
│ MRK │ 6.583 % │ 8.205 % │ 6.583 % │ 6.538 % │ 6.538 % │
│ MSFT │ 2.06 % │ 1.886 % │ 2.06 % │ 2.075 % │ 2.075 % │
│ PEP │ 11.24 % │ 14.175 % │ 11.24 % │ 11.301 % │ 11.301 % │
│ PFE │ 3.589 % │ 4.473 % │ 3.589 % │ 3.54 % │ 3.54 % │
│ PG │ 5.051 % │ 6.756 % │ 5.051 % │ 5.068 % │ 5.068 % │
│ RRC │ 1.277 % │ 1.238 % │ 1.277 % │ 1.235 % │ 1.235 % │
│ UNH │ 7.499 % │ 5.489 % │ 7.499 % │ 7.625 % │ 7.625 % │
│ WMT │ 8.257 % │ 7.513 % │ 8.257 % │ 8.318 % │ 8.318 % │
│ XOM │ 4.105 % │ 3.98 % │ 4.105 % │ 4.056 % │ 4.056 % │
└────────┴──────────┴──────────┴───────────┴─────────────┴─────────────┘The next plot stacks the same five allocations.
plot_stacked_bar_composition([res_hrp, res_herc, res_schur0, res_schur5, res_schur9], rd)
The last plot draws the share of the variance of the HRP portfolio that each asset carries. Read it against the HRP weights in the table above. The plot needs a variance that holds a covariance, and factory gives it the covariance of the prior.
rv = factory(Variance(), pr)plot_risk_contribution(rv, res_hrp, rd)
This page was generated using Literate.jl.