Nested Clustered
PortfolioOptimisers.NestedClusteredResult — Type
struct NestedClusteredResult{__T_pr, __T_clr, __T_wb, __T_fees, __T_resi, __T_reso, __T_cv, __T_scorer, __T_retcode, __T_w, __T_imsk, __T_fb} <: NonJuMPOptimisationResultHolds the clustering, the intra-cluster results and the inter-cluster result of a Nested Clustered Optimisation.
optimise returns it for a NestedClustered. clr holds the clustering that the algorithm found. resi holds one intra-cluster optimisation for each cluster, in cluster order. reso is the inter-cluster optimisation over the synthetic assets that the clusters define, so reso.w has one entry for each cluster. w is the product of the two weights, after the weight finaliser and the weight bounds. The product is the fourth step of Section 4.3 of [30].
Fields
pr: Prior result.
clr: Clusters result.
wb: Weight bounds.
fees: Fees estimator or result.
resi: Inner optimisation results.
reso: Outer optimisation results.
cv: Inner cross-validation scheme of the sub-portfolios, anOptCVE_Onl, ornothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio.nothinggives it the returns of the solve over the full sample.
scorer: Selects one path from the population of each sub-portfolio under aCombinatorialCrossValidation, aPredictionScoreror a function, ornothing.nothingselects withNearestQuantilePrediction. The constructor refuses a value that is notnothingwhencvis not aCombinatorialCrossValidation.
retcode: Optimisation return code.
w: Final portfolio weights over the full asset universe, zero at an asset outside the investable universe.
imsk: The Investable Mask the optimisation reduced on:trueat every asset whose prior moments were finite. It isnothingwhen every asset was investable, and that sentinel is what skips both the reduction and the expansion.investable_maskderives it once from the full-universe prior result, and the result carries it, because the reduced prior can no longer yield it.
fb: The fallback chain that answered this result: the(estimator, result)pair of every attemptoptimisemade before this one, in the order they ran, ornothingwhen the estimator it was asked of answered (seeFbChain).
Constructors
NestedClusteredResult(; pr::Option{<:AbstractPriorResult}, clr::Option{<:AbstractClusteringResult}, wb::Option{<:WeightBounds}, fees::Option{<:Fees}, resi::AbstractVector{<:NonFiniteAllocationOptimisationResult}, reso::OptimisationResult, cv::Option{<:OptCVE_Onl}, scorer::Option{<:PredictionCrossValScorer}, retcode::OptRetCode_VecOptRetCode, w::VecNum_VecVecNum, imsk::Option{<:BitVector} = nothing, fb::Option{<:OptE_Opt_FbChain}) -> NestedClusteredResultKeywords correspond to the struct's fields.
The keyword constructor expands w from the investable universe to the full asset universe with expand_investable_weights, and optimise builds the result through it. The positional constructor does not expand. set_retcode rebuilds the result through the positional constructor, so the weights expand one time only. pr, clr, wb, fees and each entry of resi belong to the investable universe, because the algorithm runs on that universe.
Related
NestedClusteredNonFiniteAllocationOptimisationResultStackingResultinvestable_reductionexpand_investable_weights
References
- [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.3, Snippet 4.
- [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 7.6.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Sections 12.3.1 and 12.3.2.
PortfolioOptimisers.NestedClustered — Type
struct NestedClustered{__T_pe, __T_cle, __T_wb, __T_fees, __T_sets, __T_opti, __T_opto, __T_cv, __T_scorer, __T_wf, __T_ex, __T_fb, __T_strict, __T_pcol, __T_cache} <: ClusteringOptimisationEstimatorClusters the assets, optimises each cluster, and then optimises across the cluster portfolios.
Nested Clustered Optimisation (NCO), stated in Section 4.3 of [30], solves one intra-cluster problem for each cluster with the inner optimiser. It then solves one inter-cluster problem with the outer optimiser, over the synthetic assets that the cluster portfolios define. Snippet 4 of that work states the inter-cluster problem through the reduced moments of the clusters. It names k-means clustering and notes that a hierarchical method also serves, and this type takes any clustering estimator. [26] Section 7.6 restates the algorithm. Sections 12.3.1 and 12.3.2 of [12] let each of the two problems take its own objective and its own constraints, and so does this type.
Mathematical definition
\[\begin{align} W_{ik} &= \begin{cases} w^{(k)}_{i} & i \in C_k\,,\\ 0 & i \notin C_k\,,\end{cases}\\ \mathbf{X}^{o} &= \mathbf{X} \mathbf{W}\,,\\ \boldsymbol{w} &= \mathbf{W} \boldsymbol{v}\,. \end{align}\]
Where:
- $C_1, \ldots, C_K$: Clusters of the assets, a partition of the $N$ assets.
- $w^{(k)}_{i}$: Weight of asset $i$ in the solution of the inner optimiser over the assets of $C_k$.
- $\mathbf{W}$: Inner weight matrix
assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio. - $\mathbf{X}$: Returns matrix $T \times N$.
- $\mathbf{X}^{o}$: Returns matrix of the synthetic assets, $T \times K$. Column $k$ holds the returns of the portfolio of cluster $C_k$.
- $\boldsymbol{v}$: Outer weight vector, the weights that the outer optimiser gives the $K$ synthetic assets.
- $\boldsymbol{w}$: Combined weights over the $N$ assets, before the weight finaliser and the weight bounds.
- $K$: Sub-portfolio count of a meta-optimiser, which is also the number of synthetic assets of its outer problem.
- $N$: Number of assets.
- $T$: Number of observations.
The outer optimiser solves for $\boldsymbol{v}$ over $\mathbf{X}^{o}$. The weight of asset $i$ in $C_k$ is $w_i = v_k w^{(k)}_{i}$. The sample mean of $\mathbf{X}^{o}$ is $\mathbf{W}^\intercal \hat{\boldsymbol{\mu}}$ and its sample covariance is $\mathbf{W}^\intercal \hat{\mathbf{\Sigma}} \mathbf{W}$, where $\hat{\boldsymbol{\mu}}$ and $\hat{\mathbf{\Sigma}}$ are the sample moments of $\mathbf{X}$. These are the reduced moments of Snippet 4 of [30], so an outer optimiser that estimates sample moments solves its inter-cluster problem.
The outer optimiser fits also when the clustering finds one cluster. Its problem then has one synthetic asset, and its own constraints set the weight of that asset, so the weight is 1 only under a budget of 1. An outer estimator that refuses one asset throws in this case. A prior with a ShrunkExpectedReturns mean is an example, because its James-Stein intensity is not defined at N = 1. A clustering can find one cluster in some folds of a cross-validation and several clusters in other folds. When the data can form one cluster, give the outer optimiser estimators that accept one asset.
Fields
pe: Prior estimator.
cle: Clusters estimator.
wb: Weight bounds estimator or weight bounds.
fees: Fees estimator.
sets: Sets used to map estimator values to assets.
opti: Inner optimiser.
opto: Outer optimiser.
cv: Inner cross-validation scheme of the sub-portfolios, anOptCVE_Onl, ornothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio.nothinggives it the returns of the solve over the full sample.
scorer: Selects one path from the population of each sub-portfolio under aCombinatorialCrossValidation, aPredictionScoreror a function, ornothing.nothingselects withNearestQuantilePrediction. The constructor refuses a value that is notnothingwhencvis not aCombinatorialCrossValidation.
wf: Weight finaliser.
ex: Parallel execution strategy.
fb: Fallback result or estimator.
strict: Whether a constraint that names an asset or a group thatsetsdoes not hold raises an error. When it isfalse, the constraint generator warns and skips the entry. Undertruethe fit also refuses an asset with a gap in the rows that a plain graph reader reads, which it otherwise drops with a warning, seegraph_reader_mask. It also refuses an asset that a distance reader of the fit cannot estimate, which it otherwise drops with a warning, seedistance_reader_refusal_mask.
pcol: Rule of the panel collapse, anAbstractPanelCollapseAlgorithm. It states how the collapse of the Asset Panel onto the sub-portfolios weighs a member at an observation where the member is outside the universe.RenormaliseActivedivides by the weight of the active members, andInactiveAsCashreads the missing weight as cash with a zero feature.
cache: OptionalReturnsBufferState, the fold context of the online step. It isnothinguntilpartial_fit!writes one, andoptimise(opt)with no returns reads it. The returns themselves are carried by the prior, which owns the rows once; this holds every other column of the returns data and the context pinned at the first step.factorycarries it unchanged andport_opt_viewslices it to the selected assets.
Constructors
NestedClustered(; pe::Onl{<:TD{<:PrE_Pr}} = EmpiricalPrior(), cle::TD{<:ClE_Cl} = ClustersEstimator(), wb::TD_Option{<:WbE_Wb} = nothing, fees::TD_Option{<:FeesE_Fees} = nothing, sets::TD_Option{<:UniverseSets} = nothing, opti::OptE_TD, opto::OptE_TD, cv::Option{<:OptCVE_Onl} = nothing, scorer::Option{<:PredictionCrossValScorer} = nothing, wf::TD{<:WeightFinaliser} = IterativeWeightFinaliser(), ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(), fb::TDO_Option{<:OptE_Opt} = nothing, strict::Bool = false, pcol::AbstractPanelCollapseAlgorithm = RenormaliseActive(), cache::Option{<:ReturnsBufferState} = nothing) -> NestedClusteredKeywords correspond to the struct's fields.
Time-dependent fields
pe, cle, wb, fees, sets, wf, opti, opto and fb can hold a TimeDependent schedule, one value for each fold. A schedule in opto or fb must bind :outermost, because no inner fold loop reads them. A schedule in opti can also bind :nearest. The inner cross-validation runs cross_val_predict(opti, …; cols = cl) for each cluster, so the inner fold loop reads opti. A :nearest schedule in opti resolves in that loop, one time for each cluster. The solve of each cluster outside the folds resolves the schedule to its default, so a :nearest schedule in opti needs an explicit default and a cv (see assert_nearest_optimiser_schedule). cv and scorer accept no schedule, because cv is the inner fold loop itself, and the construction checks read both.
An entry of a schedule in opti or opto must be an estimator, as the static value must be. The constructor refuses a vector schedule that holds a precomputed result.
Validation
optopassesassert_external_optimiserandassert_special_nco_requirements. A schedule passes when its entries and itsdefaultpass.- When
optiis notopto,optipassesassert_internal_optimiserandassert_special_nco_requirements. - When
cvis present,optipassesassert_external_optimiser. scorerisnothingunlesscvis aCombinatorialCrossValidation. The constructor throws anArgumentErrorotherwise, throughassert_meta_scorer.- A schedule in
optoorfbdoes not bind:nearest. A schedule inoptithat binds:nearesthas an explicitdefault, andcvis present. - A
wbthat is keyed by name (name_keyed), or aFeesEstimatorinfees, needssets. The constructor throws anIsNothingErrorotherwise. - A vector schedule holds no precomputed result, see
assert_time_dependent_substitution. - A reader that states no source and reads the raw returns data, beside a reader that states
x_src = ExcessSource(), logs one warning that names its path, seewarn_raw_readers. The head then builds as configured.
Propagated parameters
factory passes its arguments on to these @fprop fields:
fees: Recursively updated throughfactory.opti: Recursively updated throughfactory.opto: Recursively updated throughfactory.fb: Recursively updated throughfactory.cache: Recursively updated throughfactory.
View parameters
NestedClustered defines its own port_opt_view method rather than deriving one from field tags.
- The method reads the returns matrix
X. Whenpeholds a prior result, the method usespe.Xin place ofX, so the fields that read a matrix read the observations of the prior. pe,wb,setsandcacherecurse throughport_opt_viewwith the index alone.fees,optiandoptorecurse throughport_opt_viewwith the index and the matrix, andfbthroughview_child.clepasses through unchanged, because the clustering runs again over the viewed assets.
Observation weight parameters
obs_weights_view indexes these fields to the selected observations:
pe: Recursively indexed throughobs_weights_view.cle: Recursively indexed throughobs_weights_view.fees: Recursively indexed throughobs_weights_view.opti: Recursively indexed throughobs_weights_view.opto: Recursively indexed throughobs_weights_view.fb: Recursively indexed throughobs_weights_view.cache: Removed byobs_weights_view, because a state has no slice on the observation axis.
Related
optimiseNestedClusteredResultClusteringOptimisationEstimatorHierarchicalRiskParityStackingpredict_outer_returnsfactoryport_opt_viewobs_weights_view
References
- [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.3, Snippet 4.
- [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 7.6.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Sections 12.3.1 and 12.3.2.
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
AbstractResult, StatsBase.CovarianceEstimator}},
args...; kwargs...) -> AbstractArrayRebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.
factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.
factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.
The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.
Algorithm
The leaf method:
- Return
awith no change, and dropargs...andkwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.
The array method:
- For each element
aiofa, callfactoryonai, and forwardargs...andkwargs...with no change. - Collect the results into a new array of the shape of
a, in the order ofa, and return it.
A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.
The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.
Arguments
a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators ornothing, which it rebuilds one element at a time.args...: Positional arguments. The leaf method ignores them, and the array method forwards them.kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.
Returns
a: The input with no change, for the leaf method.v::AbstractArray: The rebuilt elements, in an array of the shape ofa, for the array method.
Examples
julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue w ┴ nothingRelated
factory(res::NonFiniteAllocationOptimisationResult, fb::Option{<:OptE_Opt_FbChain})Rebuilds a continuous optimisation result with a new fallback record fb.
A concrete result can add its own method when a rebuild needs more than a new fb. optimise is the one caller, and it gives the FbChain of the attempts that failed.
Algorithm
- Read every field of
res, in the order of the struct. - Call the constructor of the type of
reswith every field except the last, and withfbas the last field.
Validation
fbis the last field of the type ofres. Every concrete optimisation result obeys this rule, so the method checks nothing.
Related
factory(
opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
_
) -> EqualWeighted{_A, _B, _C, _D, _E, Bool} where {_A, _B, _C, _D, _E}
Returns opt unchanged.
This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.
Related
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
AbstractResult, StatsBase.CovarianceEstimator}},
args...; kwargs...) -> AbstractArrayRebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.
factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.
factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.
The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.
Algorithm
The leaf method:
- Return
awith no change, and dropargs...andkwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.
The array method:
- For each element
aiofa, callfactoryonai, and forwardargs...andkwargs...with no change. - Collect the results into a new array of the shape of
a, in the order ofa, and return it.
A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.
The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.
Arguments
a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators ornothing, which it rebuilds one element at a time.args...: Positional arguments. The leaf method ignores them, and the array method forwards them.kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.
Returns
a: The input with no change, for the leaf method.v::AbstractArray: The rebuilt elements, in an array of the shape ofa, for the array method.
Examples
julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue w ┴ nothingRelated
factory(
opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
_
) -> EqualWeighted{_A, _B, _C, _D, _E, Bool} where {_A, _B, _C, _D, _E}
Returns opt unchanged.
This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.
Related
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
nco::NestedClustered,
i,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
args...
) -> NestedClustered{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185", _K, Bool, <:AbstractPanelCollapseAlgorithm} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185"<:Transducers.Executor, _K}
Return a copy of a NestedClustered viewed onto the assets i, with the returns matrix X.
The ## View parameters subsection of NestedClustered states how the method views each field.
PortfolioOptimisers._optimise — Method
_optimise(
nco::NestedClustered,
rd::ReturnsResult;
branchorder,
str_names,
save,
rmsk,
kwargs...
) -> Any
Run the nested clustered optimisation.
optimise calls this method. The # Mathematical definition of NestedClustered states the weights that the method computes.
Algorithm
- Replace the schedules in the fields of
ncowith their static defaults, throughreset_time_dependent_estimator. - Fit the prior of
nco.petord, givingpr. The prior reads the returns in the frame that its ownx_srcstates.rdkeeps its raw returns, and the clustering reads the frame that its ownx_srcstates. The readers of the inner and outer optimisers state their own sources. - Take the element type of the weights, the fees and the weight bounds as
eltype(pr.X), givingTf. - Read the investable mask of
prwithinvestable_mask, givingimsk. - Resolve the fees of
nco.feesover the universe of the caller withfees_constraints, and place them on the investable assets withinvestable_fees_view, givingfees. - Remove the liquidation charges from
feeswithstrip_liquidation_charges, givingcfees. The result charges a forced exit one time, and no cluster holds an asset that exits. - Reduce
pr,ncoandrdto the investable assets withinvestable_reduction, givingpr,ncoandrdr. - Cluster the assets with
clusterise, givingclr, and check withassert_clustering_universethatclrcovers every asset. Collect the assets of each cluster, givingcls. - For each cluster
k, viewnco.optiandrdronto the assetscls[k]withport_opt_view, and optimise, givingresi[k]. Write its weights into columnkofwi, which is zero outside the cluster. - Predict the returns of the synthetic assets with
predict_outer_returns,cfeesand aClusterUniverse, givingrdo. Withoutnco.cvthese are the returns of the cluster portfolios over the observations ofpr. Withnco.cvthey are the out-of-sample predictions of the folds. - Rebuild the asset sets of
nco.optoover the names of the synthetic assets with_update_asset_sets. - Optimise
nco.optooverrdo, givingreso. - Resolve the weight bounds of
nco.wbover the investable assets withweight_bounds_constraints, givingwb. - Combine
wiandreso.wand finalise them withnco.wfunderwb, throughouter_optimisation_finaliser, givingretcodeandw. - Build the
NestedClusteredResult. Its keyword constructor expandswto the full asset universe.
Arguments
nco: The nested clustered optimiser.rd: The returns result to use.branchorder: The branch order of a hierarchical clustering. The method also passes it to the inner and outer optimisers.str_names: Whether the JuMP models name their variables with strings. The method passes it to the inner and outer optimisers.save: Whether the results save their JuMP models. The method passes it to the inner and outer optimisers.rmsk: The mask of a fit again after a refusal, ornothingon the first fit.kwargs: More keyword arguments for the inner and outer optimisers.
Validation
- The clustering covers every investable asset. The method throws a
DimensionMismatchotherwise. - No intra-cluster optimisation returns an efficient frontier. The method throws an
ArgumentErrorotherwise.
Returns
res::NestedClusteredResult: The combined portfolio.retcodeis anOptimisationFailurewhen an intra-cluster optimisation, the inter-cluster optimisation or the weight finalisation fails.
Related
PortfolioOptimisers.needs_previous_weights — Method
needs_previous_weights(opt::NestedClustered) -> Any
Return true when a part of opt needs the previous portfolio weights.
The parts are the schedules in the fields of opt, fees, opti, opto and fb.
References
- [12]
- D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
- [26]
- M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
- [30]
- M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019).