Nested Clustered

PortfolioOptimisers.NestedClusteredResult — Type
struct NestedClusteredResult{__T_pr, __T_clr, __T_wb, __T_fees, __T_resi, __T_reso, __T_cv, __T_scorer, __T_retcode, __T_w, __T_imsk, __T_fb} <: NonJuMPOptimisationResult

Holds the clustering, the intra-cluster results and the inter-cluster result of a Nested Clustered Optimisation.

optimise returns it for a NestedClustered. clr holds the clustering that the algorithm found. resi holds one intra-cluster optimisation for each cluster, in cluster order. reso is the inter-cluster optimisation over the synthetic assets that the clusters define, so reso.w has one entry for each cluster. w is the product of the two weights, after the weight finaliser and the weight bounds. The product is the fourth step of Section 4.3 of [30].

Fields

  • pr: Prior result.
  • clr: Clusters result.
  • wb: Weight bounds.
  • fees: Fees estimator or result.
  • resi: Inner optimisation results.
  • reso: Outer optimisation results.
  • cv: Inner cross-validation scheme of the sub-portfolios, an OptCVE_Onl, or nothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio. nothing gives it the returns of the solve over the full sample.
  • retcode: Optimisation return code.
  • w: Final portfolio weights over the full asset universe, zero at an asset outside the investable universe.
  • imsk: The Investable Mask the optimisation reduced on: true at every asset whose prior moments were finite. It is nothing when every asset was investable, and that sentinel is what skips both the reduction and the expansion. investable_mask derives it once from the full-universe prior result, and the result carries it, because the reduced prior can no longer yield it.
  • fb: The fallback chain that answered this result: the (estimator, result) pair of every attempt optimise made before this one, in the order they ran, or nothing when the estimator it was asked of answered (see FbChain).

Constructors

NestedClusteredResult(;    pr::Option{<:AbstractPriorResult},    clr::Option{<:AbstractClusteringResult},    wb::Option{<:WeightBounds},    fees::Option{<:Fees},    resi::AbstractVector{<:NonFiniteAllocationOptimisationResult},    reso::OptimisationResult,    cv::Option{<:OptCVE_Onl},    scorer::Option{<:PredictionCrossValScorer},    retcode::OptRetCode_VecOptRetCode,    w::VecNum_VecVecNum,    imsk::Option{<:BitVector} = nothing,    fb::Option{<:OptE_Opt_FbChain}) -> NestedClusteredResult

Keywords correspond to the struct's fields.

The keyword constructor expands w from the investable universe to the full asset universe with expand_investable_weights, and optimise builds the result through it. The positional constructor does not expand. set_retcode rebuilds the result through the positional constructor, so the weights expand one time only. pr, clr, wb, fees and each entry of resi belong to the investable universe, because the algorithm runs on that universe.

Related

References

  • [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.3, Snippet 4.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 7.6.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Sections 12.3.1 and 12.3.2.
source
PortfolioOptimisers.NestedClustered — Type
struct NestedClustered{__T_pe, __T_cle, __T_wb, __T_fees, __T_sets, __T_opti, __T_opto, __T_cv, __T_scorer, __T_wf, __T_ex, __T_fb, __T_strict, __T_pcol, __T_cache} <: ClusteringOptimisationEstimator

Clusters the assets, optimises each cluster, and then optimises across the cluster portfolios.

Nested Clustered Optimisation (NCO), stated in Section 4.3 of [30], solves one intra-cluster problem for each cluster with the inner optimiser. It then solves one inter-cluster problem with the outer optimiser, over the synthetic assets that the cluster portfolios define. Snippet 4 of that work states the inter-cluster problem through the reduced moments of the clusters. It names k-means clustering and notes that a hierarchical method also serves, and this type takes any clustering estimator. [26] Section 7.6 restates the algorithm. Sections 12.3.1 and 12.3.2 of [12] let each of the two problems take its own objective and its own constraints, and so does this type.

Mathematical definition

\[\begin{align} W_{ik} &= \begin{cases} w^{(k)}_{i} & i \in C_k\,,\\ 0 & i \notin C_k\,,\end{cases}\\ \mathbf{X}^{o} &= \mathbf{X} \mathbf{W}\,,\\ \boldsymbol{w} &= \mathbf{W} \boldsymbol{v}\,. \end{align}\]

Where:

  • $C_1, \ldots, C_K$: Clusters of the assets, a partition of the $N$ assets.
  • $w^{(k)}_{i}$: Weight of asset $i$ in the solution of the inner optimiser over the assets of $C_k$.
  • $\mathbf{W}$: Inner weight matrix assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio.
  • $\mathbf{X}$: Returns matrix $T \times N$.
  • $\mathbf{X}^{o}$: Returns matrix of the synthetic assets, $T \times K$. Column $k$ holds the returns of the portfolio of cluster $C_k$.
  • $\boldsymbol{v}$: Outer weight vector, the weights that the outer optimiser gives the $K$ synthetic assets.
  • $\boldsymbol{w}$: Combined weights over the $N$ assets, before the weight finaliser and the weight bounds.
  • $K$: Sub-portfolio count of a meta-optimiser, which is also the number of synthetic assets of its outer problem.
  • $N$: Number of assets.
  • $T$: Number of observations.

The outer optimiser solves for $\boldsymbol{v}$ over $\mathbf{X}^{o}$. The weight of asset $i$ in $C_k$ is $w_i = v_k w^{(k)}_{i}$. The sample mean of $\mathbf{X}^{o}$ is $\mathbf{W}^\intercal \hat{\boldsymbol{\mu}}$ and its sample covariance is $\mathbf{W}^\intercal \hat{\mathbf{\Sigma}} \mathbf{W}$, where $\hat{\boldsymbol{\mu}}$ and $\hat{\mathbf{\Sigma}}$ are the sample moments of $\mathbf{X}$. These are the reduced moments of Snippet 4 of [30], so an outer optimiser that estimates sample moments solves its inter-cluster problem.

One cluster

The outer optimiser fits also when the clustering finds one cluster. Its problem then has one synthetic asset, and its own constraints set the weight of that asset, so the weight is 1 only under a budget of 1. An outer estimator that refuses one asset throws in this case. A prior with a ShrunkExpectedReturns mean is an example, because its James-Stein intensity is not defined at N = 1. A clustering can find one cluster in some folds of a cross-validation and several clusters in other folds. When the data can form one cluster, give the outer optimiser estimators that accept one asset.

Fields

  • pe: Prior estimator.
  • cle: Clusters estimator.
  • wb: Weight bounds estimator or weight bounds.
  • fees: Fees estimator.
  • sets: Sets used to map estimator values to assets.
  • opti: Inner optimiser.
  • opto: Outer optimiser.
  • cv: Inner cross-validation scheme of the sub-portfolios, an OptCVE_Onl, or nothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio. nothing gives it the returns of the solve over the full sample.
  • wf: Weight finaliser.
  • ex: Parallel execution strategy.
  • fb: Fallback result or estimator.
  • strict: Whether a constraint that names an asset or a group that sets does not hold raises an error. When it is false, the constraint generator warns and skips the entry. Under true the fit also refuses an asset with a gap in the rows that a plain graph reader reads, which it otherwise drops with a warning, see graph_reader_mask. It also refuses an asset that a distance reader of the fit cannot estimate, which it otherwise drops with a warning, see distance_reader_refusal_mask.
  • pcol: Rule of the panel collapse, an AbstractPanelCollapseAlgorithm. It states how the collapse of the Asset Panel onto the sub-portfolios weighs a member at an observation where the member is outside the universe. RenormaliseActive divides by the weight of the active members, and InactiveAsCash reads the missing weight as cash with a zero feature.
  • cache: Optional ReturnsBufferState, the fold context of the online step. It is nothing until partial_fit! writes one, and optimise(opt) with no returns reads it. The returns themselves are carried by the prior, which owns the rows once; this holds every other column of the returns data and the context pinned at the first step. factory carries it unchanged and port_opt_view slices it to the selected assets.

Constructors

NestedClustered(;    pe::Onl{<:TD{<:PrE_Pr}} = EmpiricalPrior(),    cle::TD{<:ClE_Cl} = ClustersEstimator(),    wb::TD_Option{<:WbE_Wb} = nothing,    fees::TD_Option{<:FeesE_Fees} = nothing,    sets::TD_Option{<:UniverseSets} = nothing,    opti::OptE_TD,    opto::OptE_TD,    cv::Option{<:OptCVE_Onl} = nothing,    scorer::Option{<:PredictionCrossValScorer} = nothing,    wf::TD{<:WeightFinaliser} = IterativeWeightFinaliser(),    ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(),    fb::TDO_Option{<:OptE_Opt} = nothing,    strict::Bool = false,    pcol::AbstractPanelCollapseAlgorithm = RenormaliseActive(),    cache::Option{<:ReturnsBufferState} = nothing) -> NestedClustered

Keywords correspond to the struct's fields.

Time-dependent fields

pe, cle, wb, fees, sets, wf, opti, opto and fb can hold a TimeDependent schedule, one value for each fold. A schedule in opto or fb must bind :outermost, because no inner fold loop reads them. A schedule in opti can also bind :nearest. The inner cross-validation runs cross_val_predict(opti, …; cols = cl) for each cluster, so the inner fold loop reads opti. A :nearest schedule in opti resolves in that loop, one time for each cluster. The solve of each cluster outside the folds resolves the schedule to its default, so a :nearest schedule in opti needs an explicit default and a cv (see assert_nearest_optimiser_schedule). cv and scorer accept no schedule, because cv is the inner fold loop itself, and the construction checks read both.

An entry of a schedule in opti or opto must be an estimator, as the static value must be. The constructor refuses a vector schedule that holds a precomputed result.

Validation

  • opto passes assert_external_optimiser and assert_special_nco_requirements. A schedule passes when its entries and its default pass.
  • When opti is not opto, opti passes assert_internal_optimiser and assert_special_nco_requirements.
  • When cv is present, opti passes assert_external_optimiser.
  • scorer is nothing unless cv is a CombinatorialCrossValidation. The constructor throws an ArgumentError otherwise, through assert_meta_scorer.
  • A schedule in opto or fb does not bind :nearest. A schedule in opti that binds :nearest has an explicit default, and cv is present.
  • A wb that is keyed by name (name_keyed), or a FeesEstimator in fees, needs sets. The constructor throws an IsNothingError otherwise.
  • A vector schedule holds no precomputed result, see assert_time_dependent_substitution.
  • A reader that states no source and reads the raw returns data, beside a reader that states x_src = ExcessSource(), logs one warning that names its path, see warn_raw_readers. The head then builds as configured.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • fees: Recursively updated through factory.
  • opti: Recursively updated through factory.
  • opto: Recursively updated through factory.
  • fb: Recursively updated through factory.
  • cache: Recursively updated through factory.

View parameters

NestedClustered defines its own port_opt_view method rather than deriving one from field tags.

  • The method reads the returns matrix X. When pe holds a prior result, the method uses pe.X in place of X, so the fields that read a matrix read the observations of the prior.
  • pe, wb, sets and cache recurse through port_opt_view with the index alone.
  • fees, opti and opto recurse through port_opt_view with the index and the matrix, and fb through view_child.
  • cle passes through unchanged, because the clustering runs again over the viewed assets.

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Related

References

  • [30] M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019). Section 4.3, Snippet 4.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 7.6.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Sections 12.3.1 and 12.3.2.
source
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
                                 AbstractResult, StatsBase.CovarianceEstimator}},
        args...; kwargs...) -> AbstractArray

Rebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.

factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.

factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.

The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.

Algorithm

The leaf method:

  1. Return a with no change, and drop args... and kwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.

The array method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... with no change.
  2. Collect the results into a new array of the shape of a, in the order of a, and return it.

A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.

The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.

Arguments

  • a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators or nothing, which it rebuilds one element at a time.
  • args...: Positional arguments. The leaf method ignores them, and the array method forwards them.
  • kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.

Returns

  • a: The input with no change, for the leaf method.
  • v::AbstractArray: The rebuilt elements, in an array of the shape of a, for the array method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
factory(res::NonFiniteAllocationOptimisationResult, fb::Option{<:OptE_Opt_FbChain})

Rebuilds a continuous optimisation result with a new fallback record fb.

A concrete result can add its own method when a rebuild needs more than a new fb. optimise is the one caller, and it gives the FbChain of the attempts that failed.

Algorithm

  1. Read every field of res, in the order of the struct.
  2. Call the constructor of the type of res with every field except the last, and with fb as the last field.

Validation

  • fb is the last field of the type of res. Every concrete optimisation result obeys this rule, so the method checks nothing.

Related

source
factory(
    opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
    _
) -> RelaxedRiskBudgeting{JuMPOptimiser{__T_pe, __T_slv, __T_cbe, __T_bgt, __T_fees, __T_ret, __T_sca, __T_sc, __T_so, __T_strict, __T_cache}, _A, _B, <:RelaxedRiskBudgetingAlgorithm} where {__T_pe, __T_slv, __T_cbe, __T_bgt, __T_fees, __T_ret, __T_sca, __T_sc, __T_so, __T_strict, __T_cache, _A, _B}

Returns opt unchanged.

This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.

Related

source
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
                                 AbstractResult, StatsBase.CovarianceEstimator}},
        args...; kwargs...) -> AbstractArray

Rebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.

factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.

factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.

The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.

Algorithm

The leaf method:

  1. Return a with no change, and drop args... and kwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.

The array method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... with no change.
  2. Collect the results into a new array of the shape of a, in the order of a, and return it.

A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.

The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.

Arguments

  • a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators or nothing, which it rebuilds one element at a time.
  • args...: Positional arguments. The leaf method ignores them, and the array method forwards them.
  • kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.

Returns

  • a: The input with no change, for the leaf method.
  • v::AbstractArray: The rebuilt elements, in an array of the shape of a, for the array method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
factory(
    opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
    _
) -> RelaxedRiskBudgeting{JuMPOptimiser{__T_pe, __T_slv, __T_cbe, __T_bgt, __T_fees, __T_ret, __T_sca, __T_sc, __T_so, __T_strict, __T_cache}, _A, _B, <:RelaxedRiskBudgetingAlgorithm} where {__T_pe, __T_slv, __T_cbe, __T_bgt, __T_fees, __T_ret, __T_sca, __T_sc, __T_so, __T_strict, __T_cache, _A, _B}

Returns opt unchanged.

This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
    nco::NestedClustered,
    i,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    args...
) -> NestedClustered{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185", _K, Bool, <:AbstractPanelCollapseAlgorithm} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185"<:Transducers.Executor, _K}

Return a copy of a NestedClustered viewed onto the assets i, with the returns matrix X.

The ## View parameters subsection of NestedClustered states how the method views each field.

source
PortfolioOptimisers._optimise — Method
_optimise(
    nco::NestedClustered,
    rd::ReturnsResult;
    branchorder,
    str_names,
    save,
    rmsk,
    kwargs...
) -> Any

Run the nested clustered optimisation.

optimise calls this method. The # Mathematical definition of NestedClustered states the weights that the method computes.

Algorithm

  1. Replace the schedules in the fields of nco with their static defaults, through reset_time_dependent_estimator.
  2. Fit the prior of nco.pe to rd, giving pr. The prior reads the returns in the frame that its own x_src states. rd keeps its raw returns, and the clustering reads the frame that its own x_src states. The readers of the inner and outer optimisers state their own sources.
  3. Take the element type of the weights, the fees and the weight bounds as eltype(pr.X), giving Tf.
  4. Read the investable mask of pr with investable_mask, giving imsk.
  5. Resolve the fees of nco.fees over the universe of the caller with fees_constraints, and place them on the investable assets with investable_fees_view, giving fees.
  6. Remove the liquidation charges from fees with strip_liquidation_charges, giving cfees. The result charges a forced exit one time, and no cluster holds an asset that exits.
  7. Reduce pr, nco and rd to the investable assets with investable_reduction, giving pr, nco and rdr.
  8. Cluster the assets with clusterise, giving clr, and check with assert_clustering_universe that clr covers every asset. Collect the assets of each cluster, giving cls.
  9. For each cluster k, view nco.opti and rdr onto the assets cls[k] with port_opt_view, and optimise, giving resi[k]. Write its weights into column k of wi, which is zero outside the cluster.
  10. Predict the returns of the synthetic assets with predict_outer_returns, cfees and a ClusterUniverse, giving rdo. Without nco.cv these are the returns of the cluster portfolios over the observations of pr. With nco.cv they are the out-of-sample predictions of the folds.
  11. Rebuild the asset sets of nco.opto over the names of the synthetic assets with _update_asset_sets.
  12. Optimise nco.opto over rdo, giving reso.
  13. Resolve the weight bounds of nco.wb over the investable assets with weight_bounds_constraints, giving wb.
  14. Combine wi and reso.w and finalise them with nco.wf under wb, through outer_optimisation_finaliser, giving retcode and w.
  15. Build the NestedClusteredResult. Its keyword constructor expands w to the full asset universe.

Arguments

  • nco: The nested clustered optimiser.
  • rd: The returns result to use.
  • branchorder: The branch order of a hierarchical clustering. The method also passes it to the inner and outer optimisers.
  • str_names: Whether the JuMP models name their variables with strings. The method passes it to the inner and outer optimisers.
  • save: Whether the results save their JuMP models. The method passes it to the inner and outer optimisers.
  • rmsk: The mask of a fit again after a refusal, or nothing on the first fit.
  • kwargs: More keyword arguments for the inner and outer optimisers.

Validation

  • The clustering covers every investable asset. The method throws a DimensionMismatch otherwise.
  • No intra-cluster optimisation returns an efficient frontier. The method throws an ArgumentError otherwise.

Returns

  • res::NestedClusteredResult: The combined portfolio. retcode is an OptimisationFailure when an intra-cluster optimisation, the inter-cluster optimisation or the weight finalisation fails.

Related

source

References

[12]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[26]
M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
[30]
M. López de Prado. A robust estimator of the efficient frontier. SSRN Electronic Journal (2019).