GitHub

Meta optimisation: private API

PortfolioOptimisers.SubPortfolioUniverse — Type
abstract type SubPortfolioUniverse end

Abstract supertype for a meta-optimiser's sub-portfolio enumeration: what a sub-portfolio is, and what it sees.

A meta-optimiser solves one inner problem per sub-portfolio, predicts each sub-portfolio's returns, and hands the outer optimiser a synthetic universe with one asset per sub-portfolio. That is one module, and the two shipped meta-optimisers differ in exactly one respect, which this type names:

  • NestedClustered enumerates cluster index sets. One inner optimiser is viewed onto each cluster, and every full-universe quantity — the Prior Result, the Fees — is viewed onto it too.
  • Stacking enumerates inner optimisers. Each one sees the whole universe, so nothing is viewed.

FullUniverse and ClusterUniverse declare the two. The module reads them back through sub_portfolio_predict, sub_portfolio_view and fold_weight_matrix, and a third meta-optimiser is a third declaration rather than a third copy of the module.

Related

source
PortfolioOptimisers.ClusterUniverse — Type
struct ClusterUniverse{T<:(AbstractVector{<:AbstractVector{<:Integer}})} <: SubPortfolioUniverse

Sub-portfolios are cluster index sets, and each sees its own cluster. NestedClustered's enumeration.

One inner optimiser serves every sub-portfolio, viewed onto that sub-portfolio's assets, and an inner weight vector is as long as its cluster — which is why the outer collapse zero-pads it onto the full asset axis (see fold_weight_matrix).

Fields

  • cls: Asset indices of each sub-portfolio. They partition the universe, so a zero-padded column is the sub-portfolio's real weight on the full asset axis.

Related

source
PortfolioOptimisers.sub_portfolio_count — Function
sub_portfolio_count(u::FullUniverse, opti)
sub_portfolio_count(u::ClusterUniverse, opti)

Count the sub-portfolios.

A FullUniverse enumerates the inner optimisers, so it has as many sub-portfolios as opti holds. A ClusterUniverse enumerates the clusters, and one inner optimiser serves them all.

Arguments

  • u: Sub-portfolio enumeration.
  • opti: The meta-optimiser's inner optimiser field — a vector of optimisers for FullUniverse, one optimiser for ClusterUniverse.

Returns

  • The number of sub-portfolios.

Related

source
PortfolioOptimisers.sub_portfolio_predict — Function
sub_portfolio_predict(u::FullUniverse, opti, i, rd, cv, ex)
sub_portfolio_predict(u::ClusterUniverse, opti, i, rd, cv, ex)

Cross-validate sub-portfolio i.

One cross_val_predict call, and the enumeration says which optimiser it runs and on which assets. A FullUniverse runs opti[i] and passes no cols: the sub-portfolio is the whole universe, and the arity that takes a precomputed OptimisationResult has no cols keyword at all, so a colon would be a MethodError rather than a no-op. A ClusterUniverse runs the one inner optimiser on u.cls[i].

Arguments

  • u: Sub-portfolio enumeration.
  • opti: The meta-optimiser's inner optimiser field.
  • i: Sub-portfolio index.
  • rd: Returns data.
  • cv: Cross-validation scheme, already copied for this sub-portfolio.
  • ex: FLoops executor controlling parallelism.

Returns

  • Sub-portfolio i's cross-validation prediction result.

Related

source
PortfolioOptimisers.sub_portfolio_view — Function
sub_portfolio_view(u::FullUniverse, x, i::Integer)
sub_portfolio_view(u::ClusterUniverse, x, i::Integer)

View a full-universe quantity onto sub-portfolio i.

The quantities are the ones a sub-portfolio's predicted returns are computed from: the Prior Result and the Fees. A FullUniverse sub-portfolio holds the whole universe, so x is returned unchanged; a ClusterUniverse one restricts it through port_opt_view.

Arguments

  • u: Sub-portfolio enumeration.
  • x: Full-universe quantity, or nothing.
  • i: Sub-portfolio index.

Returns

  • x, viewed onto sub-portfolio i.

Related

source
PortfolioOptimisers.sub_portfolio_cv — Function
sub_portfolio_cv(cv)

Give one sub-portfolio its own copy of the cross-validation scheme.

The sub-portfolios are cross-validated in parallel, so a scheme that draws its splits from a random number generator must not be shared: the generator is mutable state, and two sub-portfolios advancing it at once would neither reproduce nor agree on their folds. A scheme with no rng field carries no such state and is passed through.

Arguments

  • cv: Cross-validation scheme.

Returns

  • cv, copied when it holds an rng.

Related

source
PortfolioOptimisers.outer_optimisation_finaliser — Function
outer_optimisation_finaliser(wb, wf, w_inner, w_outer)

Finalise outer optimisation weights for the NCO algorithm.

Combines inner cluster weights w_inner with outer portfolio weights w_outer, applying weight bounds wb and finalisation algorithm wf.

Arguments

  • wb: Weight bounds (optional).
  • wf: Weight finaliser.
  • w_inner: Inner (within-cluster) weights.
  • w_outer: Outer (across-cluster) weights.

Returns

  • (retcode, w): Final combined portfolio weights and return code. On failure of any sub-problem, retcode is an OptimisationFailure whose res is a named tuple (; msg, opti, opto, wb) carrying the failure summary, the inner optimisation return codes, the outer return code, and the weight-finalisation return code (including their solver trial diagnostics).

Related

source
PortfolioOptimisers.combination_weights — Function
combination_weights(scale::Nothing, w::VecNum_VecVecNum)
combination_weights(scale::VecNum, w::VecNum)
combination_weights(scale::VecNum, w::VecVecNum)

Apply a Combination Weight to a meta-optimiser's outer weights.

The outer optimiser decides how much of each sub-portfolio to hold. A Combination Weight is a fixed belief about the same quantity, so the two multiply: sub-portfolio k carries the coefficient sₖ·vₖ, and the coefficients are rescaled to the total the outer optimiser chose. Schur Complement Hierarchical Risk Parity blends its parameter bundles the same way; this is that shape, placed where a meta-optimiser that owns an outer optimiser can use it.

What the rescale buys

Only the ratios between the entries of a Combination Weight carry meaning, and the rescale is what makes that true here — a common factor cancels, so the weight needs no normalised form of its own. Three cases are then exactly inert:

  • A uniform weight gives back w, whatever it sums to.
  • A lone sub-portfolio gives back w. One element is not a combination.
  • An outer optimiser that chose a total other than one keeps it. Rescaling to one instead would silently overrule a bgt of 0.9.

Why the outer problem never sees the weight

Scaling the synthetic return columns instead would break the weight on two counts:

  • A uniform weight would stop being neutral. A common rescale of every column moves MaximumUtility's trade-off between return and risk, so the neutral setting would not be neutral.
  • Every predict_outer_returns overload would have to re-apply it. A custom one would drop it silently, and the two cross-validation methods already did — so a cross-validated run would disagree with a fold-less one on what the weight means. cv is execution control and stays that way.

Degenerate combinations

The rescale factor is sum(w) / sum(scale .* w), and it does not always exist. A zero denominator — a tilt that cancels a long-short outer allocation — makes it infinite or NaN; a zero numerator — a dollar-neutral outer allocation — makes it zero, which would collapse the portfolio. One test covers all three, and the tilted coefficients then stand unrescaled: finite, with their ratios intact. No scalar rescale can hold a zero total while tilting, so there is nothing better to return.

A denominator that is merely near zero is not degenerate. The large factor is the answer: the tilt genuinely rebalances a combination that nearly cancels. Should it overflow, finalise_weight_bounds reports an OptimisationFailure rather than a plausible-looking portfolio.

Arguments

  • scale: Combination Weight, one entry per sub-portfolio, or nothing.
  • w: Outer optimiser weights, one entry per sub-portfolio; a vector of them on an efficient frontier.

Returns

  • w unchanged when scale is nothing, otherwise the rescaled coefficients.

Related

source
PortfolioOptimisers.prepare_outer_rd — Function
prepare_outer_rd(
    rd::ReturnsResult,
    wi::AbstractMatrix{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}
) -> Tuple{Any, Any, Any, Any, Union{Nothing, AssetPanel{<:AbstractVector{var"#s316"}, Nothing, Nothing} where var"#s316"<:AbstractPanelField, AssetPanel{<:AbstractVector{var"#s316"}, <:AbstractMatrix{Bool}, <:AbstractMatrix{Bool}} where var"#s316"<:AbstractPanelField}, Matrix}

Prepares the ReturnsResult for outer optimisation, applying the inner cluster weights wi to the returns matrix rd.B, and adjusting the independent variable matrices rd.iv and rd.ivpa, and the Feature Matrix derived from rd.pnl, accordingly.

Warning

This function returns pnl in addition to the four values it returned before the Asset Panel was collapsed onto the synthetic universe, and it returns it before the returns buffer X. A custom predict_outer_returns overload written against the old tuple therefore breaks loudly — it binds pnl where it expects X and fails on the first write — rather than silently continuing to build an outer ReturnsResult with no panel and never learning that it should have one. Appending the value would not have done this: Julia's destructuring discards trailing values without complaint.

Arguments

  • rd: ReturnsResult containing the returns data.
  • wi: Inner weights matrix.

Returns

  • nb: New names for the benchmark returns columns after applying inner weights (if rd.B is a matrix).
  • B: Adjusted benchmarkreturns matrix after applying inner weights (if rd.B is a matrix).
  • iv: Adjusted independent variable matrix (if present).
  • ivpa: Adjusted independent variable per asset matrix (if present).
  • pnl: Asset Panel collapsed onto the synthetic assets (if present), see collapse_asset_panel.
  • X: Buffer for the outer returns matrix.

Related

source
PortfolioOptimisers.assert_fold_alignment — Function
assert_fold_alignment(predictions) -> VecPredRes

Assert that every sub-portfolio's fold f covers the same test period, and return the first sub-portfolio's folds.

A meta-optimiser runs the same cross-validation scheme over the same returns result for every sub-portfolio, so fold f covers the same observations whichever sub-portfolio produced it. rebuild_returns_result has relied on that silently since long before it was stated — reshape(X, :, N) is only meaningful if the N stacked return vectors line up row for row — and the per-fold weight matrix it now assembles is only well defined if it holds. This makes the invariant explicit, and it matters most on the combinatorial path, where each sub-portfolio's scorer selects a path independently.

Folds are compared on their timestamps where the returns result has a clock, and on their observation counts where it does not — which is the strongest statement available in each case, and exactly the statement reshape needs.

Arguments

Returns

  • The first sub-portfolio's per-fold PredictionResult objects, which every other sub-portfolio now agrees with.

Related

source
PortfolioOptimisers.fold_row_indices — Function
fold_row_indices(rd, pred) -> VecVecInt

Recover the rows of the original returns result each cross-validation fold covers.

The folds do not store their row indices, and they do not need to: port_opt_view slices ts with the very test_idx the fold was built from, so a fold's rd.ts is its slice of the original clock and feature_row_indices matches it straight back. Recovering rather than storing is what keeps this correct on the combinatorial path, where sort_predictions! assembles a path's folds in split order rather than chronologically: the timestamps carry whatever order actually happened, while a re-derived split would have to reproduce it.

Only a time-varying feature matrix needs this — a static one has no observation axis to slice — which is why the clock is required exactly there and nowhere else.

Arguments

  • rd: Original ReturnsResult, whose ts is the clock a time-varying rd.pnl's observation axis is parallel to.
  • pred: Per-fold PredictionResult objects from one sub-portfolio.

Returns

  • One row-index vector per fold.

Related

source
PortfolioOptimisers.fold_weight_matrix — Function
fold_weight_matrix(predictions, u::FullUniverse, f, na)
fold_weight_matrix(predictions, u::ClusterUniverse, f, na)

Lay fold f's sub-portfolio weights out as the assets × sub-portfolios matrix the outer collapse contracts against.

The sub-portfolio enumeration says which. A FullUniverse's inner optimisers see the whole universe, so their weight vectors are already full length. A ClusterUniverse's see one cluster each, so sub-portfolio i's weights are zero-padded onto u.cls[i]. The padding invents nothing: the clusters partition the universe, so a padded column is the sub-portfolio's real weight on the full asset axis.

Arguments

Returns

  • An assets × sub-portfolios weight matrix.

Related

source
PortfolioOptimisers.fold_asset_panel — Function
fold_asset_panel(pnl::Nothing, nx, wi, anchor) -> nothing
fold_asset_panel(pnl::AssetPanel, nx, wi, anchor) -> AssetPanel

Collapse the original AssetPanel onto a fold's synthetic universe, with an observation axis.

The collapse itself is collapse_asset_panel, applied to the original, unsliced panel and the fold's weights. What the fold supplies differs by shape, and the anchor says which:

  • A static panel has no observation axis, so its anchor is the fold's observation count. Its one collapsed panel is lifted across them, because the collapse is a function of this fold's weights and is therefore constant within the fold and different in the next one — which is how a static source becomes genuinely time-varying at the outer problem.
  • A time-varying panel is first cut to the fold's rows in the original clock, and comes back with an observation axis already. This is the only place a fold's absolute rows are needed, and fold_row_indices recovers them from the fold's timestamps.

Algorithm

  1. Return nothing when the carrier holds no panel.
  2. For a static panel, collapse it and lift every field to the fold's observation count with panel_field_lift, taking all-true universe masks.
  3. For a time-varying panel, view the fold's rows with port_opt_view and collapse that view.

Arguments

  • pnl: The original Asset Panel, unsliced, or nothing.
  • nx: The carrier's asset names, or nothing. Read for the square case alone.
  • wi: The fold's weights, assets × synthetic assets.
  • anchor: The fold's observation count for a static panel, its absolute rows for a time-varying one.

Returns

  • nothing, or a time-varying Asset Panel over the fold's observations.

Related

source
PortfolioOptimisers.fold_feature_anchors — Function
fold_feature_anchors(rd, pred)

Give each fold whatever fold_asset_panel needs from it: an observation count for a static panel, absolute rows for a time-varying one.

Scoping the row recovery to the shape that needs it is what keeps the clock requirement narrow. A static panel has no observation axis to align, so it runs on fold sizes alone and never asks the returns result for timestamps.

Arguments

Returns

  • One anchor per fold: an Integer for a static panel, a row-index vector for a time-varying one.

Related

source
PortfolioOptimisers.panel_field_stack — Method
panel_field_stack(fs::AbstractVector) -> AbstractPanelField

Stack one Panel Field's fold-by-fold collapses along the observation axis.

Every fold collapses the same source field onto the same synthetic universe, so the fold results agree on every axis but the observations, and the stack is a concatenation there. The observed masks stack with the values.

Algorithm

  1. Concatenate the values of every fold along the observation axis.
  2. Concatenate the observed masks the same way, or keep nothing when the field carries none.
  3. Rebuild the field with its own keyword constructor, which re-runs every guard.

Arguments

  • fs: One collapsed Panel Field per fold, in fold order.

Returns

  • The stacked Panel Field.

Related

source
PortfolioOptimisers.rebuild_asset_panel — Function
rebuild_asset_panel(rd, predictions, u, pred1)

Recompute the outer problem's AssetPanel at the cross-validation assembly seam.

Per fold, this makes the same collapse_asset_panel call prepare_outer_rd makes on the non-cross-validated path — same asset names, same weight-matrix arity, same original panel — and stacks the results down the observation axis. That shared call is the whole point: cv is execution control, so toggling it must not change what the outer optimiser measures.

The stacked panel is time-varying by construction, and takes all-true universe masks: the fold panels describe disjoint observation windows of one synthetic universe, and a synthetic asset exists in every one of them.

Arguments

Returns

  • pnl::Option{AssetPanel}: The Asset Panel on the synthetic universe, or nothing when rd carries none.

Related

source
PortfolioOptimisers.rebuild_returns_result — Function
rebuild_returns_result(rd, predictions, u)

Reconstruct a returns result from cross-validation predictions.

Combines individual fold predictions from predictions into a new ReturnsResult corresponding to the original data layout. u is the sub-portfolio enumeration — a ClusterUniverse for NestedClustered, a FullUniverse for Stacking — and it is what says whether a fold's weight vectors need padding onto the full asset axis.

Warning

u is positional and required, not a keyword with a full-universe default. A default would let a stale two-argument call keep working: correct for Stacking, and for NestedClustered silently writing every cluster's weights to the wrong rows — which yields not an error but a plausible-looking feature matrix. The one configuration that most needs the argument is the one a default would mis-serve, so the break is arranged to be loud.

The feature matrix

The folds carry none. Instead, the collapse onto the synthetic universe is recomputed here from the Feature Matrix derived from the original, unsliced rd.pnl, using the same collapse_asset_panel call prepare_outer_rd makes on the non-cross-validated path — with sq from features_are_assets flowing through unchanged, and the per-fold assets × sub-portfolios weight matrix assembled from pred[f].res.w (see rebuild_asset_panel). The fold results stack down the observation axis, giving the observations × assets × features shape the time-varying carrier takes, and the outer optimiser's default LastObservation reduces them to the most recent fold's collapse.

The inner solves are untouched: each still sees its own cluster-sliced feature matrix. What the recompute buys is that cv, which is execution control, no longer changes what the outer problem measures — and it closes the one intersection where the matrix used to be dropped altogether, a square feature matrix under NestedClustered, whose folds see cluster-sliced returns and so could never agree on a feature axis to stack.

Arguments

Returns

Related

source
PortfolioOptimisers.sub_portfolio_predictions — Function
sub_portfolio_predictions(::Type{T}, opti, u, rd, cv, ex) where {T}

Cross-validate every sub-portfolio, in parallel, over the same returns result.

One sub_portfolio_predict call per sub-portfolio, each on its own copy of the scheme (see sub_portfolio_cv). Every sub-portfolio therefore runs the same cross-validation over the same returns result, which is the invariant assert_fold_alignment states one level down.

Arguments

  • T: Element type of the prediction vector — a MultiPeriodPredictionResult per sub-portfolio on the non-combinatorial path, a PopulationPredictionResult on the combinatorial one.
  • opti: The meta-optimiser's inner optimiser field.
  • u: Sub-portfolio enumeration, a SubPortfolioUniverse.
  • rd: Returns data.
  • cv: Cross-validation scheme.
  • ex: FLoops executor controlling parallelism.

Returns

  • One prediction result per sub-portfolio.

Related

source
PortfolioOptimisers.predict_outer_returns — Function
predict_outer_returns(cv::Option{<:OptimisationCrossValidation}, opt,
                      u::SubPortfolioUniverse, rd::ReturnsResult,
                      pr::AbstractPriorResult, fees::Option{<:Fees}, wi::MatNum,
                      resi::VecOpt)
predict_outer_returns(cv::OptimisationCrossValidation{<:NonCombOptCV}, opt,
                      u::SubPortfolioUniverse, rd::ReturnsResult,
                      pr::AbstractPriorResult, fees::Option{<:Fees}, wi::MatNum,
                      resi::VecOpt)
predict_outer_returns(cv::OptimisationCrossValidation{<:CombinatorialCrossValidation},
                      opt, u::SubPortfolioUniverse, rd::ReturnsResult,
                      pr::AbstractPriorResult, fees::Option{<:Fees}, wi::MatNum,
                      resi::VecOpt)

Predict a meta-optimiser's sub-portfolio returns as the outer problem's ReturnsResult.

One module serves every meta-optimiser that owns an outer optimiser. What varies between them is the sub-portfolio enumeration, which arrives as a SubPortfolioUniverse — NestedClustered passes a ClusterUniverse, Stacking a FullUniverse — and nothing else here reads the meta-optimiser's own type.

Dispatch is on cv, which is what chooses the prediction, and a custom cross-validation scheme is therefore an overload on that first argument:

  • Fold-less. The sub-portfolios' own solves are already in resi, so each column of the synthetic universe is that solve's net returns, on the Prior Result and Fees viewed onto the sub-portfolio (sub_portfolio_view).
  • Non-combinatorial cross-validation. Each sub-portfolio is cross-validated and the folds are stacked (rebuild_returns_result), so the outer problem is measured out of sample.
  • Combinatorial cross-validation. As above, then each sub-portfolio's scorer selects one path from its population. The default is NearestQuantilePrediction.

wi holds the inner optimisers' own weights, and a Combination Weight is not applied to them and must not be applied here: it acts at the combination, after the outer solve, so that an overload cannot drop it and a cross-validated run cannot disagree with a fold-less one on what it means (see combination_weights).

Warning

A meta-optimiser calls this with its cv first and its sub-portfolio enumeration third. The two verbs this replaces — predict_outer_nco_estimator_returns and predict_outer_st_estimator_returns — were the same module written twice, and an overload of either is now a method of nothing. Rewrite it as a method of this verb, dispatching on the scheme rather than on the meta-optimiser's type parameters.

Arguments

  • cv: The meta-optimiser's cross-validation scheme.
  • opt: The meta-optimiser. Read for its inner optimisers and its executor.
  • u: Sub-portfolio enumeration.
  • rd: Returns data.
  • pr: Prior Result over the whole universe.
  • fees: Fees over the whole universe.
  • wi: Inner weights, assets × sub-portfolios.
  • resi: The sub-portfolios' own optimisation results.

Returns

  • The outer problem's ReturnsResult, one synthetic asset per sub-portfolio.

Related

source