Base meta optimisation: private API

PortfolioOptimisers.ClusterUniverse — Type
struct ClusterUniverse{__T_cls} <: SubPortfolioUniverse

Makes each cluster a sub-portfolio that sees only its own assets.

This is the enumeration of NestedClustered. One inner optimiser serves every sub-portfolio, viewed onto the assets of that sub-portfolio, so an inner weight vector is as long as its cluster. fold_weight_matrix puts zeros at the other assets.

Fields

  • cls: Asset indices of each sub-portfolio. They partition the universe, so a column with zeros at the other assets is the real weight of the sub-portfolio over the whole asset axis.

Related

source
PortfolioOptimisers.sub_portfolio_cv — Function
sub_portfolio_cv(cv)

Give one sub-portfolio its own copy of the cross-validation scheme.

The module cross-validates the sub-portfolios in parallel. A scheme that draws its splits from a random number generator holds mutable state, and two sub-portfolios that advance one generator at the same time get different folds on each run. So the method copies a scheme that has an rng field, and each copy starts from the same state. It returns a scheme with no rng field unchanged.

No scheme that the cv field of a meta-optimiser accepts (OptCVE_Onl) has an rng field, so every scheme a meta-optimiser runs today takes the second branch. A scheme with an rng field must implement Base.copy.

Arguments

  • cv: Cross-validation scheme.

Returns

  • A copy of cv when it has an rng field, otherwise cv itself.

Related

source
PortfolioOptimisers.outer_optimisation_finaliser — Function
outer_optimisation_finaliser(wb::WeightBounds, wf::WeightFinaliser, resi::VecOpt,
                             rco::OptimisationReturnCode, w::VecNum, wi::MatNum)
outer_optimisation_finaliser(wb::WeightBounds, wf::WeightFinaliser, resi::VecOpt,
                             rcos::VecOptRetCode, ws::VecVecNum, wi::MatNum)

Combine the inner and outer weights of a meta-optimiser, and finalise them under its weight bounds.

The return code reports a failure at any of the three stages: an inner solve, the outer solve, and the finalisation of the combined weights.

Mathematical definition

\[\begin{align} \boldsymbol{w} &= \mathbf{W} \boldsymbol{v}\,. \end{align}\]

Where:

  • $\boldsymbol{w}$: Combined weights over the $N$ assets, before the finalisation.
  • $\mathbf{W}$: Inner weight matrix assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio.
  • $\boldsymbol{v}$: Outer weight vector, the weights that the outer optimiser gives the $K$ synthetic assets. Stacking passes the coefficients of combination_weights in its place.
  • $N$: Number of assets.

Algorithm

  1. Combine the weights, giving w = wi * w.
  2. Finalise w with wf under wb through finalise_weight_bounds, giving retcode and the finalised w.
  3. Read the return code of every inner result, giving resi_retcodes.
  4. When an inner return code, rco or retcode is an OptimisationFailure, replace retcode with an OptimisationFailure whose res is (; msg, opti, opto, wb). msg has one line for each stage that failed, in the order inner, outer, finalisation. opti is resi_retcodes, opto is rco, and wb is the return code of step 2.
  5. Return (retcode, w). The method returns the weights of step 2 when a stage fails too.

The method for an efficient frontier runs these steps once for each pair of rcos and ws.

Arguments

  • wb: Weight bounds over the $N$ assets.
  • wf: Weight finaliser.
  • resi: The results of the inner optimisations.
  • rco: The return code of the outer optimisation.
  • w: Outer weights, one entry for each sub-portfolio.
  • rcos, ws: The return codes and outer weights of the points of an efficient frontier.
  • wi: Inner weights, assets × sub-portfolios.

Returns

  • (retcode, w): The return code and the finalised weights. The frontier method returns a vector of return codes and a vector of weight vectors.

Related

source
PortfolioOptimisers.combination_weights — Function
combination_weights(scale::Nothing, w::VecNum_VecVecNum)
combination_weights(scale::VecNum, w::VecNum)
combination_weights(scale::VecNum, w::VecVecNum)

Apply a Combination Weight to the outer weights of a meta-optimiser.

The outer optimiser decides how much of each sub-portfolio to hold. A Combination Weight is a fixed belief about the same quantity, so the two multiply, and the method rescales the products to the total that the outer optimiser chose. Schur Complement Hierarchical Risk Parity blends its parameter bundles in the same way.

The rescale makes only the ratios between the entries of a Combination Weight matter. A common factor cancels, so the weight needs no normalised form. Three cases give back w unchanged:

  • a uniform weight, whatever its sum;
  • a lone sub-portfolio, because one element is not a combination;
  • an outer total other than one keeps its total, so the method does not overrule a bgt of 0.9.

The outer problem never sees the weight. A common factor on every synthetic return column moves the trade-off between return and risk of MaximumUtility, so a uniform weight in the outer problem is not neutral. A weight in the outer problem also needs every predict_outer_returns method to apply it, and a method that drops it makes a cross-validated run disagree with a fold-less run.

Mathematical definition

\[\begin{align} c_k &= \begin{cases} \dfrac{s_k v_k}{\sum_{j=1}^{K} s_j v_j} \sum_{j=1}^{K} v_j & \text{if } \sum_{j=1}^{K} s_j v_j \neq 0 \text{ and } \sum_{j=1}^{K} v_j \neq 0\,,\\ s_k v_k & \text{otherwise}\,. \end{cases} \end{align}\]

Where:

  • $c_k$: Coefficient of sub-portfolio $k$ in the combination.
  • $s_k$: Combination Weight of sub-portfolio $k$.
  • $\boldsymbol{v}$: Outer weight vector, the weights that the outer optimiser gives the $K$ synthetic assets.
  • $K$: Sub-portfolio count of a meta-optimiser, which is also the number of synthetic assets of its outer problem.

The first case keeps the total, $\sum_k c_k = \sum_k v_k$, and the ratios, $c_k / c_j = s_k v_k / (s_j v_j)$. In the second case no scalar rescale can keep a zero total and apply the tilt at the same time, so the tilted products stand. They are finite and keep their ratios. A denominator near zero is not a degenerate case. The large factor is the correct rescale of a combination that nearly cancels.

Algorithm

  1. Multiply the weight into the outer weights, giving c = scale .* w.
  2. Divide the two totals, giving the factor f = sum(w) / sum(c).
  3. Return c when f is zero or not finite, otherwise c * f. A factor that overflows is not finite, so it takes the second case too.

A scale of nothing returns w, and a frontier applies the steps to each point.

Arguments

  • scale: Combination Weight, one entry for each sub-portfolio, or nothing.
  • w: Outer weights, one entry for each sub-portfolio, or a vector of them on an efficient frontier.

Returns

  • w when scale is nothing, otherwise the coefficients $\boldsymbol{c}$.

Related

source
PortfolioOptimisers.prepare_outer_rd — Function
prepare_outer_rd(
    rd::ReturnsResult,
    wi::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    alg::AbstractPanelCollapseAlgorithm
) -> Tuple{Any, Any, Any, Any, Union{Nothing, AssetPanel}, Matrix}

Prepare the returns data of the outer problem from the inner weights wi.

The method collapses every per-asset quantity of rd onto the synthetic assets, and it allocates the buffer that the caller fills with the synthetic returns. Benchmark returns are extensive and collapse as a weighted sum. The implied volatilities and the Asset Panel are intensive and collapse as convex combinations, so the gross exposure of a sub-portfolio does not scale them. On a time-varying panel, the implied volatilities read the active members of each row, under the rule alg of the panel collapse, as the Asset Panel does (see collapse_asset_panel).

Destructure all six returned values. Julia discards trailing values without an error, so a caller that names five binds pnl to the name it means for the buffer X, and its first write into that name fails.

Mathematical definition

\[\begin{align} \mathbf{B}^{o} &= \mathbf{B} \mathbf{W}\,,\\ V^{o}_{tk} &= \frac{\sum_{i=1}^{N} \tilde{W}_{ik} m_{ti} V_{ti}}{d_{tk}}\,,\\ a^{o}_{k} &= \frac{\sum_{i=1}^{N} \tilde{W}_{ik} m_{Ti} a_{i}}{d_{Tk}}\,. \end{align}\]

Where:

  • $\mathbf{B}$, $\mathbf{B}^{o}$: Benchmark returns, observations × assets and observations × sub-portfolios. A benchmark that is not a matrix is kept as it is.
  • $V_{ti}$, $V^{o}_{tk}$: Implied volatilities, observations × assets and observations × sub-portfolios.
  • $a_{i}$, $a^{o}_{k}$: Implied volatility risk premium adjustment, one entry for each asset and one for each sub-portfolio. A scalar adjustment is kept as it is.
  • $m_{ti}$: Active mask of the Asset Panel, $1$ when asset $i$ is in the universe at observation $t$ and $0$ otherwise. With no panel, or a static one, $m_{ti} = 1$.
  • $T$: The last observation of rd, the one at which the outer fit stands.
  • $d_{tk}$: Divisor of the rule alg, see active_weight_divisor. It is $1$ with no panel, with a static one, and under InactiveAsCash.
  • $\mathbf{W}$: Inner weight matrix assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio.
  • $\tilde{\mathbf{W}}$: Normalised inner weight matrix, $\tilde{W}_{ik} = |W_{ik}| / \sum_{j=1}^{N} |W_{jk}|$. A column of zeros stays a column of zeros.
  • $N$: Number of assets.

A column of zeros in $\mathbf{W}$ gives a column of zeros in $\mathbf{V}^{o}$ and a zero entry in $\boldsymbol{a}^{o}$.

Algorithm

  1. When rd.B is a matrix, collapse the cells that benchmark_cells reads, giving B = benchmark_cells(rd) * wi, and name its columns nb = ["_b1", …]. A gap in the benchmark or in the returns of an asset, or an inactive cell, gives zero and not NaN. Otherwise keep rd.B and rd.nb.
  2. Read the active mask of rd.pnl, giving m. It is nothing with no panel or a static one.
  3. When rd.iv is present or rd.ivpa is a vector, normalise wi with synthetic_asset_weights, giving wn. Collapse rd.iv and rd.ivpa with collapse_rate, on wn, m and alg.
  4. Collapse the Asset Panel with collapse_asset_panel under alg, giving pnl.
  5. Allocate the buffer X, observations × sub-portfolios, with the element type that promotes the element types of rd.X and wi.

Arguments

  • rd: The returns data of the meta-optimiser.
  • wi: Inner weights, assets × sub-portfolios.
  • alg: The rule of the panel collapse, the pcol field of the meta-optimiser.

Returns

  • nb: Names of the benchmark columns.
  • B: Benchmark returns.
  • iv: Implied volatilities, or nothing.
  • ivpa: Implied volatility risk premium adjustment, or nothing.
  • pnl: Asset Panel over the synthetic assets, or nothing.
  • X: Uninitialised buffer for the outer returns matrix.

Related

source
PortfolioOptimisers.outer_prior_rows — Function
outer_prior_rows(
    rd::ReturnsResult,
    pr::AbstractPriorResult
) -> ReturnsResult

View the returns data of a meta-optimiser onto the observations its Prior Result answers on.

The outer problem reads the net return of each sub-portfolio off the scenarios of the Prior Result, one row per scenario, so its returns data must hold one row per scenario too. A prior that answers on every observation leaves rd whole. A CrossSectionalFactorPrior answers on the observations it fitted, which are the last ones of rd, because its Descriptor warm-up and its exposure lag consume the first. So a prior with fewer scenarios than rd has observations takes the last rows of rd, and the timestamps, the benchmark, the Exogenous Series and the Asset Panel of the outer problem follow them. A prior that dropped the observations of a Shared Series gap from the rd that it read takes the last of the observations that it kept, see prior_data_rows.

Arguments

  • rd: The returns data of the meta-optimiser.
  • pr: The Prior Result over the whole universe.

Validation

  • size(pr.X, 1) <= size(rd.X, 1). Raises a DimensionMismatch, because a prior with more scenarios than observations states no row of rd for each of them.

Returns

  • rd::ReturnsResult: rd itself, or its view onto the size(pr.X, 1) observations that prior_data_rows pairs with the scenarios.

Related

source
PortfolioOptimisers.assert_fold_alignment — Function
assert_fold_alignment(predictions) -> VecPredRes

Check that fold f of every sub-portfolio covers the same observations, and return the folds of the first sub-portfolio.

A meta-optimiser runs the same cross-validation scheme over the same returns data for every sub-portfolio, so fold f covers the same observations for each of them. rebuild_returns_result needs this. Its reshape(X, :, N) lines up the N stacked return vectors row by row, and the weight matrix of a fold exists only when they agree. The check matters most on the combinatorial path, where the scorer of each sub-portfolio selects its path on its own.

Algorithm

  1. Read the folds of the first sub-portfolio, giving pred1, and their number nf.
  2. For each sub-portfolio, check that it has nf folds.
  3. For each fold, compare the timestamps with those of pred1 when the returns data has a clock, and the number of return entries when it has none.

The number of entries is the strongest check available without a clock, and it is the one reshape needs.

Arguments

Validation

  • Every sub-portfolio has the same number of folds as the first, else DimensionMismatch.
  • Fold f of every sub-portfolio has the timestamps of fold f of the first, or the same number of return entries when there are no timestamps, else DimensionMismatch.

Returns

Related

source
PortfolioOptimisers.fold_row_indices — Function
fold_row_indices(rd, pred) -> VecVecInt

Find the rows of the original returns data that each cross-validation fold covers.

A fold records its rows in the idx of its PredictionResult, and rebuild_fold_rates reads them there. This method finds them from the timestamps. port_opt_view slices ts with the test_idx of the fold, so the rd.ts of a fold is its slice of the original clock, and feature_row_indices finds the rows from it. This works on the combinatorial path too, where sort_predictions puts the folds of a path in split order and not in time order. The timestamps carry the order that the folds have.

Only a time-varying Asset Panel needs the rows. A static panel has no observation axis, so only the time-varying shape needs the clock.

Arguments

  • rd: The original ReturnsResult. The observation axis of a time-varying rd.pnl is parallel to its ts.
  • pred: The per-fold PredictionResult objects of one sub-portfolio.

Validation

  • rd.ts is not nothing, else IsNothingError.

Returns

  • One vector of row indices for each fold.

Related

source
PortfolioOptimisers.fold_asset_panel — Function
fold_asset_panel(pnl::AssetPanel, nx, wi, anchor, alg) -> AssetPanel

Collapse the original AssetPanel onto the synthetic assets of one fold, with an observation axis.

The collapse is collapse_asset_panel of the original panel with the weights of the fold, under the rule alg of the panel collapse. The anchor of the fold depends on the shape of the panel:

  • A static panel has no observation axis, so its anchor is the number of observations of the fold. The method repeats its one collapsed panel over them. The collapse depends on the weights of the fold, so it is constant in one fold and changes in the next, and the outer problem sees a time-varying panel.
  • For a time-varying panel, the method first cuts the panel to the rows of the fold in the original clock, and the collapse keeps that observation axis and the collapsed universe masks of those rows. The anchor is those rows, and fold_row_indices finds them from the timestamps of the fold.

Algorithm

  1. For a static panel, collapse it, giving c, and lift every field of c to the anchor observations of the fold with panel_field_lift. Both universe masks are all true.
  2. For a time-varying panel, view the rows anchor with port_opt_view and collapse that view.

rebuild_asset_panel returns before this method when the returns data has no panel.

Arguments

  • pnl: The original Asset Panel, not sliced.
  • nx: The asset names of the returns data, or nothing. Only the square case reads them.
  • wi: The weights of the fold, assets × sub-portfolios.
  • anchor: The number of observations of the fold for a static panel, its rows for a time-varying one.
  • alg: The rule of the panel collapse, the pcol field of the meta-optimiser.

Returns

  • A time-varying Asset Panel over the observations of the fold.

Related

source
PortfolioOptimisers.fold_feature_anchors — Function
fold_feature_anchors(rd, pred)

Give each fold the anchor that fold_asset_panel needs: the number of observations for a static panel, and the rows for a time-varying one.

Only a time-varying panel needs rows, so only that shape asks the returns data for timestamps. A static panel runs on the fold sizes alone.

Arguments

Returns

  • One anchor for each fold: an Integer for a static panel or no panel, a vector of row indices for a time-varying one.

Related

source
PortfolioOptimisers.panel_field_stack — Method
panel_field_stack(fs::AbstractVector{<:NumericPanelField}) -> NumericPanelField
panel_field_stack(fs::AbstractVector{<:TensorPanelField}) -> TensorPanelField

Stack the collapses of one Panel Field over the folds, along the observation axis.

Every fold collapses the same source field onto the same synthetic assets, so the fold results agree on every axis except the observations, and the stack concatenates them there. The observed masks stack with the values. collapse_panel_field turns a categorical field into a tensor field, so a fold result is never categorical.

Algorithm

  1. Concatenate the values of every fold along the observation axis.
  2. Concatenate the observed masks in the same way, or keep nothing when the field carries none. Concatenate the placeholder masks with panel_pmsk_vcat.
  3. Build the field again with its keyword constructor, which runs every check again.

Arguments

  • fs: One collapsed Panel Field for each fold, in fold order.

Returns

  • The stacked Panel Field.

Related

source
PortfolioOptimisers.rebuild_asset_panel — Function
rebuild_asset_panel(rd, predictions, u, pred1, alg)

Compute the AssetPanel of the outer problem on the cross-validated path.

For each fold, the method makes the same collapse_asset_panel call that prepare_outer_rd makes on the fold-less path, with the same asset names, the same original panel and the same rule alg, and the weights of that fold. Thus cv, which controls execution only, does not change what the outer optimiser measures.

The stacked panel is time-varying. The fold panels cover disjoint windows of one synthetic universe, and the method stacks their universe masks with their values. So a sub-portfolio with no active member at an observation is inactive there, on this path as on the fold-less one. A static panel has no inactive member, and its folds give universe masks that are all true.

Algorithm

  1. Return nothing when rd carries no panel.
  2. For each fold, build its weight matrix with fold_weight_matrix and collapse the original panel with fold_asset_panel at the anchor from fold_feature_anchors, giving ps.
  3. Stack each Panel Field over the folds with panel_field_stack, giving pf.
  4. Stack the active masks and the estimation masks of the folds, and build the panel.

Arguments

  • rd: The original ReturnsResult. The method collapses its panel without slicing it first.
  • predictions: One MultiPeriodPredictionResult for each sub-portfolio.
  • u: Sub-portfolio enumeration, a SubPortfolioUniverse.
  • pred1: The folds of the first sub-portfolio, from assert_fold_alignment. Every sub-portfolio agrees with them, so they set the fold boundaries.
  • alg: The rule of the panel collapse, the pcol field of the meta-optimiser.

Returns

  • pnl::Option{AssetPanel}: The Asset Panel over the synthetic assets, or nothing when rd carries none.

Related

source
PortfolioOptimisers.rebuild_fold_rates — Function
rebuild_fold_rates(rd, predictions, u, pred1, alg, iv, ivpa)

Compute the implied volatilities of the outer problem on the cross-validated path, under the rule of the panel collapse.

Each fold collapses rd.iv and rd.ivpa with the weights of its sub-portfolio through collapse_fold_rate, which reads the active mask and the finite cells under RenormaliseActive, whatever the rule. The fold-less path weighs the members that it reads under the rule (see prepare_outer_rd). So this method collapses the original rd.iv and rd.ivpa again for each fold, with collapse_rate on the weights of that fold, as rebuild_asset_panel does for the panel. It does this with no panel, a static one and a time-varying one: under InactiveAsCash the two rules differ wherever iv holds a cell that is not finite. The rows of a fold are the idx that its PredictionResult records, so the method needs no timestamps. Under RenormaliseActive with no active mask, the values of the predictions are already the collapse of the rule, and the method keeps them bit for bit.

Algorithm

  1. Read the active mask of rd.pnl with panel_moment_masks, giving am. It is nothing with no panel or a static one.
  2. Return iv and ivpa when rd carries neither rd.iv nor a vector rd.ivpa, or when alg is RenormaliseActive and am is nothing.
  3. For each fold, normalise its weight matrix from fold_weight_matrix with synthetic_asset_weights, and collapse the rows idx of rd.iv and the vector rd.ivpa with collapse_rate, on the rows idx of the active mask.
  4. Stack the collapsed iv of the folds in fold order, and take ivpa from the last fold, as each prediction keeps the adjustment of its last fold.

Arguments

  • rd: The original ReturnsResult.
  • predictions: One MultiPeriodPredictionResult for each sub-portfolio.
  • u: Sub-portfolio enumeration, a SubPortfolioUniverse.
  • pred1: The folds of the first sub-portfolio, from assert_fold_alignment.
  • alg: The rule of the panel collapse, the pcol field of the meta-optimiser.
  • iv: The stacked implied volatilities of the predictions, observations × sub-portfolios, or nothing.
  • ivpa: The adjustments of the predictions, one for each sub-portfolio, or nothing.

Validation

  • Every fold of pred1 records its rows in idx, else IsNothingError.

Returns

  • (iv, ivpa): The implied volatilities and the adjustments of the outer problem.

Related

source
PortfolioOptimisers.rebuild_returns_result — Function
rebuild_returns_result(rd, predictions, u, alg)

Build the returns data of the outer problem from the cross-validation predictions of the sub-portfolios.

Column k of the result is the out-of-sample prediction of sub-portfolio k, and its rows are the rows of the folds, in fold order. u is the sub-portfolio enumeration: a ClusterUniverse for NestedClustered and a FullUniverse for Stacking. It states where the weights of a fold sit on the asset axis.

u is positional and has no default. With a full-universe default, a two-argument call still runs. That is correct for Stacking, but for NestedClustered it writes the weights of every cluster to the wrong rows and gives a wrong Asset Panel with no error. alg is the rule of the panel collapse, and it has no default for the same reason: the fold-less path reads the rule of the meta-optimiser, and this path must read the same one.

The folds carry no Asset Panel. The method collapses the original rd.pnl again for each fold, with the same collapse_asset_panel call as prepare_outer_rd makes on the fold-less path, and stacks the fold results along the observation axis (see rebuild_asset_panel). The stack has the assets × features × observations shape of a time-varying panel, and a FeatureDistance with its default LastRow reads the collapse of the last fold. The inner solves still see the panel of their own cluster. The collapse of the original panel also serves a square panel under NestedClustered. Its folds see the returns of one cluster each, so their own panels have different feature axes and do not stack.

Algorithm

  1. Copy the returns X, and B and iv when present, of the first prediction. Wrap its ivpa in a vector when present. Each prediction keeps the ivpa of its last fold as one number.
  2. Check the folds with assert_fold_alignment, giving pred1.
  3. Append the returns, iv and B of every other prediction, and push its ivpa.
  4. Reshape the returns to observations × sub-portfolios, giving X.
  5. Check that the folds cover size(X, 1) observations, nobs.
  6. Build the Asset Panel with rebuild_asset_panel under alg, giving pnl.
  7. Reshape B and iv to observations × sub-portfolios, and name the benchmark columns _b1, _b2, ….
  8. Collapse iv and ivpa again under alg with rebuild_fold_rates.
  9. Build the ReturnsResult, with the asset names _1, _2, … and the factors and timestamps of the first prediction.

The method reads predictions and does not change them, so two calls on one vector give the same result.

Arguments

Validation

Returns

  • The returns data of the outer problem, one synthetic asset for each sub-portfolio.

Related

source
PortfolioOptimisers.sub_portfolio_predictions — Function
sub_portfolio_predictions(::Type{T}, opti, u, rd, cv, ex) where {T}

Cross-validate every sub-portfolio in parallel, over the same returns data.

Every sub-portfolio runs the same cross-validation over the same returns data, which is the condition that assert_fold_alignment checks.

Algorithm

  1. Allocate one slot of type T for each of the sub_portfolio_count sub-portfolios, giving predictions.
  2. For each sub-portfolio i, in parallel under ex, give it its own scheme with sub_portfolio_cv and write the result of sub_portfolio_predict to predictions[i].

Arguments

Returns

  • One prediction result for each sub-portfolio.

Related

source
PortfolioOptimisers.outer_returns_source — Function
outer_returns_source(pr::AbstractPriorResult, rd::ReturnsResult,
                     x_src::Option{<:AbstractReturnsSource})
outer_returns_source(pr::AbstractPriorResult, rd::ReturnsResult, x_src::ExcessSource)

Return the returns on which the outer problem of a meta-optimiser scores its sub-portfolios, in the raw frame of rd.

The outer problem reads the raw returns of the sub-portfolios, as a cross-validated prediction reads them, and the readers of the outer optimiser state their own sources. The returns of the Prior Result are in the frame that the prior of the meta-optimiser was fitted in, and the Prior Result records that frame, see prior_returns_source. When it records ExcessSource(), the frame is the excess over the benchmark, so this function adds the benchmark back. The source comes from the Result and not from the estimator, because three paths put a Result in place of the prior estimator: the call with no data on an online fold, a prior step of a Pipeline, and a Prior Result that the caller computed. A Prior Result that a returns-matrix method of prior fitted records no source, so the outer problem reads it as raw.

Mathematical definition

\[\begin{align} \mathbf{S} &= \begin{cases} \mathbf{X}_{\mathcal{P}} & \text{the prior reads the raw returns}\,,\\ \mathbf{X}_{\mathcal{P}} + \mathbf{B} & \text{the prior states the Excess Frame}\,. \end{cases} \end{align}\]

Where:

  • $\mathbf{S}$: Returns that the sub-portfolios are scored on, observations × assets.
  • $\mathbf{X}_{\mathcal{P}}$: Returns of the Prior Result, pr.X.
  • $\mathbf{B}$: Benchmark returns of rd. A vector benchmark adds the same value to each asset of an observation.

Arguments

  • pr: The Prior Result of the meta-optimiser.
  • rd: The returns data of the meta-optimiser, on the observations of pr.
  • x_src: The source that pr records, see prior_returns_source.

Returns

  • The matrix $\mathbf{S}$ when x_src is ExcessSource, and pr otherwise.

Related

source
PortfolioOptimisers.predict_outer_returns — Function
predict_outer_returns(cv::Nothing, opt, u::SubPortfolioUniverse, rd::ReturnsResult,
                      pr::AbstractPriorResult, fees::Option{<:Fees}, wi::MatNum,
                      resi::VecOpt)
predict_outer_returns(cv::OptCVE_Onl, opt, u::SubPortfolioUniverse,
                      rd::ReturnsResult, pr::AbstractPriorResult,
                      fees::Option{<:Fees}, wi::MatNum, resi::VecOpt)
predict_outer_returns(cv::CombinatorialCrossValidation, opt,
                      u::SubPortfolioUniverse, rd::ReturnsResult,
                      pr::AbstractPriorResult, fees::Option{<:Fees}, wi::MatNum,
                      resi::VecOpt)

Predict the returns of the sub-portfolios of a meta-optimiser as the ReturnsResult of the outer problem.

One module serves every meta-optimiser that owns an outer optimiser. The sub-portfolio enumeration is the only part that changes between them, and it arrives as u. NestedClustered passes a ClusterUniverse and Stacking a FullUniverse. No method reads the type of the meta-optimiser.

The methods dispatch on cv, which selects the prediction, so a custom cross-validation scheme is a method on the first argument:

  • Without folds, the column of each sub-portfolio is the net return of its own solve in resi, on the Prior Result and the Fees viewed onto it with sub_portfolio_view.
  • A scheme of OptCVE_Onl predicts each sub-portfolio out of sample and stacks the folds with rebuild_returns_result. An Online Scheme steps each sub-portfolio from fold to fold and does not refit it.
  • A CombinatorialCrossValidation does the same, and first the scorer of the meta-optimiser selects one path from the population of each sub-portfolio. When scorer is nothing, NearestQuantilePrediction selects it.

wi holds the weights of the inner optimisers alone. A Combination Weight acts at the combination after the outer solve, so no method applies it here (see combination_weights).

Mathematical definition

Without folds:

\[\begin{align} R_{tk} &= \boldsymbol{x}_t^\intercal \mathbf{W}_{\cdot k}\,. \end{align}\]

Where:

  • $R_{tk}$: Return of synthetic asset $k$ at observation $t$, before Fees.
  • $\boldsymbol{x}_t$: Asset returns for observation $t$, the $t$-th row of the returns matrix.
  • $\mathbf{W}$: Inner weight matrix assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio.

With Fees, calc_net_returns subtracts the fee of sub-portfolio $k$ from its column.

Algorithm

Without folds:

  1. View rd onto the observations the Prior Result answers on with outer_prior_rows. Prepare the returns data of the outer problem from it with prepare_outer_rd under the rule opt.pcol, giving nb, B, iv, ivpa, pnl and the buffer X.
  2. Take the returns that the sub-portfolios are scored on with outer_returns_source under the source that pr records, giving src. It is the Prior Result, or the returns of the Prior Result with the benchmark added back when pr records x_src = ExcessSource().
  3. For each inner result res in resi, write its net returns with calc_net_returns to column i of X, on src and Fees viewed onto sub-portfolio i.
  4. Build the ReturnsResult, with the asset names _1, _2, … and the factors and timestamps of rd.

With cross-validation:

  1. Cross-validate every sub-portfolio with sub_portfolio_predictions under the scheme cv, giving predictions.
  2. On the combinatorial path, select one path of each population with opt.scorer, or with NearestQuantilePrediction when it is nothing.
  3. Stack the predictions with rebuild_returns_result under the rule opt.pcol.

Arguments

  • cv: The cross-validation scheme of the meta-optimiser, or nothing.
  • opt: The meta-optimiser. Every method reads its pcol field, the rule of the panel collapse. The method without folds also reads the source of its prior estimator pe. The cross-validated methods read its opti and ex fields, and the combinatorial method reads its scorer field.
  • u: Sub-portfolio enumeration.
  • rd: Returns data.
  • pr: Prior Result over the whole universe.
  • fees: Fees over the whole universe.
  • wi: Inner weights, assets × sub-portfolios.
  • resi: The results of the inner optimisations.

Returns

  • The ReturnsResult of the outer problem, one synthetic asset for each sub-portfolio.

Related

source
PortfolioOptimisers.assert_meta_scorer — Function
assert_meta_scorer(cv::Option{<:OptCVE_Onl},
                   scorer::Option{<:PredictionCrossValScorer})

Refuse a scorer that the cross-validation scheme of a meta-optimiser never reads.

predict_outer_returns reads the scorer on the combinatorial path alone, where it selects one path from the population of each sub-portfolio. Under any other scheme, and with no scheme, a scorer has no effect. So the constructors of NestedClustered and Stacking refuse it, and the user learns at construction that the configuration does nothing.

Validation

  • isnothing(scorer) || isa(cv, CombinatorialCrossValidation). The function throws an ArgumentError otherwise.

Arguments

  • cv: The cv field of the meta-optimiser.
  • scorer: The scorer field of the meta-optimiser.

Returns

  • nothing.

Related

source