Base Cross Validation (a): private API

PortfolioOptimisers.NonOptimisationCrossValidationEstimator — Type
abstract type NonOptimisationCrossValidationEstimator <: CrossValidationEstimator

Abstract supertype for the cross-validation schemes that the cv field of a meta-optimiser does not take.

The cv field of NestedClustered and Stacking is bound to OptCVE_Onl, so a scheme of this branch reaches an optimiser through cross_val_predict. MultipleRandomised is the one member. It draws random asset subsets, applies an inner walk-forward to each, and backtests an optimiser on every path.

Related

source
PortfolioOptimisers.mapreduce_RetMtx — Function
mapreduce_RetMtx(rd, sym = :X)

Concatenate the series of a field over the folds of a path.

A field holds one series for a single portfolio, or one series per member for a population. The folds are concatenated in the order of rd.

Algorithm

  1. For one series per fold, concatenate the series of every fold with vcat.
  2. For one series per member, concatenate the series of member i over every fold, for each member i of the first fold.

Arguments

Returns

  • The concatenated series, or one concatenated series per member.
source
PortfolioOptimisers.quantile_by_measure — Function
quantile_by_measure(sca::Option{<:Sca_VecToScaM}, ppred::PopulationPredictionResult,
                    r::BaseRM_VecBaseRM, q::Real; r_kwargs::NamedTuple = (;),
                    q_kwargs::NamedTuple = (;))

Select the successful path in ppred whose risk under r is closest to the q-th quantile of the risks across all successful paths.

The risk of a path is its negated oriented_score, so a larger risk is worse whatever the orientation of r. q = 0 selects the best path and q = 1 the worst. The score orients each measure before it reduces a vector, so a vector of measures that disagree on orientation is admitted.

A path whose measure is not finite takes no part. It is dropped before the quantile, so one failed path cannot make the quantile non-finite and cannot be the answer.

Algorithm

  1. Check r_kwargs with assert_path_reduction.
  2. Keep the successful members of ppred with successful_members.
  3. Compute the negated oriented score of each member under r and sca with oriented_score, giving rks.
  4. Drop the members whose rks is not finite.
  5. Compute the q-th quantile of rks with Statistics.quantile, giving rkq.
  6. Return the first member whose abs(rks[i] - rkq) is the smallest.

Arguments

  • sca: Reduction of the oriented scores of a vector of risk measures in r to one score for each path, a Sca_VecToScaM, or nothing, which gives the sum of SumScalariser. Inert when r holds one measure.
  • ppred::PopulationPredictionResult: Population prediction result.
  • r::BaseRM_VecBaseRM: Risk measure for computing path risks, or a vector of them.
  • q::Real: Quantile level in [0, 1].
  • r_kwargs::NamedTuple = (;): Keyword arguments forwarded to oriented_score. It must not contain sca.
  • q_kwargs::NamedTuple = (;): Keyword arguments forwarded to Statistics.quantile.

Validation

  • r_kwargs must not contain sca, through assert_path_reduction.
  • At least one successful member must have a finite measure. With none, Statistics.quantile receives an empty vector and throws.

Returns

Related

source
PortfolioOptimisers.reconstruct_rd — Method
reconstruct_rd(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult, X, hw = nothing, w = res.w)

Reconstruct a PredictionReturnsResult from an optimisation result and returns data.

It collapses the benchmark returns, the implied volatilities and the implied volatility risk premium adjustment of rd with the weights of the fold, and pairs them with the portfolio series X.

A benchmark matrix collapses as collapse_benchmark states: against the weight path when the fold carries a HeldWeightsResult, and against the target weights when it does not. iv and ivpa are rates, so they collapse as convex combinations, against the weights of synthetic_asset_weights.

The fold does not collapse the panel of rd. A square feature matrix needs the weights of every synthetic asset at once for its second contraction, and only one weight vector is known here. rebuild_returns_result instead computes the collapse for the whole synthetic universe from the original rd.pnl and the fold weights pred[f].res.w.

Algorithm

  1. Collapse the cells of rd.B that benchmark_cells reads with collapse_benchmark, from w and hw. A gap in the benchmark or in the returns, or an inactive cell, gives zero, as the portfolio series zeroes it.
  2. When rd.iv is given or rd.ivpa is a vector, compute the convex weights cw of w with synthetic_asset_weights, one vector per member under a population.
  3. Collapse rd.iv and a vector rd.ivpa with collapse_fold_rate, which reads the active mask of rd.pnl and the finite cells, as the collapse of a meta-optimiser does. A scalar ivpa stays as it is.
  4. Under a population, repeat a scalar ivpa once per member.
  5. Build the PredictionReturnsResult from nx, X, the factor block, nb, the collapsed values and ts.

Arguments

  • res::NonFiniteAllocationOptimisationResult: Fitted optimisation result.
  • rd::ReturnsResult: Original returns data.
  • X: Portfolio returns (vector or vector of vectors).
  • hw: Held Weights record of the fold, or nothing.
  • w: The weights the fold's series was formed from. It defaults to res.w, and a fold that viewed its window at the Investable Mask passes the view instead, so the collapse reads the same asset axis rd carries.

Returns

  • PredictionReturnsResult with updated benchmark returns, implied volatilities and implied volatility risk premium adjustment.

Related

source
PortfolioOptimisers.investable_fold_view — Function
investable_fold_view(imsk::Nothing, w, rd::ReturnsResult, fees::Option{<:Fees})
investable_fold_view(imsk::BitVector, w, rd::ReturnsResult, fees::Option{<:Fees})

View a fold's weights and test window at the Investable Mask, and pass its fees through.

An optimisation reduces to the Investable Mask at its entry and expands the solved weights back to the caller's universe, so the weight of an asset that the fit found non-investable is 0. The fold views the weights and the window at the mask together, before anything reads the window. A dead column is therefore never read, and filter_held_gaps runs over the investable columns alone.

The fees are not viewed. A result carries the objects of the universe it solved on, so res.fees is on the investable universe already, and a second view would index its per-asset rates by positions of the full universe. The function returns the fees unchanged so that the fold receives the three values it scores with from one call.

Algorithm

  1. With no mask, give (w, rd, fees) unchanged.
  2. With a mask, give the weights viewed at the mask with investable_weights_view, rd viewed at the columns findall(imsk) with port_opt_view, and fees unchanged.

Arguments

  • imsk: The Investable Mask, or nothing.
  • w: The fold's target weights, or a population of them.
  • rd: The returns result to use.
  • fees: Fees the fold is charged, on the investable universe, or nothing.

Returns

  • (w, rd, fees): The weights and the window reduced to the investable assets, or unchanged, and the fees as given.

Related

source
PortfolioOptimisers.fold_fees — Function
fold_fees(res::OptimisationResult, fees::Nothing, X::MatNum)
fold_fees(res::OptimisationResult, fees::Fees, X::MatNum)

Resolve the fee a fold-taking consumer charges: the result's own, or a caller's viewed at the result's Investable Mask.

The three fold-taking consumers, calc_net_asset_returns, risk_contribution and factor_risk_contribution on a PredictionResult, take an optional fees, so a caller can score a stored fold under a fee of their own. The two fees are on different universes. The fit reduced the fee of the result, so its five per-asset fields are on the investable axis and its two liquidation charges, lq and flq, are on the complement. A caller's fee is on the caller's universe, as a caller's rd is, so it takes the same view that a fee takes at the fit.

Without the view, a per-asset rate on the full universe met the reduced weights with a BoundsError, and a liquidation charge on the full universe charged every position as though it had been liquidated.

Algorithm

  1. With no caller fee, give the fee of the result through extract_fees.
  2. With a caller fee, view it with investable_fees_view. The per-asset fields are sliced to the mask and the two liquidation charges to its complement, which is found from the width of X. The two liquidation charges are removed when the mask is nothing, because no asset left.

Arguments

  • res::OptimisationResult: The fold's optimisation result, carrying the mask and its own fee.
  • fees: A caller's Fees on the caller's universe, or nothing to charge the result's own.
  • X: The fold's expanded record, observations × assets on the caller's universe. Only its width is read.

Returns

  • fees::Option{<:Fees}: The fee on the two axes the result's mask leaves, or nothing.

Related

source
PortfolioOptimisers.threads_weights — Function
threads_weights(pws::Nothing, pred::PredictionResult)
threads_weights(pws::AbstractPreviousWeightsSource, pred::PredictionResult)

Say whether a fold's prediction carries weights the next fold can be handed.

previous_weights reads the weights, and this function says whether the fold has any to read. The sequential loops, run_folds and online_folds, hand a fold on only when this holds, so a failed fold is never read and the last threadable fold is read instead.

The weights that are read decide the test. The target weights are finite exactly when every return code is an OptimisationSuccess. The held weights are finite when the drift ran, and held_start_weights lets it run on a failed fold that was handed previous weights. So under a source, a failed fold that held its book is threaded, and only a fold with nothing to hold is skipped.

Algorithm

  1. With no source, hold when the fold's return code is a success, every member's under a population.
  2. With a source, hold when the fold's held weights are all finite, every member's under a population.

Arguments

  • pws: Previous-weights source of the scheme, or nothing.
  • pred: Prediction result of the fold.

Returns

  • Bool: Whether the fold's weights can be threaded.

Related

source
PortfolioOptimisers.held_start_weights — Function
held_start_weights(retcode::OptimisationSuccess, w::VecNum_VecVecNum, w_prev)
held_start_weights(retcode::OptimisationFailure, w::VecNum_VecVecNum, w_prev::Nothing)
held_start_weights(retcode::OptimisationFailure, w::VecNum, w_prev::VecNum)
held_start_weights(retcode::OptimisationFailure, w::VecVecNum, w_prev::VecNum)
held_start_weights(retcode::OptimisationFailure, w::VecVecNum, w_prev::VecVecNum)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::Nothing)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecNum)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecVecNum)

Name the weights a fold's drift starts from: its own on a solved fold, the previous weights on a failed one.

A fold that could not rebalance holds what it held. So under a Weight Drift or a Previous-Weights Source, a failed fold drifts the weights it was handed and not its NaN target. The return code decides, read per member under a population, and the previous weights are one vector for every member or one vector per member. A population solved under one return code, such as the frontier of one JuMPOptimisationResult, reads that code for every member. A failed fold or member with no previous weights keeps its NaN weights, and held_weights_result records NaN for it without a drift.

Algorithm

The method that Julia selects is the algorithm. A population with one return code takes the same answer for every member, so its methods agree with the per-member methods under that code repeated once per member.

  1. On an OptimisationSuccess, give w, one vector or a population.
  2. With no w_prev, give w.
  3. On an OptimisationFailure over one vector, give w_prev.
  4. On an OptimisationFailure over a population, give w_prev to each member: the one vector repeated once per member, or a copy of the vector of one w_prev per member.
  5. Over a population with one return code per member, give each member its own vector on a success, and on a failure the one w_prev vector or its own entry of w_prev.

Arguments

  • retcode: Return code of the fold, or one per member of the population.
  • w: Target weights of the fold, on the universe the fold is scored on.
  • w_prev: Previous weights the fold was handed, on the same universe, or nothing.

Validation

  • A per-member w_prev that a failure reads has one entry per member of w, else a DimensionMismatch is raised. A success reads no w_prev, so it does not check it.
  • A vector of return codes with one w_prev vector has one code per member of w, else a DimensionMismatch is raised.

Returns

Related

source
PortfolioOptimisers.fold_factor_returns — Function
fold_factor_returns(imsk, rd::Nothing, pred::PredictionResult)
fold_factor_returns(imsk, rd::ReturnsResult, pred::PredictionResult)

Resolve the returns result a fold's factor loadings are fitted from, on the live assets of the fold.

The fold-taking factor_risk_contribution reads rd by dispatch. A caller's rd is on the caller's universe, as it is for the value-level method, so it is viewed at the Investable Mask of the result. With nothing, the function builds a returns result from the fold itself. The nx of the fold is on the reduced axis already.

Algorithm

  1. On a caller's rd, view it at imsk with investable_returns_view.
  2. On nothing, view the asset returns hw.X of the fold at imsk, and build a ReturnsResult from them, the nx of the fold, the factor block nf, F, the Exogenous Series ne, E and the timestamps ts that reconstruct_rd kept. The benchmark and the implied volatilities of the fold are on the portfolio axis, so they do not line up with hw.X and the result does not carry them.

Arguments

  • imsk: The result's Investable Mask, or nothing.
  • rd: Returns result on the caller's universe, or nothing.
  • pred: Single-fold prediction result carrying a HeldWeightsResult.

Returns

  • rd::ReturnsResult: The returns result on the live assets.

Related

source
PortfolioOptimisers.collapse_benchmark — Method
collapse_benchmark(B::Nothing, w::VecNum_VecVecNum, hw)
collapse_benchmark(B::VecNum, w::VecNum, hw)
collapse_benchmark(B::VecNum, w::VecVecNum, hw)
collapse_benchmark(B::MatNum, w::VecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecNum, hw::HeldWeightsResult)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::HeldWeightsResult)

Collapse a fold's benchmark asset returns into a benchmark return series.

A benchmark that is already a series passes through. A benchmark matrix is contracted with the weights of the fold, and dispatch on the pair (B, w) chooses the method.

The Held Weights record of the fold chooses the weights. Without one, the matrix collapses against the target weights, which is what a fold that ran no drift held. With one, it collapses row by row against the weight path, as the portfolio series does. A caller that compares the two series, for a tracking error for instance, then compares two series formed the same way.

Algorithm

  1. On nothing, give nothing.
  2. On a benchmark series, give it back, repeated once per member under a population.
  3. On a matrix with no record, give B * w, once per member under a population.
  4. On a matrix with a record, give vec(sum(B ⊙ U; dims = 2)) for the fold's weight path U, once per member under a population.

Arguments

  • B: Benchmark returns of the fold: nothing, a series, or an observations × assets matrix.
  • w: Target weights of the fold, or a population of them.
  • hw: Held Weights record of the fold, or nothing.

Returns

  • The benchmark return series, or a vector of them under a population, or nothing.

Related

source
PortfolioOptimisers.ruined_retcodes — Function
ruined_retcodes(retcode::VecOptRetCode, ruined::VecInt)

Set the return code of every ruined member of a population to an OptimisationFailure.

The failure payload names the member and the reason, so a reader of res.retcode finds why the member left the run. A member that is not named keeps the code its own optimisation gave it.

Algorithm

  1. Walk the codes with their positions, and replace the code of a named member with a failure that states the reason.

Arguments

  • retcode: Return codes of the population, one per member.
  • ruined: Indices of the members whose drifted wealth is not positive.

Returns

  • VecOptRetCode: The codes, with the ruined members failed.

Related

source
PortfolioOptimisers.mark_ruined_members — Function
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::Nothing)
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::VecInt)

Rebuild a fold's optimisation result so that its ruined members carry a failure code.

A failure code is enough to drop the member. The library reads a vector of return codes with any(x -> isa(x, OptimisationFailure), …), and the cross-validation path keeps a path only when isa(y.res.retcode, OptimisationSuccess) holds, so a failed member takes its path out of the run.

Algorithm

  1. With no ruined member, and with an empty set of them, give res unchanged. Nothing is rebuilt on the ordinary path.
  2. Otherwise rebuild res through set_retcode, with the codes ruined_retcodes makes.

Arguments

  • res: Optimisation result of the fold.
  • ruined: Indices of the members whose drifted wealth is not positive, or nothing.

Returns

  • NonFiniteAllocationOptimisationResult: The result, rebuilt only when a member was ruined.

Related

source
PortfolioOptimisers.warn_ruined_members — Function
warn_ruined_members(wd::AbstractWeightDrift, args...)
warn_ruined_members(wd::Nothing, ruined::Nothing, n::Integer)
warn_ruined_members(wd::Nothing, ruined::VecInt, n::Integer)

Warn once when a drift dropped members of a population, and the return series did not.

When the series of a fold is drifted, calc_net_returns(w::VecVecNum, X::MatNum, fees, wd::AbstractWeightDrift, obs) runs the same drift over the same window and warns. A fold that drifts only its held weights runs no such call, so this function warns instead. Either way the fold warns once.

Algorithm

  1. With a drifted series, say nothing. The series already warned.
  2. With no ruined member, say nothing.
  3. Otherwise warn, and name the count and the members.

Arguments

  • wd: Weight drift of the scheme, or nothing.
  • ruined: Indices of the ruined members, or nothing.
  • n: Number of members of the population.

Returns

  • nothing.

Related

source
PortfolioOptimisers.cv_live_assets — Function
cv_live_assets(rd::AbstractReturnsResult) -> Any

Return the positions of the assets in the Coverage Universe of a cross-validation window.

This is the sibling of cv_nobs for the asset axis. A fold that draws a subset of assets draws it from these positions, so it never draws a column that could not have been traded.

The two data-level methods differ only in how they reach the numeric matrix. Returns data holds it directly, and price data holds a TimeArray. The two mask-level methods take the nothing sentinel of an all-covered window and the mask of a window with gaps.

Algorithm

  1. Derive the Coverage Universe of the window's asset matrix and its panel with coverage_mask.
  2. Return every position of the asset axis on the nothing sentinel.
  3. Return findall(cmsk) otherwise.

Arguments

  • data: Returns-level or price-level data (Prices_RR).
  • cmsk: The Coverage Universe, or nothing.
  • N: The number of assets of the window.

Validation

  • At least one asset must be in the Coverage Universe of the window. coverage_mask throws an IsEmptyError on a window in which every asset is dead.

Returns

  • live::Vector{Int}: The positions of the covered assets, in increasing order.

Related

source