GitHub

Base Cross Validation (a): private API

PortfolioOptimisers.mapreduce_RetMtx — Function
mapreduce_RetMtx(rd, sym = :X)

Concatenate the series of a field over the folds of a path.

A field holds one series for a single portfolio, or one series per member for a population. The folds are concatenated in the order of rd.

Algorithm

  1. For one series per fold, concatenate the series of every fold with vcat.
  2. For one series per member, concatenate the series of member i over every fold, for each member i of the first fold.

Arguments

Returns

  • The concatenated series, or one concatenated series per member.
source
PortfolioOptimisers.quantile_by_measure — Function
quantile_by_measure(ppred::PopulationPredictionResult, r::BaseRM_VecBaseRM, q::Real;
                    r_kwargs::NamedTuple = (;), q_kwargs::NamedTuple = (;),
                    sign::Integer = 1)

Select the successful path in ppred whose expected risk under r is closest to the q-th quantile of the risk distribution across all successful paths.

A path whose measure is not finite takes no part. It is dropped before the quantile, so one failed path cannot make the quantile non-finite and cannot be the answer.

sort_by_measure calls bigger_is_better for its direction, so it throws on a vector of measures that disagree on polarity. This function takes an explicit sign instead. The caller gives the one fact that bigger_is_better cannot infer, so a mixed vector is admitted.

Algorithm

  1. Keep the successful members of ppred with successful_members.
  2. Compute sign times the expected risk of each member under r, giving rks.
  3. Drop the members whose rks is not finite.
  4. Compute the q-th quantile of rks with Statistics.quantile, giving rkq.
  5. Return the first member whose abs(rks[i] - rkq) is the smallest.

Arguments

  • ppred::PopulationPredictionResult: Population prediction result.
  • r::BaseRM_VecBaseRM: Risk measure for computing path risks, or a vector of them. A vector is scalarised by r_kwargs.sca, defaulting to SumScalariser.
  • q::Real: Quantile level in [0, 1].
  • r_kwargs::NamedTuple = (;): Keyword arguments forwarded to expected_risk.
  • q_kwargs::NamedTuple = (;): Keyword arguments forwarded to Statistics.quantile.
  • sign::Integer = 1: Orientation of the risk scale. Use 1 when a larger risk is worse, -1 when it is better. This is what lets a mixed vector through, see below.

Validation

  • At least one successful member must have a finite measure. With none, Statistics.quantile receives an empty vector and throws.

Returns

Related

source
PortfolioOptimisers.reconstruct_rd — Method
reconstruct_rd(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult, X, hw = nothing, w = res.w)

Reconstruct a PredictionReturnsResult from an optimisation result and returns data.

It collapses the benchmark returns, the implied volatilities and the implied volatility risk premium adjustment of rd with the weights of the fold, and pairs them with the portfolio series X.

A benchmark matrix collapses as collapse_benchmark states: against the weight path when the fold carries a HeldWeightsResult, and against the target weights when it does not. iv and ivpa are rates, so they collapse as convex combinations, against the weights of synthetic_asset_weights.

The fold does not collapse the panel of the carrier. A square feature matrix needs the weights of every synthetic asset at once for its second contraction, and only one weight vector is known here. rebuild_returns_result instead computes the collapse for the whole synthetic universe from the original rd.pnl and the fold weights pred[f].res.w.

Algorithm

  1. Collapse rd.B with collapse_benchmark, from w and hw.
  2. When rd.iv is given or rd.ivpa is a vector, compute the convex weights cw of w with synthetic_asset_weights, one vector per member under a population.
  3. Collapse rd.iv to rd.iv * cw, and a vector rd.ivpa to dot(rd.ivpa, cw). A scalar ivpa stays as it is.
  4. Under a population, repeat a scalar ivpa once per member.
  5. Build the PredictionReturnsResult from nx, X, the factor block, nb, the collapsed values and ts.

Arguments

  • res::NonFiniteAllocationOptimisationResult: Fitted optimisation result.
  • rd::ReturnsResult: Original returns data.
  • X: Portfolio returns (vector or vector of vectors).
  • hw: Held Weights record of the fold, or nothing.
  • w: The weights the fold's series was formed from. It defaults to res.w, and a fold that viewed its window at the Investable Mask passes the view instead, so the collapse reads the same asset axis rd carries.

Returns

  • PredictionReturnsResult with updated benchmark returns, implied volatilities and implied volatility risk premium adjustment.

Related

source
PortfolioOptimisers.investable_fold_view — Function
investable_fold_view(imsk::Nothing, w, rd::ReturnsResult, fees::Option{<:Fees})
investable_fold_view(imsk::BitVector, w, rd::ReturnsResult, fees::Option{<:Fees})

View a fold's weights and test window at the Investable Mask, and pass its fees through.

An optimisation reduces to the Investable Mask at its entry and expands the solved weights back to the caller's universe, so the weight of an asset that the fit found non-investable is 0. The fold views the weights and the window at the mask together, before anything reads the window. A dead column is therefore never read, and filter_held_gaps runs over the investable columns alone.

The fees are not viewed. A result carries the objects of the universe it solved on, so res.fees is on the investable universe already, and a second view would index its per-asset rates by positions of the full universe. The function returns the fees unchanged so that the fold receives the three values it scores with from one call.

Algorithm

  1. With no mask, give (w, rd, fees) unchanged.
  2. With a mask, give the weights viewed at the mask with investable_weights_view, rd viewed at the columns findall(imsk) with port_opt_view, and fees unchanged.

Arguments

  • imsk: The Investable Mask, or nothing.
  • w: The fold's target weights, or a population of them.
  • rd: The returns result to use.
  • fees: Fees the fold is charged, on the investable universe, or nothing.

Returns

  • (w, rd, fees): The weights and the window reduced to the investable assets, or unchanged, and the fees as given.

Related

source
PortfolioOptimisers.fold_fees — Function
fold_fees(res::OptimisationResult, fees::Nothing, X::MatNum)
fold_fees(res::OptimisationResult, fees::Fees, X::MatNum)

Resolve the fee a fold-taking consumer charges: the result's own, or a caller's viewed at the result's Investable Mask.

The three fold-taking consumers, calc_net_asset_returns, risk_contribution and factor_risk_contribution on a PredictionResult, take an optional fees, so a caller can score a stored fold under a fee of their own. The two fees are on different universes. The fit reduced the fee of the result, so its five per-asset fields are on the investable axis and its two liquidation carriers are on the complement. A caller's fee is on the caller's universe, as a caller's rd is, so it takes the same view that a fee takes at the fit.

Without the view, a per-asset rate on the full universe met the reduced weights with a BoundsError, and a carrier on the full universe was charged as though every position had been liquidated.

Algorithm

  1. With no caller fee, give the fee of the result through extract_fees.
  2. With a caller fee, view it with investable_fees_view. The per-asset fields are sliced to the mask and the carriers to its complement, which is found from the width of X. The carriers are removed when the mask is nothing, because no asset left.

Arguments

  • res::OptimisationResult: The fold's optimisation result, carrying the mask and its own fee.
  • fees: A caller's Fees on the caller's universe, or nothing to charge the result's own.
  • X: The fold's expanded record, observations × assets on the caller's universe. Only its width is read.

Returns

  • fees::Option{<:Fees}: The fee on the two axes the result's mask leaves, or nothing.

Related

source
PortfolioOptimisers.threads_weights — Function
threads_weights(pws::Nothing, pred::PredictionResult)
threads_weights(pws::AbstractPreviousWeightsSource, pred::PredictionResult)

Say whether a fold's prediction carries weights the next fold can be handed.

previous_weights reads the weights, and this function says whether the fold has any to read. The sequential loops, run_folds and online_folds, hand a fold on only when this holds, so a failed fold is never read and the last threadable fold is read instead.

The weights that are read decide the test. The target weights are finite exactly when every return code is an OptimisationSuccess. The held weights are finite when the drift ran, and held_start_weights lets it run on a failed fold that was handed previous weights. So under a source, a failed fold that held its book is threaded, and only a fold with nothing to hold is skipped.

Algorithm

  1. With no source, hold when the fold's return code is a success, every member's under a population.
  2. With a source, hold when the fold's held weights are all finite, every member's under a population.

Arguments

  • pws: Previous-weights source of the scheme, or nothing.
  • pred: Prediction result of the fold.

Returns

  • Bool: Whether the fold's weights can be threaded.

Related

source
PortfolioOptimisers.held_start_weights — Function
held_start_weights(retcode::OptimisationSuccess, w::VecNum, w_prev)
held_start_weights(retcode::OptimisationFailure, w::VecNum, w_prev::Nothing)
held_start_weights(retcode::OptimisationFailure, w::VecNum, w_prev::VecNum)
held_start_weights(retcode::OptimisationReturnCode, w::VecVecNum, w_prev)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::Nothing)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecNum)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecVecNum)

Name the weights a fold's drift starts from: its own on a solved fold, the previous weights on a failed one.

A fold that could not rebalance holds what it held. So under a Weight Drift or a Previous-Weights Source, a failed fold drifts the weights it was handed and not its NaN target. The return code decides, read per member under a population, and the previous weights are one vector for every member or one vector per member. A population solved under one return code, such as the frontier of one JuMPOptimisationResult, reads that code for every member. A failed fold or member with no previous weights keeps its NaN weights, and held_weights_result records NaN for it without a drift.

Algorithm

  1. On an OptimisationSuccess, give w.
  2. On an OptimisationFailure over one vector, give w_prev, or w when there is none.
  3. Over a population with one return code, repeat that code once per member.
  4. Over a population with no w_prev, give w.
  5. Over a population with one w_prev vector, repeat it once per member.
  6. Over a population with one w_prev per member, give each member its own vector on a success and its entry of w_prev on a failure.

Arguments

  • retcode: Return code of the fold, or one per member of the population.
  • w: Target weights of the fold, on the universe the fold is scored on.
  • w_prev: Previous weights the fold was handed, on the same universe, or nothing.

Validation

  • A per-member w_prev has one entry per member of w, else a DimensionMismatch is raised.

Returns

Related

source
PortfolioOptimisers.fold_factor_returns — Function
fold_factor_returns(imsk, rd::Nothing, pred::PredictionResult)
fold_factor_returns(imsk, rd::ReturnsResult, pred::PredictionResult)

Resolve the returns result a fold's factor loadings are fitted from, on the live assets of the fold.

The fold-taking factor_risk_contribution reads rd by dispatch. A caller's rd is on the caller's universe, as it is for the value-level method, so it is viewed at the Investable Mask of the result. With nothing, the function builds a returns result from the fold itself. The nx of the fold is on the reduced axis already.

Algorithm

  1. On a caller's rd, view it at imsk with investable_returns_view.
  2. On nothing, view the asset returns hw.X of the fold at imsk, and build a ReturnsResult from them, the nx of the fold and the factor block nf, F that reconstruct_rd kept.

Arguments

  • imsk: The result's Investable Mask, or nothing.
  • rd: Returns result on the caller's universe, or nothing.
  • pred: Single-fold prediction result carrying a HeldWeightsResult.

Returns

  • rd::ReturnsResult: The returns result on the live assets.

Related

source
PortfolioOptimisers.collapse_benchmark — Method
collapse_benchmark(B::Nothing, w::VecNum_VecVecNum, hw)
collapse_benchmark(B::VecNum, w::VecNum, hw)
collapse_benchmark(B::VecNum, w::VecVecNum, hw)
collapse_benchmark(B::MatNum, w::VecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecNum, hw::HeldWeightsResult)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::HeldWeightsResult)

Collapse a fold's benchmark asset returns into a benchmark return series.

A benchmark that is already a series passes through. A benchmark matrix is contracted with the weights of the fold, and dispatch on the pair (B, w) chooses the method.

The Held Weights record of the fold chooses the weights. Without one, the matrix collapses against the target weights, which is what a fold that ran no drift held. With one, it collapses row by row against the weight path, as the portfolio series does. A caller that compares the two series, for a tracking error for instance, then compares two series formed the same way.

Algorithm

  1. On nothing, give nothing.
  2. On a benchmark series, give it back, repeated once per member under a population.
  3. On a matrix with no record, give B * w, once per member under a population.
  4. On a matrix with a record, give vec(sum(B ⊙ U; dims = 2)) for the fold's weight path U, once per member under a population.

Arguments

  • B: Benchmark returns of the fold: nothing, a series, or an observations × assets matrix.
  • w: Target weights of the fold, or a population of them.
  • hw: Held Weights record of the fold, or nothing.

Returns

  • The benchmark return series, or a vector of them under a population, or nothing.

Related

source
PortfolioOptimisers.ruined_retcodes — Function
ruined_retcodes(retcode::VecOptRetCode, ruined::VecInt)

Set the return code of every ruined member of a population to an OptimisationFailure.

The failure payload names the member and the reason, so a reader of res.retcode finds why the member left the run. A member that is not named keeps the code its own optimisation gave it.

Algorithm

  1. Walk the codes with their positions, and replace the code of a named member with a failure that states the reason.

Arguments

  • retcode: Return codes of the population, one per member.
  • ruined: Indices of the members whose drifted wealth is not positive.

Returns

  • VecOptRetCode: The codes, with the ruined members failed.

Related

source
PortfolioOptimisers.mark_ruined_members — Function
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::Nothing)
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::VecInt)

Rebuild a fold's optimisation result so that its ruined members carry a failure code.

A failure code is enough to drop the member. The library reads a vector of return codes with any(x -> isa(x, OptimisationFailure), …), and the cross-validation path keeps a path only when isa(y.res.retcode, OptimisationSuccess) holds, so a failed member takes its path out of the run.

Algorithm

  1. With no ruined member, and with an empty set of them, give res unchanged. Nothing is rebuilt on the ordinary path.
  2. Otherwise rebuild res through set_retcode, with the codes ruined_retcodes makes.

Arguments

  • res: Optimisation result of the fold.
  • ruined: Indices of the members whose drifted wealth is not positive, or nothing.

Returns

  • NonFiniteAllocationOptimisationResult: The result, rebuilt only when a member was ruined.

Related

source
PortfolioOptimisers.warn_ruined_members — Function
warn_ruined_members(wd::AbstractWeightDrift, args...)
warn_ruined_members(wd::Nothing, ruined::Nothing, n::Integer)
warn_ruined_members(wd::Nothing, ruined::VecInt, n::Integer)

Warn once when a drift dropped members of a population, and the return series did not.

When the series of a fold is drifted, calc_net_returns(w::VecVecNum, X::MatNum, fees, wd::AbstractWeightDrift, obs) runs the same drift over the same window and warns. A fold that drifts only its held weights runs no such call, so this function warns instead. Either way the fold warns once.

Algorithm

  1. With a drifted series, say nothing. The series already warned.
  2. With no ruined member, say nothing.
  3. Otherwise warn, and name the count and the members.

Arguments

  • wd: Weight drift of the scheme, or nothing.
  • ruined: Indices of the ruined members, or nothing.
  • n: Number of members of the population.

Returns

  • nothing.

Related

source
PortfolioOptimisers.cv_live_assets — Function
cv_live_assets(rd::AbstractReturnsResult) -> Any

Return the positions of the assets in the Coverage Universe of a cross-validation window.

This is the sibling of cv_nobs for the asset axis. A fold that draws a subset of assets draws it from these positions, so it never draws a column that could not have been traded.

The two data-level methods differ only in how they reach the numeric matrix. A returns carrier holds it directly, and a price carrier holds a TimeArray. The two mask-level methods take the nothing sentinel of an all-covered window and the mask of a window with gaps.

Algorithm

  1. Derive the Coverage Universe of the window's asset matrix and its panel with coverage_mask.
  2. Return every position of the asset axis on the nothing sentinel.
  3. Return findall(cmsk) otherwise.

Arguments

  • data: Returns-level or price-level data (Prices_RR).
  • cmsk: The Coverage Universe, or nothing.
  • N: The number of assets of the window.

Validation

  • At least one asset must be in the Coverage Universe of the window. coverage_mask throws an IsEmptyError on a window in which every asset is dead.

Returns

  • live::Vector{Int}: The positions of the covered assets, in increasing order.

Related

source