GitHub

Base Cross Validation: private API

PortfolioOptimisers.Fold — Type
struct Fold{T1, T2, T3, T4, T5, T6, T7}

One fold of a cross-validation scheme, as fold_loop hands it to its callback.

The record is the fold loop's whole hand-off. est and rd are already resolved: the asset view is taken, every TimeDependent schedule is swapped for its fold-i value, and the previous fold's weights are threaded in. train and test are this fold's own windows, so a callback never indexes train_idx/test_idx itself. w_prev is the weights that were threaded, handed over a second time so a fold whose solve fails can hold them: held_start_weights reads it inside predict.

train === nothing says the estimator holds its window. The online arm of the loop, online_folds, hands its callback a Fold of that shape: est has already folded every row of the fold's training window through partial_fit!, so a callback reads it out — optimise(est) with no returns — rather than fitting it over rows the record does not carry. fit_and_predict takes train_idx = nothing for exactly that, so every entry point's callback is unchanged across the two arms.

The type is immutable and every field is concretely typed at the construction site, so the record costs nothing at run time. fold_loop is the only site that builds one, which is why there is no keyword constructor.

Fields

  • i: Index of the fold within the scheme's split enumeration (1-based).
  • n: Number of folds in the enumeration.
  • est: The fold's resolved estimator: asset-viewed, schedule-swapped, weights-threaded.
  • rd: The fold's (possibly asset-viewed) input data.
  • train: The fold's training indices, or nothing when the estimator holds its window.
  • test: The fold's test indices.
  • w_prev: The previous weights threaded into est, or nothing when there are none.

Related

source
PortfolioOptimisers.mapreduce_RetMtx — Function
mapreduce_RetMtx(rd, sym = :X)

Concatenate return matrices from a vector of PredictionReturnsResult objects.

Internal helper that vertically concatenates the field sym across all elements of rd. Handles both single-asset (vector) and multi-asset (vector of vectors) return data.

Arguments

Returns

  • Concatenated return matrix or vector of vectors.
source
PortfolioOptimisers.quantile_by_measure — Function
quantile_by_measure(ppred::PopulationPredictionResult, r::BaseRM_VecBaseRM, q::Real;
                    r_kwargs::NamedTuple = (;), q_kwargs::NamedTuple = (;),
                    sign::Integer = 1)

Select the successful path in ppred whose expected risk under r is closest to the q-th quantile of the risk distribution across all successful paths.

Arguments

  • ppred::PopulationPredictionResult: Population prediction result.
  • r::BaseRM_VecBaseRM: Risk measure for computing path risks, or a vector of them. A vector is scalarised by r_kwargs.sca, defaulting to SumScalariser.
  • q::Real: Quantile level in [0, 1].
  • r_kwargs::NamedTuple = (;): Keyword arguments forwarded to expected_risk.
  • q_kwargs::NamedTuple = (;): Keyword arguments forwarded to Statistics.quantile.
  • sign::Integer = 1: Orientation of the risk scale. Use 1 when a larger risk is worse, -1 when it is better. This is what lets a mixed vector through, see below.

Returns

The quantile is taken over the finite members

A path whose measure is not finite takes no part: it is dropped before the quantile, so one failed path cannot make the quantile itself non-finite and cannot be returned as the answer. A population in which no member has a finite measure reaches Statistics.quantile with an empty vector, which refuses.

A mixed vector is accepted here, and refused by its sibling

sort_by_measure calls bigger_is_better for its rev flag, so it throws on a vector whose elements disagree on polarity. This function takes an explicit sign instead, so the caller has already supplied the one thing bigger_is_better cannot infer, and a mixed vector is admitted.

Related

source
PortfolioOptimisers.fit_and_predict — Function
fit_and_predict(opt, rd::ReturnsResult, cv::NonSeqCVER; cols, ex, id) -> MultiPeriodPredictionResult
fit_and_predict(opt, rd::ReturnsResult, cv::CombCVER; cols, ex) -> PopulationPredictionResult
fit_and_predict(opt, rd::ReturnsResult; train_idx = nothing, test_idx, cols) -> PredictionResult
fit_and_predict(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult; test_idx, cols) -> PredictionResult

Fit an optimisation estimator on training data and predict on test data using cross-validation.

The three-argument method (opt, rd, cv) performs full cross-validated prediction over all folds of cv. The two-argument methods operate on a single pre-defined train/test split or on a pre-existing result.

The estimator form takes train_idx = nothing to mean the estimator holds its window: it reads the estimator out through optimise(opt) instead of fitting it over port_opt_view(rd, train_idx, cols), and predicts over test_idx as before. That is the read-out of the online arm of fold_loop, and it is also a public entry for a hand-stepped estimator — one warmed up with update_online_estimator and folded with partial_fit! — so fit_and_predict(opt, rd; test_idx) on a stepped estimator equals fit_and_predict(opt, rd; train_idx, test_idx) on the cold one over the same rows. The two arms are fit_fold_result's, and the method lives beside them.

Arguments

  • opt: Optimisation estimator or an existing optimisation result.
  • rd::ReturnsResult: Returns data.
  • cv::NonSeqCVER: Non-sequential cross-validation estimator (e.g. KFold or CombinatorialCrossValidation).
  • cv::CombCVER: Combinatorial cross-validation estimator or result (CombinatorialCrossValidation).
  • train_idx::Option{<:VecInt}: Training indices, or nothing to read a stepped estimator out.
  • test_idx: Test indices (vector or vector of vectors).
  • cols: Column selector (default : for all assets).

Returns

Details

Related

source
PortfolioOptimisers.reconstruct_rd — Method
reconstruct_rd(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult, X, hw = nothing, w = res.w)

Reconstruct a PredictionReturnsResult from an optimisation result and returns data.

Computes the benchmark returns, the implied volatilities and the implied volatility risk premium adjustment from the optimisation result weights and the original returns data.

The benchmark collapse follows the fold's weight-drift setting whenever rd.B is a matrix. A fold that carries a HeldWeightsResult collapses the matrix row by row against its weight path, the same convention its portfolio series is scored under; a fold that carries none collapses it against the target weights, as before. collapse_benchmark is the verb, and it reads the pair by dispatch.

No feature matrix

The fold does not collapse the carrier's panel. Only one weight vector is in scope here, which is not enough to collapse a square feature matrix — the second contraction needs every synthetic asset's weights at once — so a fold-side collapse could only serve one of the two shapes, and the two paths into the outer problem would disagree on what a square carrier means. rebuild_returns_result instead recomputes the collapse for the whole synthetic universe at once, from the original rd.pnl and the fold weights it reaches through pred[f].res.w.

Arguments

  • res::NonFiniteAllocationOptimisationResult: Fitted optimisation result.
  • rd::ReturnsResult: Original returns data.
  • X: Portfolio returns (vector or vector of vectors).
  • hw: Held Weights record of the fold, or nothing.
  • w: The weights the fold's series was formed from. It defaults to res.w, and a fold that viewed its window at the Investable Mask passes the view instead, so the collapse reads the same asset axis rd carries.

Returns

  • PredictionReturnsResult with updated benchmark returns, implied volatilities and implied volatility risk premium adjustment.

Related

source
PortfolioOptimisers.sort_predictions! — Method
sort_predictions!(res::VecVecInt, predictions::VecPredRes) -> VecPredRes
sort_predictions!(res::CrossValidationResult, predictions::VecPredRes) -> VecPredRes

Sort prediction results to match the order of test indices.

Reorders predictions so that they align with the original time ordering of test_idx. The key is the first observation of each fold, so the folds come back in the order the timeline visits them.

Arguments

  • res:

    • ::VecVecInt: Vector of test index vectors.
    • ::CrossValidationResult: Cross validation result object, uses the test indices stored in res.test_idx.

  • predictions: Vector of prediction results, one per fold, in split order.

Validation

  • Every element of test_idx holds unique indices.

Returns

  • Sorted predictions vector.

Details

Related

source
PortfolioOptimisers.cv_sequential_info — Function
cv_sequential_info()

Build the informational message emitted when a cross-validation run runs its folds sequentially. run_folds is the only site that emits it, and it is the sequential loop, so the message states the two facts that sent the run there rather than quoting a value back.

The two facts are the conjunction fold_loop computes. The fold enumeration of the scheme is a timeline (folds_are_time_ordered), and the estimator needs the previous fold's weights (needs_previous_weights). Either one alone leaves the folds independent. Time dependence is neither of them: a TimeDependent schedule is known for every fold before the loop starts, so fold_loop resolves it in parallel.

Returns

  • String: The message.

Related

source
PortfolioOptimisers.parallel_folds — Function
parallel_folds(fit_fold, n::Integer, ex::FLoops.Transducers.Executor,
               ::Type{ElT} = PredictionResult)

Run n cross-validation folds in parallel, filling predictions[i] = fit_fold(i) for i in 1:n over executor ex. ElT is the per-fold result element type (a single PredictionResult for time-ordered schemes, a Vector{PredictionResult} for the multi-path combinatorial scheme).

This is the sibling of run_folds, and the two divide the work by name. A fold here takes no previous fold, so fit_fold takes the fold index alone. fold_loop decides which of the two runs, and neither one re-decides.

ElT is a positional ::Type{ElT} argument, not a keyword, so a method always specialises on it and Vector{ElT}(undef, n) stays a compile-time construction. As a keyword its value only survives constant propagation, which one forwarding hop is enough to lose.

Related

source
PortfolioOptimisers.run_folds — Function
run_folds(fit_fold, n::Integer, ::Type{ElT} = PredictionResult; pws = nothing)

Run n cross-validation folds in order, filling predictions[i] = fit_fold(i, prev) for i in 1:n, and emit cv_sequential_info. prev is the last fold whose weights the next fold can be handed: fold 1 takes nothing, because it has no fold behind it, and after fold i the loop advances prev to predictions[i] only when threads_weights holds of it, so a fold whose solve failed is skipped over and the fold before it is read instead. The caller uses prev to thread its weights into fold i. ElT is the per-fold result element type, and pws is the scheme's Previous-Weights Source, which decides what threads_weights tests.

This is the sequential loop, and it does that one job. fold_loop is the only site that calls it, and it calls it only when the folds are a timeline and the estimator needs the previous fold's weights. The loop therefore neither re-decides nor takes an executor: its sibling parallel_folds owns the other case.

ElT is a positional ::Type{ElT} argument for the reason given in parallel_folds.

Related

source
PortfolioOptimisers.advance_previous_fold — Function
advance_previous_fold(pws, prev, pred::PredictionResult)

Give the fold the next fold is handed: pred when threads_weights holds of it, prev otherwise.

One line shared by run_folds and online_folds, so the two sequential loops advance by the same rule.

Related

source
advance_previous_fold(pws, prev, step::NamedTuple)

Hand the next fold what the last one held, when the fold's result is a step record and not a prediction.

The kernel's step record carries no weights, so there is nothing to thread and prev stays where it was. This is the arm online_folds reaches under covariance_forecast_evaluation.

Related

source
PortfolioOptimisers.folds_are_time_ordered — Function
folds_are_time_ordered(cv)

Return true if the fold enumeration of a cross-validation scheme is a timeline.

This is one half of the conjunction fold_loop computes, so a scheme states for itself whether its folds carry history. A walk-forward, a multiple-randomised path, and any scheme with no more specific method enumerate their folds in time order. Fold i has fold i - 1 behind it, so the loop may run the folds in order and thread the previous fold's weights. needs_previous_weights is the other half, and it decides whether the loop does so.

A NonSeqCVER scheme answers false. A k-fold and a combinatorial enumeration are not timelines: each fold is independent of the others, so no fold has a previous fold, and the loop runs them in parallel. A KFold training window holds rows that follow its test window, so a quantity measured against another fold's weights is not a backtest reading.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv and never needs the value. folds_are_time_ordered(::Any) answers nothing too, which is what fold_loop receives from a call site that holds no scheme.

Related

source
PortfolioOptimisers.fold_evaluation — Function
fold_evaluation(cv)

Read the evaluation switches of a cross-validation scheme, in one named tuple.

Every scheme entry point reads the same settings before it runs its folds, and each scheme states them for itself through a method of its own. A scheme that carries none of them, and a call site that holds a split result rather than the scheme that made it, reach the fallback and get today's behaviour: no drift, no drifted previous weights, no fee-clock override, and no stored weight path.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv, exactly as folds_are_time_ordered does.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::KFold)

Read the evaluation switches of a KFold.

The folds of this scheme are not a timeline, so it carries no Previous-Weights Source and the tuple names nothing for it. There is no previous fold whose weights a fold of this scheme could inherit.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::CombinatorialCrossValidation)

Read the evaluation switches of a CombinatorialCrossValidation.

The folds of this scheme are not a timeline, so it carries no Previous-Weights Source and the tuple names nothing for it. There is no previous fold whose weights a fold of this scheme could inherit.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::IndexWalkForward)

Read the evaluation switches of a IndexWalkForward.

The folds of this scheme are a timeline, so it carries both weight switches and states both of them here, beside the Fee Clock of its realised series.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::DateWalkForward)

Read the evaluation switches of a DateWalkForward.

The folds of this scheme are a timeline, so it carries both weight switches and states both of them here, beside the Fee Clock of its realised series.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::MultipleRandomised)

Read the evaluation switches of a MultipleRandomised.

The scheme carries no switch of its own. Each of its paths is an inner walk-forward, so it inherits every switch from the scheme in its cv field, and a caller sets them there.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
PortfolioOptimisers.fold_fit — Function
fold_fit(cv)

Read the Fold Fit of a cross-validation scheme: how fold_loop fits each fold.

nothing means a refit from the fold's training window, which is the released behaviour and what every scheme answers unless it states otherwise. A walk-forward carries the switch in its ff field and answers it through a method of its own; a MultipleRandomised forwards to the walk-forward it wraps; a split result, a k-fold, a combinatorial scheme and a call site that holds no scheme reach this fallback. An OnlineStep sends the loop down its online arm.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv, exactly as fold_evaluation and folds_are_time_ordered do, and the arm that cannot run is eliminated.

Related

source
fold_fit(cv::IndexWalkForward)
fold_fit(cv::DateWalkForward)

Read the Fold Fit of a walk-forward: its ff field.

The two walk-forwards are the schemes that carry the switch, because only a timeline has a previous fold to thread an estimator from, and each states it here beside fold_evaluation.

Returns

  • ff::Option{<:AbstractFoldFit}: The Fold Fit, or nothing for a refit every fold.

Related

source
fold_fit(cv::MultipleRandomised)

Read the Fold Fit of a MultipleRandomised.

The scheme carries no switch of its own. Each of its paths is an inner walk-forward, so it inherits the Fold Fit from the scheme in its cv field, as it inherits its evaluation switches. Under an OnlineStep each path slices the estimator to its asset subset once and threads it through the path's folds.

Returns

  • ff::Option{<:AbstractFoldFit}: The inner walk-forward's Fold Fit.

Related

source
PortfolioOptimisers.investable_fold_view — Function
investable_fold_view(imsk::Nothing, w, rd::ReturnsResult, fees::Option{<:Fees})
investable_fold_view(imsk::BitVector, w, rd::ReturnsResult, fees::Option{<:Fees})

View a fold's weights and test window at the Investable Mask, and pass its fees through.

An optimisation reduces to the Investable Mask at its entry and expands the solved weights back to the caller's universe, so the weight of an asset the fit found non-investable is 0. That rule carries to the window those weights are scored on: the fold views the weights and the window together, before anything reads the window, so a dead column is never read at all and the Held Gap filter of filter_held_gaps runs over the investable columns alone.

The fees are not viewed. A result carries the objects of the universe it solved on beside the mask, so res.fees is on the investable universe already and a second view would index its per-asset rates by positions of the full universe. The fees ride along so the verb hands the fold the three things it scores with in one call.

A result whose mask is nothing views nothing, which is what keeps a universe with nothing to exclude on the path it took before the mask existed.

Arguments

  • imsk: The Investable Mask, or nothing.
  • w: The fold's target weights, or a population of them.
  • rd: The returns result to use.
  • fees: Fees the fold is charged, on the investable universe, or nothing.

Returns

  • (w, rd, fees): The weights and the window reduced to the investable assets, or unchanged, and the fees as given.

Related

source
PortfolioOptimisers.fold_fees — Function
fold_fees(res::OptimisationResult, fees::Nothing, X::MatNum)
fold_fees(res::OptimisationResult, fees::Fees, X::MatNum)

Resolve the fee a fold-taking consumer charges: the result's own, or a caller's viewed at the result's Investable Mask.

The three fold-taking consumers — calc_net_asset_returns, risk_contribution and factor_risk_contribution on a PredictionResult — take an optional fees so a caller can score a stored fold under a fee of their own. The two fees they can meet live on different universes. The result's own fee was reduced at the fit's door, so its five per-asset fields sit on the investable axis and its two liquidation carriers on the complement, and investable_fold_view hands it through unviewed. A caller's fee is stated on the caller's universe, as a caller's rd is, so it takes the same door a fee takes at the fit: investable_fees_view slices the per-asset fields to the mask and the carriers to its complement, deriving the complement from the width of the expanded record X, and strips the carriers when the mask is nothing, because nothing left.

The split is by dispatch on fees. A nothing reads the result's fee through extract_fees; a Fees is the caller's and is viewed. Without the view a per-asset rate stated on the full universe met the reduced weights with a BoundsError, and a full-universe carrier was charged as though every position had been liquidated.

Arguments

  • res::OptimisationResult: The fold's optimisation result, carrying the mask and its own fee.
  • fees: A caller's Fees on the caller's universe, or nothing to charge the result's own.
  • X: The fold's expanded record, observations × assets on the caller's universe. Only its width is read.

Returns

  • fees::Option{<:Fees}: The fee on the two axes the result's mask leaves, or nothing.

Related

source
PortfolioOptimisers.threads_weights — Function
threads_weights(pws::Nothing, pred::PredictionResult)
threads_weights(pws::AbstractPreviousWeightsSource, pred::PredictionResult)

Say whether a fold's prediction carries weights the next fold can be handed.

The other half of the Previous-Weights Source seam: previous_weights reads the weights, and this verb says whether the fold has any to read. The sequential loops, run_folds and online_folds, advance the fold they hand on only when it holds, so a failed fold is never the one read and the last threadable fold is read instead. What is read decides what is tested: the target weights are finite exactly when every member's return code is an OptimisationSuccess, and the held weights are finite when the drift ran, which after held_start_weights it does on a failed fold too, so a source advances past a failed fold that held its book and stops only at one with nothing to hold.

Algorithm

  1. With no source, hold when the fold's return code is a success, every member's under a population.
  2. With a source, hold when the fold's held weights are all finite, every member's under a population.

Arguments

  • pws: Previous-weights source of the scheme, or nothing.
  • pred: Prediction result of the fold.

Returns

  • Bool: Whether the fold's weights can be threaded.

Related

source
PortfolioOptimisers.held_start_weights — Function
held_start_weights(retcode::OptimisationSuccess, w::VecNum, w_prev)
held_start_weights(retcode::OptimisationFailure, w::VecNum, w_prev::Nothing)
held_start_weights(retcode::OptimisationFailure, w::VecNum, w_prev::VecNum)
held_start_weights(retcode::OptimisationReturnCode, w::VecVecNum, w_prev)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::Nothing)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecNum)
held_start_weights(retcode::VecOptRetCode, w::VecVecNum, w_prev::VecVecNum)

Name the weights a fold's drift starts from: its own on a solved fold, the previous weights on a failed one.

A fold that could not rebalance holds what it held, so under a Weight Drift or a Previous-Weights Source a failed fold drifts the weights it was handed rather than its NaN target. The choice is by return code, read per member under a population, and the previous weights are one vector for every member or one per member. A population solved under one return code — a frontier the one JuMPOptimisationResult carries — reads that code for every member. A failed fold with no previous weights keeps its NaN weights, and held_weights_result records NaN for it without drifting.

Algorithm

  1. On an OptimisationSuccess, give w.
  2. On an OptimisationFailure over one vector, give w_prev, or w when there is none.
  3. Over a population, give, for each member, its own vector on a success and the previous weights on a failure: the one vector when w_prev is one, its own entry when w_prev is one per member. One return code over a population is that code for every member.

Arguments

  • retcode: Return code of the fold, or one per member of the population.
  • w: Target weights of the fold, on the universe the fold is scored on.
  • w_prev: Previous weights the fold was handed, on the same universe, or nothing.

Validation

  • A per-member w_prev has one entry per member of w, else a DimensionMismatch is raised.

Returns

Related

source
PortfolioOptimisers.fold_factor_returns — Function
fold_factor_returns(imsk, rd::Nothing, pred::PredictionResult)
fold_factor_returns(imsk, rd::ReturnsResult, pred::PredictionResult)

Resolve the returns result a fold's factor loadings are fitted from, on the live assets of the fold.

The fold-taking factor_risk_contribution reads rd by dispatch. A caller's rd is stated on the caller's universe, as it is at the value-level door, and is viewed at the result's Investable Mask through investable_returns_view. nothing builds one from the fold itself: the fold's asset returns viewed at the same mask, beside the factor block reconstruct_rd carried through and the fold's own nx, which is the reduced axis already.

Arguments

  • imsk: The result's Investable Mask, or nothing.
  • rd: Returns result on the caller's universe, or nothing.
  • pred: Single-fold prediction result carrying a HeldWeightsResult.

Returns

  • rd::ReturnsResult: The returns result on the live assets.

Related

source
PortfolioOptimisers.collapse_benchmark — Method
collapse_benchmark(B::Nothing, w::VecNum_VecVecNum, hw)
collapse_benchmark(B::VecNum, w::VecNum, hw)
collapse_benchmark(B::VecNum, w::VecVecNum, hw)
collapse_benchmark(B::MatNum, w::VecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::Nothing)
collapse_benchmark(B::MatNum, w::VecNum, hw::HeldWeightsResult)
collapse_benchmark(B::MatNum, w::VecVecNum, hw::HeldWeightsResult)

Collapse a fold's benchmark asset returns into a benchmark return series.

A benchmark that is already a series passes through. A benchmark matrix is contracted with the fold's own weights, and the method is chosen by the pair (B, w), so nothing is tested at run time.

The fold's Held Weights record picks the reading. Without one the matrix collapses against the target weights, which is the library's original behaviour and what a fold that ran no drift keeps. With one it collapses row by row against the weight path, so the benchmark follows the same convention the portfolio series follows and a caller comparing the two — a tracking error, for instance — compares two series scored the same way.

Algorithm

  1. On nothing, give nothing.
  2. On a benchmark series, give it back, repeated once per member under a population.
  3. On a matrix with no record, give B * w, once per member under a population.
  4. On a matrix with a record, give vec(sum(B ⊙ U; dims = 2)) for the fold's weight path U, once per member under a population.

Arguments

  • B: Benchmark returns of the fold: nothing, a series, or an observations × assets matrix.
  • w: Target weights of the fold, or a population of them.
  • hw: Held Weights record of the fold, or nothing.

Returns

  • The benchmark return series, or a vector of them under a population, or nothing.

Related

source
PortfolioOptimisers.ruined_retcodes — Function
ruined_retcodes(retcode::VecOptRetCode, ruined::VecInt)

Set the return code of every ruined member of a population to an OptimisationFailure.

The failure payload names the member and the reason, so a reader of res.retcode finds why the member left the run. A member that is not named keeps the code its own optimisation gave it.

Algorithm

  1. Walk the codes with their positions, and replace the code of a named member with a failure that states the reason.

Arguments

  • retcode: Return codes of the population, one per member.
  • ruined: Indices of the members whose drifted wealth is not positive.

Returns

  • VecOptRetCode: The codes, with the ruined members failed.

Related

source
PortfolioOptimisers.mark_ruined_members — Function
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::Nothing)
mark_ruined_members(res::NonFiniteAllocationOptimisationResult, ruined::VecInt)

Rebuild a fold's optimisation result so that its ruined members carry a failure code.

The drop needs no machinery of its own. The library already folds a vector of return codes with any(x -> isa(x, OptimisationFailure), …), and the cross-validation path already filters a path on isa(y.res.retcode, OptimisationSuccess), so a failed member takes the path it is on out of the run.

Algorithm

  1. With no ruined member, and with an empty set of them, give res unchanged. Nothing is rebuilt on the ordinary path.
  2. Otherwise rebuild res through set_retcode, with the codes ruined_retcodes makes.

Arguments

  • res: Optimisation result of the fold.
  • ruined: Indices of the members whose drifted wealth is not positive, or nothing.

Returns

  • NonFiniteAllocationOptimisationResult: The result, rebuilt only when a member was ruined.

Related

source
PortfolioOptimisers.warn_ruined_members — Function
warn_ruined_members(wd::AbstractWeightDrift, args...)
warn_ruined_members(wd::Nothing, ruined::Nothing, n::Integer)
warn_ruined_members(wd::Nothing, ruined::VecInt, n::Integer)

Warn once when a drift dropped members of a population, and the return series did not.

A fold whose series is drifted is warned about by calc_net_returns(w::VecVecNum, X::MatNum, fees, wd::AbstractWeightDrift, obs), which runs the same drift over the same window. A fold that drifts only its held weights has no such site, so the warning is raised here instead. Either way the fold warns once.

Algorithm

  1. With a drifted series, say nothing. The series already warned.
  2. With no ruined member, say nothing.
  3. Otherwise warn, and name the count and the members.

Arguments

  • wd: Weight drift of the scheme, or nothing.
  • ruined: Indices of the ruined members, or nothing.
  • n: Number of members of the population.

Returns

  • nothing.

Related

source
PortfolioOptimisers.fold_loop — Function
fold_loop(fit_fold, est, n::Integer, ex::FLoops.Transducers.Executor,
          ::Type{ElT} = PredictionResult; rd, train_idx, test_idx,
          path_id = nothing, cv = nothing, fold_view = nothing)

Run the n folds of a cross-validation scheme over est, and resolve each fold's estimator before the callback sees it.

This is the fold loop of the package. Every cross-validation entry point goes through it: the optimiser-level schemes and the Pipeline ones alike. For fold i it does four steps.

  1. It takes the fold's view of (est, rd) through fold_view. An asset-resampling scheme gives one; the other schemes do not.
  2. It swaps every TimeDependent schedule for its fold-i value against a TimeDependentContext, if est is_time_dependent. The swap runs first, so a freshly swapped-in per-fold entry also gets the weights of step 3.
  3. It threads the previous fold's weights in through factory, if est needs_previous_weights.
  4. It calls fit_fold(fold), with fold a Fold.

The callback takes the one Fold record, so a call site names what it reads (fold.est, fold.train) instead of relying on the position of an argument.

assert_time_dependent_fold_count runs once, before the loop, and so does assert_batch_entry when the scheme declares no Fold Fit: the batch arms refit every fold from its training window and run no warm-up, so an Online anywhere in est would reach prior(pe, X) unresolved, and it is refused by name instead — once, here, rather than up to once per fold on the workers of ex.

This is also the one site that decides how the folds run, and it has three arms. The online arm, online_folds, is taken first, when the scheme declares a Fold Fit (fold_fit): the loop then warms one estimator up on the first training window, folds each fold's new rows into it, and hands the callback a Fold whose train is nothing — steps 2 and 3 run on a per-fold copy of the threaded estimator, so a schedule and the previous weights still reach the fold. Otherwise a run is sequential only when two facts hold at once: the fold enumeration of cv is a timeline (folds_are_time_ordered), and est needs the previous fold's weights (needs_previous_weights). The conjunction routes through run_folds. Every other case routes through parallel_folds, because a fold with no fold behind it, or a fold whose estimator reads no previous weights, is independent of the other folds. No loop re-decides.

cv is the scheme, and the loop reads its three per-type predicates rather than a value a call site computes. All are decided by the types of cv and est, so inference folds the conjunction and eliminates the arm that cannot run. A Bool keyword cannot do this: its value survives only by constant propagation, which one call hop loses, and the sequential arm is then inferred even where it can never run. The two path-level sites enumerate an inner walk-forward; the optimiser's passes the MultipleRandomised it runs, which forwards its Fold Fit, and the Pipeline's holds no scheme and omits cv; folds_are_time_ordered(nothing) answers true.

ElT is the per-fold result element type: a single PredictionResult for a time-ordered scheme, a Vector{PredictionResult} for the multi-path combinatorial scheme. It is positional for the reason given in parallel_folds.

Returns

  • predictions::Vector{ElT}: One result per fold, in split order — the new folds only under a Resume.
  • opt: The estimator the online arm threaded, folded through the last training end, or nothing from the batch arms. An online walk-forward's Result carries it.

Related

source
PortfolioOptimisers.assert_unshuffled_folds — Function
assert_unshuffled_folds(cv, train_idx)

Assert that the cross-validation scheme cv enumerates unshuffled folds.

Two checks, applied to every scheme:

  1. cv must not declare a set shuffle field. The check is by hasfield, so it holds for a user-defined scheme too — no scheme in this package has such a field.
  2. Every fold's training indices must increase strictly. A scheme may leave gaps (purging, embargoing and the combinatorial splits all do), but it must never reorder rows.

A shuffled fold breaks the timeline that the fold loop, the TimeDependentContext schedules and the rolling transforms all read the fold's rows in.

Related

source
PortfolioOptimisers.cv_live_assets — Function
cv_live_assets(rd::AbstractReturnsResult) -> Any

Return the positions of the assets in the Coverage Universe of a cross-validation window.

The sibling of cv_nobs for the asset axis. It reads the numeric asset matrix of the window and the window's AssetPanel, hands both to coverage_mask, and turns the answer into the positions themselves, so a fold that draws over assets draws over the live ones and never over a column it could not have traded.

Four methods, and both branches are dispatch rather than a condition. The two data-level methods differ only in how the numeric matrix is reached: a returns carrier holds it directly, and a price carrier holds a TimeArray. The two mask-level methods take the nothing sentinel of an all-covered window and the mask of a gapped one.

An all-dead window throws an IsEmptyError where the mask is derived.

Algorithm

  1. Derive the Coverage Universe of the window's asset matrix and its panel with coverage_mask.
  2. Return every position of the asset axis on the nothing sentinel.
  3. Return findall(cmsk) otherwise.

Arguments

  • data: Returns-level or price-level data (Prices_RR).
  • cmsk: The Coverage Universe, or nothing.
  • N: The number of assets of the window.

Validation

  • At least one asset must be in the Coverage Universe of the window.

Returns

  • live::Vector{Int}: The positions of the covered assets, in increasing order.

Related

source