GitHub

Base Cross Validation (b): private API

PortfolioOptimisers.Fold — Type
struct Fold{T1, T2, T3, T4, T5, T6, T7}

One fold of a cross-validation scheme, as fold_loop hands it to its callback.

The record holds everything the fold loop gives its callback. est and rd are already resolved: the loop took the asset view, swapped every TimeDependent schedule for its value at fold i, and threaded in the weights of the previous fold. train and test are the windows of this fold, so a callback never indexes train_idx or test_idx itself. w_prev holds the weights that were threaded, a second time, so that a failed fold can hold them. held_start_weights reads it inside predict.

train === nothing says the estimator holds its window. The online arm of the loop, online_folds, hands its callback a Fold of that shape. est has already folded every row of the training window through partial_fit!, so a callback reads it out with optimise(est) and no returns, in place of a fit over rows the record does not carry. fit_and_predict takes train_idx = nothing for that case, so the callback of every entry point is the same under both arms.

The type is immutable and every field is concretely typed at the construction site, so the record costs nothing at run time. fold_loop is the only site that builds one, which is why there is no keyword constructor.

Fields

  • i: Index of the fold within the scheme's split enumeration (1-based).
  • n: Number of folds in the enumeration.
  • est: The fold's resolved estimator: asset-viewed, schedule-swapped, weights-threaded.
  • rd: The fold's (possibly asset-viewed) input data.
  • train: The fold's training indices, or nothing when the estimator holds its window.
  • test: The fold's test indices.
  • w_prev: The previous weights threaded into est, or nothing when there are none.

Related

source
PortfolioOptimisers.fit_and_predict — Function
fit_and_predict(opt, rd::ReturnsResult, cv::NonSeqCVER; cols, ex, id) -> MultiPeriodPredictionResult
fit_and_predict(opt, rd::ReturnsResult, cv::CombCVER; cols, ex) -> PopulationPredictionResult
fit_and_predict(opt, rd::ReturnsResult; train_idx = nothing, test_idx, cols) -> PredictionResult
fit_and_predict(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult; test_idx, cols) -> PredictionResult

Fit an optimisation estimator on training data and predict on test data using cross-validation.

The method over a scheme cv predicts over every fold of cv. The methods without a scheme predict over one train/test split, or apply a result that is already fitted.

A combinatorial cv takes its own method, because its folds recombine into several paths and not one. That method regroups the fold predictions by path, and returns one MultiPeriodPredictionResult per path in a PopulationPredictionResult.

The estimator form reads train_idx = nothing as the estimator holds its window. It reads the estimator out through optimise(opt) in place of a fit over port_opt_view(rd, train_idx, cols), and predicts over test_idx as before. The online arm of fold_loop reads a fold out this way. It is also the public entry for an estimator stepped by hand, warmed up with update_online_estimator and folded with partial_fit!. fit_and_predict(opt, rd; test_idx) on a stepped estimator equals fit_and_predict(opt, rd; train_idx, test_idx) on the cold one over the same rows. The two arms belong to fit_fold_result, and the method is defined beside them.

The result form is fitted already, so it ignores train_idx and every keyword that predict does not take. The fold loop passes the same keywords to an estimator and to a result.

Algorithm

For a scheme cv that is not combinatorial:

  1. Split rd with cv, giving train_idx and test_idx.
  2. Check that the folds are not shuffled with assert_unshuffled_folds.
  3. Read the evaluation settings of cv with fold_evaluation, and the drift of the Held Weights record with held_weights_drift.
  4. Run the folds with fold_loop. Each fold fits its estimator on its training rows and predicts over its test rows, giving one PredictionResult per fold, in split order.
  5. Concatenate the folds into a MultiPeriodPredictionResult carrying id.

Arguments

  • opt: Optimisation estimator or an existing optimisation result.
  • rd::ReturnsResult: Returns data.
  • cv::NonSeqCVER: Non-sequential cross-validation estimator or result, such as KFold.
  • cv::CombCVER: Combinatorial cross-validation estimator or result (CombinatorialCrossValidation).

Keyword Arguments

  • train_idx::Option{<:VecInt}: Training indices, or nothing to read a stepped estimator out.
  • test_idx: Test indices, one vector or one vector per fold.
  • cols = :: Column selector.
  • ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(): Executor of the folds that run in parallel.
  • id = nothing: Identifier of the path.
  • wd, hwd, fa, store_weight_path, strict, w_prev: The keywords of predict, which the methods over one split pass on.

Returns

Related

source
PortfolioOptimisers.sort_predictions — Method
sort_predictions(res::VecVecInt, predictions::VecPredRes) -> VecPredRes
sort_predictions(res::CrossValidationResult, predictions::VecPredRes) -> VecPredRes

Sort the prediction results of the folds by the first observation of their test windows.

The function returns a sorted copy, and does not change predictions. The key is the first entry of each test window, so the folds come back in time order when each window starts at its earliest row, which is true of every scheme in the library.

CombinatorialCrossValidationResult has its own method with a different shape and a different job. Its folds do not form one timeline, so that method takes a vector of per-split vectors and regroups them by path into MultiPeriodPredictionResults.

Algorithm

  1. Check that every test window holds unique indices.
  2. Compute the permutation that sorts the test windows by their first entry, giving idx.
  3. Return predictions[idx].

Arguments

  • res:

    • ::VecVecInt: Vector of test index vectors.
    • ::CrossValidationResult: Cross validation result object, uses the test indices stored in res.test_idx.

  • predictions: Vector of prediction results, one per fold, in split order.

Validation

  • Every test window holds unique indices, else an ArgumentError is raised.

Returns

  • A new vector of the predictions, sorted.

Related

source
PortfolioOptimisers.cv_sequential_info — Function
cv_sequential_info()

Build the informational message emitted when a cross-validation run runs its folds sequentially. run_folds is the only site that emits it, and it is the sequential loop, so the message states the two facts that sent the run there rather than quoting a value back.

The two facts are the conjunction fold_loop computes. The fold enumeration of the scheme is a timeline (folds_are_time_ordered), and the estimator needs the previous fold's weights (needs_previous_weights). Either one alone leaves the folds independent. Time dependence is neither of them: a TimeDependent schedule is known for every fold before the loop starts, so fold_loop resolves it in parallel.

Returns

  • String: The message.

Related

source
PortfolioOptimisers.parallel_folds — Function
parallel_folds(fit_fold, n::Integer, ex::FLoops.Transducers.Executor,
               ::Type{ElT} = PredictionResult)

Run n cross-validation folds in parallel over the executor ex. ElT is the element type of the result of one fold: a single PredictionResult for a time-ordered scheme, and a Vector{PredictionResult} for the multi-path combinatorial scheme.

This is the sibling of run_folds. A fold here takes no previous fold, so fit_fold takes the fold index alone. fold_loop decides which of the two runs.

ElT is a positional ::Type{ElT} argument, not a keyword, so a method always specialises on it and Vector{ElT}(undef, n) is built at compile time. The value of a keyword survives only by constant propagation, which one forwarding call can lose.

Algorithm

  1. Allocate predictions = Vector{ElT}(undef, n).
  2. For each i in 1:n, in parallel over ex, set predictions[i] = fit_fold(i).

Related

source
PortfolioOptimisers.run_folds — Function
run_folds(fit_fold, n::Integer, ::Type{ElT} = PredictionResult; pws = nothing)

Run n cross-validation folds in order, and emit cv_sequential_info.

prev is the last fold whose weights the next fold can be handed. Fold 1 takes nothing, because it has no fold behind it. A failed fold is skipped, and the fold before it stays prev. The caller uses prev to thread its weights into fold i. pws is the Previous-Weights Source of the scheme, and it decides what threads_weights tests.

fold_loop is the only caller, and it calls this loop only when the folds are a timeline and the estimator needs the weights of the previous fold. The loop therefore takes no executor, and its sibling parallel_folds runs every other case. ElT is a positional ::Type{ElT} argument for the reason that parallel_folds gives.

Algorithm

  1. Emit cv_sequential_info as an @info message.
  2. Allocate predictions = Vector{ElT}(undef, n), and set prev = nothing.
  3. For each i in 1:n in order, set predictions[i] = fit_fold(i, prev).
  4. After fold i, set prev with advance_previous_fold: predictions[i] when threads_weights holds of it, and prev unchanged otherwise.

Related

source
PortfolioOptimisers.advance_previous_fold — Function
advance_previous_fold(pws, prev, pred::PredictionResult)

Give the fold the next fold is handed: pred when threads_weights holds of it, prev otherwise.

run_folds and online_folds both call it, so the two sequential loops advance by the same rule.

Related

source
advance_previous_fold(pws, prev, step::NamedTuple)

Hand the next fold what the last fold held, when the result of the fold is a step record and not a prediction.

The step record of the kernel carries no weights, so the method threads nothing and returns prev unchanged. online_folds reaches this method under covariance_forecast_evaluation.

Related

source
PortfolioOptimisers.folds_are_time_ordered — Function
folds_are_time_ordered(cv)

Return true if the fold enumeration of a cross-validation scheme is a timeline.

This is one half of the conjunction fold_loop computes, so a scheme states for itself whether its folds carry history. A walk-forward, a multiple-randomised path, and any scheme with no more specific method enumerate their folds in time order. Fold i has fold i - 1 behind it, so the loop may run the folds in order and thread the previous fold's weights. needs_previous_weights is the other half, and it decides whether the loop does so.

A NonSeqCVER scheme answers false. A k-fold and a combinatorial enumeration are not timelines: each fold is independent of the others, so no fold has a previous fold, and the loop runs them in parallel. A KFold training window holds rows that follow its test window, so a quantity measured against another fold's weights is not a backtest reading.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv and never needs the value. The fallback over ::Any also answers true for nothing, which is what fold_loop receives from a call site that holds no scheme.

Related

source
PortfolioOptimisers.fold_evaluation — Function
fold_evaluation(cv)

Read the evaluation switches of a cross-validation scheme, in one named tuple.

Every scheme entry point reads the same settings before it runs its folds, and each scheme states them through a method of its own. A scheme that carries none of them, and a call site that holds a split result in place of the scheme that made it, reach the fallback. The fallback sets no drift, no drifted previous weights, no override of the fee clock, no stored weight path, and a Held Gap that warns.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv, exactly as folds_are_time_ordered does.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::KFold)

Read the evaluation switches of a KFold.

The folds of this scheme are not a timeline, so it carries no Previous-Weights Source and the tuple names nothing for it. There is no previous fold whose weights a fold of this scheme could inherit.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::CombinatorialCrossValidation)

Read the evaluation switches of a CombinatorialCrossValidation.

The folds of this scheme are not a timeline, so it carries no Previous-Weights Source and the tuple names nothing for it. There is no previous fold whose weights a fold of this scheme could inherit.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::IndexWalkForward)

Read the evaluation switches of a IndexWalkForward.

The folds of this scheme are a timeline, so it carries both weight switches and states both of them here, beside the Fee Clock of its realised series.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::DateWalkForward)

Read the evaluation switches of a DateWalkForward.

The folds of this scheme are a timeline, so it carries both weight switches and states both of them here, beside the Fee Clock of its realised series.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::HindsightSplit)

Read the evaluation switches of a HindsightSplit.

The folds of this scheme are a timeline, so it carries both weight switches and states both of them here, beside the Fee Clock of its realised series.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
fold_evaluation(cv::MultipleRandomised)

Read the evaluation switches of a MultipleRandomised.

The scheme carries no switch of its own. Each of its paths is an inner walk-forward, so it inherits every switch from the scheme in its cv field, and a caller sets them there.

Returns

  • (; wd, pws, fa, store_weight_path, strict): The Weight Drift, the Previous-Weights Source, the Fee Clock of the fold's realised series, the flag that stores a fold's weight path, and the flag that makes a Held Gap raise rather than warn.

Related

source
PortfolioOptimisers.folds_are_stepped — Function
folds_are_stepped(cv)

Whether fold_loop fits each fold of cv by the online step, or by a refit from the fold's training window.

false is a refit every fold, which is what every scheme answers unless it is an Online Scheme: an Online around a walk-forward, built by OnlineIndexWalkForward, OnlineDateWalkForward or OnlineHindsightSplit, answers true through a method of its own, and a MultipleRandomised answers for the walk-forward it wraps. A plain scheme, a split result, and a call site that holds no scheme reach this fallback.

The method is per type and takes the scheme itself, so inference reads the answer from the type of cv, exactly as fold_evaluation and folds_are_time_ordered do, and the arm that cannot run is eliminated.

Related

source
folds_are_stepped(cv::MultipleRandomised)

Whether the paths of a MultipleRandomised are fitted by the online step.

The scheme has no online form of its own. Each of its paths is an inner walk-forward, so it answers for the scheme in its cv field, as it does for its evaluation switches: MultipleRandomised(OnlineIndexWalkForward(…); …) steps, and each path slices the estimator to its asset subset once and threads it through the path's folds.

Returns

  • flag::Bool: Whether the inner walk-forward is an Online Scheme.

Related

source
PortfolioOptimisers.fold_loop — Function
fold_loop(fit_fold, est, n::Integer, ex::FLoops.Transducers.Executor,
          ::Type{ElT} = PredictionResult; rd, train_idx, test_idx,
          path_id = nothing, cv = nothing, fold_view = nothing, pws = nothing)

Run the n folds of a cross-validation scheme over est, and resolve the estimator of each fold before the callback sees it.

This is the fold loop of the package. Every cross-validation entry point goes through it, the optimiser-level schemes and the Pipeline ones alike. The callback takes the one Fold record, so a call site names what it reads (fold.est, fold.train) and does not rely on the position of an argument.

The loop is also the one site that decides how the folds run, and it has three arms. The online arm, online_folds, runs when the scheme is an Online Scheme. It warms one estimator up on the first training window, folds the new rows of each fold into it, and hands the callback a Fold whose train is nothing. Steps 5 and 6 below run on a per-fold copy of the threaded estimator, and a schedule that the step reads is resolved earlier in the same fold through online_step_fold. Otherwise the folds run in order only when the folds of cv are a timeline and est needs the weights of the previous fold. Every other case runs in parallel, because such a fold does not depend on the other folds.

The loop reads the three predicates of cv by type, so inference folds the choice and removes the arms that cannot run. A Bool keyword cannot do this: its value survives only by constant propagation, which one call can lose, and the sequential arm is then inferred where it can never run. Of the two path-level sites, the optimiser passes the MultipleRandomised it runs, which answers for the walk-forward it wraps, and the Pipeline omits cv, for which folds_are_time_ordered(nothing) answers true.

ElT is the element type of the result of one fold: a single PredictionResult for a time-ordered scheme, and a Vector{PredictionResult} for the multi-path combinatorial scheme. It is positional for the reason that parallel_folds gives.

Algorithm

  1. When est is_time_dependent, check the number of folds against its schedules with assert_time_dependent_fold_count.
  2. When cv is not an Online Scheme (folds_are_stepped), refuse an Online anywhere in est with assert_batch_entry. The batch arms refit every fold from its training window and run no warm-up, so the loop refuses it once here and not once per fold on the workers of ex.
  3. Choose the arm: online_folds for an Online Scheme; run_folds when folds_are_time_ordered holds of cv and est needs_previous_weights; and parallel_folds otherwise.
  4. For fold i, take the view of (est, rd) that fold_view(i) gives, or (est, rd) when fold_view is nothing. An asset-resampling scheme gives a view.
  5. Read the previous weights with previous_weights, giving w_prev, and swap every TimeDependent schedule for its value at fold i against a TimeDependentContext. The swap runs first, so a per-fold entry that it swaps in also receives the weights of step 6.
  6. When w_prev is not nothing and est needs previous weights, thread them in through factory. one_previous_portfolio refuses the population of a frontier, because no one portfolio of it is the previous one.
  7. Call fit_fold with the Fold of fold i.

Returns

  • predictions::Vector{ElT}: One result per fold, in split order. Under a Resume, the new folds only.
  • opt: The estimator that the online arm threaded, folded through the last training end, or nothing from the batch arms. The Result of an online walk-forward carries it.

Related

source
PortfolioOptimisers.assert_unshuffled_folds — Function
assert_unshuffled_folds(cv, train_idx)

Assert that the cross-validation scheme cv enumerates unshuffled folds.

A shuffled fold breaks the time order in which the fold loop, the TimeDependentContext schedules and the rolling transforms read the rows of the fold. A scheme may leave gaps, as purging, embargoing and the combinatorial splits do, but it must never reorder rows.

Arguments

  • cv: The cross-validation scheme, or any value.
  • train_idx: The training indices of every fold.

Validation

  • cv does not have a shuffle field set to true. The check reads the field by hasfield, so it holds for a scheme a user defines. No scheme of the library has such a field.
  • The training indices of every fold increase strictly.
  • Either failure raises an ArgumentError.

Returns

  • nothing.

Related

source