Online fold loop: private API

A scheme built by OnlineIndexWalkForward, OnlineDateWalkForward or OnlineHindsightSplit is an online scheme. With an online scheme, PortfolioOptimisers.fold_loop does not refit the estimator on each training window. It fits one estimator on the first training window. At each later fold it adds to that estimator only the rows that the training window gained, by default with partial_fit!.

The loop then gives the callback a PortfolioOptimisers.Fold whose train field is nothing, because the estimator already holds the training rows. Where a batch fold refits, this fold calls optimise(opt) with no returns. At fold i the estimator holds every row up to the end of the training window. The batch expanding-window fold reads the same rows. The weights of each fold are therefore those of the batch walk-forward, to the tolerance of the moment estimates and of the solver.

PortfolioOptimisers.fit_fold_result — Function
fit_fold_result(opt::NonFiniteAllocationOptimisationEstimator, rd::ReturnsResult, train_idx::VecInt, cols)
fit_fold_result(opt::NonFiniteAllocationOptimisationEstimator, rd::ReturnsResult, ::Nothing, cols)

Fits the estimator of one fold. Dispatch on train_idx selects the arm.

These are the two arms of the estimator form of fit_and_predict. A training window selects the refit that every fold of the batch arms runs, optimise(opt, port_opt_view(rd, train_idx, cols)). nothing means that the estimator holds its window, because the online arm of fold_loop folded every row of it. This arm calls optimise(opt) with no returns. That call rebuilds the returns data from the state and runs the batch path over it. When cols is not :, the caller takes the asset view of opt before either arm, so it slices a stepped estimator and a cold one by asset in the same way.

Arguments

  • opt: The estimator of the fold, already viewed by asset.
  • rd: The returns data of the fold. The arm with no training window does not read it.
  • train_idx: The training window of the fold, or nothing for a stepped estimator.
  • cols: The asset columns of the fold. The arm with no training window does not read them.

Validation

  • On the arm with no training window, everything that optimise(opt) refuses. A cold estimator holds no window, and optimise(opt) refuses it with an ArgumentError.

Returns

  • res::OptimisationResult: The fitted result of the fold.

Related

source
PortfolioOptimisers.thread_online_folds! — Function
thread_online_folds!(predictions, fit_fold, est, folds, prev; rd, train_idx, test_idx, n, last_end, pws, path_id)

Folds and reads out the folds folds of a walk-forward, and threads est from one fold to the next.

The online arm and the resumed arm of fold_loop share this body, so a resume continues the loop that started the run.

Algorithm

  1. Read is_time_dependent on est once, giving td_flag.
  2. For each fold i in folds, take the last row of its training window, giving stop.
  3. When stop > last_end, build the context of the step through step_context, with the weights that previous_weights reads from prev, giving ctx. Fold the rows (last_end + 1):stop of rd into est through online_step_fold, and set last_end to stop. Otherwise the fold gained no row, which is the case of fold 1 after the warm-up, and est does not change.
  4. Call fit_fold(i, prev, est, rd, nothing), which makes the per-fold copy as the batch arms make it, giving the prediction of fold i. Store it at predictions[i - first(folds) + 1].
  5. Advance prev past the prediction through advance_previous_fold.
  6. After the last fold, return est.

Arguments

  • predictions: The vector that receives the predictions, with one slot per fold of folds.
  • fit_fold: The per-fold resolution and callback that fold_loop builds.
  • est: The estimator, folded through last_end.
  • folds: The indices of the folds to run, a contiguous range of the enumeration of the scheme.
  • prev: The prediction of the last threadable fold before first(folds), or nothing.
  • rd: The returns or the prices of the walk-forward.
  • train_idx: The training windows of every fold, in split order.
  • test_idx: The test windows of every fold, in split order. The context of each fold reads them.
  • n: The number of folds that the scheme enumerates. The context of each fold reads it.
  • last_end: The last row that est has folded.
  • pws: The Previous-Weights Source of the scheme, or nothing.
  • path_id: The path that the folds belong to, or nothing. Only the context of a fold reads it.

Returns

  • est: The estimator, folded through last(train_idx[last(folds)]), with its schedules unresolved.

Related

source
PortfolioOptimisers.online_folds — Function
online_folds(fit_fold, est, n::Integer, ::Type{ElT}, path_id = nothing; rd, train_idx, test_idx, fold_view, pws)

Runs the n folds of a walk-forward through the online step, and threads one estimator from fold to fold.

This is the online arm of fold_loop, which the loop takes when the scheme is an Online Scheme (folds_are_stepped). The arm logs no message. The caller chooses an Online Scheme by name through its constructor, so the arm has no accident to report. run_folds is different, because it reports a sequential run that the loop chose on its own. A Resume in the estimator slot takes the method of Resume, which skips the warm-up and the folds that the Result holds, and runs the same body from the fold after them.

Fold for fold, the run reaches the weights of the batch expanding walk-forward, to the tolerance of the moment layer and of the solver. A purged scheme folds a row when the training window reaches it, and it never drops a row. So fold i has folded the rows 1:last(train_idx[i]), which are the rows that the batch expanding fold reads, and the call with no data rebuilds the training window of the batch fold.

Algorithm

  1. Refuse, through assert_online_entry, an estimator that is not the configuration alone. The check refuses a partial-fit state at entry, because the loop starts cold. It also refuses a TimeDependent schedule on a field that carries a state, because a schedule replaces the value through which the loop threads the state.
  2. Refuse, through assert_online_fee_source, an OnlinePortfolioSelection head whose fees carry a turnover term when the scheme threads no Previous-Weights Source.
  3. Under a multiple-randomised path, take the asset view of the path through fold_view(1), giving est and rd. A path is one asset subset crossed with the folds of the walk-forward, and a view slices a state by asset, so the arm threads the sliced estimator.
  4. Resolve every Online wrapper through update_online_estimator, giving est.
  5. Build the context of fold 1 through step_context, giving ctx. Fold the first training window train_idx[1] into est through online_step_fold. This is the warm-up. The loop folds no more rows for fold 1, so the warm-up is the step that runs inside the schedule entries of fold 1.
  6. Run the folds 1:n through thread_online_folds! from last_end = last(train_idx[1]), giving predictions and the threaded est.

Arguments

  • fit_fold: The per-fold resolution and callback that fold_loop builds. It takes (i, prev, est, rd, train): the index of the fold, the prediction of the last threadable fold or nothing, the estimator to resolve, rd, and the training window, which is nothing here. The arm advances prev through advance_previous_fold, as run_folds advances it, so a failed step does not move the previous weights.
  • est: The estimator, as the caller gave it to the loop.
  • n: The number of folds.
  • ElT: The element type of the per-fold result. It is positional for the reason that parallel_folds gives.
  • path_id: The path that the folds belong to, or nothing. Only the context of a fold reads it.
  • rd: The returns or the prices of the walk-forward.
  • train_idx: The training windows of every fold, in split order.
  • test_idx: The test windows of every fold, in split order. The context of each fold reads them.
  • fold_view: The asset view of a multiple-randomised path, or nothing.
  • pws: The Previous-Weights Source of the scheme, or nothing. It decides what threads_weights tests, and assert_online_fee_source reads it.

Validation

Returns

  • predictions::Vector{ElT}: One prediction per fold, in split order.
  • est: The threaded estimator, folded through last(train_idx[n]), for the Result to carry.

Related

source
online_folds(fit_fold, r::Resume, n::Integer, ::Type{ElT}, path_id = nothing; rd, train_idx, test_idx, fold_view, pws)

Runs the resumed arm of fold_loop, which continues an online walk-forward from the folds that its Result holds.

The loop takes this arm when the estimator slot holds a Resume. The arm resolves the declaration at entry, so no verb below the loop meets it. The new folds run through thread_online_folds!, the body that the online arm runs, so a resume continues exactly the loop that started the run. The TimeDependentContext of a schedule carries the i and n of the enumeration over the extended history. Every step checks the context that the state pinned, such as the asset names, a static panel and the column presence, as the one-shot run checks it.

Algorithm

  1. Refuse, through assert_online_fee_source, a head whose fee cannot be measured without a Previous-Weights Source, as the one-shot online arm refuses it.
  2. Find the index of the last fold that the Result holds through resume_fold_count, giving n_old.
  3. Refuse an rd that adds no fold, through assert_resume_folds.
  4. Take the last row of the training window of fold n_old, giving last_end.
  5. Refuse a partial last held fold, through assert_resume_full_fold.
  6. Advance over every fold of res.pred through advance_previous_fold, as the one-shot loop advanced over them, giving prev, the last threadable fold.
  7. Copy every state of res.opt through copy_states.
  8. Run the folds (n_old + 1):n on the copy from last_end through thread_online_folds!, giving predictions and the threaded est.

Arguments

  • fit_fold: The per-fold resolution and callback that fold_loop builds.
  • r: The declaration, holding the Result to continue.
  • n: The number of folds that the scheme enumerates over the extended history.
  • path_id: The path that the folds belong to, or nothing. Only the context of a fold reads it.
  • rd: The full history extended.
  • train_idx: The training windows of every fold, in split order.
  • test_idx: The test windows of every fold, in split order.
  • fold_view: Not read, because a multiple-randomised path never reaches this arm.
  • pws: The Previous-Weights Source of the scheme, or nothing.

Validation

Returns

  • predictions::Vector{ElT}: One prediction per new fold, in split order.
  • est: The threaded estimator, folded through last(train_idx[n]).

Related

source
PortfolioOptimisers.online_step_fold — Method
online_step_fold(est, ctx::Option{<:TimeDependentContext}, rd::Prices_RR)

Folds the rows that a fold gained into the estimator that the online arm threads. The step reads each of its schedules at the entry of that fold.

The default is partial_fit!, and every family except one takes it. assert_stateless_schedule lets a schedule reach only a field that carries no state. On every other family the step reads none of those fields. The call with no data that fits the fold reads them, and it resolves them itself. For example, the programme that this call solves reads a scheduled weight bound, and the buffer that the step fills never reads it. OnlinePortfolioSelection is the exception. Its step is the optimisation, because the step projects every row onto set, and a leader in its rule tree solves at every row. So the head has its own method, which resolves the schedules on set and in the rule tree.

ctx is nothing when the estimator carries no schedule, which is the usual case. The loop then builds no context.

Arguments

  • est: The estimator that the loop threads, with its schedules unresolved.
  • ctx: The context of the fold, or nothing.
  • rd: The rows that the fold gained, as returns, or as the prices that a Pipeline threads.

Validation

Returns

  • est: The estimator, with the rows of the fold folded in and its schedules unresolved.

Related

source
PortfolioOptimisers.one_previous_portfolio — Function
one_previous_portfolio(est, w_prev::VecNum)
one_previous_portfolio(est, w_prev::VecVecNum)

Returns the one portfolio that a fold reads as its previous weights, and refuses a population.

fold_loop passes the weights of the previous fold to factory when est needs_previous_weights. A term that reads them measures the trade from one portfolio. A turnover, a tracking error, a fee, and a custom constraint or objective are such terms. A frontier sweep gives a population, with one portfolio per sweep point. Each fold resolves the span of its frontier from its own data, so point k at one fold and point k at the next fold sit at different return levels. No portfolio of the population is the previous one. So the loop refuses the combination by name, and it does not charge the term against a portfolio that it would have to choose. Fold 1 reads no previous weights, so the refusal comes after the solve of fold 1. The batch arms and the online arm refuse alike.

Arguments

  • est: The estimator of the fold. The error names its type.
  • w_prev: The weights that the previous fold gives.

Validation

  • w_prev is one portfolio. The second method throws an ArgumentError, because a population holds more than one.

Returns

  • w_prev::VecNum: The weights, unchanged.

Related

source
PortfolioOptimisers.step_context — Function
step_context(td::Bool, i::Integer, n::Integer, rd, train_idx, test_idx, w_prev, path_id)

Returns the context against which the online arm resolves the schedules that the step of fold i reads, or nothing when td is false.

A usual run carries no schedule. online_folds and thread_online_folds! each read is_time_dependent once per call, and not once per fold. The estimator that they thread keeps its schedules unresolved from fold to fold, so one answer holds for every fold.

The per-fold copy of fold_loop builds the same record later in the fold, before the callback fits it. Both places call the constructor of TimeDependentContext with the same arguments, so the step and the call with no data of fold i resolve a schedule against equal records.

Arguments

  • td: Whether the threaded estimator carries a schedule.
  • i, n, rd, train_idx, test_idx, w_prev, path_id: As the constructor of TimeDependentContext takes them.

Returns

  • ctx::Option{<:TimeDependentContext}: The context of the fold, or nothing.

Related

source