Online fold loop: private API
A scheme built by OnlineIndexWalkForward, OnlineDateWalkForward or OnlineHindsightSplit is an online scheme. With an online scheme, PortfolioOptimisers.fold_loop does not refit the estimator on each training window. It fits one estimator on the first training window. At each later fold it adds to that estimator only the rows that the training window gained, by default with partial_fit!.
The loop then gives the callback a PortfolioOptimisers.Fold whose train field is nothing, because the estimator already holds the training rows. Where a batch fold refits, this fold calls optimise(opt) with no returns. At fold i the estimator holds every row up to the end of the training window. The batch expanding-window fold reads the same rows. The weights of each fold are therefore those of the batch walk-forward, to the tolerance of the moment estimates and of the solver.
PortfolioOptimisers.fit_fold_result — Function
fit_fold_result(opt::NonFiniteAllocationOptimisationEstimator, rd::ReturnsResult, train_idx::VecInt, cols)
fit_fold_result(opt::NonFiniteAllocationOptimisationEstimator, rd::ReturnsResult, ::Nothing, cols)Fits the estimator of one fold. Dispatch on train_idx selects the arm.
These are the two arms of the estimator form of fit_and_predict. A training window selects the refit that every fold of the batch arms runs, optimise(opt, port_opt_view(rd, train_idx, cols)). nothing means that the estimator holds its window, because the online arm of fold_loop folded every row of it. This arm calls optimise(opt) with no returns. That call rebuilds the returns data from the state and runs the batch path over it. When cols is not :, the caller takes the asset view of opt before either arm, so it slices a stepped estimator and a cold one by asset in the same way.
Arguments
opt: The estimator of the fold, already viewed by asset.rd: The returns data of the fold. The arm with no training window does not read it.train_idx: The training window of the fold, ornothingfor a stepped estimator.cols: The asset columns of the fold. The arm with no training window does not read them.
Validation
- On the arm with no training window, everything that
optimise(opt)refuses. A cold estimator holds no window, andoptimise(opt)refuses it with anArgumentError.
Returns
res::OptimisationResult: The fitted result of the fold.
Related
PortfolioOptimisers.thread_online_folds! — Function
thread_online_folds!(predictions, fit_fold, est, folds, prev; rd, train_idx, test_idx, n, last_end, pws, path_id)Folds and reads out the folds folds of a walk-forward, and threads est from one fold to the next.
The online arm and the resumed arm of fold_loop share this body, so a resume continues the loop that started the run.
Algorithm
- Read
is_time_dependentonestonce, givingtd_flag. - For each fold
iinfolds, take the last row of its training window, givingstop. - When
stop > last_end, build the context of the step throughstep_context, with the weights thatprevious_weightsreads fromprev, givingctx. Fold the rows(last_end + 1):stopofrdintoestthroughonline_step_fold, and setlast_endtostop. Otherwise the fold gained no row, which is the case of fold 1 after the warm-up, andestdoes not change. - Call
fit_fold(i, prev, est, rd, nothing), which makes the per-fold copy as the batch arms make it, giving the prediction of foldi. Store it atpredictions[i - first(folds) + 1]. - Advance
prevpast the prediction throughadvance_previous_fold. - After the last fold, return
est.
Arguments
predictions: The vector that receives the predictions, with one slot per fold offolds.fit_fold: The per-fold resolution and callback thatfold_loopbuilds.est: The estimator, folded throughlast_end.folds: The indices of the folds to run, a contiguous range of the enumeration of the scheme.prev: The prediction of the last threadable fold beforefirst(folds), ornothing.rd: The returns or the prices of the walk-forward.train_idx: The training windows of every fold, in split order.test_idx: The test windows of every fold, in split order. The context of each fold reads them.n: The number of folds that the scheme enumerates. The context of each fold reads it.last_end: The last row thatesthas folded.pws: The Previous-Weights Source of the scheme, ornothing.path_id: The path that the folds belong to, ornothing. Only the context of a fold reads it.
Returns
est: The estimator, folded throughlast(train_idx[last(folds)]), with its schedules unresolved.
Related
PortfolioOptimisers.online_folds — Function
online_folds(fit_fold, est, n::Integer, ::Type{ElT}, path_id = nothing; rd, train_idx, test_idx, fold_view, pws)Runs the n folds of a walk-forward through the online step, and threads one estimator from fold to fold.
This is the online arm of fold_loop, which the loop takes when the scheme is an Online Scheme (folds_are_stepped). The arm logs no message. The caller chooses an Online Scheme by name through its constructor, so the arm has no accident to report. run_folds is different, because it reports a sequential run that the loop chose on its own. A Resume in the estimator slot takes the method of Resume, which skips the warm-up and the folds that the Result holds, and runs the same body from the fold after them.
Fold for fold, the run reaches the weights of the batch expanding walk-forward, to the tolerance of the moment layer and of the solver. A purged scheme folds a row when the training window reaches it, and it never drops a row. So fold i has folded the rows 1:last(train_idx[i]), which are the rows that the batch expanding fold reads, and the call with no data rebuilds the training window of the batch fold.
Algorithm
- Refuse, through
assert_online_entry, an estimator that is not the configuration alone. The check refuses a partial-fit state at entry, because the loop starts cold. It also refuses aTimeDependentschedule on a field that carries a state, because a schedule replaces the value through which the loop threads the state. - Refuse, through
assert_online_fee_source, anOnlinePortfolioSelectionhead whose fees carry a turnover term when the scheme threads no Previous-Weights Source. - Under a multiple-randomised path, take the asset view of the path through
fold_view(1), givingestandrd. A path is one asset subset crossed with the folds of the walk-forward, and a view slices a state by asset, so the arm threads the sliced estimator. - Resolve every
Onlinewrapper throughupdate_online_estimator, givingest. - Build the context of fold 1 through
step_context, givingctx. Fold the first training windowtrain_idx[1]intoestthroughonline_step_fold. This is the warm-up. The loop folds no more rows for fold 1, so the warm-up is the step that runs inside the schedule entries of fold 1. - Run the folds
1:nthroughthread_online_folds!fromlast_end = last(train_idx[1]), givingpredictionsand the threadedest.
Arguments
fit_fold: The per-fold resolution and callback thatfold_loopbuilds. It takes(i, prev, est, rd, train): the index of the fold, the prediction of the last threadable fold ornothing, the estimator to resolve,rd, and the training window, which isnothinghere. The arm advancesprevthroughadvance_previous_fold, asrun_foldsadvances it, so a failed step does not move the previous weights.est: The estimator, as the caller gave it to the loop.n: The number of folds.ElT: The element type of the per-fold result. It is positional for the reason thatparallel_foldsgives.path_id: The path that the folds belong to, ornothing. Only the context of a fold reads it.rd: The returns or the prices of the walk-forward.train_idx: The training windows of every fold, in split order.test_idx: The test windows of every fold, in split order. The context of each fold reads them.fold_view: The asset view of a multiple-randomised path, ornothing.pws: The Previous-Weights Source of the scheme, ornothing. It decides whatthreads_weightstests, andassert_online_fee_sourcereads it.
Validation
- Everything that
assert_online_entryandassert_online_fee_sourcerefuse.
Returns
predictions::Vector{ElT}: One prediction per fold, in split order.est: The threaded estimator, folded throughlast(train_idx[n]), for the Result to carry.
Related
online_folds(fit_fold, r::Resume, n::Integer, ::Type{ElT}, path_id = nothing; rd, train_idx, test_idx, fold_view, pws)Runs the resumed arm of fold_loop, which continues an online walk-forward from the folds that its Result holds.
The loop takes this arm when the estimator slot holds a Resume. The arm resolves the declaration at entry, so no verb below the loop meets it. The new folds run through thread_online_folds!, the body that the online arm runs, so a resume continues exactly the loop that started the run. The TimeDependentContext of a schedule carries the i and n of the enumeration over the extended history. Every step checks the context that the state pinned, such as the asset names, a static panel and the column presence, as the one-shot run checks it.
Algorithm
- Refuse, through
assert_online_fee_source, a head whose fee cannot be measured without a Previous-Weights Source, as the one-shot online arm refuses it. - Find the index of the last fold that the Result holds through
resume_fold_count, givingn_old. - Refuse an
rdthat adds no fold, throughassert_resume_folds. - Take the last row of the training window of fold
n_old, givinglast_end. - Refuse a partial last held fold, through
assert_resume_full_fold. - Advance over every fold of
res.predthroughadvance_previous_fold, as the one-shot loop advanced over them, givingprev, the last threadable fold. - Copy every state of
res.optthroughcopy_states. - Run the folds
(n_old + 1):non the copy fromlast_endthroughthread_online_folds!, givingpredictionsand the threadedest.
Arguments
fit_fold: The per-fold resolution and callback thatfold_loopbuilds.r: The declaration, holding the Result to continue.n: The number of folds that the scheme enumerates over the extended history.path_id: The path that the folds belong to, ornothing. Only the context of a fold reads it.rd: The full history extended.train_idx: The training windows of every fold, in split order.test_idx: The test windows of every fold, in split order.fold_view: Not read, because a multiple-randomised path never reaches this arm.pws: The Previous-Weights Source of the scheme, ornothing.
Validation
- Everything that
assert_online_fee_source,resume_fold_count,assert_resume_foldsandassert_resume_full_foldrefuse.
Returns
predictions::Vector{ElT}: One prediction per new fold, in split order.est: The threaded estimator, folded throughlast(train_idx[n]).
Related
PortfolioOptimisers.online_step_fold — Method
online_step_fold(est, ctx::Option{<:TimeDependentContext}, rd::Prices_RR)Folds the rows that a fold gained into the estimator that the online arm threads. The step reads each of its schedules at the entry of that fold.
The default is partial_fit!, and every family except one takes it. assert_stateless_schedule lets a schedule reach only a field that carries no state. On every other family the step reads none of those fields. The call with no data that fits the fold reads them, and it resolves them itself. For example, the programme that this call solves reads a scheduled weight bound, and the buffer that the step fills never reads it. OnlinePortfolioSelection is the exception. Its step is the optimisation, because the step projects every row onto set, and a leader in its rule tree solves at every row. So the head has its own method, which resolves the schedules on set and in the rule tree.
ctx is nothing when the estimator carries no schedule, which is the usual case. The loop then builds no context.
Arguments
est: The estimator that the loop threads, with its schedules unresolved.ctx: The context of the fold, ornothing.rd: The rows that the fold gained, as returns, or as the prices that aPipelinethreads.
Validation
- Everything that
partial_fit!refuses onestandrd.
Returns
est: The estimator, with the rows of the fold folded in and its schedules unresolved.
Related
PortfolioOptimisers.one_previous_portfolio — Function
one_previous_portfolio(est, w_prev::VecNum)
one_previous_portfolio(est, w_prev::VecVecNum)Returns the one portfolio that a fold reads as its previous weights, and refuses a population.
fold_loop passes the weights of the previous fold to factory when est needs_previous_weights. A term that reads them measures the trade from one portfolio. A turnover, a tracking error, a fee, and a custom constraint or objective are such terms. A frontier sweep gives a population, with one portfolio per sweep point. Each fold resolves the span of its frontier from its own data, so point k at one fold and point k at the next fold sit at different return levels. No portfolio of the population is the previous one. So the loop refuses the combination by name, and it does not charge the term against a portfolio that it would have to choose. Fold 1 reads no previous weights, so the refusal comes after the solve of fold 1. The batch arms and the online arm refuse alike.
Arguments
est: The estimator of the fold. The error names its type.w_prev: The weights that the previous fold gives.
Validation
w_previs one portfolio. The second method throws anArgumentError, because a population holds more than one.
Returns
w_prev::VecNum: The weights, unchanged.
Related
PortfolioOptimisers.step_context — Function
step_context(td::Bool, i::Integer, n::Integer, rd, train_idx, test_idx, w_prev, path_id)Returns the context against which the online arm resolves the schedules that the step of fold i reads, or nothing when td is false.
A usual run carries no schedule. online_folds and thread_online_folds! each read is_time_dependent once per call, and not once per fold. The estimator that they thread keeps its schedules unresolved from fold to fold, so one answer holds for every fold.
The per-fold copy of fold_loop builds the same record later in the fold, before the callback fits it. Both places call the constructor of TimeDependentContext with the same arguments, so the step and the call with no data of fold i resolve a schedule against equal records.
Arguments
td: Whether the threaded estimator carries a schedule.i,n,rd,train_idx,test_idx,w_prev,path_id: As the constructor ofTimeDependentContexttakes them.
Returns
ctx::Option{<:TimeDependentContext}: The context of the fold, ornothing.
Related