Pipeline
The Pipeline estimator fits a list of steps as one estimator. The steps can be data preparation, prior estimation, phylogeny, uncertainty sets, constraint generation and optimisation. What a step computes replaces the matching setting of the optimiser at the end of the pipeline. For each step that the pipeline does not have, the optimiser computes the value itself, as it does outside a pipeline.
PortfolioOptimisers.Pipeline — Type
struct Pipeline{__T_names, __T_steps, __T_cache} <: AbstractPipelineEstimatorRuns an ordered list of steps as one end-to-end portfolio workflow.
The steps run from left to right over a PipelineContext. A step is an ordinary estimator. It is a preprocessing, prior, phylogeny, uncertainty-set, constraint or optimisation estimator, a nested Pipeline, or a PipelineStep wrapper. pipe_writes and pipe_reads give the slots that the family of a step writes and reads. fit runs the steps in order, and the computed slots replace the configuration of the last optimiser, see inject_context. When the pipeline has no step for a stage, the optimiser computes that value itself, so each stage is optional.
A last optimiser is optional too. A prior-only pipeline is valid, but a prediction needs weights.
Algorithm
The keyword constructor runs these steps.
- Split each element of
stepsinto its explicit name, ornothing, and its estimator, givingexplicitandests. - Count the steps that write each slot, giving
counts. The count includes the steps with an explicit name. - Name each step, giving
names. A step keeps its explicit name. A step whose slot no other step writes takes the slot name, such as"prior". Every other step takes the slot name and its position among the steps of that slot, such as"prices_1"and"prices_2". - Call the positional constructor, which runs each check of
## Validation. - Log the warning of
warn_raw_readerswhen it applies.
The positional constructor runs the checks, so a rebuild through rebuild_struct and a lens write through Accessors.set run them too. A lens write that changes the steps to an order that the keyword constructor refuses is refused. The keyword constructor alone logs the warning, so a rebuild at each online step does not log it again.
Fields
names: The name of each step, in the order ofsteps.
steps: The step estimators, in the order of execution.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
Constructors
Pipeline(; steps::Union{Tuple, AbstractVector}, cache::Option{<:AbstractPartialFitState} = nothing) -> PipelineGive the steps in the order of execution. Each element is a step estimator or a "name" => estimator pair.
The steps keep the form that you give, a Tuple or a vector, and the names and the fitted results of a PipelineResult take the same form, see steps_form. The two forms give the same results. A Tuple compiles the methods of each pipeline for its own step types. A vector compiles them for its element type, which can cost less time to the first result when you build many pipelines. The positional constructor takes names and steps as a Tuple or a vector too.
Validation
!isempty(steps). Raises anIsEmptyError.pipe_writesis defined for each step. Raises anArgumentError.- A
TrainTestSplitis the first step only, seeassert_split_position. - An optimisation step is the last step only, see
assert_opt_last. It writes the:optslot, and no step runs after it. - Each step declares the slot and the target that it writes, see
assert_step_declarations. - A nested pipeline changes no Data Slot that the outer pipeline does not receive, see
assert_nested_data_slots. - Each constraint step resolves to one routing target, see
assert_constraint_targets. - The last optimiser can receive each target that an earlier step writes, see
assert_routable. - A prior step that states
x_src = ExcessSource()does not feed an optimiser that tracks aDataBenchmarkTracking, seeassert_pipeline_data_benchmark. - An earlier step writes each slot that a step reads, or the pipeline input fills it with
pricesorreturns, seeassert_slot_order. Raises anArgumentError. - No step writes a slot that makes stale a slot that an earlier step wrote, see
PIPELINE_INVALIDATESandassert_slot_order. A step that rewrites:returnsafter a prior, phylogeny, uncertainty or constraint step leaves that result on an old asset universe. Raises anArgumentError. - The step names are unique. Raises an
ArgumentError.
The steps whose values the last optimiser receives join the group of that optimiser. When a reader of the group states x_src = ExcessSource(), and a reader at x_src = nothing reads the raw returns data, the constructor logs one warning that names the path of that reader, see warn_raw_readers and pipeline_frames. The pipeline then builds as configured. The group also decides the gap drop, as in a head: when a reader of the group states ExcessSource(), each step of the group and the optimiser read the returns data without the observations at which a vector benchmark has a gap, in batch and online, see pipeline_gap_sources. Each reader still reads the frame that its own x_src states.
Online form
A Pipeline takes the online step. partial_fit! runs the steps in order, and folds each block of observations through them into the row owner. The row owner is the prior step, or the optimiser step when the pipeline has no prior step. fit(pipe) with no data returns the fitted PipelineResult. Each step before the row owner is one of three kinds.
- A row-local step folds its rows and emits the transformed rows.
PricesToReturns,PriceGapFillwith aCarriedPrice, andMissingDataFilteratrow_thr = 1are row-local. - A universe-only step folds nothing. An
AbstractAssetSelectorand the column filter of aMissingDataFilterare universe-only.fit(pipe)with no data fits such a step again over the rows of the row owner, and applies its universe as a view. - A window-valued step, and every other step that writes a Data Slot, stops the warm-up with an error that names it.
Online(pipe)is the declared refit that accepts such a step.
A cap on the row owner alone, Online(pe; max_history = w), is a window that counts the rows of the row owner. A row-local step before it folds a state across the front edge of that window, so the warm-up refuses that pair with an error that names it. The rolling window through a Pipeline is Online(pipe; max_history = w), which fits every step again over the window. cache holds the Fold Context of the Pipeline when a prior step owns the rows, or the buffer of the input data that Online(pipe) seeds. It is nothing until a step writes one. See partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{PipelineBufferState, ReturnsBufferState}}}, data::Prices_RR) and fit(pipe::Pipeline).
Examples
julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> pipe.names("returns", "prior", "opt")Related
PortfolioOptimisers.PipelineResult — Type
struct PipelineResult{__T_names, __T_results, __T_ctx} <: AbstractPipelineResultHolds the fitted result of each step of a pipeline, and the last Pipeline Context.
The context is a PipelineContext. Its slots hold the computed data, prior, phylogeny, uncertainty sets, constraints and optimisation result of the Pipeline.
Read the result of a step by its name with getindex, as in res["prior"], or by its position through the results field, as in res.results[2]. A name that no step has raises an ArgumentError that suggests the nearest name. An integer index keeps the length-1 container behaviour of the package, so res[1] === res. The w property returns the weights of the optimisation result, res.ctx.opt.w. It raises a PropertyPathError when the pipeline has no optimisation result.
Fields
names: The name of each step, in the order ofresults.
results: The fitted result of each step, in step order.
ctx: The lastPipelineContext.
Constructors
PipelineResult(; names::Union{Tuple{Vararg{String}}, AbstractVector{<:String}}, results::Union{Tuple, AbstractVector}, ctx::PipelineContext) -> PipelineResultKeywords correspond to the struct's fields. fit builds the Result, and Accessors.set builds it again through this constructor.
Related
StatsAPI.fit — Method
StatsAPI.fit(pipe::Pipeline, data::Prices_RR) -> PipelineResultFit a Pipeline on price data or returns data.
fit has no folds, so each TimeDependent schedule step resolves to its explicit default, see reset_time_dependent_estimator. A schedule with no default raises a TimeDependentDefaultError. Backtest such a pipeline with cross_val_predict, whose folds resolve the schedule. Inside a fold loop the reset changes nothing, because the loop first replaces each schedule with its value for the fold.
Algorithm
- When the pipeline has a schedule step, reset each schedule to its
default. - Fill the slot of the input type, giving
ctx. APricesResultfillsprices, and aReturnsResultfillsreturns. So a pipeline for returns data has no price steps, because a price step raises anIsNothingErrorwhen thepricesslot is empty. - Find the source of the gap drop of each step with
pipeline_gap_sources. The steps whose values the last optimiser receives, and the optimiser, drop a gap of a vector benchmark when a reader of their group statesExcessSource(), as the readers of one head do. - For each step in order, inject
ctxinto an optimisation step withmaybe_inject_step. Then run the step withrun_group_stepunder its source, giving its fitted result and the newctx. - Return the step names, the fitted results in step order and the last
ctxas aPipelineResult.
Arguments
pipe: The pipeline.data: The input data, aPricesResultor aReturnsResult.
Returns
res::PipelineResult: The named fitted result of each step, and the last context.
Examples
julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4), [100.0 101.0; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> res = fit(pipe, PricesResult(; X = X));julia> res.w2-element Vector{Float64}: 0.5 0.5Related
StatsAPI.predict — Function
predict(res::PipelineResult, data::AbstractPricesResult,
test_idx = Colon(), cols = Colon()) -> PredictionResult
predict(res::PipelineResult, data::AbstractPricesResult,
test_idxs::VecVecInt, cols = Colon()) -> Vector{<:PredictionResult}
predict(res::PipelineResult, data::AbstractReturnsResult,
test_idx = Colon(), cols = Colon()) -> PredictionResult
predict(res::PipelineResult, data::AbstractReturnsResult,
test_idxs::VecVecInt, cols = Colon()) -> Vector{<:PredictionResult}Apply a fitted pipeline to a new data window, and return the PredictionResult that the weight-level functions read.
test_idx selects the rows of the window, and cols selects its asset columns. The fitted preprocessing steps change the window in step order. They use the universe, the imputation parameters and the returns conversion of the training window, so no statistic of the test window changes the window. The weight-level predict then reads the window, so the scorers and the risk measures apply without a change.
A vector of index vectors predicts on each window and returns one result per window. The cross-validation functions read this shape.
Algorithm
- Read the optimisation result
optof the pipeline. - Take the rows
test_idxand the columnscolsofdatawithport_opt_view, giving the window. Returns data with:for both is the window as it is. - Apply the fitted steps to the window with
apply_fitted_steps, givingrd. For price data, check thatrdis returns data. - Check the asset axis of
rdwithassert_universe_aligned. - Return
predict(opt, rd), with the keyword arguments. For returns data, apply the steps withapply_fitted_steps_keepingin step 3, and recordtest_idxon the prediction withpipeline_fold_predictionwhen every step keeps the observations, seekeeps_observations. Price data records no rows, becausePricesToReturnsdrops the first price row, so the positions of the returns are not those oftest_idx. Price data carries timestamps, and the timestamps name the rows instead.
Arguments
res: The fittedPipelineResult.data: Price data or returns data that contains the window, aPricesResultor aReturnsResult.test_idx: The rows ofdatain the window, as integer indices or:for all rows. Price data also takes a vector of timestamps.test_idxs: Several such windows, as a vector of index vectors.cols: The columns ofdatain the window, as integer indices or:for all assets. The columns must give the training universe. So a subset of the columns needs a fit on the same subset, asfit_and_predict(pipe, data, train_idx, test_idx, cols)does.
Keyword Arguments
wd,hwd,fa,store_weight_path,strict,w_prev: The keywords ofpredict(res::NonFiniteAllocationOptimisationResult, rd::ReturnsResult), passed on.
Validation
!isnothing(res.ctx.opt). Raises anIsNothingError.- For price data, the fitted steps convert the window to returns, so the pipeline contains a
PricesToReturnsstep. Raises anArgumentError. - The window passes
assert_universe_aligned.
Returns
pred::PredictionResult: The weight-level prediction on the changed window, or one such result per window when several are given.
Related
PortfolioOptimisers.fit_predict — Method
fit_predict(pipe::Pipeline, data::Prices_RR) -> PredictionResultFit a pipeline on data, and predict with it at once.
When the pipeline starts with a TrainTestSplit, the prediction uses the test window of the split, which no fitted step saw. Otherwise the prediction uses data itself, in sample. So one call with a split fits on the training rows and scores on the test rows.
Algorithm
- Fit
pipeondatawithfit, givingres. - Read the test window of the split with
holdout_window, givingtest. - Predict with
resontest, or ondatawhentestisnothing.
Arguments
pipe: The pipeline.data::Prices_RR: Price data or returns data.
Returns
pred::PredictionResult: The prediction on the test window when the pipeline has a split, and ondataotherwise.
Related
PortfolioOptimisers.port_opt_view — Method
port_opt_view(pipe::AbstractPipelineEstimator, i, args...; kwargs...)Refuse an asset view of a Pipeline, or of any other AbstractPipelineEstimator.
A meta-optimiser such as NestedClustered, Stacking or SubsetResampling takes a port_opt_view of its inner estimator to build a portfolio over a subset of the assets. The asset universe of a pipeline is fitted state, because a missing-data filter selects it from the training window. So an asset view before the fit has no defined meaning. A meta-optimiser cannot wrap a Pipeline, but it can be the optimisation step of one.
Validation
- Each call raises an
ArgumentError.
Related
PortfolioOptimisers.implicit_constraint_target — Function
implicit_constraint_target(_::WeightBounds) -> Symbol
Return the routing target that a constraint result names by its type alone, or nothing.
Four result types name one optimiser field each. A WeightBounds goes to :wb, a LinearConstraint to :lcse, a phylogeny constraint result to :ple, and a RiskBudget to :rkb. Every other type returns nothing, and needs a target beside it, see TargetedConstraint.
This function is the only part of the fan-out that the type decides. It also decides whether the value of a step needs a wrapper. add_constraint_result wraps a value only when the value cannot name its own target, so the constraints slot holds a bare result where it can.
Arguments
c: A constraint result.
Returns
target::Union{Nothing, Symbol}: One ofPIPELINE_ROUTING_TARGETS, ornothing.
Related
Holdout splitting
A TrainTestSplit step keeps a test window aside before any other step runs. It must be the first step, because a fitted step before it would see the rows of the test window. You cannot use it with cross-validation, which makes its own training and test windows. fit_predict(pipe, data) predicts on the test window that the split kept aside.
PortfolioOptimisers.assert_split_position — Function
assert_split_position(ests)
Validate that a TrainTestSplit is the first step of a Pipeline, and that no nested step contains one.
The split keeps the test window away from every fitted step. A stateful step before the split reads the rows of the test window. A MissingDataFilter selects the universe from them, and a PriceGapFill computes fill values from them, so the fitted state carries test data into the training workflow. At the first position, no step runs before the split. The check refuses a split inside a nested pipeline at any position, because holdout_window reads the steps of the outer pipeline only.
Arguments
ests: The step estimators.
Validation
- A
TrainTestSplitstep is at index 1. Raises anArgumentError. - No nested
Pipelineand noPipelineStepcontains aTrainTestSplit. Raises anArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.has_split — Function
has_split(_) -> Bool
Return whether a step is a TrainTestSplit, or contains one.
The search enters each nested Pipeline and each PipelineStep. A split inside a nested pipeline sees data that an outer step already changed, and the first position of the split exists to prevent that. The same search tells whether a whole pipeline has a Holdout Split, which the cross-validation functions check before they run.
Returns
truefor aTrainTestSplit, and for aPipelineor aPipelineStepthat contains one.falsefor every other step.
Related
PortfolioOptimisers.assert_no_holdout — Function
assert_no_holdout(pipe::Pipeline)
Refuse a Pipeline that contains a TrainTestSplit in the cross-validation functions.
A Holdout Split and a cross-validation scheme are two methods of evaluation. The scheme sets the training and the test window of each fold. A split in the pipeline takes a second holdout from the training window of each fold, and keeps a test window that no function reads. So each fold loses training data without a message. One call uses one method of evaluation.
Arguments
pipe: The pipeline.
Validation
!has_split(pipe). Raises anArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.holdout_window — Function
holdout_window(res::PipelineResult) -> Any
Return the test window that the TrainTestSplit step of a fitted pipeline keeps, or nothing when the pipeline has no split.
The function reads the steps of the outer pipeline only. assert_split_position keeps a split there.
Arguments
res: The fittedPipelineResult.
Returns
- The
testwindow of theTrainTestSplitResult, ornothing.
Related