Pipeline

The Pipeline estimator fits a list of steps as one estimator. The steps can be data preparation, prior estimation, phylogeny, uncertainty sets, constraint generation and optimisation. What a step computes replaces the matching setting of the optimiser at the end of the pipeline. For each step that the pipeline does not have, the optimiser computes the value itself, as it does outside a pipeline.

PortfolioOptimisers.Pipeline — Type
struct Pipeline{__T_names, __T_steps, __T_cache} <: AbstractPipelineEstimator

Runs an ordered list of steps as one end-to-end portfolio workflow.

The steps run from left to right over a PipelineContext. A step is an ordinary estimator. It is a preprocessing, prior, phylogeny, uncertainty-set, constraint or optimisation estimator, a nested Pipeline, or a PipelineStep wrapper. pipe_writes and pipe_reads give the slots that the family of a step writes and reads. fit runs the steps in order, and the computed slots replace the configuration of the last optimiser, see inject_context. When the pipeline has no step for a stage, the optimiser computes that value itself, so each stage is optional.

A last optimiser is optional too. A prior-only pipeline is valid, but a prediction needs weights.

Algorithm

The keyword constructor runs these steps.

  1. Split each element of steps into its explicit name, or nothing, and its estimator, giving explicit and ests.
  2. Count the steps that write each slot, giving counts. The count includes the steps with an explicit name.
  3. Name each step, giving names. A step keeps its explicit name. A step whose slot no other step writes takes the slot name, such as "prior". Every other step takes the slot name and its position among the steps of that slot, such as "prices_1" and "prices_2".
  4. Call the positional constructor, which runs each check of ## Validation.
  5. Log the warning of warn_raw_readers when it applies.

The positional constructor runs the checks, so a rebuild through rebuild_struct and a lens write through Accessors.set run them too. A lens write that changes the steps to an order that the keyword constructor refuses is refused. The keyword constructor alone logs the warning, so a rebuild at each online step does not log it again.

Fields

  • names: The name of each step, in the order of steps.
  • steps: The step estimators, in the order of execution.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.

Constructors

Pipeline(; steps::Union{Tuple, AbstractVector},           cache::Option{<:AbstractPartialFitState} = nothing) -> Pipeline

Give the steps in the order of execution. Each element is a step estimator or a "name" => estimator pair.

The steps keep the form that you give, a Tuple or a vector, and the names and the fitted results of a PipelineResult take the same form, see steps_form. The two forms give the same results. A Tuple compiles the methods of each pipeline for its own step types. A vector compiles them for its element type, which can cost less time to the first result when you build many pipelines. The positional constructor takes names and steps as a Tuple or a vector too.

Validation

  • !isempty(steps). Raises an IsEmptyError.
  • pipe_writes is defined for each step. Raises an ArgumentError.
  • A TrainTestSplit is the first step only, see assert_split_position.
  • An optimisation step is the last step only, see assert_opt_last. It writes the :opt slot, and no step runs after it.
  • Each step declares the slot and the target that it writes, see assert_step_declarations.
  • A nested pipeline changes no Data Slot that the outer pipeline does not receive, see assert_nested_data_slots.
  • Each constraint step resolves to one routing target, see assert_constraint_targets.
  • The last optimiser can receive each target that an earlier step writes, see assert_routable.
  • A prior step that states x_src = ExcessSource() does not feed an optimiser that tracks a DataBenchmarkTracking, see assert_pipeline_data_benchmark.
  • An earlier step writes each slot that a step reads, or the pipeline input fills it with prices or returns, see assert_slot_order. Raises an ArgumentError.
  • No step writes a slot that makes stale a slot that an earlier step wrote, see PIPELINE_INVALIDATES and assert_slot_order. A step that rewrites :returns after a prior, phylogeny, uncertainty or constraint step leaves that result on an old asset universe. Raises an ArgumentError.
  • The step names are unique. Raises an ArgumentError.

The steps whose values the last optimiser receives join the group of that optimiser. When a reader of the group states x_src = ExcessSource(), and a reader at x_src = nothing reads the raw returns data, the constructor logs one warning that names the path of that reader, see warn_raw_readers and pipeline_frames. The pipeline then builds as configured. The group also decides the gap drop, as in a head: when a reader of the group states ExcessSource(), each step of the group and the optimiser read the returns data without the observations at which a vector benchmark has a gap, in batch and online, see pipeline_gap_sources. Each reader still reads the frame that its own x_src states.

Online form

A Pipeline takes the online step. partial_fit! runs the steps in order, and folds each block of observations through them into the row owner. The row owner is the prior step, or the optimiser step when the pipeline has no prior step. fit(pipe) with no data returns the fitted PipelineResult. Each step before the row owner is one of three kinds.

  • A row-local step folds its rows and emits the transformed rows. PricesToReturns, PriceGapFill with a CarriedPrice, and MissingDataFilter at row_thr = 1 are row-local.
  • A universe-only step folds nothing. An AbstractAssetSelector and the column filter of a MissingDataFilter are universe-only. fit(pipe) with no data fits such a step again over the rows of the row owner, and applies its universe as a view.
  • A window-valued step, and every other step that writes a Data Slot, stops the warm-up with an error that names it. Online(pipe) is the declared refit that accepts such a step.

A cap on the row owner alone, Online(pe; max_history = w), is a window that counts the rows of the row owner. A row-local step before it folds a state across the front edge of that window, so the warm-up refuses that pair with an error that names it. The rolling window through a Pipeline is Online(pipe; max_history = w), which fits every step again over the window. cache holds the Fold Context of the Pipeline when a prior step owns the rows, or the buffer of the input data that Online(pipe) seeds. It is nothing until a step writes one. See partial_fit!(pipe::Pipeline{<:Any, <:Any, <:Option{<:Union{PipelineBufferState, ReturnsBufferState}}}, data::Prices_RR) and fit(pipe::Pipeline).

Examples

julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> pipe.names("returns", "prior", "opt")

Related

source
PortfolioOptimisers.PipelineResult — Type
struct PipelineResult{__T_names, __T_results, __T_ctx} <: AbstractPipelineResult

Holds the fitted result of each step of a pipeline, and the last Pipeline Context.

The context is a PipelineContext. Its slots hold the computed data, prior, phylogeny, uncertainty sets, constraints and optimisation result of the Pipeline.

Read the result of a step by its name with getindex, as in res["prior"], or by its position through the results field, as in res.results[2]. A name that no step has raises an ArgumentError that suggests the nearest name. An integer index keeps the length-1 container behaviour of the package, so res[1] === res. The w property returns the weights of the optimisation result, res.ctx.opt.w. It raises a PropertyPathError when the pipeline has no optimisation result.

Fields

  • names: The name of each step, in the order of results.
  • results: The fitted result of each step, in step order.

Constructors

PipelineResult(;    names::Union{Tuple{Vararg{String}}, AbstractVector{<:String}},    results::Union{Tuple, AbstractVector},    ctx::PipelineContext) -> PipelineResult

Keywords correspond to the struct's fields. fit builds the Result, and Accessors.set builds it again through this constructor.

Related

source
StatsAPI.fit — Method
StatsAPI.fit(pipe::Pipeline, data::Prices_RR) -> PipelineResult

Fit a Pipeline on price data or returns data.

fit has no folds, so each TimeDependent schedule step resolves to its explicit default, see reset_time_dependent_estimator. A schedule with no default raises a TimeDependentDefaultError. Backtest such a pipeline with cross_val_predict, whose folds resolve the schedule. Inside a fold loop the reset changes nothing, because the loop first replaces each schedule with its value for the fold.

Algorithm

  1. When the pipeline has a schedule step, reset each schedule to its default.
  2. Fill the slot of the input type, giving ctx. A PricesResult fills prices, and a ReturnsResult fills returns. So a pipeline for returns data has no price steps, because a price step raises an IsNothingError when the prices slot is empty.
  3. Find the source of the gap drop of each step with pipeline_gap_sources. The steps whose values the last optimiser receives, and the optimiser, drop a gap of a vector benchmark when a reader of their group states ExcessSource(), as the readers of one head do.
  4. For each step in order, inject ctx into an optimisation step with maybe_inject_step. Then run the step with run_group_step under its source, giving its fitted result and the new ctx.
  5. Return the step names, the fitted results in step order and the last ctx as a PipelineResult.

Arguments

Returns

  • res::PipelineResult: The named fitted result of each step, and the last context.

Examples

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4),                     [100.0 101.0; 102.0 103.0; 101.0 104.0; 103.0 102.0], ["A", "B"]);julia> pipe = Pipeline(; steps = (PricesToReturns(), EmpiricalPrior(), EqualWeighted()));julia> res = fit(pipe, PricesResult(; X = X));julia> res.w2-element Vector{Float64}: 0.5 0.5

Related

source
StatsAPI.predict — Function
predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult

predict(res::PipelineResult, data::AbstractPricesResult,
                      test_idxs::VecVecInt, cols = Colon()) -> Vector{<:PredictionResult}

predict(res::PipelineResult, data::AbstractReturnsResult,
                      test_idx = Colon(), cols = Colon()) -> PredictionResult

predict(res::PipelineResult, data::AbstractReturnsResult,
                      test_idxs::VecVecInt, cols = Colon()) -> Vector{<:PredictionResult}

Apply a fitted pipeline to a new data window, and return the PredictionResult that the weight-level functions read.

test_idx selects the rows of the window, and cols selects its asset columns. The fitted preprocessing steps change the window in step order. They use the universe, the imputation parameters and the returns conversion of the training window, so no statistic of the test window changes the window. The weight-level predict then reads the window, so the scorers and the risk measures apply without a change.

A vector of index vectors predicts on each window and returns one result per window. The cross-validation functions read this shape.

Algorithm

  1. Read the optimisation result opt of the pipeline.
  2. Take the rows test_idx and the columns cols of data with port_opt_view, giving the window. Returns data with : for both is the window as it is.
  3. Apply the fitted steps to the window with apply_fitted_steps, giving rd. For price data, check that rd is returns data.
  4. Check the asset axis of rd with assert_universe_aligned.
  5. Return predict(opt, rd), with the keyword arguments. For returns data, apply the steps with apply_fitted_steps_keeping in step 3, and record test_idx on the prediction with pipeline_fold_prediction when every step keeps the observations, see keeps_observations. Price data records no rows, because PricesToReturns drops the first price row, so the positions of the returns are not those of test_idx. Price data carries timestamps, and the timestamps name the rows instead.

Arguments

  • res: The fitted PipelineResult.
  • data: Price data or returns data that contains the window, a PricesResult or a ReturnsResult.
  • test_idx: The rows of data in the window, as integer indices or : for all rows. Price data also takes a vector of timestamps.
  • test_idxs: Several such windows, as a vector of index vectors.
  • cols: The columns of data in the window, as integer indices or : for all assets. The columns must give the training universe. So a subset of the columns needs a fit on the same subset, as fit_and_predict(pipe, data, train_idx, test_idx, cols) does.

Keyword Arguments

Validation

  • !isnothing(res.ctx.opt). Raises an IsNothingError.
  • For price data, the fitted steps convert the window to returns, so the pipeline contains a PricesToReturns step. Raises an ArgumentError.
  • The window passes assert_universe_aligned.

Returns

  • pred::PredictionResult: The weight-level prediction on the changed window, or one such result per window when several are given.

Related

source
PortfolioOptimisers.fit_predict — Method
fit_predict(pipe::Pipeline, data::Prices_RR) -> PredictionResult

Fit a pipeline on data, and predict with it at once.

When the pipeline starts with a TrainTestSplit, the prediction uses the test window of the split, which no fitted step saw. Otherwise the prediction uses data itself, in sample. So one call with a split fits on the training rows and scores on the test rows.

Algorithm

  1. Fit pipe on data with fit, giving res.
  2. Read the test window of the split with holdout_window, giving test.
  3. Predict with res on test, or on data when test is nothing.

Arguments

  • pipe: The pipeline.
  • data::Prices_RR: Price data or returns data.

Returns

  • pred::PredictionResult: The prediction on the test window when the pipeline has a split, and on data otherwise.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(pipe::AbstractPipelineEstimator, i, args...; kwargs...)

Refuse an asset view of a Pipeline, or of any other AbstractPipelineEstimator.

A meta-optimiser such as NestedClustered, Stacking or SubsetResampling takes a port_opt_view of its inner estimator to build a portfolio over a subset of the assets. The asset universe of a pipeline is fitted state, because a missing-data filter selects it from the training window. So an asset view before the fit has no defined meaning. A meta-optimiser cannot wrap a Pipeline, but it can be the optimisation step of one.

Validation

  • Each call raises an ArgumentError.

Related

source
PortfolioOptimisers.implicit_constraint_target — Function
implicit_constraint_target(_::WeightBounds) -> Symbol

Return the routing target that a constraint result names by its type alone, or nothing.

Four result types name one optimiser field each. A WeightBounds goes to :wb, a LinearConstraint to :lcse, a phylogeny constraint result to :ple, and a RiskBudget to :rkb. Every other type returns nothing, and needs a target beside it, see TargetedConstraint.

This function is the only part of the fan-out that the type decides. It also decides whether the value of a step needs a wrapper. add_constraint_result wraps a value only when the value cannot name its own target, so the constraints slot holds a bare result where it can.

Arguments

  • c: A constraint result.

Returns

Related

source

Holdout splitting

A TrainTestSplit step keeps a test window aside before any other step runs. It must be the first step, because a fitted step before it would see the rows of the test window. You cannot use it with cross-validation, which makes its own training and test windows. fit_predict(pipe, data) predicts on the test window that the split kept aside.

PortfolioOptimisers.assert_split_position — Function
assert_split_position(ests)

Validate that a TrainTestSplit is the first step of a Pipeline, and that no nested step contains one.

The split keeps the test window away from every fitted step. A stateful step before the split reads the rows of the test window. A MissingDataFilter selects the universe from them, and a PriceGapFill computes fill values from them, so the fitted state carries test data into the training workflow. At the first position, no step runs before the split. The check refuses a split inside a nested pipeline at any position, because holdout_window reads the steps of the outer pipeline only.

Arguments

  • ests: The step estimators.

Validation

  • A TrainTestSplit step is at index 1. Raises an ArgumentError.
  • No nested Pipeline and no PipelineStep contains a TrainTestSplit. Raises an ArgumentError.

Returns

  • nothing.

Related

source
PortfolioOptimisers.has_split — Function
has_split(_) -> Bool

Return whether a step is a TrainTestSplit, or contains one.

The search enters each nested Pipeline and each PipelineStep. A split inside a nested pipeline sees data that an outer step already changed, and the first position of the split exists to prevent that. The same search tells whether a whole pipeline has a Holdout Split, which the cross-validation functions check before they run.

Returns

  • true for a TrainTestSplit, and for a Pipeline or a PipelineStep that contains one. false for every other step.

Related

source
PortfolioOptimisers.assert_no_holdout — Function
assert_no_holdout(pipe::Pipeline)

Refuse a Pipeline that contains a TrainTestSplit in the cross-validation functions.

A Holdout Split and a cross-validation scheme are two methods of evaluation. The scheme sets the training and the test window of each fold. A split in the pipeline takes a second holdout from the training window of each fold, and keeps a test window that no function reads. So each fold loses training data without a message. One call uses one method of evaluation.

Arguments

  • pipe: The pipeline.

Validation

  • !has_split(pipe). Raises an ArgumentError.

Returns

  • nothing.

Related

source