Base Pipeline

A pipeline holds a whole workflow as a list of steps that run from left to right. The steps can be price preprocessing, the conversion of prices to returns, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation and optimisation. Each step can read what the steps before it computed. Cross-validation and hyperparameter tuning then cover the whole workflow, data preparation included, not the optimiser alone.

To add a pipeline of another shape, subtype AbstractPipelineEstimator and AbstractPipelineResult, and implement fit and predict. The # Interfaces section of each type lists the methods that stay on Pipeline alone.

PortfolioOptimisers.AbstractPipelineEstimator — Type
abstract type AbstractPipelineEstimator <: AbstractEstimator

Abstract supertype for all pipeline estimator types.

A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over a PipelineContext. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included.

All concrete pipeline estimators should subtype AbstractPipelineEstimator.

Interfaces

To add a pipeline estimator, subtype AbstractPipelineEstimator and implement the following method. Its result subtypes AbstractPipelineResult and follows the interface of that type.

fit

  • fit(pipe::MyPipeline, data::Prices_RR) -> AbstractPipelineResult: Fits the workflow on data.

Arguments

  • pipe: The instance of the subtype.
  • data::Prices_RR: Price data or returns data.

Returns

  • res::AbstractPipelineResult: The fitted result of the workflow.

A pipeline that folds observations one at a time also implements fit(pipe::MyPipeline) with no data. It reads the fitted result out of the partial-fit state of the pipeline.

With fit, the subtype gets three methods that the library defines on AbstractPipelineEstimator:

  • The method fit_and_predict(pipe, data, train_idx, test_idx, cols) fits the pipeline over the rows train_idx and predicts over test_idx. When train_idx is nothing, it calls fit(pipe) with no data.
  • The methods of port_opt_view and optimise raise an ArgumentError. A pipeline is a workflow, not an optimiser, so a meta-optimiser cannot hold one.

The shape of Pipeline is a linear list of steps, not a graph of steps. A subtype can have a different shape, which the library does not define. These functions read the steps, the names or the cache of a Pipeline, so their methods stay on Pipeline alone:

  • The cross-validation and the search: cross_val_predict, search_cross_validation, pipeline_lens and fit_predict.
  • The online updates: partial_fit!, fit(pipe) with no data, update_online_estimator and held_timestamps.
  • The schedules: is_time_dependent, needs_previous_weights, factory and assert_time_dependent_fold_count.
  • A pipeline as the step of another pipeline: run_step, pipe_reads and pipe_writes.

Examples

julia> struct MyPipeline{T} <: PortfolioOptimisers.AbstractPipelineEstimator           opt::T       endjulia> struct MyPipelineResult{T} <: PortfolioOptimisers.AbstractPipelineResult           res::T       endjulia> function StatsBase.fit(pipe::MyPipeline, rd::ReturnsResult)           return MyPipelineResult(optimise(pipe.opt, rd))       endjulia> function StatsBase.predict(res::MyPipelineResult, rd::ReturnsResult, args...; kwargs...)           return predict(res.res, rd, args...; kwargs...)       endjulia> rd = ReturnsResult(; nx = ["A", "B", "C"],                          X = [0.01 0.02 -0.01; -0.02 0.01 0.03; 0.03 -0.01 0.02;                               0.01 0.0 -0.02; -0.01 0.02 0.01; 0.02 0.01 0.0]);julia> pipe = MyPipeline(InverseVolatility());julia> fit(pipe, rd).res.w3-element Vector{Float64}: 0.2787151260257324 0.44390013800777395 0.2773847359664937julia> pred = PortfolioOptimisers.fit_and_predict(pipe, rd, 1:4, 5:6);julia> pred.res.w3-element Vector{Float64}: 0.28877552525205813 0.461137533888465 0.25008694085947697

Related

source
PortfolioOptimisers.AbstractPipelineResult — Type
abstract type AbstractPipelineResult <: AbstractResult

Abstract supertype for all pipeline result types.

Pipeline results carry the fitted per-step results of executing an AbstractPipelineEstimator, the final PipelineContext, and the terminal optimisation result when one exists.

All concrete pipeline results should subtype AbstractPipelineResult.

Interfaces

To add a pipeline result, subtype AbstractPipelineResult and implement the following method. The fit of an AbstractPipelineEstimator returns it.

predict

  • predict(res::MyPipelineResult, data::Prices_RR, test_idx, cols; kwargs...) -> PredictionResult: Applies the fitted workflow to the rows test_idx and the columns cols of data.

Arguments

  • res: The instance of the subtype.
  • data::Prices_RR: Price data or returns data.
  • test_idx: The rows to predict over. A vector of windows gives one prediction for each window.
  • cols: The columns of data.

Keyword Arguments

  • The keywords of the predict of an optimisation result: wd, hwd, fa, store_weight_path, strict and w_prev.

Returns

  • pred::PredictionResult: The prediction, or a vector of predictions for a vector of windows.

With predict, the subtype gets the method fit_and_predict(res, data, train_idx, test_idx, cols) that the library defines on AbstractPipelineResult. It ignores train_idx, and calls predict(res, data, test_idx, cols) with the keywords above. PipelineResult keeps its two methods for a vector of windows, because a method on AbstractPipelineResult with that signature is ambiguous with the predict of PipelineResult.

Examples

julia> struct MyPipeline{T} <: PortfolioOptimisers.AbstractPipelineEstimator           opt::T       endjulia> struct MyPipelineResult{T} <: PortfolioOptimisers.AbstractPipelineResult           res::T       endjulia> function StatsBase.fit(pipe::MyPipeline, rd::ReturnsResult)           return MyPipelineResult(optimise(pipe.opt, rd))       endjulia> function StatsBase.predict(res::MyPipelineResult, rd::ReturnsResult, args...; kwargs...)           return predict(res.res, rd, args...; kwargs...)       endjulia> rd = ReturnsResult(; nx = ["A", "B", "C"],                          X = [0.01 0.02 -0.01; -0.02 0.01 0.03; 0.03 -0.01 0.02;                               0.01 0.0 -0.02; -0.01 0.02 0.01; 0.02 0.01 0.0]);julia> res = fit(MyPipeline(InverseVolatility()), rd);julia> predict(res, rd, 5:6).rd.X2-element Vector{Float64}: 0.008864698859563092 0.010013303900592387julia> PortfolioOptimisers.fit_and_predict(res, rd, nothing, 5:6).rd.X2-element Vector{Float64}: 0.008864698859563092 0.010013303900592387

Related

source
PortfolioOptimisers.PipelineStep — Type
struct PipelineStep{__T_est, __T_reads, __T_writes, __T_target} <: AbstractEstimator

Explicit pipeline step wrapper — used when a step's slots or its routing intent must be stated rather than inferred.

Most estimators are used as pipeline steps directly: their family determines which PipelineContext slots they read and write via pipe_reads/pipe_writes. PipelineStep covers the two cases that dispatch alone cannot settle:

  • Slots dispatch cannot infer: a custom callable, or an estimator routed to a nonstandard slot. reads and writes supply what the family would otherwise declare. This includes a bare-callable TimeDependent schedule of optimisers (TimeDependent(ctx -> optimiser)), whose output kind is not in its type: it enters via PipelineStep(; est = td, writes = :opt) and its output is type-checked when the fold loop swaps it in (see TD_OptE_Opt_Inferable).
  • Routing intent dispatch must not guess: an uncertainty-set estimator writes the uncertainty slot either way, so the slot is never in doubt; what the wrapper declares through target is which parameters you want bounded — :mu, :sigma, or :both. Since ucs derives both halves from a single fit, this is a statement of intent, not a disambiguation, and every populated half must reach the optimiser or inject_context rejects it.

A step that reads the returns, such as a prior, a clustering or an uncertainty set with its own prior, states the source of its returns on its own reader, with the x_src field of the reader. So PipelineStep(; est = EmpiricalPrior(; x_src = ExcessSource()), writes = :prior) fits the prior on the returns in excess of the benchmark, X - B. The wrapper holds no source. The steps whose values the last optimiser receives join the group of that optimiser, so the Pipeline warns about a reader that states no source beside a reader of the Excess Frame, see group_frames.

Fields

  • est: The wrapped step: an estimator or a callable.
  • reads: Slots the step requires to be populated before it runs (a subset of PIPELINE_SLOTS, as a tuple or a vector, which the step keeps as given).

Constructors

PipelineStep(;    est::Union{AbstractEstimator, Function},    writes::Symbol,    reads::Union{Tuple{Vararg{Symbol}}, AbstractVector{<:Symbol}} = (),    target::Option{Symbol} = nothing,) -> PipelineStep

Keywords correspond to the struct's fields.

Validation

  • writes in PIPELINE_SLOTS.
  • all(r -> r in PIPELINE_SLOTS, reads).
  • isnothing(target) || target in PIPELINE_STEP_TARGETS.
  • A TimeDependent est must be an optimiser-position schedule (TD_OptE_Opt) and declare writes = :opt: schedules of non-optimiser families are not steppable — a per-fold prior/constraint/… is spelled as a TimeDependent field of the optimisation step instead.

Examples

julia> ps = PipelineStep(; est = NormalUncertaintySet(), reads = (:returns,),                         writes = :uncertainty, target = :mu);julia> PortfolioOptimisers.pipe_writes(ps):uncertaintyjulia> PortfolioOptimisers.pipe_reads(ps)(:returns,)julia> PipelineStep(; est = EmpiricalPrior(; x_src = ExcessSource()), writes = :prior).est.x_srcExcessSource()

Related

source
PortfolioOptimisers.pipe_constraint_targets — Function
pipe_constraint_targets(
    _::AbstractConstraintEstimator
) -> Tuple{Symbol}

The routing targets a constraint family's step can write.

This is the one declaration of the estimator → target half of the routing of a Pipeline slot into an optimiser, and it has three readers, which is why it exists as a table rather than as scattered knowledge:

The tuple's length is the contract:

  • One target: the family names its destination, and the step needs no annotation.
  • Several: the destination is a real choice the result cannot express, so the step must name one through its PipelineStep wrapper. This is the same rule an uncertainty-set step follows.
  • None: the family has no value to contribute to the constraints slot, so it is not a step. JuMPConstraintEstimator is the case — it is configuration for the model, not a computation over data.

Arguments

  • ce: A constraint estimator.

Returns

Examples

julia> PortfolioOptimisers.pipe_constraint_targets(WeightBoundsEstimator())(:wb,)

Related

source
PortfolioOptimisers.obs_weights_view — Method
obs_weights_view(ps::PipelineStep, i) -> PipelineStep

Return the step with the observation weights of its estimator cut to the observations i, through obs_weights_fields.

A step of an Excess group of a Pipeline reads the returns with each gap of a vector benchmark dropped, so the fixed observation weights of its estimator must lose the same observations, see group_returns_view. A callable step comes back with no change.

Related

source