Base Pipeline
A pipeline holds a whole workflow as a list of steps that run from left to right. The steps can be price preprocessing, the conversion of prices to returns, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation and optimisation. Each step can read what the steps before it computed. Cross-validation and hyperparameter tuning then cover the whole workflow, data preparation included, not the optimiser alone.
To add a pipeline of another shape, subtype AbstractPipelineEstimator and AbstractPipelineResult, and implement fit and predict. The # Interfaces section of each type lists the methods that stay on Pipeline alone.
PortfolioOptimisers.AbstractPipelineEstimator — Type
abstract type AbstractPipelineEstimator <: AbstractEstimatorAbstract supertype for all pipeline estimator types.
A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over a PipelineContext. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included.
All concrete pipeline estimators should subtype AbstractPipelineEstimator.
Interfaces
To add a pipeline estimator, subtype AbstractPipelineEstimator and implement the following method. Its result subtypes AbstractPipelineResult and follows the interface of that type.
fit
fit(pipe::MyPipeline, data::Prices_RR) -> AbstractPipelineResult: Fits the workflow ondata.
Arguments
pipe: The instance of the subtype.data::Prices_RR: Price data or returns data.
Returns
res::AbstractPipelineResult: The fitted result of the workflow.
A pipeline that folds observations one at a time also implements fit(pipe::MyPipeline) with no data. It reads the fitted result out of the partial-fit state of the pipeline.
With fit, the subtype gets three methods that the library defines on AbstractPipelineEstimator:
- The method
fit_and_predict(pipe, data, train_idx, test_idx, cols)fits the pipeline over the rowstrain_idxand predicts overtest_idx. Whentrain_idxisnothing, it callsfit(pipe)with no data. - The methods of
port_opt_viewandoptimiseraise anArgumentError. A pipeline is a workflow, not an optimiser, so a meta-optimiser cannot hold one.
The shape of Pipeline is a linear list of steps, not a graph of steps. A subtype can have a different shape, which the library does not define. These functions read the steps, the names or the cache of a Pipeline, so their methods stay on Pipeline alone:
- The cross-validation and the search:
cross_val_predict,search_cross_validation,pipeline_lensandfit_predict. - The online updates:
partial_fit!,fit(pipe)with no data,update_online_estimatorandheld_timestamps. - The schedules:
is_time_dependent,needs_previous_weights,factoryandassert_time_dependent_fold_count. - A pipeline as the step of another pipeline:
run_step,pipe_readsandpipe_writes.
Examples
julia> struct MyPipeline{T} <: PortfolioOptimisers.AbstractPipelineEstimator opt::T endjulia> struct MyPipelineResult{T} <: PortfolioOptimisers.AbstractPipelineResult res::T endjulia> function StatsBase.fit(pipe::MyPipeline, rd::ReturnsResult) return MyPipelineResult(optimise(pipe.opt, rd)) endjulia> function StatsBase.predict(res::MyPipelineResult, rd::ReturnsResult, args...; kwargs...) return predict(res.res, rd, args...; kwargs...) endjulia> rd = ReturnsResult(; nx = ["A", "B", "C"], X = [0.01 0.02 -0.01; -0.02 0.01 0.03; 0.03 -0.01 0.02; 0.01 0.0 -0.02; -0.01 0.02 0.01; 0.02 0.01 0.0]);julia> pipe = MyPipeline(InverseVolatility());julia> fit(pipe, rd).res.w3-element Vector{Float64}: 0.2787151260257324 0.44390013800777395 0.2773847359664937julia> pred = PortfolioOptimisers.fit_and_predict(pipe, rd, 1:4, 5:6);julia> pred.res.w3-element Vector{Float64}: 0.28877552525205813 0.461137533888465 0.25008694085947697Related
PortfolioOptimisers.AbstractPipelineResult — Type
abstract type AbstractPipelineResult <: AbstractResultAbstract supertype for all pipeline result types.
Pipeline results carry the fitted per-step results of executing an AbstractPipelineEstimator, the final PipelineContext, and the terminal optimisation result when one exists.
All concrete pipeline results should subtype AbstractPipelineResult.
Interfaces
To add a pipeline result, subtype AbstractPipelineResult and implement the following method. The fit of an AbstractPipelineEstimator returns it.
predict
predict(res::MyPipelineResult, data::Prices_RR, test_idx, cols; kwargs...) -> PredictionResult: Applies the fitted workflow to the rowstest_idxand the columnscolsofdata.
Arguments
res: The instance of the subtype.data::Prices_RR: Price data or returns data.test_idx: The rows to predict over. A vector of windows gives one prediction for each window.cols: The columns ofdata.
Keyword Arguments
- The keywords of the
predictof an optimisation result:wd,hwd,fa,store_weight_path,strictandw_prev.
Returns
pred::PredictionResult: The prediction, or a vector of predictions for a vector of windows.
With predict, the subtype gets the method fit_and_predict(res, data, train_idx, test_idx, cols) that the library defines on AbstractPipelineResult. It ignores train_idx, and calls predict(res, data, test_idx, cols) with the keywords above. PipelineResult keeps its two methods for a vector of windows, because a method on AbstractPipelineResult with that signature is ambiguous with the predict of PipelineResult.
Examples
julia> struct MyPipeline{T} <: PortfolioOptimisers.AbstractPipelineEstimator opt::T endjulia> struct MyPipelineResult{T} <: PortfolioOptimisers.AbstractPipelineResult res::T endjulia> function StatsBase.fit(pipe::MyPipeline, rd::ReturnsResult) return MyPipelineResult(optimise(pipe.opt, rd)) endjulia> function StatsBase.predict(res::MyPipelineResult, rd::ReturnsResult, args...; kwargs...) return predict(res.res, rd, args...; kwargs...) endjulia> rd = ReturnsResult(; nx = ["A", "B", "C"], X = [0.01 0.02 -0.01; -0.02 0.01 0.03; 0.03 -0.01 0.02; 0.01 0.0 -0.02; -0.01 0.02 0.01; 0.02 0.01 0.0]);julia> res = fit(MyPipeline(InverseVolatility()), rd);julia> predict(res, rd, 5:6).rd.X2-element Vector{Float64}: 0.008864698859563092 0.010013303900592387julia> PortfolioOptimisers.fit_and_predict(res, rd, nothing, 5:6).rd.X2-element Vector{Float64}: 0.008864698859563092 0.010013303900592387Related
PortfolioOptimisers.PipelineStep — Type
struct PipelineStep{__T_est, __T_reads, __T_writes, __T_target} <: AbstractEstimatorExplicit pipeline step wrapper — used when a step's slots or its routing intent must be stated rather than inferred.
Most estimators are used as pipeline steps directly: their family determines which PipelineContext slots they read and write via pipe_reads/pipe_writes. PipelineStep covers the two cases that dispatch alone cannot settle:
- Slots dispatch cannot infer: a custom callable, or an estimator routed to a nonstandard slot.
readsandwritessupply what the family would otherwise declare. This includes a bare-callableTimeDependentschedule of optimisers (TimeDependent(ctx -> optimiser)), whose output kind is not in its type: it enters viaPipelineStep(; est = td, writes = :opt)and its output is type-checked when the fold loop swaps it in (seeTD_OptE_Opt_Inferable). - Routing intent dispatch must not guess: an uncertainty-set estimator writes the
uncertaintyslot either way, so the slot is never in doubt; what the wrapper declares throughtargetis which parameters you want bounded —:mu,:sigma, or:both. Sinceucsderives both halves from a single fit, this is a statement of intent, not a disambiguation, and every populated half must reach the optimiser orinject_contextrejects it.
A step that reads the returns, such as a prior, a clustering or an uncertainty set with its own prior, states the source of its returns on its own reader, with the x_src field of the reader. So PipelineStep(; est = EmpiricalPrior(; x_src = ExcessSource()), writes = :prior) fits the prior on the returns in excess of the benchmark, X - B. The wrapper holds no source. The steps whose values the last optimiser receives join the group of that optimiser, so the Pipeline warns about a reader that states no source beside a reader of the Excess Frame, see group_frames.
Fields
est: The wrapped step: an estimator or a callable.
reads: Slots the step requires to be populated before it runs (a subset ofPIPELINE_SLOTS, as a tuple or a vector, which the step keeps as given).
writes: Slot the step writes (one ofPIPELINE_SLOTS).
target: Optional routing annotation for heterogeneous slots (one ofPIPELINE_STEP_TARGETS, ornothing).
Constructors
PipelineStep(; est::Union{AbstractEstimator, Function}, writes::Symbol, reads::Union{Tuple{Vararg{Symbol}}, AbstractVector{<:Symbol}} = (), target::Option{Symbol} = nothing,) -> PipelineStepKeywords correspond to the struct's fields.
Validation
writes in PIPELINE_SLOTS.all(r -> r in PIPELINE_SLOTS, reads).isnothing(target) || target in PIPELINE_STEP_TARGETS.- A
TimeDependentestmust be an optimiser-position schedule (TD_OptE_Opt) and declarewrites = :opt: schedules of non-optimiser families are not steppable — a per-fold prior/constraint/… is spelled as aTimeDependentfield of the optimisation step instead.
Examples
julia> ps = PipelineStep(; est = NormalUncertaintySet(), reads = (:returns,), writes = :uncertainty, target = :mu);julia> PortfolioOptimisers.pipe_writes(ps):uncertaintyjulia> PortfolioOptimisers.pipe_reads(ps)(:returns,)julia> PipelineStep(; est = EmpiricalPrior(; x_src = ExcessSource()), writes = :prior).est.x_srcExcessSource()Related
PortfolioOptimisers.pipe_constraint_targets — Function
pipe_constraint_targets(
_::AbstractConstraintEstimator
) -> NTuple{6, Symbol}
The routing targets a constraint family's step can write.
This is the one declaration of the estimator → target half of the routing of a Pipeline slot into an optimiser, and it has three readers, which is why it exists as a table rather than as scattered knowledge:
run_stepresolves the target and pairs it with the computed value.pipe_required_targetshands it toassert_routable, so a step whose target the terminal optimiser cannot receive is refused when thePipelineis built rather than after the first fold has run.PipelineStepvalidates a declared target against it.
The tuple's length is the contract:
- One target: the family names its destination, and the step needs no annotation.
- Several: the destination is a real choice the result cannot express, so the step must name one through its
PipelineStepwrapper. This is the same rule an uncertainty-set step follows. - None: the family has no value to contribute to the
constraintsslot, so it is not a step.JuMPConstraintEstimatoris the case — it is configuration for the model, not a computation over data.
Arguments
ce: A constraint estimator.
Returns
- A tuple of routing targets.
Examples
julia> PortfolioOptimisers.pipe_constraint_targets(WeightBoundsEstimator())(:wb,)Related
PortfolioOptimisers.obs_weights_view — Method
obs_weights_view(ps::PipelineStep, i) -> PipelineStepReturn the step with the observation weights of its estimator cut to the observations i, through obs_weights_fields.
A step of an Excess group of a Pipeline reads the returns with each gap of a vector benchmark dropped, so the fixed observation weights of its estimator must lose the same observations, see group_returns_view. A callable step comes back with no change.
Related