Base Pipeline

A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over an accumulating context. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included. See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Abstract types

PortfolioOptimisers.AbstractPipelineEstimator Type

abstract type AbstractPipelineEstimator <: AbstractEstimator

Abstract supertype for all pipeline estimator types in PortfolioOptimisers.jl.

A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over a PipelineContext. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included.

All concrete pipeline estimators should subtype AbstractPipelineEstimator.

See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Related

• AbstractEstimator

• AbstractPipelineResult

• PipelineContext

• PipelineStep

source

PortfolioOptimisers.AbstractPipelineResult Type

abstract type AbstractPipelineResult <: AbstractResult

Abstract supertype for all pipeline result types in PortfolioOptimisers.jl.

Pipeline results carry the fitted per-step results of executing an AbstractPipelineEstimator, the final PipelineContext, and the terminal optimisation result when one exists.

All concrete pipeline results should subtype AbstractPipelineResult.

Related

• AbstractResult

• AbstractPipelineEstimator

source

The preprocessing estimator and result hierarchies, the price-level PricesResult container, and their fit/apply verbs are not pipeline concepts — they are documented under Pre-processing.

Context and slots

PortfolioOptimisers.PipelineContext Type

struct PipelineContext{__T_prices, __T_returns, __T_prior, __T_phylogeny, __T_uncertainty, __T_constraints, __T_opt} <: AbstractResult

The accumulating blackboard threaded through a pipeline's steps.

A PipelineContext holds one typed slot per stage of the workflow. Steps run in user-given order; each reads the slots it needs and writes the slot its estimator family produces. Heterogeneous slots (uncertainty, constraints) hold one result or a vector of results whose elements are routed to their optimiser targets by result type.

Internal machinery — not part of the user-facing API.

Fields

• prices: Price-level data (AbstractPricesResult).

• returns: Returns-level data (AbstractReturnsResult).

• prior: Prior statistics (AbstractPriorResult).

• phylogeny: Phylogeny structure (AbstractPhylogenyResult).

• uncertainty: Uncertainty set results as an explicit mu/sigma pair (PipelineUncertaintySets).

• constraints: Constraint result(s) (AbstractConstraintResult or a vector thereof).

• opt: Terminal optimisation result (OptimisationResult).

Constructors

PipelineContext(;
    prices::Option{<:AbstractPricesResult} = nothing,
    returns::Option{<:AbstractReturnsResult} = nothing,
    prior::Option{<:AbstractPriorResult} = nothing,
    phylogeny::Option{<:AbstractPhylogenyResult} = nothing,
    uncertainty::Option{<:PipelineUncertaintySets} = nothing,
    constraints::Option{<:Union{<:AbstractConstraintResult, <:AbstractVector{<:AbstractConstraintResult}}} = nothing,
    opt::Option{<:OptimisationResult} = nothing,
) -> PipelineContext

Keywords correspond to the struct's fields.

Related

• PIPELINE_SLOTS

• AbstractPipelineEstimator

• PipelineStep

source

PortfolioOptimisers.PIPELINE_SLOTS Constant

const PIPELINE_SLOTS = (:prices, :returns, :prior, :phylogeny, :uncertainty, :constraints, :opt)

The named slots of a PipelineContext. Each pipeline step reads the slots it needs and writes the slot its estimator family produces.

source

PortfolioOptimisers.PIPELINE_INVALIDATES Constant

const PIPELINE_INVALIDATES = (prices = (:returns, :prior, :phylogeny, :uncertainty, :constraints),
                              returns = (:prior, :phylogeny, :uncertainty, :constraints))

The PipelineContext slots each written slot invalidates.

Writing a data slot makes every slot derived from that data stale: a prior, phylogeny, uncertainty set, or constraint result computed on one asset universe does not match a later, different one. Pipeline rejects such an ordering at construction rather than letting a stale, asset-misdimensioned result reach inject_context.

A slot filled by the pipeline input rather than by a step is not "written", so the usual MissingDataFilter → Imputer → PricesToReturns → … ordering is unaffected. Slots absent from this table (prior, phylogeny, uncertainty, constraints, opt) invalidate nothing.

Related

• PIPELINE_SLOTS

• Pipeline

• PipelineContext

source

PortfolioOptimisers.pipe_reads Function

pipe_reads(est) -> Tuple{}

Return the PipelineContext slots a pipeline step requires to be populated before it runs.

These are the required reads used for construction-time dependency validation, not every slot the step may consume. Slots an estimator can compute internally when absent (for example a phylogeny-constraint estimator's own phylogeny) are not listed.

Arguments

• est: The step estimator.

Returns

• slots::Tuple{Vararg{Symbol}}: Subset of PIPELINE_SLOTS.

Examples

julia> PortfolioOptimisers.pipe_reads(EmpiricalPrior())
(:returns,)

Related

• pipe_writes

• PIPELINE_SLOTS

• PipelineStep

source

PortfolioOptimisers.pipe_writes Function

pipe_writes(est) -> Symbol

Return the PipelineContext slot a pipeline step writes.

The estimator's family determines the slot via dispatch on the existing abstract-type taxonomy. Estimators whose family cannot be inferred must be wrapped in a PipelineStep; the fallback method throws an ArgumentError saying so.

Arguments

• est: The step estimator.

Returns

• slot::Symbol: One of PIPELINE_SLOTS.

Examples

julia> PortfolioOptimisers.pipe_writes(EmpiricalPrior())
:prior

Related

• pipe_reads

• PIPELINE_SLOTS

• PipelineStep

source

pipe_writes(::TrainTestSplit) = :split
pipe_reads(::TrainTestSplit) = ()

A TrainTestSplit narrows whichever data slot the pipeline input filled, so the slot it writes is not a property of its type.

:split is a sentinel, deliberately not a member of PIPELINE_SLOTS: it names the step (pipe.names reads "split"), invalidates nothing, and satisfies nothing. That is sound only because a split is pinned to the first position of a Pipeline, where both data slots are already available from the input and no derived slot exists to invalidate. Which data slot is actually rewritten — :prices or :returns — is decided at run time by run_step.

Related

• TrainTestSplit

• assert_split_position

• PIPELINE_SLOTS

source

PortfolioOptimisers.PipelineStep Type

struct PipelineStep{__T_est, __T_reads, __T_writes, __T_target} <: AbstractEstimator

Explicit pipeline step wrapper — used when a step's slots or its routing intent must be stated rather than inferred.

Most estimators are used as pipeline steps directly: their family determines which PipelineContext slots they read and write via pipe_reads/pipe_writes. PipelineStep covers the two cases that dispatch alone cannot settle:

• Slots dispatch cannot infer: a custom callable, or an estimator routed to a nonstandard slot. reads and writes supply what the family would otherwise declare. This includes a bare-callable TimeDependent schedule of optimisers (TimeDependent(ctx -> optimiser)), whose output kind is not in its type: it enters via PipelineStep(; est = td, writes = :opt) and its output is type-checked when the fold loop swaps it in (see TD_OptE_Opt_Inferable).

• Routing intent dispatch must not guess: an uncertainty-set estimator writes the uncertainty slot either way, so the slot is never in doubt; what the wrapper declares through target is which parameters you want bounded — :mu, :sigma, or :both. Since ucs derives both halves from a single fit, this is a statement of intent, not a disambiguation, and every populated half must reach the optimiser or inject_context rejects it.

Fields

• est: The wrapped step: an estimator or a callable.

• reads: Slot the step requires to be populated before it runs (subset of PIPELINE_SLOTS).

• writes: Slot the step writes (one of PIPELINE_SLOTS).

• target: Optional routing annotation for heterogeneous slots (for uncertainty sets: :mu, :sigma, or :both).

Constructors

PipelineStep(;
    est,
    writes::Symbol,
    reads::Tuple{Vararg{Symbol}} = (),
    target::Option{Symbol} = nothing,
) -> PipelineStep

Keywords correspond to the struct's fields.

Validation

• writes in PIPELINE_SLOTS.

• all(r -> r in PIPELINE_SLOTS, reads).

• A TimeDependent est must be an optimiser-position schedule (TD_OptE_Opt) and declare writes = :opt: schedules of non-optimiser families are not steppable — a per-fold prior/constraint/… is spelled as a TimeDependent field of the optimisation step instead.

Examples

julia> ps = PipelineStep(; est = NormalUncertaintySet(), reads = (:returns,),
                         writes = :uncertainty, target = :mu);

julia> PortfolioOptimisers.pipe_writes(ps)
:uncertainty

julia> PortfolioOptimisers.pipe_reads(ps)
(:returns,)

Related

• pipe_reads

• pipe_writes

• PIPELINE_SLOTS

• AbstractPipelineEstimator

source

PortfolioOptimisers.PipelineUncertaintySets Type

struct PipelineUncertaintySets{__T_mu, __T_sigma} <: AbstractResult

The mu/sigma pair held by the uncertainty slot of a PipelineContext.

A computed uncertainty-set result cannot always reveal which parameter it bounds (a BoxUncertaintySet may bound either the mean or the covariance), so the slot stores the two targets explicitly. Uncertainty-set steps declare their target through a PipelineStep wrapper (target = :mu, target = :sigma, or target = :both); a narrowed step fills its half of the pair and leaves the other untouched, so :mu and :sigma steps compose, while :both derives the two halves from a single ucs call.

Fields

• mu: Uncertainty set bounding the expected returns vector (AbstractUncertaintySetResult).

• sigma: Uncertainty set bounding the covariance matrix (AbstractUncertaintySetResult).

Constructors

PipelineUncertaintySets(;
    mu::Option{<:AbstractUncertaintySetResult} = nothing,
    sigma::Option{<:AbstractUncertaintySetResult} = nothing,
) -> PipelineUncertaintySets

Keywords correspond to the struct's fields.

Related

• PipelineContext

• PipelineStep

source

PortfolioOptimisers.TD_OptE_Opt_Inferable Type

const TD_OptE_Opt_Inferable = Union{TimeDependent{<:AbstractVector{<:OptE_Opt}},
                                    TimeDependent{<:TimeDependentOptimiserCallable}}

The TimeDependent optimiser-position schedule forms whose pipeline slot is inferable from their type: a vector schedule whose entries are all optimisers or precomputed results, and a declared TimeDependentOptimiserCallable functor. Both are optimisation steps, so they write :opt and read :returns like any OptimisationEstimator step.

The other two forms of TD_OptE_Opt — a bare ctx -> optimiser and a PreviousWeightsFunction wrapping one — declare nothing in their type, so they keep the cannot-infer throw of pipe_writes and enter via PipelineStep(; est = td, writes = :opt), their output type-checked when the fold loop swaps it in (see assert_time_dependent_optimiser).

Related

• TD_OptE_Opt

• pipe_writes

• PipelineStep

source