GitHub

Base Pipeline

A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over an accumulating context. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included. See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Abstract types

PortfolioOptimisers.AbstractPipelineEstimator — Type
abstract type AbstractPipelineEstimator <: AbstractEstimator

Abstract supertype for all pipeline estimator types.

A pipeline reifies an end-to-end workflow — price preprocessing, prices-to-returns conversion, returns preprocessing, prior estimation, phylogeny, uncertainty sets, constraint generation, and optimisation — as an ordered list of steps executed left-to-right over a PipelineContext. Pipelines widen the cross-validation and hyperparameter-tuning boundary to the entire workflow, data preparation included.

All concrete pipeline estimators should subtype AbstractPipelineEstimator.

See docs/adr/0028-pipeline-workflow-estimator.md for the design rationale.

Related

source

The preprocessing estimator and result hierarchies, the price-level PricesResult container, and their fit/apply verbs are not pipeline concepts — they are documented under Pre-processing.

Context and slots

PortfolioOptimisers.PipelineContext — Type
struct PipelineContext{__T_prices, __T_returns, __T_prior, __T_phylogeny, __T_uncertainty, __T_constraints, __T_opt} <: AbstractResult

The accumulating blackboard threaded through a pipeline's steps.

A PipelineContext holds one typed slot per stage of the workflow. Steps run in user-given order; each reads the slots it needs and writes the slot its estimator family produces. Heterogeneous slots (uncertainty, constraints) hold one result or a vector of results whose elements are routed to their optimiser targets by result type.

Internal machinery — not part of the user-facing API.

Fields

Constructors

PipelineContext(;    prices::Option{<:AbstractPricesResult} = nothing,    returns::Option{<:AbstractReturnsResult} = nothing,    prior::Option{<:AbstractPriorResult} = nothing,    phylogeny::Option{<:AbstractPhylogenyResult} = nothing,    uncertainty::Option{<:PipelineUncertaintySets} = nothing,    constraints::Option{<:Union{<:AbstractConstraintResult, <:AbstractVector{<:AbstractConstraintResult}}} = nothing,    opt::Option{<:OptimisationResult} = nothing,) -> PipelineContext

Keywords correspond to the struct's fields.

Related

source
PortfolioOptimisers.PIPELINE_DATA_SLOTS — Constant
const PIPELINE_DATA_SLOTS = (:prices, :returns)

The PIPELINE_SLOTS whose write changes the asset universe — equivalently, the two slots a Pipeline input can fill directly. Writing one of these makes every slot derived from it stale, which is exactly what PIPELINE_INVALIDATES is derived from. Every other slot is computed from the data and reorders nothing, so writing it invalidates nothing.

Related

source
PortfolioOptimisers.PIPELINE_SLOTS — Constant
const PIPELINE_SLOTS = fieldnames(PipelineContext)

The named slots of a PipelineContext, in field order: (:prices, :returns, :prior, :phylogeny, :uncertainty, :constraints, :opt). Each pipeline step reads the slots it needs and writes the slot its estimator family produces.

The list is derived from the struct rather than retyped, so a new slot cannot be added to PipelineContext and forgotten here.

Related

source
PortfolioOptimisers.PIPELINE_INVALIDATES — Constant
PIPELINE_INVALIDATES

The PipelineContext slots each written slot invalidates, derived from PIPELINE_SLOTS order and PIPELINE_DATA_SLOTS:

(prices = (:returns, :prior, :phylogeny, :uncertainty, :constraints), returns = (:prior, :phylogeny, :uncertainty, :constraints))

Writing a data slot makes every slot derived from that data stale: a prior, phylogeny, uncertainty set, or constraint result computed on one asset universe does not match a later, different one. Pipeline rejects such an ordering at construction rather than letting a stale, asset-misdimensioned result reach inject_context.

The derivation: only a data slot invalidates, and it invalidates every slot after it in PIPELINE_SLOTS except the terminal :opt. :opt is the workflow's output — nothing derives from it, so a stale :opt is never read by a later step; it is excluded from the invalidatable set by construction. A slot filled by the pipeline input rather than by a step is not "written", so the usual MissingDataFilter → Imputer → PricesToReturns → … ordering is unaffected, and a non-data write (prior, phylogeny, uncertainty, constraints, opt) invalidates nothing.

Related

source
PortfolioOptimisers.PIPELINE_ROUTING_TARGETS — Constant
const PIPELINE_ROUTING_TARGETS = (:pe, :cle, :wb, :lcse, :cte, :ple, :lt, :st, :slt,
                                 :sst, :sglt, :sgst, :smtx, :sgmtx, :rkb, :mu_ucs,
                                 :sigma_ucs)

The destinations inject_context can deliver a computed slot to.

Routing targets are finer than PIPELINE_SLOTS and address a different audience. A slot is pipeline-author vocabulary: it names a stage of the workflow, and a step's estimator family decides which slot it writes. A target is optimiser-author vocabulary: it names a destination, and an optimiser's fields decide which targets it accepts. Users writing pipelines never name a target.

The fan-out from slots to targets is the Pipeline's job, and it is where the slot-level heterogeneity is resolved:

  • prior → :pe.
  • phylogeny → :cle, when the result is a clustering structure.
  • uncertainty → :mu_ucs and :sigma_ucs, one per populated half of the PipelineUncertaintySets pair.
  • constraints → the target the element carries when it is a TargetedConstraint, and otherwise the one its result type names: :wb, :lcse, :ple or :rkb. Elements reaching an accumulating target are packed into a vector in write order; a second element reaching any other target is an error.

All but :mu_ucs and :sigma_ucs are named after the field they land in, using this package's shared field vocabulary (see field_dict), so pipe_route needs no per-optimiser declaration — the target lands in the like-named field of whichever optimiser has one. The two exceptions carry validation policy and name no plain field. :rkb names a field one level down (rba.rkb) and is declared per optimiser by @pipe_route_rkb.

Note that :cle and :ple are one letter apart and come from the same phylogeny slot; they are not interchangeable. :cle is a clustering structure the optimiser uses to build a hierarchy, :ple is a phylogeny constraint result. That is why only one of them is optional below.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.PIPELINE_OPTIONAL_TARGETS — Constant
const PIPELINE_OPTIONAL_TARGETS = (:pe, :cle)

The routing targets an optimiser may have no home for without it being an error.

The asymmetry is the whole of the injection policy, and it turns on whether dropping the value changes the answer:

  • :pe and :cle do not. An optimiser with no pe field either needs no prior (EqualWeighted) or computes an equivalent one internally, which is what ADR 0028 means by every stage being optional. A JuMPOptimiser has no cle field because phylogeny reaches it as constraint results — generated from returns, not from this slot — so the structure is genuinely surplus to it.
  • Everything else does. A weight bound, linear constraint, phylogeny constraint or uncertainty set that reaches no optimiser field would silently change the solved portfolio, so it is an error — the same reason PipelineStep's target rejects an uncertainty half it cannot place.

Note the cost that is paid: no step reads the prior or phylogeny slots — inject_context is their only consumer — so a step writing a slot the terminal optimiser cannot receive is wasted computation, silently. That is a performance trap rather than a correctness one, which is why it is tolerated rather than rejected, but it is the reason to keep this list short.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.PIPELINE_ACCUMULATING_TARGETS — Constant
const PIPELINE_ACCUMULATING_TARGETS = (:lcse, :cte, :ple, :slt, :sst, :sglt, :sgst,
                                      :smtx, :sgmtx)

The routing targets that accept more than one computed result.

Two steps writing one of these compose, and accumulate_constraint_values says how. A second write to any other target is refused instead, because those fields hold one value and the second would silently replace the first.

The rule for membership is that one step per estimator must reach the optimiser with the value the estimator vector would have produced, so what a target accepts is read off what constraint generation does when handed several estimators at once. That gives two shapes:

  • Packed. :lcse, :ple, :slt, :sst, :sglt, :sgst, :smtx and :sgmtx hold a vector, and generation returns one result per estimator. :lcse and :ple are order-free blocks. The other six are positional: entry i belongs to scenario or group block i, paired with the corresponding entry of scard or sgcarde, so write order is block order. A count that does not match is not a silent mis-pairing — JuMPOptimiser validates those lengths against each other when the routed value is absorbed, so a wrong number of steps fails at injection with a DimensionMismatch.
  • Folded. :cte holds one LinearConstraint. Its field does take a vector, but a vector of CentralityConstraint estimators, and centrality_constraints appends every row of every estimator into a single result. Separate steps therefore merge rather than pack, which is what makes a pipeline of n centrality steps agree with one cte field holding n estimators.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.PIPELINE_STEP_TARGETS — Constant
const PIPELINE_STEP_TARGETS = (:mu, :sigma, :both, :lt, :st, :slt, :sst, :sglt, :sgst,
                              :smtx, :sgmtx)

The routing annotations a PipelineStep's target field may carry.

A target is routing intent, not a slot. It is needed by exactly the steps whose destination their own result cannot express, and there are two such cases:

  • An uncertainty-set step says which parameters it is meant to bound — :mu, :sigma or :both — and pipe_required_targets turns that into the routing targets :mu_ucs and :sigma_ucs.
  • A constraint step whose family names several targets says which one, naming the routing target directly. The families and their targets are declared by pipe_constraint_targets, and the annotations above are exactly the union of the ones that name more than one.

nothing is the remaining accepted value and means "no annotation", which is what every step of every other family carries.

The allowlist is applied at construction, beside the one on writes, so a mistyped target is refused where it is written rather than by run_uncertainty_step after the pipeline has already fitted the steps before it. run_uncertainty_step and run_constraint_step keep their own checks, because an unwrapped estimator reaches them with target = nothing and because an annotation legal for one family is not legal for another.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.PIPELINE_THRESHOLD_TARGETS — Constant
const PIPELINE_THRESHOLD_TARGETS = (:lt, :st, :slt, :sst, :sglt, :sgst)

The six routing targets a Threshold can land in.

A buy-in threshold is one number per asset, and a JuMPOptimiser holds six of them: the long and short thresholds of the plain cardinality constraint (lt, st), and of its scenario and group variants (slt, sst, sglt, sgst). The result carries nothing that says which, so a threshold step must be told — see pipe_constraint_targets.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.TargetedConstraint — Type
struct TargetedConstraint{__T_target, __T_res} <: AbstractConstraintResult

A computed constraint value paired with the routing target it must land in.

The constraints slot is heterogeneous, and for three families the result type alone names the destination: a WeightBounds can only be :wb, a LinearConstraint only :lcse, a phylogeny constraint result only :ple. The other families are not so lucky. A Threshold has six homes on a JuMPOptimiser (lt, st, slt, sst, sglt, sgst), a centrality result is a LinearConstraint that belongs in cte rather than lcse, and an asset-sets matrix is not a constraint result at all.

TargetedConstraint carries the answer with the value. The target comes from pipe_constraint_targets when the family names exactly one, and from the PipelineStep wrapper when it names several — so the same declaration that assert_routable checks at construction is the one constraint_targets reads at injection, and the two cannot disagree.

A step wraps only what needs wrapping. When the value's own type already names the target — see implicit_constraint_target — the slot keeps the bare result, so reading ctx.constraints still shows what constraint generation returned.

Internal machinery — not part of the user-facing API.

Fields

  • res: The computed value.

Constructors

TargetedConstraint(;    target::Symbol,    res) -> TargetedConstraint

Keywords correspond to the struct's fields.

Validation

  • target in PIPELINE_ROUTING_TARGETS.

Related

source
PortfolioOptimisers.assert_opt_last — Function
assert_opt_last(ests)

Validate that an optimisation step, if present, is the last step of a Pipeline.

A pipeline's optimiser writes the terminal :opt slot — the workflow's output. Nothing is derived from :opt, so a step running after an optimiser could only strand those weights: a later data or estimator step would leave :opt computed on a since-changed context, and no later step reads :opt to catch it. Pinning the optimiser last keeps :opt genuinely terminal, which is also what lets PIPELINE_INVALIDATES omit it from the invalidatable slots. A terminal optimiser is optional (a prior-only pipeline is legal); when absent the rule is vacuous.

Validation

  • No step writes :opt unless it is the final step. A nested Pipeline reports the slot its own last step writes and is validated at its own construction, so a non-terminal optimiser hidden inside one is caught there.

Related

source
PortfolioOptimisers.pipe_reads — Function
pipe_reads(est) -> Tuple{Symbol}

Return the PipelineContext slots a pipeline step requires to be populated before it runs.

These are the required reads used for construction-time dependency validation, not every slot the step may consume. Slots an estimator can compute internally when absent (for example a phylogeny-constraint estimator's own phylogeny) are not listed.

Constraint estimators read :returns — the minimal UniverseSets a bare constraint step resolves against is built from the returns' names (see pipeline_asset_sets). An ExposureConstraintEstimator additionally reads :prior, because the basis it re-bases through is the prior's regression loadings.

Arguments

  • est: The step estimator.

Returns

Examples

julia> PortfolioOptimisers.pipe_reads(EmpiricalPrior())(:returns,)

Related

source
PortfolioOptimisers.pipe_writes — Function
pipe_writes(est) -> Symbol

Return the PipelineContext slot a pipeline step writes.

The estimator's family determines the slot via dispatch on the existing abstract-type taxonomy. Estimators whose family cannot be inferred must be wrapped in a PipelineStep; the fallback method throws an ArgumentError saying so.

Arguments

  • est: The step estimator.

Returns

Examples

julia> PortfolioOptimisers.pipe_writes(EmpiricalPrior()):prior

Related

source
pipe_writes(::TrainTestSplit) = :split
pipe_reads(::TrainTestSplit) = ()

A TrainTestSplit narrows whichever data slot the pipeline input filled, so the slot it writes is not a property of its type.

:split is a sentinel, deliberately not a member of PIPELINE_SLOTS: it names the step (pipe.names reads "split"), invalidates nothing, and satisfies nothing. That is sound only because a split is pinned to the first position of a Pipeline, where both data slots are already available from the input and no derived slot exists to invalidate. Which data slot is actually rewritten — :prices or :returns — is decided at run time by run_step.

Related

source
PortfolioOptimisers.pipe_constraint_targets — Function
pipe_constraint_targets(
    _::AbstractConstraintEstimator
) -> Tuple{Symbol}

The routing targets a constraint family's step can write.

This is the one declaration of the estimator → target half of the seam, and it has three readers, which is why it exists as a table rather than as scattered knowledge:

The tuple's length is the contract:

  • One target: the family names its destination, and the step needs no annotation.
  • Several: the destination is a real choice the result cannot express, so the step must name one through its PipelineStep wrapper. This is the same rule an uncertainty-set step follows.
  • None: the family has no value to contribute to the constraints slot, so it is not a step. JuMPConstraintEstimator is the case — it is configuration for the model, not a computation over data.

Arguments

  • ce: A constraint estimator.

Returns

Examples

julia> PortfolioOptimisers.pipe_constraint_targets(WeightBoundsEstimator())(:wb,)

Related

source
PortfolioOptimisers.PipelineStep — Type
struct PipelineStep{__T_est, __T_reads, __T_writes, __T_target} <: AbstractEstimator

Explicit pipeline step wrapper — used when a step's slots or its routing intent must be stated rather than inferred.

Most estimators are used as pipeline steps directly: their family determines which PipelineContext slots they read and write via pipe_reads/pipe_writes. PipelineStep covers the two cases that dispatch alone cannot settle:

  • Slots dispatch cannot infer: a custom callable, or an estimator routed to a nonstandard slot. reads and writes supply what the family would otherwise declare. This includes a bare-callable TimeDependent schedule of optimisers (TimeDependent(ctx -> optimiser)), whose output kind is not in its type: it enters via PipelineStep(; est = td, writes = :opt) and its output is type-checked when the fold loop swaps it in (see TD_OptE_Opt_Inferable).
  • Routing intent dispatch must not guess: an uncertainty-set estimator writes the uncertainty slot either way, so the slot is never in doubt; what the wrapper declares through target is which parameters you want bounded — :mu, :sigma, or :both. Since ucs derives both halves from a single fit, this is a statement of intent, not a disambiguation, and every populated half must reach the optimiser or inject_context rejects it.

Fields

  • est: The wrapped step: an estimator or a callable.
  • reads: Slots the step requires to be populated before it runs (a subset of PIPELINE_SLOTS, as a tuple).

Constructors

PipelineStep(;    est::Union{<:AbstractEstimator, <:Function},    writes::Symbol,    reads::Tuple{Vararg{Symbol}} = (),    target::Option{Symbol} = nothing,) -> PipelineStep

Keywords correspond to the struct's fields.

Validation

  • writes in PIPELINE_SLOTS.
  • all(r -> r in PIPELINE_SLOTS, reads).
  • isnothing(target) || target in PIPELINE_STEP_TARGETS.
  • A TimeDependent est must be an optimiser-position schedule (TD_OptE_Opt) and declare writes = :opt: schedules of non-optimiser families are not steppable — a per-fold prior/constraint/… is spelled as a TimeDependent field of the optimisation step instead.

Examples

julia> ps = PipelineStep(; est = NormalUncertaintySet(), reads = (:returns,),                         writes = :uncertainty, target = :mu);julia> PortfolioOptimisers.pipe_writes(ps):uncertaintyjulia> PortfolioOptimisers.pipe_reads(ps)(:returns,)

Related

source
PortfolioOptimisers.PipelineUncertaintySets — Type
struct PipelineUncertaintySets{__T_mu, __T_sigma} <: AbstractResult

The mu/sigma pair held by the uncertainty slot of a PipelineContext.

A computed uncertainty-set result cannot always reveal which parameter it bounds (a BoxUncertaintySet may bound either the mean or the covariance), so the slot stores the two targets explicitly. Uncertainty-set steps declare their target through a PipelineStep wrapper (target = :mu, target = :sigma, or target = :both); a narrowed step fills its half of the pair and leaves the other untouched, so :mu and :sigma steps compose, while :both derives the two halves from a single ucs call.

Fields

Constructors

PipelineUncertaintySets(;    mu::Option{<:AbstractUncertaintySetResult} = nothing,    sigma::Option{<:AbstractUncertaintySetResult} = nothing,) -> PipelineUncertaintySets

Keywords correspond to the struct's fields.

Related

source
PortfolioOptimisers.TD_OptE_Opt_Inferable — Type
const TD_OptE_Opt_Inferable = Union{TimeDependent{<:AbstractVector{<:OptE_Opt}},
                                    TimeDependent{<:TimeDependentOptimiserCallable}}

The TimeDependent optimiser-position schedule forms whose pipeline slot is inferable from their type: a vector schedule whose entries are all optimisers or precomputed results, and a declared TimeDependentOptimiserCallable functor. Both are optimisation steps, so they write :opt and read :returns like any OptimisationEstimator step.

The other two forms of TD_OptE_Opt — a bare ctx -> optimiser and a PreviousWeightsFunction wrapping one — declare nothing in their type, so they keep the cannot-infer throw of pipe_writes and enter via PipelineStep(; est = td, writes = :opt), their output type-checked when the fold loop swaps it in (see assert_time_dependent_optimiser).

Related

source