Pipeline: private API

PortfolioOptimisers.first_duplicate — Function
first_duplicate(xs) -> Any

Return the first element of xs that repeats an earlier element.

The error for a repeated step name shows this element, and not the whole collection. Only the failure path of the Pipeline constructor calls this function.

Arguments

  • xs: A collection of names.

Returns

  • The first element that repeats an earlier one, or nothing when every element is unique.

Related

source
PortfolioOptimisers.assert_step_declarations — Function
assert_step_declarations(ests)

Validate that each step of a Pipeline declares the slot and the target that the step writes when it runs.

run_step selects the method for a PipelineStep that wraps an estimator from the family of that estimator. So the family decides the slot that the step writes, and the writes field does not. An uncertainty-set step computes the halves that its target names, and it stops the fit when it has no such target. The constructor calls this check, so a wrong declaration stops the construction and not the first fold.

Arguments

  • ests: The step estimators.

Validation

  • An uncertainty-set estimator is wrapped in a PipelineStep whose target is :mu, :sigma or :both. Raises an ArgumentError.
  • A PipelineStep that wraps a preprocessing, prior, phylogeny, uncertainty-set, constraint or optimisation estimator declares the slot of that family in writes, see pipe_writes. Raises an ArgumentError.

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_nested_data_slots — Function
assert_nested_data_slots(ests)

Validate that a nested Pipeline step changes no Data Slot that the outer pipeline does not receive.

A nested pipeline gives one slot to the outer Pipeline Context, the slot of its last step. Assume that a step inside it rewrites :prices or :returns, and that its last step writes a different slot. Then the outer pipeline receives a result for assets that its own Data Slot does not hold. The weights and the prior then describe two different universes, and a prediction fails at assert_universe_aligned.

Arguments

  • ests: The step estimators.

Validation

  • A nested Pipeline whose last step writes a slot outside PIPELINE_DATA_SLOTS contains no step that writes a Data Slot. Raises an ArgumentError. Put such a step in the outer pipeline.

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_slot_order — Function
assert_slot_order(ests)

Validate that each step of a Pipeline reads only slots that are filled, and makes stale no slot that an earlier step wrote.

The check walks the steps in order, through foreach_step. slots holds the slot that each step writes, see pipe_writes. last_slot_writer reads it to find the last earlier step that writes a slot, so the message can name that step.

Arguments

  • ests: The step estimators.

Validation

  • Each slot that a step reads, see pipe_reads, is a Data Slot, see PIPELINE_DATA_SLOTS, or the slot of an earlier step. Raises an ArgumentError.
  • No slot that the write of a step makes stale, see PIPELINE_INVALIDATES, is the slot of an earlier step. Raises an ArgumentError.

Returns

  • nothing.

Related

source
PortfolioOptimisers.last_slot_writer — Function
last_slot_writer(slots, s::Symbol, k::Integer) -> Any

Return the position of the last step before step k that writes the slot s.

assert_slot_order calls this function with the slot that each step writes, so it needs no set of the filled slots and no dictionary of the writers.

Arguments

  • slots: The slot that each step writes, in step order, see pipe_writes.
  • s: The slot to find.
  • k: The position of the step that reads s or makes it stale.

Returns

  • j::Union{Nothing, Int}: The largest j < k with slots[j] === s, or nothing when no earlier step writes s.

Related

source

The container form of the steps

The steps of a pipeline are a Tuple or a vector, as the user gives them. These functions walk the steps and build new containers in that form, so the checks of the constructor allocate nothing on a Tuple.

PortfolioOptimisers.foreach_step — Function
foreach_step(f, ests::Tuple) -> nothing
foreach_step(f, ests::AbstractVector) -> nothing

Call f(e, i) on each step e of a Pipeline and its position i, in step order.

The steps of a Tuple have different types. A for loop over them reads each step at a type that inference cannot know, so each call of f dispatches at run time and boxes the step. The Tuple method walks the steps through map over the steps and their positions, which Julia unrolls, so each call of f dispatches at compile time and allocates nothing. The AbstractVector method walks the steps with a loop, at the element type that the vector holds.

Arguments

  • f: A function of a step and its position. Its value is ignored.
  • ests: The step estimators.

Returns

  • nothing.

Related

source
PortfolioOptimisers.map_steps — Function
map_steps(f, ests::Tuple) -> Tuple
map_steps(f, ests::AbstractVector) -> Vector

Return f(e) for each step e of a Pipeline, in the container form of ests.

The Tuple method calls map. The AbstractVector method uses a comprehension, which takes the element type of its values as map does. Over a vector of two steps of an abstract element type, map takes from 1.6 to 10 times the time of the comprehension, measured on the maps of update_online_estimator, copy_states and reset_time_dependent_estimator. A pipeline maps over its steps at each fold of a prediction and at each online step, so the vector form takes the comprehension.

Arguments

  • f: A function of one step.
  • ests: The step estimators.

Returns

  • The values of f, a Tuple when ests is a Tuple, and a vector otherwise.

Related

source
PortfolioOptimisers.steps_form — Function
steps_form(ests::Tuple, v::AbstractVector) -> Tuple
steps_form(ests::AbstractVector, v::AbstractVector) -> Vector

Return v in the container form of the steps ests of a Pipeline.

The user chooses the form of the steps, a Tuple or a vector, and the library does not convert one to the other. A Tuple of steps of different types compiles a method for each pipeline, and a vector of steps compiles one method for each element type, which can cost less time to the first result. The names of the steps and the fitted results of a PipelineResult take the same form as the steps. The AbstractVector method narrows v to the element type of its values.

Arguments

  • ests: The step estimators, which give the form.
  • v: A vector with one entry for each step.

Returns

  • Tuple(v) when ests is a Tuple, and a new vector of the values of v otherwise.

Related

source

Injection

The pipeline turns every slot it computed into routing targets and passes them to the optimiser. The optimiser chooses the field that receives the value, through pipe_route.

PortfolioOptimisers.inject_context — Function
inject_context(
    opt::OptimisationEstimator,
    ctx::PipelineContext
) -> OptimisationEstimator

Replace the configuration of an optimisation step with the computed slots of the Pipeline Context, immediately before the step runs.

This function is the pipeline half of the routing. It turns the slots into a flat sequence of routing targets. To do this, it finds which halves of the uncertainty pair are present, which result types the constraints slot holds, and how many of each. Then it gives each target to pipe_route, which finds the field that receives the target. The optimiser owns that choice, so a field rename is a local change.

unroutable_target handles a target that an optimiser has no field for. :pe and :cle pass without a change, and every other target raises an error. So a naive optimiser or a meta-optimiser accepts a computed prior that it can use, and refuses an uncertainty set that it cannot use.

Algorithm

  1. Return opt unchanged when the prior, phylogeny, uncertainty and constraints slots are all nothing.
  2. Route the prior slot to :pe.
  3. Route the phylogeny slot to :cle when it is an AbstractClusteringResult. A phylogeny result of another type has no :cle target, so this step skips it.
  4. Route the mean half of the uncertainty slot to :mu_ucs, and its covariance half to :sigma_ucs, when the half is present.
  5. Route each target => value pair of constraint_targets to its target.
  6. Return the rebuilt optimiser opt′.

Arguments

  • opt: The optimisation step estimator.
  • ctx: The pipeline context.

Returns

  • opt′: The estimator that runs. It is opt itself when no slot applies, and a rebuilt copy otherwise.

Related

source
PortfolioOptimisers.constraint_target_of — Function
constraint_target_of(c) -> Symbol

Return the routing target that one element of the constraints slot goes to.

run_step pairs a value that its type cannot place with its target, and this function reads that target. implicit_constraint_target places every other element.

Arguments

  • c: One element of the constraints slot.

Validation

  • A Threshold without a target raises an ArgumentError. It names six optimiser fields, so its type cannot place it. The error names the declaration that places it.
  • A result of any other type that names no target raises an ArgumentError. The function refuses it here, before it reaches an optimiser.

Returns

Related

source
PortfolioOptimisers.accumulate_constraint_values — Function
accumulate_constraint_values(
    _::Val,
    vals
) -> LinearConstraint

Combine the values that reached one accumulating routing target.

The default puts the values in a vector, in write order. Each field that holds one result per estimator reads this shape.

:cte is different. Its field takes a vector of CentralityConstraint estimators, and centrality_constraints appends every row of every estimator into one LinearConstraint. So the values of separate steps merge through merge_linear_constraints. Then n centrality steps in a Pipeline give the optimiser the value that one cte field with n estimators gives.

constraint_targets calls this function with more than one value only, because it unwraps a single value first.

Arguments

  • ::Val{target}: The routing target that the values reached.
  • vals: The values, in write order.

Validation

  • For :cte, each value is a LinearConstraint. Raises an ArgumentError.

Returns

  • The combined value.

Related

source
PortfolioOptimisers.constraint_targets — Function
constraint_targets(cs) -> Any

Split the constraints slot into routing targets.

Algorithm

  1. For each element c of the constraints slot, in write order, read its target with constraint_target_of and its value with constraint_value_of.
  2. Collect the values of each target into its group, in write order. The groups follow the order of the first write to each target, and each group takes the element type of its values.
  3. Return the value itself for a group of one value. This is the scalar shape that the fields accept outside a pipeline. For a larger group, combine the values with accumulate_constraint_values. It puts them in a vector in write order, or merges them into one constraint for :cte.

Arguments

  • cs: The constraints slot.

Validation

  • A second value for a target outside PIPELINE_ACCUMULATING_TARGETS raises an ArgumentError. Such a field holds one value, and the second value replaces the first without a message.

Returns

  • A vector of target => value pairs, in the order of the first write to each target.

Related

source
PortfolioOptimisers.maybe_inject_step — Function
maybe_inject_step(est, ::PipelineContext) = est
maybe_inject_step(opt::OptimisationEstimator, ctx::PipelineContext)
maybe_inject_step(ps::PipelineStep{<:OptimisationEstimator}, ctx::PipelineContext)

Return the step that runs, with the Pipeline Context injected when the step is an optimiser.

A step that is not an optimiser runs unchanged. An optimiser takes the context through inject_context. For a PipelineStep that wraps an optimiser, the function builds a new PipelineStep around the injected optimiser, with the same reads, writes and target.

Arguments

  • est: A step estimator.
  • opt: An optimisation step estimator.
  • ps: A PipelineStep. A PipelineStep that wraps no optimiser runs unchanged.
  • ctx: The pipeline context.

Returns

  • est: The step estimator, unchanged.
  • opt′: The optimiser with the configuration that the context replaced.
  • ps′: The pipeline step with the injected optimiser.
source
maybe_inject_step(
    res::NonFiniteAllocationOptimisationResult,
    ctx::PipelineContext
) -> NonFiniteAllocationOptimisationResult

Apply the injection rules to a precomputed optimisation result in the optimisation step.

The predict-only fold of a mixed TimeDependent schedule puts a result in the optimisation step. A result is solved, so it has no configuration to replace. It follows the rule of inject_context for an optimiser without a field for the target. The computed prior and phylogeny slots pass, because the result has its own. A computed uncertainty set or constraint must not disappear without a solve, so a populated uncertainty or constraints slot stops the fit.

Arguments

  • res: The precomputed optimisation result.
  • ctx: The pipeline context.

Validation

  • isnothing(ctx.uncertainty). Raises an ArgumentError.
  • isnothing(ctx.constraints). Raises an ArgumentError.

Returns

  • res, unchanged.

Related

source
PortfolioOptimisers.pipe_required_targets — Function
pipe_required_targets(
    ps::PipelineStep
) -> Union{Tuple{}, Tuple{Symbol}, Tuple{Symbol, Symbol}}

Return the routing targets of a step that are known before any step runs.

Two kinds of step have such targets. An uncertainty-set step declares the parameters that it bounds in the target field of its PipelineStep. A constraint step takes its target from its family through pipe_constraint_targets, or from the target field of its PipelineStep when the family names several. run_step resolves the target from the same declaration, so this function returns the target that the step writes.

Every other step returns an empty tuple. A callable step declares no family, and a precomputed result names its target only by its type.

Arguments

  • est: A step estimator.

Returns

  • (:mu_ucs,), (:sigma_ucs,) or (:mu_ucs, :sigma_ucs) for a step that writes :uncertainty with the target :mu, :sigma or :both.
  • A tuple of one routing target for a constraint step.
  • () for every other step. This includes a callable step that writes :uncertainty with no such target.

Related

source
PortfolioOptimisers.assert_routable — Function
assert_routable(ests)

Refuse at construction a pipeline whose last optimiser cannot receive a target that an earlier step writes.

Without this check, inject_context finds a target that it cannot route when the optimisation step runs. Under cross_val_predict, that is after the fold loop fitted every earlier step of the first fold. The check asks the optimiser through pipe_accepts, so it follows the fields that the optimiser has.

The check is structural. It finds whether the optimiser family can receive the target, and not whether this configuration accepts the value. A JuMPOptimiser always accepts :mu_ucs, but one with a return estimator other than ArithmeticReturn fails at injection. pipe_route owns that condition.

Algorithm

  1. Read the optimiser opt of the last step with pipe_terminal_optimiser. Stop when there is none. A TimeDependent schedule and a precomputed result have none, and the fold loop resolves them later.
  2. Read the targets of each earlier step with pipe_required_targets.
  3. Check each target with pipe_accepts.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.

Validation

  • pipe_accepts(opt, Val(target)) holds for each target of each earlier step. Raises an ArgumentError.

Returns

  • nothing.

Related

source
PortfolioOptimisers.assert_constraint_targets — Function
assert_constraint_targets(ests)

Validate that each constraint step of a Pipeline resolves to one routing target.

The check calls resolve_constraint_target on each constraint step. run_step makes the same call when the step runs, so three failures move from the fold loop to the constructor. The first is a family that computes nothing for the constraints slot. The second is a family that names several targets when the step names none. The third is a declared target that another family owns.

Arguments

  • ests: The step estimators.

Validation

  • The family of each constraint step declares at least one target, see pipe_constraint_targets.
  • When the family declares several targets, the target of the PipelineStep names one of them.

Returns

  • nothing.

Related

source

The group of a pipeline

A Pipeline joins the group of its last optimiser, so its constructor warns on a reader that states no source beside a reader of the excess returns. A nested Pipeline adds the frames of its last step to the outer group.

PortfolioOptimisers.group_frames — Method
group_frames(pipe::Pipeline) -> ReturnsFrames
reader_frames(p::Pipeline, path::AbstractString, refit::Bool) -> ReturnsFrames

The frame walk of a Pipeline.

A Pipeline joins the group of its last optimiser, see pipeline_frames, and its constructor warns on that group with warn_raw_readers. A nested Pipeline is a step of an outer one, and the outer optimiser receives the value of its last step, so it adds the frames of that step.

Arguments

  • pipe, p: The pipeline.
  • path: The path of the value from the head, for example "pe", "cle" or "r[2].pe". A warning names a reader by its path.
  • refit: true when a regression reader at x_src = nothing in the group fits its regression on the raw returns data. refits_regression finds the value.

Returns

  • fr::ReturnsFrames: The frames of the group of pipe, or the frames of the last step of p at the path "<path>.steps[n]".

Related

source

Prediction

To predict with a fitted pipeline, the pipeline applies its fitted preprocessing steps to the new data window. The steps keep what they learned on the training window: the universe, the imputation parameters and the conversion to returns. The pipeline then predicts with the same functions as an optimisation result. Cross-validation also accepts price data, through Prices_RR. Each fold then fits the whole pipeline on its own training window, so no preprocessing step learns anything from the test window.

PortfolioOptimisers.apply_fitted_step — Function
apply_fitted_step(fitted, data) -> data′

Apply one fitted step of a pipeline to a data window for a prediction.

A preprocessing step changes the window at its own data level. A price-level fitted object, an AbstractPricesPreprocessingResult or an AbstractPricesPreprocessingEstimator, changes a price window. A returns-level fitted object changes a returns window. PricesToReturns converts a price window to returns.

A fitted object of the other data level returns the window unchanged. So a pipeline that was fitted on prices can predict on a returns window, and assert_universe_aligned then checks the asset axis of that window. The result of every other step, such as a prior, a phylogeny, an uncertainty set, a constraint or an optimisation result, returns the window unchanged. A nested PipelineResult applies its own steps with apply_fitted_steps.

Arguments

Returns

  • data′: The changed data window, or data itself.

Related

source
PortfolioOptimisers.apply_fitted_steps — Function
apply_fitted_steps(
    results::Union{Tuple, AbstractVector},
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> Any

Apply the fitted steps of a pipeline to a data window, in step order.

Algorithm

  1. For each fitted result f in results, in step order, replace data with apply_fitted_step(f, data), see apply_fitted_step.
  2. Return the last data.

Arguments

  • results: The fitted result of each step, from a PipelineResult.
  • data: The data window.

Returns

  • data′: The changed data window. It is returns data when the steps include a PricesToReturns step.

Related

source
PortfolioOptimisers.assert_universe_aligned — Function
assert_universe_aligned(
    res::PipelineResult,
    rd::AbstractReturnsResult
)

Check that a test window and the training window come from the same ingestion.

The last weights use the asset axis of the training window. A test window with a different asset set, or with the same set from a different universe statement, misaligns the weights and the returns. This check reports that case with an error that names both universes. Without the check, the case shows as a dimension mismatch inside the risk calculation.

The check reads equality of nx, which checks the asset axis. check_asset_panel binds the asset axis of an AssetPanel to the nx of the result that holds it, so equality of nx also compares the axes of the panels. The check does not compare panel presence. The ingestion layer emits no panel for a window with no gap and no Panel Field, so a gapless training window and a gapped test window of one ingestion differ in panel presence.

Two other parts of the library hold the rest of the alignment. The ingestion layer fixes the asset axis before the split, and port_opt_view slices it. So each window of each fold carries every asset, and no policy of the layer drops a row or a column. An asset that is present in both windows and not investable in one is a Held Gap, which filter_held_gaps reads from the weights and the returns under the strictness policy.

Two cases remain, and the error message names both. The first is returns data built outside the ingestion layer. The second is a step of the caller that changes the asset set.

Arguments

  • res: The fitted PipelineResult.
  • rd: The returns of the test window, after the fitted steps.

Validation

  • When the fitted context has no returns slot, the check passes.
  • rd.nx == train.nx, where train is the returns slot of the fitted context. Raises an ArgumentError.

Returns

  • nothing.

Related

source