Step execution: private API

A run_step method is the only code that connects an estimator family to a pipeline. It reads the slots of the PipelineContext that its estimator needs, and calls the function that the family uses outside a pipeline: prior for a prior estimator, clusterise for a clustering estimator, optimise for an optimiser, and fit_preprocessing and apply_preprocessing for a preprocessing estimator. It then writes the slot that the family produces.

The estimators do not depend on the pipeline code, and their docstrings are with their own families. The preprocessing estimators, for example, are on the preprocessing page.

PortfolioOptimisers.run_step — Function
run_step(est, ctx::PipelineContext, ps::Option{PipelineStep} = nothing) -> (fitted, ctx′)

Execute one pipeline step: fit est on the PipelineContext slots it reads and return the fitted object together with a new context whose written slot is updated.

A PipelineStep checks the slots that it declares, then runs its estimator with itself in the slot ps. A bare step runs with ps = nothing. Three estimators read the wrapper. A callable writes its value to the slot that ps.writes names. An uncertainty-set estimator reads the parameters that it bounds from ps.target, and a bare one is refused. A constraint estimator resolves its routing target from ps.target, or from its family when it runs bare. Every other estimator ignores the slot.

Each estimator family dispatches to its native verb — prior for prior estimators, clusterise/phylogeny_matrix for phylogeny estimators, optimise for optimisation estimators, fit_preprocessing/apply_preprocessing for preprocessing estimators. The fitted object is what apply_preprocessing later uses to transform unseen data windows; for non-preprocessing steps it is the step's ordinary result.

Estimators whose family is not steppable throw an ArgumentError directing the caller to PipelineStep.

A step that reads the returns reads them from the source that its own reader states in its x_src field, as the same reader does in the slot of an optimiser. A PipelineStep wrapper holds no source.

Arguments

  • est: The step estimator (or a PipelineStep wrapper).
  • ctx: The pipeline context.
  • ps: The wrapper that holds est, or nothing for a bare step.

Returns

  • (fitted, ctx′): The fitted object and the updated context.

Related

source
run_step(
    res::NonFiniteAllocationOptimisationResult,
    ctx::PipelineContext
) -> Tuple{NonFiniteAllocationOptimisationResult, PipelineContext}
run_step(
    res::NonFiniteAllocationOptimisationResult,
    ctx::PipelineContext,
    ::Union{Nothing, PipelineStep}
) -> Tuple{NonFiniteAllocationOptimisationResult, PipelineContext}

Execute a precomputed optimisation result as the optimisation step: there is nothing to solve, so the result is written to the opt slot as-is and the fold predicts with its weights.

This is how a mixed TimeDependent schedule runs its result entries: the fold-loop swap (update_time_dependent_estimator) replaces the schedule step with entry i, which may be an estimator (the fold optimises) or a result (the fold only predicts). A result cannot consume computed context slots — see maybe_inject_step for the fail-closed injection rules.

Related

source
run_step(
    ue::AbstractUncertaintySetEstimator,
    ctx::PipelineContext,
    ps::PipelineStep
) -> Tuple{Any, PipelineContext}

Execute an uncertainty-set step pinned to a target and merge its result into the uncertainty slot.

The target comes from the PipelineStep wrapper:

  • :mu computes mu_ucs and fills the mean half.
  • :sigma computes sigma_ucs and fills the covariance half.
  • :both computes ucs, which derives both halves from one fit — sharing the prior and, for the sampling algorithms, the simulation draws — and is therefore cheaper than the two narrowed calls.

A narrowed step fills its half of the PipelineUncertaintySets pair and leaves the other untouched, so separate :mu and :sigma steps compose. Every populated half must reach the optimiser: each becomes its own routing target — :mu_ucs and :sigma_ucs — and neither is optional, so a set that cannot be routed is rejected rather than dropped and :both requires an optimiser with an ArithmeticReturn and an UncertaintySetVariance risk measure.

Which slot the step reads is decided by reads_prior_result. An estimator with a prior of its own is fitted from the returns slot. One that reads a prior result — an AbstractPriorUncertaintySetEstimator, or a returns-data estimator with pe = nothing — is fitted from the prior slot instead, so a prior step must come earlier; the returns are not read at all, so a pipeline that writes :prior from a precomputed result needs no :returns slot for such a step.

Arguments

  • ue: The uncertainty-set estimator.
  • ctx: The pipeline context; requires the returns slot, or the prior slot for an estimator that reads a prior result.
  • ps: The wrapper of ue. Its target is :mu, :sigma, or :both; anything else throws an ArgumentError.

Returns

Related

source
run_step(
    ce::AbstractConstraintEstimator,
    ctx::PipelineContext
) -> Tuple{Any, PipelineContext}
run_step(
    ce::AbstractConstraintEstimator,
    ctx::PipelineContext,
    target::Union{Nothing, Symbol}
) -> Tuple{Any, PipelineContext}

Execute a constraint step and append its value, paired with its routing target, to the constraints slot.

Every constraint family that computes a value is a step, and every value it computes has a declared destination — that is the whole of the rule, and pipe_constraint_targets is where it is written down. The destination travels with the value as a TargetedConstraint rather than being re-derived from the result type at injection, so a family whose result type names no unique field (a Threshold, a centrality LinearConstraint, an asset-sets matrix) routes as cleanly as one whose does.

Arguments

  • ce: The constraint estimator.
  • ctx: The pipeline context.
  • target: The PipelineStep target, or nothing for an unwrapped step. The method for a PipelineStep passes the target of the wrapper.

Returns

  • (res, ctx′): The computed value and the updated context. res is the bare value, so what a fitted pipeline reports is the constraint result itself rather than the routing wrapper.

Related

source
run_step(
    tts::TrainTestSplit,
    ctx::PipelineContext
) -> Tuple{TrainTestSplitResult, PipelineContext}
run_step(
    tts::TrainTestSplit,
    ctx::PipelineContext,
    ::Union{Nothing, PipelineStep}
) -> Tuple{TrainTestSplitResult, PipelineContext}

Execute a TrainTestSplit step: replace the input data slot with the training window, and return the fitted TrainTestSplitResult holding both windows.

The split runs at whichever level the pipeline input supplied — prices when the pipeline was fed price-level data, returns otherwise — so the same estimator serves both. Every later step therefore fits on the training window alone, which is the whole point of pinning the split to the first position.

Related

source
run_step(o::Online, ::PipelineContext)
run_step(
    o::Online,
    ::PipelineContext,
    ::Union{Nothing, PipelineStep}
)

Refuses an Online step that reaches a fit with no folds, with an error that names it.

The warm-up of the online arm resolves a wrapper, and fit(pipe, data) runs no warm-up. A wrapper that reaches it has no buffer to seed and no step to fold. A fit of the wrapped estimator as a plain one would return a batch answer under an online declaration, so the method refuses the step instead.

Validation

  • The method always throws an ArgumentError. The message names the wrapped estimator and the two ways out: an Online Scheme, or the plain estimator.

Related

source
PortfolioOptimisers.uncertainty_step_source — Function
uncertainty_step_source(
    ue::AbstractUncertaintySetEstimator,
    ctx::PipelineContext
) -> Union{Tuple{Any, Tuple{Any, Any}}, Tuple{AbstractUncertaintySetEstimator, Tuple{Any}}}

Read the slot an uncertainty-set step fits from, as the arguments its verb takes.

An estimator that reads a prior result — reads_prior_result answers true — is fitted from the prior slot, and never touches the returns; every other estimator is fitted from the returns slot, whose X and F its verb takes. The estimator reads the source that its own prior states, returns_source of ue.pe, as the verb over a ReturnsResult reads it. The observations of a Shared Series gap that the fit reads are dropped first with shared_series_fit: a gap of a vector benchmark under ExcessSource(), and a gap of the factor returns when the prior of the estimator reads them. Each fixed observation weight in the tree of the estimator loses the same observations, so the function returns the cut estimator too. When the optimiser receives the set, the returns slot already has the gaps of its group dropped, see run_group_step, so a raw prior in an excess group reads the same observations as the other readers of the group. The asset returns are then read with benchmark_relative_returns under the same source. The required slot is checked here, so a missing one is refused with an error that names it before any fit. The predicate reads the type, so the branch folds.

Arguments

  • ue: The uncertainty-set estimator.
  • ctx: The pipeline context.

Validation

Returns

  • ue::AbstractUncertaintySetEstimator: The estimator that the fit reads, cut to the observations that the fit keeps. A prior-reading estimator comes back with no change.
  • src::Tuple: (ctx.prior,) for a prior-reading estimator, (X, F) otherwise, where X is the asset returns from the source of ue.pe and both hold the observations that the fit keeps.

Related

source
PortfolioOptimisers.resolve_constraint_target — Function
resolve_constraint_target(
    ce::AbstractConstraintEstimator,
    target::Union{Nothing, Symbol}
) -> Symbol

Resolve the routing target a constraint step writes.

The family declares its targets through pipe_constraint_targets and the step's PipelineStep wrapper supplies target. The three outcomes are the three tuple lengths that declaration can have — no target means the family is not a step, one means the step needs no annotation, several mean it must carry one.

Arguments

  • ce: The constraint estimator.
  • target: The step's declared target, or nothing when it carries no annotation.

Returns

  • target::Symbol: The resolved routing target.

Related

source
PortfolioOptimisers.pipeline_asset_sets — Function
pipeline_asset_sets(
    ctx::PipelineContext,
    est
) -> UniverseSets{String, String, String, String, String, String, String, Dict{K, V}} where {K<:AbstractString, V}

Build the UniverseSets a constraint-generation step needs from the universe names of the context's returns slot.

Every axis the returns declare is declared here: nx always, and nf whenever the returns carry factors. nf is the default tfkey, the time-series factor axis, and that is the right axis because the returns carry F and nothing else: a cross-sectional model's factors are the exposures its own fit was built from, and no returns result carries them. An ExposureConstraintEstimator step resolves its names against this axis, and taking it from rd.nf is what makes it agree with the loadings by construction — the columns of rr.M are the factors the regression was fitted on, which are the columns of rd.F. A step whose loadings are a CrossSectionalFactorModel therefore finds no cfkey entry here and says so through factor_universe; such a mandate needs sets a caller supplies.

Constraint estimators referencing groups beyond the plain universe names cannot be satisfied by this minimal set; precompute their result instead, or wrap a callable in a PipelineStep that supplies richer sets.

Arguments

  • ctx: The pipeline context; requires the returns slot.
  • est: The step about to run, used in the error message.

Returns

  • sets::UniverseSets: Universe sets whose nx entry holds the asset names, plus an nf entry holding the factor names when the returns carry them.

Related

source
PortfolioOptimisers.add_constraint_result — Function
add_constraint_result(ctx::PipelineContext, res::AbstractConstraintResult) -> PipelineContext
add_constraint_result(ctx::PipelineContext, res::AbstractVector) -> PipelineContext
add_constraint_result(ctx::PipelineContext, ::Nothing) -> PipelineContext

Append a constraint result to the constraints slot of the context.

The slot accumulates: the first result is stored as-is, later results widen it into a Vector{AbstractConstraintResult} preserving step order.

Two shapes constraint generation can return are absorbed rather than rejected, because both are ordinary outcomes of a step rather than errors:

  • nothing, which is what linear_constraints returns when every row was dropped — a non-strict run whose names were all unknown, or a re-basis the loadings annihilated. The step contributed no constraint, so the slot is left untouched; the slot's job is to carry constraints, not to re-diagnose a condition generation already decided was recoverable.
  • a vector, which is what a step wrapping a vector of estimators returns. Its elements are appended individually, so they reach constraint_targets as siblings of every other step's result rather than as a nested vector it has no case for.

Arguments

Returns

  • ctx′::PipelineContext: The updated context.

Related

source

The frame of a step

A step that reads the returns reads them from the source that its own reader states in its x_src field. A PipelineStep holds no source. The steps whose values the last optimiser receives join the group of that optimiser, and the readers that those values replace leave it. The Pipeline constructor warns on that group, and it refuses a prior step that fits on the excess returns and feeds an optimiser with a data benchmark.

PortfolioOptimisers.reader_frames — Method
reader_frames(ps::PipelineStep, path::AbstractString, refit::Bool) -> ReturnsFrames

Return the frames that the estimator of a PipelineStep adds to the group of a Pipeline.

The wrapper holds no source. The estimator that it wraps states its own, so the walk enters the field est.

Arguments

  • ps: The step wrapper.
  • path: The path of the value from the head, for example "pe", "cle" or "r[2].pe". A warning names a reader by its path.
  • refit: true when a regression reader at x_src = nothing in the group fits its regression on the raw returns data. refits_regression finds the value.

Returns

  • fr::ReturnsFrames: The frames of ps.est, at the path "<path>.est".

Related

source
PortfolioOptimisers.step_prior — Function
step_prior(step) -> Option{<:AbstractPriorEstimator}

Return the prior estimator that a step of a Pipeline fits into the prior slot, or nothing.

The answer sees through a PipelineStep and an Online wrapper. A nested Pipeline answers for its last step. A callable and a precomputed Prior Result state no source, so they answer nothing.

Internal machinery — not part of the user-facing API.

Arguments

  • step: A step of a Pipeline.

Returns

  • pe::Option{<:AbstractPriorEstimator}: The prior estimator of the step, or nothing.

Related

source
PortfolioOptimisers.pipe_step_targets — Function
pipe_step_targets(step) -> Tuple

Return the routing targets that the value of a step of a Pipeline fills in the last optimiser.

inject_context puts the prior slot into pe, and a clustering result in the phylogeny slot into cle. Every other target comes from pipe_required_targets. A network step writes a phylogeny matrix, which reaches no target.

Internal machinery — not part of the user-facing API.

Algorithm

  1. A step that writes :prior fills :pe.
  2. A step that writes :phylogeny fills :cle when it is a clustering estimator or a clustering result, and nothing otherwise.
  3. Every other step fills the targets that pipe_required_targets gives.

Arguments

  • step: A step of a Pipeline.

Returns

  • targets::Tuple: The routing targets of the value of the step.

Related

source
PortfolioOptimisers.pipe_target_paths — Function
pipe_target_paths(x, ::Val{target}) -> Vector{String}

Return the paths, from the optimiser x, of the slots that a value at a routing target replaces.

The walk follows pipe_route. A target goes into the field of the same name, or into the configuration that @pipe_delegates names. :mu_ucs goes into ret.ucs of a JuMPOptimiser, and :sigma_ucs goes into the ucs of each UncertaintySetVariance in the field r. The frame walk of a Pipeline leaves these slots out of the group of the optimiser, because the value of the step replaces the reader that they hold.

Internal machinery — not part of the user-facing API.

Algorithm

  1. :sigma_ucs: the path "r.ucs", or "r[i].ucs" for each element i of a vector that is an UncertaintySetVariance. Only an optimiser that @pipe_route_sigma_ucs declares accepts this target, and each one has a field r.
  2. A target with a field of the same name in x: the name of the field.
  3. Otherwise the paths in the configuration of x, after its field name, see pipe_config_paths.

Arguments

  • x: The optimiser or the optimiser configuration.
  • ::Val{target}: The routing target.

Returns

  • paths::Vector{String}: The paths of the replaced slots. It is empty when x has no slot for the target.

Related

source
PortfolioOptimisers.pipe_config_paths — Function
pipe_config_paths(x, t::Val) -> Vector{String}

Return the paths of pipe_target_paths in the configuration of x, after the name of the field that holds it, or an empty vector when x delegates to no configuration.

A target that the ConstraintBlock of x accepts goes into the block, as pipe_route puts it there. So its path starts with the field that pipe_block_field names, such as "cbe.ple", which is the path that the frame walk gives to the slot.

Internal machinery — not part of the user-facing API.

Arguments

  • x: The optimiser.
  • t: The routing target.

Returns

  • paths::Vector{String}: The paths from x.

Related

source
PortfolioOptimisers.pipe_prior_estimator — Function
pipe_prior_estimator(x)

Return the prior that the optimiser x holds in the field that receives the :pe routing target, or nothing when it has none.

The walk follows pipe_route: the field pe of x, or of the configuration that @pipe_delegates names.

Internal machinery — not part of the user-facing API.

Arguments

  • x: The optimiser or the optimiser configuration.

Returns

  • pe: The prior estimator, the Prior Result or the schedule in the field, or nothing.

Related

source
PortfolioOptimisers.cut_frames — Function
cut_frames(x, path::AbstractString, refit::Bool, cut::AbstractVector{<:AbstractString}) -> ReturnsFrames

Return the frames that x adds to a group, without the slots at the paths in cut.

The walk of a Pipeline calls this function on its last optimiser. A step value replaces each slot in cut, so the reader that the slot holds never runs. The function enters only the values on the way to a cut slot, field by field and element by element, and gives every other value to reader_frames. The optimiser itself is at the empty path, and it always enters, because a head has no method of reader_frames.

Internal machinery — not part of the user-facing API.

Algorithm

  1. A value at a path in cut adds nothing.
  2. A value at a path that no cut path starts below adds reader_frames(x, path, refit).
  3. Otherwise join the frames of each element of a vector, or of each field, with cut_children.

Arguments

  • x: The value to walk: a reader of the returns, a value that holds readers, a vector of them, or any other value.
  • path: The path of the value from the head, for example "pe", "cle" or "r[2].pe". A warning names a reader by its path.
  • refit: true when a regression reader at x_src = nothing in the group fits its regression on the raw returns data. refits_regression finds the value.
  • cut: The paths of the replaced slots, from pipe_target_paths.

Returns

  • fr::ReturnsFrames: The frames of x without the cut slots.

Related

source
PortfolioOptimisers.cut_children — Function
cut_children(x, path::AbstractString, refit::Bool, cut::AbstractVector{<:AbstractString})
cut_children(x::AbstractVector, path::AbstractString, refit::Bool,
             cut::AbstractVector{<:AbstractString})

Return the frames of each field of x, or of each element of a vector, through cut_frames.

cut_frames calls this function on a value on the way to a cut slot. A field is at the path that join_path gives, and an element is at "<path>[i]", as reader_frames names it.

Internal machinery — not part of the user-facing API.

Arguments

  • x: The value to walk: a reader of the returns, a value that holds readers, a vector of them, or any other value.
  • path: The path of the value from the head, for example "pe", "cle" or "r[2].pe". A warning names a reader by its path.
  • refit: true when a regression reader at x_src = nothing in the group fits its regression on the raw returns data. refits_regression finds the value.
  • cut: The paths of the replaced slots.

Returns

  • fs::Vector{ReturnsFrames}: The frames of each field or element, in order.

Related

source
PortfolioOptimisers.pipe_terminal_optimiser — Function
pipe_terminal_optimiser(
    ests
) -> Union{Nothing, OptimisationEstimator}

Return the optimiser of the last step of a Pipeline, or nothing when the pipeline has none that a construction check can read.

The checks that compare the earlier steps with the optimiser read it here: assert_routable, assert_pipeline_data_benchmark and pipeline_frames.

Algorithm

  1. Read the optimiser from the last step, without its PipelineStep wrapper.
  2. Return nothing when the pipeline has one step, or when the optimiser is not an OptimisationEstimator. A TimeDependent schedule and a precomputed result are such steps, and the fold loop resolves them later. The type is checked first, because fit also reads the optimiser for the gap drop of the group, see pipeline_gap_sources, and in a fold of a mixed schedule the last step is a precomputed result, whose written slot pipe_writes cannot infer.
  3. Return the optimiser when its step writes :opt, and nothing otherwise.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.

Returns

  • opt::Option{<:OptimisationEstimator}: The optimiser, or nothing.

Related

source
PortfolioOptimisers.pipeline_frames — Function
pipeline_frames(ests) -> ReturnsFrames

Return the frames of the group that the steps of a Pipeline join: the group of its last optimiser.

inject_context puts the value of each step into a slot of the last optimiser, so the step reads the returns for that optimiser. The steps whose values the optimiser receives join its group, and the readers that those values replace leave it. A network step writes a phylogeny matrix that no slot receives, so it stays out. A prior step decides refit for the optimiser when the optimiser receives its value, because that prior then replaces the prior of the optimiser. A step reads the prior slot, so the last prior step decides refit for the steps. With no prior step a step reads a Prior Result that the input gave, which reads as raw. group_frames of a Pipeline calls this function.

Internal machinery — not part of the user-facing API.

Algorithm

  1. Read the optimiser opt of the last step with pipe_terminal_optimiser. Answer that the group holds no reader when there is none.
  2. For each earlier step, keep the targets of pipe_step_targets that opt accepts (pipe_accepts). The value of a step with no such target does not reach opt.
  3. Find refit for the steps and for opt with pipeline_refits.
  4. Walk each step that opt receives with reader_frames, at the path "steps[i]".
  5. Walk opt without the slots that the steps replace with terminal_frames, and join every part with merge_frames.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.

Returns

  • fr::ReturnsFrames: The frames of the group of the last optimiser, with the steps that it receives.

Related

source
PortfolioOptimisers.pipeline_received — Function
pipeline_received(ests, opt) -> Vector{Vector{Symbol}}

Return, for each step of a Pipeline before the last, the routing targets of the step that the last optimiser opt accepts.

A step with no such target does not reach opt, so it stays out of the group of opt.

Internal machinery — not part of the user-facing API.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.
  • opt: The optimiser of the last step.

Returns

  • received::Vector{Vector{Symbol}}: The targets that opt receives from each earlier step.

Related

source
PortfolioOptimisers.pipeline_gap_sources — Function
pipeline_gap_sources(ests) -> Vector{Option{ExcessSource}}

Return, for each step of a Pipeline, the source under which the Pipeline drops a gap of a vector benchmark before the step runs.

The steps whose values the last optimiser receives join its group, see pipeline_frames. A head drops a gap of a vector benchmark when any reader of its group states ExcessSource, see group_source, so every reader of the head pairs the same observations. The Pipeline applies the same rule: each step of the group, and the optimiser, reads the returns data with the gaps of the group dropped. A raw reader of the group loses the observations too. The rule decides which observations a step reads, and not its frame: each reader still reads the frame that its own x_src states. A step outside the group gets nothing, so it drops by the rule of its own reader alone.

Internal machinery — not part of the user-facing API.

Algorithm

  1. Read the optimiser opt of the last step with pipe_terminal_optimiser. When there is none, answer nothing for every step, except for a Pipeline of one optimiser step: that step is a head alone, so it gets group_source of its optimiser.
  2. Find the source of the group with group_source of pipeline_frames, giving gsrc.
  3. Answer gsrc for each step that opt receives (pipeline_received) and for the last step, and nothing for every other step.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.

Returns

  • gsrcs::Vector{Option{ExcessSource}}: One source for each step.

Related

source
PortfolioOptimisers.run_group_step — Function
run_group_step(step, ctx::PipelineContext, gsrc::Nothing) -> (fitted, ctx′)
run_group_step(step, ctx::PipelineContext, gsrc::ExcessSource) -> (fitted, ctx′)

Run one step of a Pipeline with the gaps of the group of its optimiser dropped from the returns data that the step reads.

The step reads the returns slot with each observation at which a vector benchmark has a gap dropped, through shared_series_view under gsrc. The returns data is then put back in the slot, so the next step, and the context of the PipelineResult, hold the returns data that the caller gave. A step of the group never writes the returns slot, because the optimiser receives no Data Slot. The reader of the step then drops a gap of the factor returns, and a gap of the benchmark under its own source, as it does in a head. Both drops are idempotent on a view with no gap left.

Internal machinery — not part of the user-facing API.

Algorithm

The method that Julia selects is the algorithm.

  1. gsrc is nothing: run the step with run_step.
  2. gsrc is ExcessSource: view the returns slot with group_returns_view, run the step on that context, and put the returns data back.

Arguments

Returns

  • (fitted, ctx′): The fitted object and the updated context.

Related

source
PortfolioOptimisers.group_returns_view — Function
group_returns_view(step, rd::ReturnsResult, gsrc::ExcessSource) -> Tuple{Any, ReturnsResult}
group_returns_view(step, rd, gsrc::ExcessSource)

Return the step and the returns data that a step of the group of a Pipeline reads: rd with each gap of a vector benchmark dropped, and step with each fixed observation weight of the same observations dropped, see shared_series_fit.

A slot that holds no ReturnsResult, such as an empty slot before a step that reads prices, is returned unchanged with the step.

Internal machinery — not part of the user-facing API.

Related

source
PortfolioOptimisers.pipeline_refits — Function
pipeline_refits(ests, received, opt) -> Tuple{Bool, Bool}

Return refit for the steps of a Pipeline and refit for its last optimiser, see refits_regression.

A step reads the prior slot, so the last prior step decides refit for the steps. With no prior step a step reads a Prior Result that the input gave, which reads as raw. A prior step decides refit for the optimiser when the optimiser receives its value, because that prior then replaces the prior of the optimiser.

Internal machinery — not part of the user-facing API.

Algorithm

  1. Find the last prior step with step_prior. refit for the steps is refits_regression of its prior, or false when there is none.
  2. Find the last step whose targets in received hold :pe. refit for opt is refits_regression of its prior, or of the prior of opt (pipe_prior_estimator) when there is none.

Arguments

  • ests: The step estimators, with the optimisation step last.
  • received: The routing targets that opt receives from each earlier step.
  • opt: The optimiser of the last step.

Returns

  • (step_refit, opt_refit)::Tuple{Bool, Bool}: refit for the steps and for opt.

Related

source
PortfolioOptimisers.terminal_frames — Function
terminal_frames(step, n::Integer, opt, received, refit::Bool) -> ReturnsFrames

Return the frames of the last optimiser of a Pipeline, without the slots that the earlier steps replace, at the path of the last step.

Internal machinery — not part of the user-facing API.

Algorithm

  1. Find the paths of the slots that the earlier steps replace with pipe_target_paths.
  2. When there are none, take group_frames of opt. Otherwise walk opt without those slots with cut_frames.
  3. Put the path of the last step, "steps[n]" or "steps[n].est" for a PipelineStep, before each raw path.

Arguments

  • step: The last step, opt or its wrapper.
  • n: The position of the last step.
  • opt: The optimiser of the last step.
  • received: The routing targets that opt receives from each earlier step.
  • refit: true when a regression reader at x_src = nothing in the group fits its regression on the raw returns data. refits_regression finds the value.

Returns

  • fr::ReturnsFrames: The frames of opt, with paths from the pipeline.

Related

source
PortfolioOptimisers.pipe_data_benchmark_path — Function
pipe_data_benchmark_path(x) -> Option{<:String}

Return the path of the first DataBenchmarkTracking in a measure that reads the prior of the optimiser x, or nothing.

Every tracking form compares pr.X * w with its benchmark series, so the prior that the measure reads decides its frame. The measures that read the prior of a head are the tracking errors tr of its configuration and the risk measures r, ri and ro of the head. The walk follows the path of the :pe routing target through pipe_config_field, so it reads the fields that hold the prior and the fields beside them. It does not enter an inner optimiser or a fallback, because each one checks its own prior.

Internal machinery — not part of the user-facing API.

Algorithm

  1. For each of the fields tr, r, ri and ro of x, in order, return the path that data_benchmark_path finds below it, after the name of the field.
  2. Otherwise return the answer for the ConstraintBlock of x, after the name of its field, see pipe_block_field. The block holds the tracking errors tr of a JuMPOptimiser.
  3. An x with no block returns the answer for its configuration, after the name of its field. An x with neither answers for nothing, which is nothing.

Arguments

  • x: The optimiser or the optimiser configuration.

Returns

  • path::Option{<:String}: The path of the data benchmark from x, or nothing.

Related

source
PortfolioOptimisers.assert_pipeline_data_benchmark — Function
assert_pipeline_data_benchmark(ests) -> Nothing

Refuse at construction a Pipeline whose prior step states x_src = ExcessSource() when the optimiser that receives the prior tracks a DataBenchmarkTracking.

The optimiser receives the Prior Result of the step in its field pe. In the Excess Frame pr.X holds X - B, so a data benchmark gives the series (X - B) * w - B, and the benchmark is subtracted two times. The constructor of the optimiser refuses this pair for its own prior. It cannot see the source of the step, because the step value replaces its prior only when the pipeline runs. So the Pipeline constructor refuses it. The configuration alone decides the refusal. A prior step that holds a precomputed Prior Result, or a callable, states no source, so it reads as raw and passes.

Internal machinery — not part of the user-facing API.

Algorithm

  1. Read the optimiser opt of the last step with pipe_terminal_optimiser. Stop when there is none, or when it has no field for the :pe routing target.
  2. Find the last prior step, whose Prior Result the prior slot holds when opt runs, with step_prior. Stop when there is none, or when its source is not ExcessSource().
  3. Find a data benchmark in a measure of opt that reads its prior with pipe_data_benchmark_path.

Arguments

  • ests: The step estimators, with the optimisation step last, see assert_opt_last.

Validation

  • The last prior step does not state x_src = ExcessSource(), or the optimiser holds no DataBenchmarkTracking in a measure that reads its prior. An ArgumentError that names the step, the field and the two remedies is thrown otherwise.

Returns

  • nothing.

Related

source

A TrainTestSplit is the one step whose written slot does not follow from its type. It narrows whichever data slot the pipeline input filled, the prices or the returns. pipe_writes returns :split for it, which is not a slot, so the pipeline treats the step as one that writes nothing. The docstrings of run_step, pipe_reads and pipe_writes cover the run_step method of the split and its slot declarations.