Pipeline: private API
PortfolioOptimisers.first_duplicate — Function
first_duplicate(xs) -> Any
Return the first element of xs that repeats an earlier element.
The error for a repeated step name shows this element, and not the whole collection. Only the failure path of the Pipeline constructor calls this function.
Arguments
xs: A collection of names.
Returns
- The first element that repeats an earlier one, or
nothingwhen every element is unique.
Related
PortfolioOptimisers.assert_step_declarations — Function
assert_step_declarations(ests)
Validate that each step of a Pipeline declares the slot and the target that the step writes when it runs.
run_step selects the method for a PipelineStep that wraps an estimator from the family of that estimator. So the family decides the slot that the step writes, and the writes field does not. An uncertainty-set step computes the halves that its target names, and it stops the fit when it has no such target. The constructor calls this check, so a wrong declaration stops the construction and not the first fold.
Arguments
ests: The step estimators.
Validation
- An uncertainty-set estimator is wrapped in a
PipelineStepwhosetargetis:mu,:sigmaor:both. Raises anArgumentError. - A
PipelineStepthat wraps a preprocessing, prior, phylogeny, uncertainty-set, constraint or optimisation estimator declares the slot of that family inwrites, seepipe_writes. Raises anArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.assert_nested_data_slots — Function
assert_nested_data_slots(ests)
Validate that a nested Pipeline step changes no Data Slot that the outer pipeline does not receive.
A nested pipeline gives one slot to the outer Pipeline Context, the slot of its last step. Assume that a step inside it rewrites :prices or :returns, and that its last step writes a different slot. Then the outer pipeline receives a result for assets that its own Data Slot does not hold. The weights and the prior then describe two different universes, and a prediction fails at assert_universe_aligned.
Arguments
ests: The step estimators.
Validation
- A nested
Pipelinewhose last step writes a slot outsidePIPELINE_DATA_SLOTScontains no step that writes a Data Slot. Raises anArgumentError. Put such a step in the outer pipeline.
Returns
nothing.
Related
PortfolioOptimisers.assert_slot_order — Function
assert_slot_order(ests)
Validate that each step of a Pipeline reads only slots that are filled, and makes stale no slot that an earlier step wrote.
The check walks the steps in order, through foreach_step. slots holds the slot that each step writes, see pipe_writes. last_slot_writer reads it to find the last earlier step that writes a slot, so the message can name that step.
Arguments
ests: The step estimators.
Validation
- Each slot that a step reads, see
pipe_reads, is a Data Slot, seePIPELINE_DATA_SLOTS, or the slot of an earlier step. Raises anArgumentError. - No slot that the write of a step makes stale, see
PIPELINE_INVALIDATES, is the slot of an earlier step. Raises anArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.last_slot_writer — Function
last_slot_writer(slots, s::Symbol, k::Integer) -> Any
Return the position of the last step before step k that writes the slot s.
assert_slot_order calls this function with the slot that each step writes, so it needs no set of the filled slots and no dictionary of the writers.
Arguments
slots: The slot that each step writes, in step order, seepipe_writes.s: The slot to find.k: The position of the step that readssor makes it stale.
Returns
j::Union{Nothing, Int}: The largestj < kwithslots[j] === s, ornothingwhen no earlier step writess.
Related
The container form of the steps
The steps of a pipeline are a Tuple or a vector, as the user gives them. These functions walk the steps and build new containers in that form, so the checks of the constructor allocate nothing on a Tuple.
PortfolioOptimisers.foreach_step — Function
foreach_step(f, ests::Tuple) -> nothing
foreach_step(f, ests::AbstractVector) -> nothingCall f(e, i) on each step e of a Pipeline and its position i, in step order.
The steps of a Tuple have different types. A for loop over them reads each step at a type that inference cannot know, so each call of f dispatches at run time and boxes the step. The Tuple method walks the steps through map over the steps and their positions, which Julia unrolls, so each call of f dispatches at compile time and allocates nothing. The AbstractVector method walks the steps with a loop, at the element type that the vector holds.
Arguments
f: A function of a step and its position. Its value is ignored.ests: The step estimators.
Returns
nothing.
Related
PortfolioOptimisers.map_steps — Function
map_steps(f, ests::Tuple) -> Tuple
map_steps(f, ests::AbstractVector) -> VectorReturn f(e) for each step e of a Pipeline, in the container form of ests.
The Tuple method calls map. The AbstractVector method uses a comprehension, which takes the element type of its values as map does. Over a vector of two steps of an abstract element type, map takes from 1.6 to 10 times the time of the comprehension, measured on the maps of update_online_estimator, copy_states and reset_time_dependent_estimator. A pipeline maps over its steps at each fold of a prediction and at each online step, so the vector form takes the comprehension.
Arguments
f: A function of one step.ests: The step estimators.
Returns
- The values of
f, aTuplewhenestsis aTuple, and a vector otherwise.
Related
PortfolioOptimisers.steps_form — Function
steps_form(ests::Tuple, v::AbstractVector) -> Tuple
steps_form(ests::AbstractVector, v::AbstractVector) -> VectorReturn v in the container form of the steps ests of a Pipeline.
The user chooses the form of the steps, a Tuple or a vector, and the library does not convert one to the other. A Tuple of steps of different types compiles a method for each pipeline, and a vector of steps compiles one method for each element type, which can cost less time to the first result. The names of the steps and the fitted results of a PipelineResult take the same form as the steps. The AbstractVector method narrows v to the element type of its values.
Arguments
ests: The step estimators, which give the form.v: A vector with one entry for each step.
Returns
Tuple(v)whenestsis aTuple, and a new vector of the values ofvotherwise.
Related
Injection
The pipeline turns every slot it computed into routing targets and passes them to the optimiser. The optimiser chooses the field that receives the value, through pipe_route.
PortfolioOptimisers.inject_context — Function
inject_context(
opt::OptimisationEstimator,
ctx::PipelineContext
) -> OptimisationEstimator
Replace the configuration of an optimisation step with the computed slots of the Pipeline Context, immediately before the step runs.
This function is the pipeline half of the routing. It turns the slots into a flat sequence of routing targets. To do this, it finds which halves of the uncertainty pair are present, which result types the constraints slot holds, and how many of each. Then it gives each target to pipe_route, which finds the field that receives the target. The optimiser owns that choice, so a field rename is a local change.
unroutable_target handles a target that an optimiser has no field for. :pe and :cle pass without a change, and every other target raises an error. So a naive optimiser or a meta-optimiser accepts a computed prior that it can use, and refuses an uncertainty set that it cannot use.
Algorithm
- Return
optunchanged when theprior,phylogeny,uncertaintyandconstraintsslots are allnothing. - Route the
priorslot to:pe. - Route the
phylogenyslot to:clewhen it is anAbstractClusteringResult. A phylogeny result of another type has no:cletarget, so this step skips it. - Route the mean half of the
uncertaintyslot to:mu_ucs, and its covariance half to:sigma_ucs, when the half is present. - Route each
target => valuepair ofconstraint_targetsto its target. - Return the rebuilt optimiser
opt′.
Arguments
opt: The optimisation step estimator.ctx: The pipeline context.
Returns
opt′: The estimator that runs. It isoptitself when no slot applies, and a rebuilt copy otherwise.
Related
PortfolioOptimisers.constraint_results — Function
constraint_results(_::Nothing) -> Tuple{}
Return the elements of the constraints slot as one iterable.
Arguments
x:nothing, oneAbstractConstraintResult, or a vector of them.
Returns
- An iterable of constraint results. It is empty for
nothing, and holds one element for one result.
Related
PortfolioOptimisers.constraint_target_of — Function
constraint_target_of(c) -> Symbol
Return the routing target that one element of the constraints slot goes to.
run_step pairs a value that its type cannot place with its target, and this function reads that target. implicit_constraint_target places every other element.
Arguments
c: One element of theconstraintsslot.
Validation
- A
Thresholdwithout a target raises anArgumentError. It names six optimiser fields, so its type cannot place it. The error names the declaration that places it. - A result of any other type that names no target raises an
ArgumentError. The function refuses it here, before it reaches an optimiser.
Returns
target::Symbol: One ofPIPELINE_ROUTING_TARGETS.
Related
PortfolioOptimisers.constraint_value_of — Function
constraint_value_of(c) -> Any
Return the value that one element of the constraints slot delivers, without its routing wrapper.
Arguments
c: One element of theconstraintsslot.
Returns
- The value to route.
Related
PortfolioOptimisers.accumulate_constraint_values — Function
accumulate_constraint_values(
_::Val,
vals
) -> LinearConstraint
Combine the values that reached one accumulating routing target.
The default puts the values in a vector, in write order. Each field that holds one result per estimator reads this shape.
:cte is different. Its field takes a vector of CentralityConstraint estimators, and centrality_constraints appends every row of every estimator into one LinearConstraint. So the values of separate steps merge through merge_linear_constraints. Then n centrality steps in a Pipeline give the optimiser the value that one cte field with n estimators gives.
constraint_targets calls this function with more than one value only, because it unwraps a single value first.
Arguments
::Val{target}: The routing target that the values reached.vals: The values, in write order.
Validation
- For
:cte, each value is aLinearConstraint. Raises anArgumentError.
Returns
- The combined value.
Related
PortfolioOptimisers.constraint_targets — Function
constraint_targets(cs) -> Any
Split the constraints slot into routing targets.
Algorithm
- For each element
cof theconstraintsslot, in write order, read its target withconstraint_target_ofand its value withconstraint_value_of. - Collect the values of each target into its group, in write order. The groups follow the order of the first write to each target, and each group takes the element type of its values.
- Return the value itself for a group of one value. This is the scalar shape that the fields accept outside a pipeline. For a larger group, combine the values with
accumulate_constraint_values. It puts them in a vector in write order, or merges them into one constraint for:cte.
Arguments
cs: Theconstraintsslot.
Validation
- A second value for a target outside
PIPELINE_ACCUMULATING_TARGETSraises anArgumentError. Such a field holds one value, and the second value replaces the first without a message.
Returns
- A vector of
target => valuepairs, in the order of the first write to each target.
Related
PortfolioOptimisers.maybe_inject_step — Function
maybe_inject_step(est, ::PipelineContext) = est
maybe_inject_step(opt::OptimisationEstimator, ctx::PipelineContext)
maybe_inject_step(ps::PipelineStep{<:OptimisationEstimator}, ctx::PipelineContext)Return the step that runs, with the Pipeline Context injected when the step is an optimiser.
A step that is not an optimiser runs unchanged. An optimiser takes the context through inject_context. For a PipelineStep that wraps an optimiser, the function builds a new PipelineStep around the injected optimiser, with the same reads, writes and target.
Arguments
est: A step estimator.opt: An optimisation step estimator.ps: APipelineStep. APipelineStepthat wraps no optimiser runs unchanged.ctx: The pipeline context.
Returns
est: The step estimator, unchanged.opt′: The optimiser with the configuration that the context replaced.ps′: The pipeline step with the injected optimiser.
maybe_inject_step(
res::NonFiniteAllocationOptimisationResult,
ctx::PipelineContext
) -> NonFiniteAllocationOptimisationResult
Apply the injection rules to a precomputed optimisation result in the optimisation step.
The predict-only fold of a mixed TimeDependent schedule puts a result in the optimisation step. A result is solved, so it has no configuration to replace. It follows the rule of inject_context for an optimiser without a field for the target. The computed prior and phylogeny slots pass, because the result has its own. A computed uncertainty set or constraint must not disappear without a solve, so a populated uncertainty or constraints slot stops the fit.
Arguments
res: The precomputed optimisation result.ctx: The pipeline context.
Validation
isnothing(ctx.uncertainty). Raises anArgumentError.isnothing(ctx.constraints). Raises anArgumentError.
Returns
res, unchanged.
Related
PortfolioOptimisers.pipe_required_targets — Function
pipe_required_targets(
ps::PipelineStep
) -> Union{Tuple{}, Tuple{Symbol}, Tuple{Symbol, Symbol}}
Return the routing targets of a step that are known before any step runs.
Two kinds of step have such targets. An uncertainty-set step declares the parameters that it bounds in the target field of its PipelineStep. A constraint step takes its target from its family through pipe_constraint_targets, or from the target field of its PipelineStep when the family names several. run_step resolves the target from the same declaration, so this function returns the target that the step writes.
Every other step returns an empty tuple. A callable step declares no family, and a precomputed result names its target only by its type.
Arguments
est: A step estimator.
Returns
(:mu_ucs,),(:sigma_ucs,)or(:mu_ucs, :sigma_ucs)for a step that writes:uncertaintywith the target:mu,:sigmaor:both.- A tuple of one routing target for a constraint step.
()for every other step. This includes a callable step that writes:uncertaintywith no such target.
Related
PortfolioOptimisers.assert_routable — Function
assert_routable(ests)
Refuse at construction a pipeline whose last optimiser cannot receive a target that an earlier step writes.
Without this check, inject_context finds a target that it cannot route when the optimisation step runs. Under cross_val_predict, that is after the fold loop fitted every earlier step of the first fold. The check asks the optimiser through pipe_accepts, so it follows the fields that the optimiser has.
The check is structural. It finds whether the optimiser family can receive the target, and not whether this configuration accepts the value. A JuMPOptimiser always accepts :mu_ucs, but one with a return estimator other than ArithmeticReturn fails at injection. pipe_route owns that condition.
Algorithm
- Read the optimiser
optof the last step withpipe_terminal_optimiser. Stop when there is none. ATimeDependentschedule and a precomputed result have none, and the fold loop resolves them later. - Read the targets of each earlier step with
pipe_required_targets. - Check each target with
pipe_accepts.
Arguments
ests: The step estimators, with the optimisation step last, seeassert_opt_last.
Validation
pipe_accepts(opt, Val(target))holds for each target of each earlier step. Raises anArgumentError.
Returns
nothing.
Related
PortfolioOptimisers.assert_constraint_targets — Function
assert_constraint_targets(ests)
Validate that each constraint step of a Pipeline resolves to one routing target.
The check calls resolve_constraint_target on each constraint step. run_step makes the same call when the step runs, so three failures move from the fold loop to the constructor. The first is a family that computes nothing for the constraints slot. The second is a family that names several targets when the step names none. The third is a declared target that another family owns.
Arguments
ests: The step estimators.
Validation
- The family of each constraint step declares at least one target, see
pipe_constraint_targets. - When the family declares several targets, the
targetof thePipelineStepnames one of them.
Returns
nothing.
Related
The group of a pipeline
A Pipeline joins the group of its last optimiser, so its constructor warns on a reader that states no source beside a reader of the excess returns. A nested Pipeline adds the frames of its last step to the outer group.
PortfolioOptimisers.group_frames — Method
group_frames(pipe::Pipeline) -> ReturnsFrames
reader_frames(p::Pipeline, path::AbstractString, refit::Bool) -> ReturnsFramesThe frame walk of a Pipeline.
A Pipeline joins the group of its last optimiser, see pipeline_frames, and its constructor warns on that group with warn_raw_readers. A nested Pipeline is a step of an outer one, and the outer optimiser receives the value of its last step, so it adds the frames of that step.
Arguments
pipe,p: The pipeline.path: The path of the value from the head, for example"pe","cle"or"r[2].pe". A warning names a reader by its path.refit:truewhen a regression reader atx_src = nothingin the group fits its regression on the raw returns data.refits_regressionfinds the value.
Returns
fr::ReturnsFrames: The frames of the group ofpipe, or the frames of the last step ofpat the path"<path>.steps[n]".
Related
Prediction
To predict with a fitted pipeline, the pipeline applies its fitted preprocessing steps to the new data window. The steps keep what they learned on the training window: the universe, the imputation parameters and the conversion to returns. The pipeline then predicts with the same functions as an optimisation result. Cross-validation also accepts price data, through Prices_RR. Each fold then fits the whole pipeline on its own training window, so no preprocessing step learns anything from the test window.
PortfolioOptimisers.apply_fitted_step — Function
apply_fitted_step(fitted, data) -> data′Apply one fitted step of a pipeline to a data window for a prediction.
A preprocessing step changes the window at its own data level. A price-level fitted object, an AbstractPricesPreprocessingResult or an AbstractPricesPreprocessingEstimator, changes a price window. A returns-level fitted object changes a returns window. PricesToReturns converts a price window to returns.
A fitted object of the other data level returns the window unchanged. So a pipeline that was fitted on prices can predict on a returns window, and assert_universe_aligned then checks the asset axis of that window. The result of every other step, such as a prior, a phylogeny, an uncertainty set, a constraint or an optimisation result, returns the window unchanged. A nested PipelineResult applies its own steps with apply_fitted_steps.
Arguments
fitted: The fitted result of one step, from aPipelineResult.data: The data window, anAbstractPricesResultor anAbstractReturnsResult.
Returns
data′: The changed data window, ordataitself.
Related
PortfolioOptimisers.apply_fitted_steps — Function
apply_fitted_steps(
results::Union{Tuple, AbstractVector},
data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> Any
Apply the fitted steps of a pipeline to a data window, in step order.
Algorithm
- For each fitted result
finresults, in step order, replacedatawithapply_fitted_step(f, data), seeapply_fitted_step. - Return the last
data.
Arguments
results: The fitted result of each step, from aPipelineResult.data: The data window.
Returns
data′: The changed data window. It is returns data when the steps include aPricesToReturnsstep.
Related
PortfolioOptimisers.assert_universe_aligned — Function
assert_universe_aligned(
res::PipelineResult,
rd::AbstractReturnsResult
)
Check that a test window and the training window come from the same ingestion.
The last weights use the asset axis of the training window. A test window with a different asset set, or with the same set from a different universe statement, misaligns the weights and the returns. This check reports that case with an error that names both universes. Without the check, the case shows as a dimension mismatch inside the risk calculation.
The check reads equality of nx, which checks the asset axis. check_asset_panel binds the asset axis of an AssetPanel to the nx of the result that holds it, so equality of nx also compares the axes of the panels. The check does not compare panel presence. The ingestion layer emits no panel for a window with no gap and no Panel Field, so a gapless training window and a gapped test window of one ingestion differ in panel presence.
Two other parts of the library hold the rest of the alignment. The ingestion layer fixes the asset axis before the split, and port_opt_view slices it. So each window of each fold carries every asset, and no policy of the layer drops a row or a column. An asset that is present in both windows and not investable in one is a Held Gap, which filter_held_gaps reads from the weights and the returns under the strictness policy.
Two cases remain, and the error message names both. The first is returns data built outside the ingestion layer. The second is a step of the caller that changes the asset set.
Arguments
res: The fittedPipelineResult.rd: The returns of the test window, after the fitted steps.
Validation
- When the fitted context has no
returnsslot, the check passes. rd.nx == train.nx, wheretrainis thereturnsslot of the fitted context. Raises anArgumentError.
Returns
nothing.
Related