Stacking

PortfolioOptimisers.StackingResult — Type
struct StackingResult{__T_pr, __T_wb, __T_fees, __T_resi, __T_reso, __T_cv, __T_scorer, __T_retcode, __T_w, __T_imsk, __T_fb} <: NonJuMPOptimisationResult

Holds the inner results, the outer result and the stacked weights of a stacking optimisation.

optimise returns it for a Stacking. resi holds one result for each inner optimiser, in the order of opti. reso is the result of the outer optimiser on the synthetic assets, so reso.w holds one entry for each inner optimiser, not one for each asset. w holds the stacked weights over the assets.

Fields

  • pr: Prior result.
  • wb: Weight bounds.
  • fees: Fees estimator or result.
  • resi: Inner optimisation results.
  • reso: Outer optimisation results.
  • cv: Inner cross-validation scheme of the sub-portfolios, an OptCVE_Onl, or nothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio. nothing gives it the returns of the solve over the full sample.
  • retcode: Optimisation return code.
  • w: Stacked portfolio weights over the full asset universe, after the weight finaliser. An outer efficient frontier gives one weight vector for each point.
  • imsk: The Investable Mask the optimisation reduced on: true at every asset whose prior moments were finite. It is nothing when every asset was investable, and that sentinel is what skips both the reduction and the expansion. investable_mask derives it once from the full-universe prior result, and the result carries it, because the reduced prior can no longer yield it.
  • fb: The fallback chain that answered this result: the (estimator, result) pair of every attempt optimise made before this one, in the order they ran, or nothing when the estimator it was asked of answered (see FbChain).

Constructors

StackingResult(;    pr::Option{<:AbstractPriorResult},    wb::Option{<:WeightBounds},    fees::Option{<:Fees},    resi::AbstractVector{<:NonFiniteAllocationOptimisationResult},    reso::OptimisationResult,    cv::Option{<:OptCVE_Onl},    scorer::Option{<:PredictionCrossValScorer},    retcode::OptRetCode_VecOptRetCode,    w::VecNum_VecVecNum,    imsk::Option{<:BitVector} = nothing,    fb::Option{<:OptE_Opt_FbChain}) -> StackingResult

Keywords correspond to the struct's fields. The keyword constructor expands w from the assets of the Investable Mask imsk onto the full asset universe, through expand_investable_weights. An asset outside the mask gets a zero weight. _optimise returns through this constructor. The positional constructor does not expand w, so set_retcode and factory rebuild a result and do not expand it a second time.

Related

References

  • [246] D. H. Wolpert. Stacked generalization. Neural Networks 5, 241–259 (1992).
source
PortfolioOptimisers.Stacking — Type
struct Stacking{__T_pe, __T_wb, __T_fees, __T_sets, __T_scale, __T_opti, __T_opto, __T_cv, __T_scorer, __T_wf, __T_ex, __T_fb, __T_strict, __T_pcol, __T_cache} <: BaseStackingOptimisationEstimator

Combines several inner optimisers through one outer optimiser that allocates across their portfolios.

Stacking solves each inner optimiser of opti on the whole sample. The portfolio of each inner optimiser is one synthetic asset, and the outer optimiser opto allocates across these synthetic assets. The stacked weights are the inner weights, combined by the outer weights. This is the stacked generalisation of [246], with portfolios in place of predictors.

cv selects the returns of the synthetic assets that opto reads. With cv, they are out-of-sample returns. Each observation comes from inner solves that did not see it, which is the level-one data of [246]. Without cv, they are the in-sample returns of the full-sample inner solves. Then opto reads the observations that fitted the inner weights, and it does not see the returns of an inner optimiser on data that it did not fit.

Mathematical definition

Without cv:

\[\begin{align} \mathbf{R}_{\cdot k} &= \mathbf{X} \mathbf{W}_{\cdot k} - F(\mathbf{W}_{\cdot k})\,,\\ \boldsymbol{v} &= \mathcal{O}(\mathbf{R})\,,\\ c_k &= \begin{cases} \dfrac{s_k v_k}{\sum_{j=1}^{K} s_j v_j} \sum_{j=1}^{K} v_j & \text{if } \sum_{j=1}^{K} s_j v_j \neq 0 \text{ and } \sum_{j=1}^{K} v_j \neq 0\,,\\ s_k v_k & \text{otherwise}\,, \end{cases}\\ \boldsymbol{w} &= \mathbf{W} \boldsymbol{c}\,. \end{align}\]

Where:

  • $\mathbf{R}$: Returns matrix of the synthetic assets, $T \times K$, with column $k$ for inner optimiser $k$.
  • $\mathbf{X}$: Returns matrix $T \times N$.
  • $\mathbf{W}$: Inner weight matrix assets × sub-portfolios. Column $k$ holds the weights of sub-portfolio $k$ over all $N$ assets, and zero at an asset outside the sub-portfolio. Sub-portfolio $k$ is inner optimiser $k$, solved on the whole sample.
  • $F(\boldsymbol{w})$: Fee series $T \times 1$ of the portfolio, zero when fees is nothing. See calc_net_returns. Here $\boldsymbol{w}$ is $\mathbf{W}_{\cdot k}$.
  • $\mathcal{O}$: The outer optimiser, which maps the returns matrix of the synthetic assets to their weights.
  • $\boldsymbol{v}$: Outer weight vector, the weights that the outer optimiser gives the $K$ synthetic assets.
  • $c_k$: Coefficient of sub-portfolio $k$ in the combination.
  • $s_k$: Combination Weight of sub-portfolio $k$. Without a Combination Weight, $\boldsymbol{c} = \boldsymbol{v}$.
  • $K$: Sub-portfolio count of a meta-optimiser, which is also the number of synthetic assets of its outer problem. Here it is the number of inner optimisers.
  • $\boldsymbol{w}$: Stacked weights over the $N$ assets.
  • $T$: Number of observations.
  • $N$: Number of assets.

With cv, row $t$ of $\mathbf{R}_{\cdot k}$ is the return at observation $t$ of inner optimiser $k$, solved on the training observations of the fold whose test window holds $t$. Then $\mathbf{R}$ has one row for each observation that a test window holds. $\mathbf{W}$ is the full-sample matrix in both cases.

$\mathcal{O}$ reads $\mathbf{R}$, which does not depend on $s_k$. So the Combination Weight acts at the combination alone, and a run with cv applies it in the same way as a run without cv. The ratios $c_k / c_j = s_k v_k / (s_j v_j)$ hold in both cases of $c_k$, and the first case keeps the total, $\sum_k c_k = \sum_k v_k$.

Fields

  • pe: Prior estimator.
  • wb: Weight bounds estimator or weight bounds.
  • fees: Fees estimator.
  • sets: Sets used to map estimator values to assets.
  • scale: Combination Weight of the inner optimisers, one entry for each element of opti, or nothing. Entry $k$ is the weight that inner optimiser $k$ carries in the combination that the answer of opto defines. Only the ratios between the entries matter, because combination_weights rescales the tilted coefficients to the total of opto. So a common factor cancels. A uniform weight and the weight of a lone inner optimiser change nothing. nothing keeps the answer of opto.
  • opti: Inner optimiser.
  • opto: Outer optimiser.
  • cv: Inner cross-validation scheme of the sub-portfolios, an OptCVE_Onl, or nothing. A scheme gives the outer optimiser the out-of-sample returns of each sub-portfolio. nothing gives it the returns of the solve over the full sample.
  • wf: Weight finaliser.
  • ex: Parallel execution strategy.
  • fb: Fallback result or estimator.
  • strict: Whether a constraint that names an asset or a group that sets does not hold raises an error. When it is false, the constraint generator warns and skips the entry.
  • pcol: Rule of the panel collapse, an AbstractPanelCollapseAlgorithm. It states how the collapse of the Asset Panel onto the sub-portfolios weighs a member at an observation where the member is outside the universe. RenormaliseActive divides by the weight of the active members, and InactiveAsCash reads the missing weight as cash with a zero feature.
  • cache: Optional ReturnsBufferState, the fold context of the online step. It is nothing until partial_fit! writes one, and optimise(opt) with no returns reads it. The returns themselves are carried by the prior, which owns the rows once; this holds every other column of the returns data and the context pinned at the first step. factory carries it unchanged and port_opt_view slices it to the selected assets.

Constructors

Stacking(;    pe::Onl{<:TD{<:PrE_Pr}} = EmpiricalPrior(),    wb::TD_Option{<:WbE_Wb} = nothing,    fees::TD_Option{<:FeesE_Fees} = nothing,    sets::TD_Option{<:UniverseSets} = nothing,    scale::TD_Option{<:VecNum} = nothing,    opti::Union{AbstractVector, TD_VecOptE_Opt},    opto::OptE_TD,    cv::Option{<:OptCVE_Onl} = nothing,    scorer::Option{<:PredictionCrossValScorer} = nothing,    wf::TD{<:WeightFinaliser} = IterativeWeightFinaliser(),    ex::FLoops.Transducers.Executor = FLoops.ThreadedEx(),    fb::TDO_Option{<:OptE_Opt} = nothing,    strict::Bool = false,    pcol::AbstractPanelCollapseAlgorithm = RenormaliseActive(),    cache::Option{<:ReturnsBufferState} = nothing) -> Stacking

Keywords correspond to the struct's fields.

Time-dependent fields

pe, wb, fees, sets, scale, wf, opto and fb can hold a TimeDependent schedule. No inner fold loop of Stacking reads them, so the fold loop that reaches the Stacking resolves them. A schedule in opto or fb must bind :outermost. opti takes a schedule at two levels:

  • Element, as in opti = [static, TimeDependent(…)]. The inner cross-validation solves each element on its own, so an element schedule can bind :nearest. Such a schedule needs an explicit default and a cv that is not nothing, because the full-sample solve of the element always resolves it to its default. See assert_nearest_optimiser_schedule.
  • Field, as in opti = TimeDependent([[…], […]]), one vector of inner optimisers for each fold. See TD_VecOptE_Opt. The schedule must bind :outermost. The inner cross-validation receives the elements and never the field, and a vector that changes with the fold changes the number and the order of the synthetic assets that opto reads.

Validation

  • When opti is a vector: !isempty(opti). The constructor throws an IsEmptyError otherwise.
  • Every element of opti is an OptE_Opt_TD. The keyword constructor throws an ArgumentError otherwise, through erase_optimiser_vector. Both constructors store a vector opti as a Vector{OptE_Opt_TD}, so two Stacking estimators that differ only in their inner optimisers have one type.
  • An element schedule of opti that binds :nearest has an explicit default, and cv !== nothing. The constructor throws a TimeDependentDefaultError or an ArgumentError otherwise, through assert_nearest_optimiser_schedule.
  • When opti is a TimeDependent: opti.bind !== :nearest. The constructor throws an ArgumentError otherwise.
  • When scale is a vector: all(isfinite, scale), and length(scale) == length(opti) when opti is a vector. The constructor throws an IsNonFiniteError or a DimensionMismatch otherwise.
  • A schedule in opto or fb has bind !== :nearest. The constructor throws an ArgumentError otherwise.
  • scorer is nothing unless cv is a CombinatorialCrossValidation. The constructor throws an ArgumentError otherwise, through assert_meta_scorer.
  • opto passes assert_external_optimiser, and every element of opti passes it when cv !== nothing. The constructor throws an ArgumentError otherwise. Such an optimiser runs on the synthetic assets or on the training rows of a fold, which it does not know in advance. So it cannot hold a precomputed prior or a precomputed constraint.
  • When wb is keyed by name (name_keyed) or fees is a FeesEstimator: !isnothing(sets). The constructor throws an IsNothingError otherwise.
  • Each entry and the default of each schedule give a valid Stacking. assert_time_dependent_substitution runs the constructor on each of them.
  • A reader that states no source and reads the raw returns data, beside a reader that states x_src = ExcessSource(), logs one warning that names its path, see warn_raw_readers. The head then builds as configured.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • fees: Recursively updated through factory.
  • opti: Recursively updated through factory.
  • opto: Recursively updated through factory.
  • fb: Recursively updated through factory.
  • cache: Carried unchanged through factory.

View parameters

Stacking defines its own port_opt_view method rather than deriving one from field tags.

  • The method reads the returns matrix X as its third argument. When pe holds a prior result, the method replaces X with pe.X, so it views the children against the observations of that prior.
  • pe, wb, sets and cache recurse through port_opt_view with the index alone.
  • fees, opti and opto recurse through port_opt_view with that matrix.
  • fb recurses through view_child with that matrix.
  • scale stays unchanged. It holds one entry for each inner optimiser, not one for each asset.

Observation weight parameters

obs_weights_view indexes these fields to the selected observations:

Related

References

  • [246] D. H. Wolpert. Stacked generalization. Neural Networks 5, 241–259 (1992).
source
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
                                 AbstractResult, StatsBase.CovarianceEstimator}},
        args...; kwargs...) -> AbstractArray

Rebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.

factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.

factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.

The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.

Algorithm

The leaf method:

  1. Return a with no change, and drop args... and kwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.

The array method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... with no change.
  2. Collect the results into a new array of the shape of a, in the order of a, and return it.

A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.

The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.

Arguments

  • a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators or nothing, which it rebuilds one element at a time.
  • args...: Positional arguments. The leaf method ignores them, and the array method forwards them.
  • kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.

Returns

  • a: The input with no change, for the leaf method.
  • v::AbstractArray: The rebuilt elements, in an array of the shape of a, for the array method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
factory(res::NonFiniteAllocationOptimisationResult, fb::Option{<:OptE_Opt_FbChain})

Rebuilds a continuous optimisation result with a new fallback record fb.

A concrete result can add its own method when a rebuild needs more than a new fb. optimise is the one caller, and it gives the FbChain of the attempts that failed.

Algorithm

  1. Read every field of res, in the order of the struct.
  2. Call the constructor of the type of res with every field except the last, and with fb as the last field.

Validation

  • fb is the last field of the type of res. Every concrete optimisation result obeys this rule, so the method checks nothing.

Related

source
factory(
    opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
    _
) -> EqualWeighted{_A, _B, _C, _D, _E, Bool} where {_A, _B, _C, _D, _E}

Returns opt unchanged.

This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.

Related

source
PortfolioOptimisers.factory — Method
factory(a, args...; kwargs...) -> a
factory(a::AbstractArray{<:Union{Nothing, AbstractEstimator, AbstractAlgorithm,
                                 AbstractResult, StatsBase.CovarianceEstimator}},
        args...; kwargs...) -> AbstractArray

Rebuild an estimator, an algorithm or a result with runtime values. These two methods are the leaf and the array step of the traversal.

factory and port_opt_view are the two propagation mechanisms of this library. factory passes runtime values, such as prior moments, observation weights and previous portfolio weights, down a tree of composed structs. port_opt_view passes an index selection, a subset of the assets, down the same tree.

factory is a traversal, so it owns the identity on every leaf, as fmap in Functors.jl and adapt_structure in Adapt.jl do. A value that has no method of its own comes back with no change: a data field, a scalar, nothing, a set of observation weights and a covariance estimator of StatsBase. So a @propagatable struct sends each @fprop field to factory directly, whatever the field holds.

The array method is the one forwarding contract for every propagation field that holds an array of estimators, algorithms, results or covariance estimators. It applies factory to each element, and it forwards args... and kwargs... with no change. A family that accepts an array of them needs no method of its own. A family that needs more than the forward defines its own, more specific method. A concrete element type, through concrete_typed_array_if_abstract, is an example. An array of data, such as a vector of numbers, is a leaf, and it comes back with no change and with no copy.

Algorithm

The leaf method:

  1. Return a with no change, and drop args... and kwargs.... This method is the leaf of the recursion. Because of it, a call of the verb on a type with no tags returns the value and raises no error.

The array method:

  1. For each element ai of a, call factory on ai, and forward args... and kwargs... with no change.
  2. Collect the results into a new array of the shape of a, in the order of a, and return it.

A @propagatable struct always has a generated method that is more specific than the leaf method. If the struct has at least one @fprop or @wprop field, that method builds the struct again with its keyword constructor. It sends each @fprop field through factory and each @wprop field through _wprop. If the struct has no such field, that method returns x.

The leaf method does not fill a nothing field. A weights field that must take the incoming weights when it holds nothing carries @wprop, and it goes to _wprop.

Arguments

  • a: Any value, which the leaf method returns with no change. For the array method, an array of estimators, algorithms, results, covariance estimators or nothing, which it rebuilds one element at a time.
  • args...: Positional arguments. The leaf method ignores them, and the array method forwards them.
  • kwargs...: Keyword arguments. The leaf method ignores them, and the array method forwards them.

Returns

  • a: The input with no change, for the leaf method.
  • v::AbstractArray: The rebuilt elements, in an array of the shape of a, for the array method.

Examples

julia> factory(nothing, 1, 2; x = 3)julia> factory([1.0, 2.0], 1, 2; x = 3)2-element Vector{Float64}: 1.0 2.0julia> factory(MeanValue())MeanValue  w ┴ nothing

Related

source
factory(
    opt::Union{NonFiniteAllocationOptimisationEstimator, NonFiniteAllocationOptimisationResult},
    _
) -> EqualWeighted{_A, _B, _C, _D, _E, Bool} where {_A, _B, _C, _D, _E}

Returns opt unchanged.

This is the fallback factory of an optimisation estimator or result. An estimator whose parameters change at each step of an optimisation adds its own method. A precomputed result in a field of an estimator, for example in fb, reaches this method when factory gives the estimator the weights of the previous fold. A precomputed result does not change.

Related

source
PortfolioOptimisers.port_opt_view — Method
port_opt_view(
    st::Stacking,
    i,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    args...
) -> Stacking{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185", _K, Bool, <:AbstractPanelCollapseAlgorithm} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, var"#s185"<:Transducers.Executor, _K}

Return a view of the Stacking st sliced to the asset indices i.

When st.pe holds a prior result, the view reads the X of that prior in place of the X it receives. The struct's ## View parameters states what each field does.

Related

source
PortfolioOptimisers._optimise — Method
_optimise(
    st::Stacking,
    rd::ReturnsResult;
    branchorder,
    str_names,
    save,
    kwargs...
) -> Any

Run the stacking optimisation.

optimise calls this method. The struct Stacking states the mathematics.

Algorithm

  1. Resolve every TimeDependent field of st to its static default, with reset_time_dependent_estimator.
  2. Fit the prior pr with st.pe on rd, and take the element type Tf of the weights, fees and bounds as eltype(pr.X). The prior reads the returns in the frame that its own x_src states. rd keeps its raw returns, and the readers of the inner and outer optimisers state their own sources.
  3. Find the Investable Mask imsk of pr. Resolve the fees fees on the full universe, and place them on the assets of imsk with investable_fees_view.
  4. Remove the two liquidation charges, lq and flq, from fees, giving cfees, the fees that the returns of the synthetic assets pay.
  5. Reduce pr, st and rd to the assets of imsk with investable_reduction, giving rdr. X is the returns matrix of the reduced pr.
  6. Solve each inner optimiser of st.opti on rdr under st.ex, giving the inner results resi. Write the weights of result $k$ to column $k$ of wi.
  7. Build the returns of the synthetic assets, rdo, with predict_outer_returns under st.cv.
  8. Solve st.opto on rdo, giving the outer result reso.
  9. Resolve the weight bounds wb from st.wb and st.sets.
  10. Apply the Combination Weight st.scale to reso.w with combination_weights.
  11. Combine the coefficients with wi, and finalise the weights under wb and st.wf with outer_optimisation_finaliser, giving retcode and w.
  12. Return a StackingResult. Its keyword constructor expands w onto the full asset universe through imsk.

Arguments

  • st: The stacking optimiser.
  • rd: The returns result to use.
  • branchorder: Passed to each inner optimiser and to the outer optimiser. The branch order of a clusterisation.
  • str_names: Passed to each inner optimiser and to the outer optimiser. When true, the optimisation uses string names for the assets.
  • save: Passed to each inner optimiser and to the outer optimiser. When true, a JuMP result keeps its model.
  • kwargs: Passed to each inner optimiser and to the outer optimiser.

Validation

  • No inner optimiser returns an efficient frontier. The method throws an ArgumentError otherwise.

Returns

  • res::StackingResult: The stacked portfolio. retcode is an OptimisationFailure when an inner solve, the outer solve or the weight finaliser failed. An outer efficient frontier gives one weight vector and one return code for each point.

Related

source

References

[246]
D. H. Wolpert. Stacked generalization. Neural Networks 5, 241–259 (1992).