Investable universe: private API

investable_reduction(
    pr::AbstractPriorResult,
    opt::AbstractOptimisationEstimator,
    rd::ReturnsResult
) -> Union{Tuple{BitVector, Union{HighOrderPrior{<:AbstractPriorResult}, LowOrderPrior{var"#s185", _A, var"#s1851", <:AbstractMatrix{var"#s137"}} where {var"#s137"<:(Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}), var"#s185"<:AbstractMatrix{var"#s137"}, _A, var"#s137"<:(Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}), var"#s1851"<:AbstractVector{var"#s137"}, var"#s137"<:(Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar})}}, Any, Union{ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, Nothing} where {_A, _B, _C, _D, _E, _F, _G, _H, _I}, ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, AssetPanel{__T_pf, __T_amsk, __T_emsk}} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, __T_pf, __T_amsk, __T_emsk}}}, Tuple{Nothing, AbstractPriorResult, AbstractOptimisationEstimator, ReturnsResult}}

Derive the Investable Mask of a fitted prior, and reduce the prior, the optimiser and the returns data to the assets it keeps.

A Prior Estimator fits on the Coverage Universe and returns a result on the full asset universe, in which an asset that it could not estimate carries NaN in mu and on the diagonal of sigma. No code after the fit can solve over such an asset. So the reduction happens once, at the entry of the optimiser, and the same index slices every constraint that the caller stated over the full universe.

Every optimisation family reduces through this one verb. That is why it is bound to AbstractOptimisationEstimator and lives here rather than beside the JuMP prelude: the hierarchical, naive and meta files load before that prelude and reach this verb without a back reference. A meta-optimiser composes two masks, its own at its entry and the mask of each inner head inside the solve of that head, and both expand.

The branch is dispatch rather than a condition, over three methods. The first derives the mask. The nothing method is the all-investable path and returns its arguments unchanged. The BitVector method takes the three views. A universe with nothing to exclude therefore costs the derivation of the mask and nothing more: no view, no copy of the data and no expansion.

The optimiser is viewed with pr.X, the unreduced returns matrix, because port_opt_view slices a tracking estimator against it by the same asset index. The prior is reduced after the optimiser, so the view reads the full matrix.

The mask stays a BitVector because the expansion needs the length of the full universe, and nothing else carries that length once the prior is reduced. The three views take findall(imsk) instead, which is the integer index that every other caller of port_opt_view passes.

Algorithm

1. Derive the Investable Mask from the fitted prior with investable_mask.
2. Return the mask, the prior, the optimiser and the returns data unchanged when the mask is nothing.
3. Otherwise read the departed names ni off the unreduced rd.nx with non_investable_names, and announce them once with announce_non_investable.
4. Take a port_opt_view of each of the three at findall(imsk).
5. Declare the Non-Investable Axis on the viewed optimiser with non_investable_universe, and return it beside the mask and the other two views. The axis is declared after the view because the view drops it, so a name-keyed constraint stated for a departed asset resolves here and nowhere deeper.

Arguments

• pr: Prior result.
• opt::AbstractOptimisationEstimator: The optimisation estimator, holding every constraint estimator the caller stated over the full universe.
• rd: The returns result to use.

Validation

• At least one asset must be investable. investable_mask throws an IsEmptyError otherwise.

Returns

• (imsk, pr, opt, rd): The Investable Mask and the three reduced to it, or nothing and the three unchanged.

Related

• investable_mask
• expand_investable_weights
• port_opt_view
• non_investable_names
• non_investable_universe
• announce_non_investable

coverage_reduction(
    opt::AbstractOptimisationEstimator,
    rd::ReturnsResult
) -> Union{Tuple{BitVector, Any, Union{ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, Nothing} where {_A, _B, _C, _D, _E, _F, _G, _H, _I}, ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, AssetPanel{__T_pf, __T_amsk, __T_emsk}} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, __T_pf, __T_amsk, __T_emsk}}}, Tuple{Nothing, AbstractOptimisationEstimator, ReturnsResult}}

Derive the Coverage Universe of a window, and reduce a prior-free optimiser and its returns data to the assets it keeps.

This is the twin of investable_reduction for a head that fits no prior. No Prior Result exists to derive an Investable Mask from, so the head derives the Coverage Universe of its own window through the verb that the priors use. An asset is kept when its return is finite and the active mask of the AssetPanel is true at every row of the window. A stale finite price during an inactive spell is therefore outside the mask, and the asset gets a zero weight.

Downstream, the two masks are one object. The head carries the Coverage Universe as the imsk of its result, and the keyword constructor of the result expands the weights through expand_investable_weights, as it does in every other family. So every optimisation result of the library carries a mask, and a reader meets one idiom.

The branch is dispatch rather than a condition, over three methods, as in investable_reduction. The first derives the mask. The nothing method is the all-covered path and returns its arguments unchanged. The BitVector method takes the two views. The optimiser is viewed with rd.X, the unreduced returns matrix, because port_opt_view slices an estimator against it by the same asset index.

Algorithm

1. Derive the Coverage Universe cmsk of rd.X and rd.pnl with coverage_mask, with dims = 1, because a ReturnsResult holds its observations along the rows.
2. Return the mask, the optimiser and the returns data unchanged when the mask is nothing.
3. Otherwise read the departed names ni off the unreduced rd.nx with non_investable_names, and announce them once with announce_non_investable.
4. Take a port_opt_view of the optimiser and of the returns data at findall(cmsk).
5. Declare the Non-Investable Axis on the viewed optimiser with non_investable_universe, and return it beside the mask and the viewed returns data.

Arguments

• opt::AbstractOptimisationEstimator: The optimisation estimator, holding every constraint estimator the caller stated over the full universe.
• rd: The returns result to use.

Validation

• At least one asset must be in the Coverage Universe. coverage_mask throws an IsEmptyError otherwise, so every prior-free head refuses an all-dead window in one place.

Returns

• (cmsk, opt, rd): The Coverage Universe and the two reduced to it, or nothing and the two unchanged.

Related

• coverage_mask
• investable_reduction
• expand_investable_weights
• port_opt_view

expand_investable_weights(
    _::Nothing,
    w::Union{Nothing, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}}}
) -> Union{Nothing, AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}, AbstractVector{<:AbstractVector{<:Union{var"#s136", var"#s53"} where {var"#s136"<:Number, var"#s53"<:AbstractJuMPScalar}}}}

Expand a solved weight vector from the investable subset back onto the full asset universe.

The optimiser solves over the assets that the Investable Mask keeps, so its weight vector is shorter than the universe that the caller stated. This puts each solved weight back at its own asset and writes a zero at every other asset. A zero is what a non-investable asset holds, because the optimiser could not trade it.

A failed solve carries NaN at every solved position. The expansion keeps two cases apart: NaN where the optimiser tried and failed, and zero where it could not try. It does not flatten both to zero.

The nothing mask returns the weights unchanged, so nothing is copied when every asset is investable. A nothing weight vector stays nothing, which is what a naive head records when its finaliser gave up. The vector-of-vectors method serves the efficient-frontier route, which records one weight vector per sweep point. JuMPOptimisationSolution carries its own methods beside the JuMP prelude, and they delegate to the plain weight vector here, so one length check and one message serve every family.

The keyword constructor of each optimisation result is the caller, and every family builds its result through it. The positional constructor never expands, because every rebuild of a return code goes through it, and a second pass would expand twice.

Mathematical definition

\[\begin{align} \tilde{w}_{i} &= \begin{cases} w_{k(i)} & \text{if } m_{i} = 1\,, \\ 0 & \text{if } m_{i} = 0\,, \end{cases} \quad i = 1, \ldots, N\,, \\ k(i) &= \sum_{j=1}^{i} m_{j}\,. \end{align}\]

Where:

• $\tilde{w}_{i}$: Weight of asset $i$ on the full asset universe.
• $w_{k}$: Solved weight at position $k$ of the investable assets, in their order in the full universe.
• $m_{i}$: Entry $i$ of the Investable Mask, $1$ when asset $i$ is investable.
• $k(i)$: Position of asset $i$ among the investable assets.
• $N$: Number of assets.

The expansion keeps the sum of the weights, $\sum_{i=1}^{N} \tilde{w}_{i} = \sum_{k} w_{k}$, so a budget stated over the investable assets holds over the full universe.

Arguments

• imsk: The Investable Mask the optimisation reduced on: true at every asset whose prior moments were finite. It is nothing when every asset was investable, and that sentinel is what skips both the reduction and the expansion. investable_mask derives it once from the full-universe prior result, and the result carries it, because the reduced prior can no longer yield it.
• w: The weights the optimisation solved over the investable universe: one weight vector, or a vector of them on the efficient-frontier route.

Validation

• The solved weight vector must hold one weight per investable asset.

Returns

• w: The weights, or the vector of them, on the full asset universe.

Related

• investable_mask
• investable_reduction
• set_retcode

assert_internal_optimiser(
    _::NonFiniteAllocationOptimisationResult
)

Accept a precomputed optimisation result where an optimiser must run on assets that it did not choose.

A meta-optimiser calls assert_internal_optimiser on an inner optimiser, and a cross-validation entry calls it on the optimiser of each fold. The methods for estimators refuse a precomputed input that a subset of the assets cannot reuse. A result holds no estimator to refit, so it passes. An asset-subset view of it still throws, in port_opt_view.

Returns

• nothing.

Related

• assert_external_optimiser
• port_opt_view
• NestedClustered
• NonFiniteAllocationOptimisationResult

assert_external_optimiser(
    _::NonFiniteAllocationOptimisationResult
)

Accept a precomputed optimisation result where an optimiser must run on a universe that it does not know in advance.

A meta-optimiser calls assert_external_optimiser on its outer optimiser, which solves over synthetic assets, and a cross-validation entry calls it on the optimiser of each fold. The methods for estimators refuse a precomputed prior or constraint that such a universe cannot reuse. A result holds no estimator to refit, so it passes.

Returns

• nothing.

Related

• assert_internal_optimiser
• NestedClustered
• NonFiniteAllocationOptimisationResult

non_investable_universe(opt, ni::VecStr)

Declare the Non-Investable Axis on the UniverseSets that an optimisation estimator carries.

A door calls this after it has viewed the estimator, and no other code calls it. An axis declared before the view would not survive it: port_opt_view(::UniverseSets, i) drops the axis, so that a cluster of a nested optimisation does not inherit the departures of its parent and charge each of them again. So the door reads the departed names off the unreduced returns data, takes the view, and declares the axis on the viewed estimator.

The generic method returns opt unchanged. That is correct for every estimator that carries no sets, because it has no axis to declare. A head that owns a sets field writes one method beside its own port_opt_view, and a head that reaches one through a nested optimiser forwards to the method of that optimiser. Each method is one line, and it is written per type rather than derived by reflection, for the same reason as port_opt_view: the type states what its field means.

Arguments

• opt: The optimisation estimator, already reduced to the Investable Mask.
• ni: The names the mask left out, from non_investable_names.

Returns

• opt: The estimator, carrying the Non-Investable Axis where it carries a UniverseSets.

Related

• non_investable_sets
• non_investable_names
• investable_reduction
• coverage_reduction
• port_opt_view