Investable universe: private API
PortfolioOptimisers.investable_reduction — Method
investable_reduction(pr::AbstractPriorResult, opt::AbstractOptimisationEstimator,
rd::ReturnsResult, imsk::Nothing)
investable_reduction(pr::AbstractPriorResult, opt::AbstractOptimisationEstimator,
rd::ReturnsResult, imsk::BitVector = investable_mask(pr))Derive the Investable Mask of a fitted prior, and reduce the prior, the optimiser and the returns data to the assets it keeps.
A Prior Estimator fits on the Coverage Universe and returns a result on the full asset universe, in which an asset that it could not estimate carries NaN in mu and on the diagonal of sigma. No code after the fit can solve over such an asset. So the reduction happens once, at the entry of the optimiser, and the same index slices every constraint that the caller stated over the full universe.
Every optimisation family reduces through this one verb. That is why it is bound to AbstractOptimisationEstimator and lives here rather than beside the JuMP prelude: the hierarchical, naive and meta files load before that prelude and reach this verb without a back reference. A meta-optimiser composes two masks, its own at its entry and the mask of each inner head inside the solve of that head, and both expand.
The branch is dispatch rather than a condition, over two methods. A call without the mask derives it with the default of the last slot, and dispatches to one of them. The nothing method is the all-investable path and returns its arguments unchanged. The BitVector method takes the three views. A universe with nothing to exclude therefore costs the derivation of the mask and nothing more: no view, no copy of the data and no expansion.
The optimiser is viewed with pr.X, the unreduced returns matrix, because port_opt_view slices a tracking estimator against it by the same asset index. The prior is reduced after the optimiser, so the view reads the full matrix.
The mask stays a BitVector because the expansion needs the length of the full universe, and nothing else carries that length once the prior is reduced. The three views take findall(imsk) instead, which is the integer index that every other caller of port_opt_view passes.
Algorithm
- Take the Investable Mask
imsk. By default,investable_maskderives it from the fitted prior. - Check that every
UniverseSetsin the tree of the optimiser names the columns of the unreducedrd, in order, withassert_universe_axis_order. Both methods that take a mask do it, because a caller can pass a mask that it composed itself. - Return the mask, the prior, the optimiser and the returns data unchanged when the mask is
nothing. - Otherwise read the departed names
nioff the unreducedrd.nxwithnon_investable_names, and announce them once withannounce_non_investable. - Take a
port_opt_viewof each of the three atfindall(imsk). - Declare the Non-Investable Axis on the viewed optimiser with
non_investable_universe, and return it beside the mask and the other two views. The axis is declared after the view because the view drops it, so a name-keyed constraint stated for a departed asset resolves here and nowhere deeper.
Arguments
pr: Prior result.opt::AbstractOptimisationEstimator: The optimisation estimator, holding every constraint estimator the caller stated over the full universe.rd: The returns result to use.imsk: The Investable Mask ofpr, ornothingwhen every asset is investable. A meta-optimiser passes the mask that it composed itself.
Validation
- At least one asset must be investable.
investable_maskthrows anIsEmptyErrorotherwise. - The rules of
assert_universe_axis_order.
Returns
(imsk, pr, opt, rd): The Investable Mask and the three reduced to it, ornothingand the three unchanged.
Related
PortfolioOptimisers.coverage_reduction — Method
coverage_reduction(
opt::AbstractOptimisationEstimator,
rd::ReturnsResult
) -> Union{Tuple{BitVector, Any, ReturnsResult}, Tuple{Nothing, AbstractOptimisationEstimator, ReturnsResult}}
Derive the Coverage Universe of a window, and reduce a prior-free optimiser and its returns data to the assets it keeps.
This is the twin of investable_reduction for a head that fits no prior. No Prior Result exists to derive an Investable Mask from, so the head derives the Coverage Universe of its own window through the verb that the priors use. An asset is kept when its return is finite and the active mask of the AssetPanel is true at every row of the window. A stale finite price during an inactive spell is therefore outside the mask, and the asset gets a zero weight.
Downstream, the two masks are one object. The head carries the Coverage Universe as the imsk of its result, and the keyword constructor of the result expands the weights through expand_investable_weights, as it does in every other family. So every optimisation result of the library carries a mask, and a reader meets one idiom.
The branch is dispatch rather than a condition, over three methods, as in investable_reduction. The first derives the mask. The nothing method is the all-covered path and returns its arguments unchanged. The BitVector method takes the two views. The optimiser is viewed with rd.X, the unreduced returns matrix, because port_opt_view slices an estimator against it by the same asset index.
Algorithm
- Derive the Coverage Universe
cmskofrd.Xandrd.pnlwithcoverage_mask, withdims = 1, because aReturnsResultholds its observations along the rows. - Check the universes of the optimiser against the unreduced
rdwithassert_universe_axis_order, as ininvestable_reduction. - Return the mask, the optimiser and the returns data unchanged when the mask is
nothing. - Otherwise read the departed names
nioff the unreducedrd.nxwithnon_investable_names, and announce them once withannounce_non_investable. - Take a
port_opt_viewof the optimiser and of the returns data atfindall(cmsk). - Declare the Non-Investable Axis on the viewed optimiser with
non_investable_universe, and return it beside the mask and the viewed returns data.
Arguments
opt::AbstractOptimisationEstimator: The optimisation estimator, holding every constraint estimator the caller stated over the full universe.rd: The returns result to use.
Validation
- At least one asset must be in the Coverage Universe.
coverage_maskthrows anIsEmptyErrorotherwise, so every prior-free head refuses an all-dead window in one place. - The rules of
assert_universe_axis_order.
Returns
(cmsk, opt, rd): The Coverage Universe and the two reduced to it, ornothingand the two unchanged.
Related
PortfolioOptimisers.expand_investable_weights — Function
expand_investable_weights(
_::Nothing,
w::Union{Nothing, AbstractVector{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:AbstractVector{<:Union{Number, AbstractJuMPScalar}}}}
) -> Union{Nothing, AbstractVector{<:Union{Number, AbstractJuMPScalar}}, AbstractVector{<:AbstractVector{<:Union{Number, AbstractJuMPScalar}}}}
Expand a solved weight vector from the investable subset back onto the full asset universe.
The optimiser solves over the assets that the Investable Mask keeps, so its weight vector is shorter than the universe that the caller stated. This puts each solved weight back at its own asset and writes a zero at every other asset. A zero is what a non-investable asset holds, because the optimiser could not trade it.
A failed solve carries NaN at every solved position. The expansion keeps two cases apart: NaN where the optimiser tried and failed, and zero where it could not try. It does not flatten both to zero.
The nothing mask returns the weights unchanged, so nothing is copied when every asset is investable. A nothing weight vector stays nothing, which is what a naive head records when its finaliser gave up. The vector-of-vectors method serves the efficient-frontier route, which records one weight vector per sweep point.
A matrix is a per-asset history over the investable assets, observations × investable assets: the weight path or the asset returns of a HeldWeightsResult, which expand_held_weights expands on the way out of a fold. The column of a non-investable asset is zero, which is what filter_held_gaps writes over its whole NaN column anyway. A drift over a zero column at a zero weight moves no wealth, so the expanded history rebuilds the same weight path that the reduced one did. A vector of matrices is the population of the efficient-frontier route, and each member expands as one matrix. JuMPOptimisationSolution carries its own methods beside the JuMP prelude, and they delegate to the plain weight vector here, so one length check and one message serve every family.
The keyword constructor of each optimisation result is the caller, and every family builds its result through it. The positional constructor never expands, because every rebuild of a return code goes through it, and a second pass would expand twice.
Mathematical definition
\[\begin{align} \tilde{w}_{i} &= \begin{cases} w_{k(i)} & \text{if } m_{i} = 1\,, \\ 0 & \text{if } m_{i} = 0\,, \end{cases} \quad i = 1, \ldots, N\,, \\ k(i) &= \sum_{j=1}^{i} m_{j}\,. \end{align}\]
Where:
- $\tilde{w}_{i}$: Weight of asset $i$ on the full asset universe.
- $w_{k}$: Solved weight at position $k$ of the investable assets, in their order in the full universe.
- $m_{i}$: Entry $i$ of the Investable Mask, $1$ when asset $i$ is investable.
- $k(i)$: Position of asset $i$ among the investable assets.
- $N$: Number of assets.
The expansion keeps the sum of the weights, $\sum_{i=1}^{N} \tilde{w}_{i} = \sum_{k} w_{k}$, so a budget stated over the investable assets holds over the full universe.
Arguments
imsk: The Investable Mask the optimisation reduced on:trueat every asset whose prior moments were finite. It isnothingwhen every asset was investable, and that sentinel is what skips both the reduction and the expansion.investable_maskderives it once from the full-universe prior result, and the result carries it, because the reduced prior can no longer yield it.w: The weights the optimisation solved over the investable universe: one weight vector, or a vector of them on the efficient-frontier route. Or a history over the investable assets,observations × investable assets, or a vector of them.
Validation
- The solved weight vector must hold one weight per investable asset, and a history one column per investable asset.
Returns
w: The weights, the history, or the vector of them, on the full asset universe.
Related
PortfolioOptimisers.assert_internal_optimiser — Method
assert_internal_optimiser(
_::NonFiniteAllocationOptimisationResult
)
Accept a precomputed optimisation result where an optimiser must run on assets that it did not choose.
A meta-optimiser calls assert_internal_optimiser on an inner optimiser, and a cross-validation entry calls it on the optimiser of each fold. The methods for estimators refuse a precomputed input that a subset of the assets cannot reuse. A result holds no estimator to refit, so it passes. An asset-subset view of it still throws, in port_opt_view.
Returns
nothing.
Related
PortfolioOptimisers.assert_external_optimiser — Method
assert_external_optimiser(
_::NonFiniteAllocationOptimisationResult
)
Accept a precomputed optimisation result where an optimiser must run on a universe that it does not know in advance.
A meta-optimiser calls assert_external_optimiser on its outer optimiser, which solves over synthetic assets, and a cross-validation entry calls it on the optimiser of each fold. The methods for estimators refuse a precomputed prior or constraint that such a universe cannot reuse. A result holds no estimator to refit, so it passes.
Returns
nothing.
Related
PortfolioOptimisers.non_investable_universe — Function
non_investable_universe(opt, ni::VecStr)Declare the Non-Investable Axis on the UniverseSets that an optimisation estimator carries.
investable_reduction and coverage_reduction call this after they have viewed the estimator, and no other code calls it. An axis declared before the view would not survive it: port_opt_view(::UniverseSets, i) drops the axis, so that a cluster of a nested optimisation does not inherit the departures of its parent and charge each of them again. So each of the two reads the departed names off the unreduced returns data, takes the view, and declares the axis on the viewed estimator.
The generic method returns opt unchanged. That is correct for every estimator that carries no sets, because it has no axis to declare. A head that owns a sets field writes one method beside its own port_opt_view, and a head that reaches one through a nested optimiser forwards to the method of that optimiser. Each method is one line, and it is written per type rather than derived by reflection, for the same reason as port_opt_view: the type states what its field means.
Arguments
opt: The optimisation estimator, already reduced to the Investable Mask.ni: The names the mask left out, fromnon_investable_names.
Returns
opt: The estimator, carrying the Non-Investable Axis where it carries aUniverseSets.
Related
PortfolioOptimisers.distance_reader_entry_mask — Function
distance_reader_entry_mask(x, pr::AbstractPriorResult, rd::ReturnsResult, strict::Bool;
rmsk::Option{BitVector} = nothing) -> Option{BitVector}Compose the Investable Mask of a fit that reads distances, at its entry.
Every fit that clusters or builds a phylogeny composes the mask of its prior with two rules before investable_reduction: the assets that a FeatureDistance can read, feature_readable_mask, and the assets that a plain graph reader can read, graph_reader_mask. A distance reader can still refuse an asset after the reduction. The fit then fits again from the same prior result, and passes the mask of distance_reader_refusal_mask as rmsk. That mask already holds the two rules, so this function returns it as it is, and the two rules do not warn a second time.
Algorithm
- Return
rmskwhen it is aBitVector. - Otherwise compose
investable_mask(pr)withfeature_readable_mask, then withgraph_reader_mask.
Arguments
x: The distance readers of the fit, one estimator or aTupleof them.pr: Prior result.rd: The returns result to use.strict: Thestrictfield of the optimiser.rmsk: The mask of a fit again after a refusal, ornothingon the first fit.
Returns
imsk::Option{BitVector}: The composed mask, ornothingwhen every asset stays.
Related
PortfolioOptimisers.shared_series_solve — Function
shared_series_solve(opt, rd::ReturnsResult, args...; kwargs...) -> OptimisationResult
shared_series_solve(opt, args...; kwargs...) -> OptimisationResultSolve opt once on the observations that it keeps after a Shared Series gap, and record the cut on the prior that the solve fits.
Every reader inside a solve pairs the observations of the Prior Result with the observations of the returns data: the outer problem of a meta-optimiser, the implied volatilities of an ImpliedVolatility, the probabilities of a scenario. So the drop happens once, before the solve, and every reader sees the same observations. Each attempt of the fallback chain drops by its own rule, because a fallback is a head with its own group and prior.
The prior that the solve fits then sees no gap, and it counts its rows among the kept observations. So the function maps that prior back to the observations of rd with shared_series_record: its marks ss_gaps, and the row key of its factor model block. The prior of the result then states the same rows as prior(pe, rd) states for the same prior estimator. A prior that is already a result keeps its own record, see fits_prior.
Algorithm
The method that Julia selects is the algorithm.
- A
ReturnsResultfirst: find the kept observations withoptimiser_shared_series_rows, which reads the group ofoptand its prior, seegroup_source. Solveoptwith_optimiseon the cut ofoptimiser_shared_series_view. When the cut drops an observation, the solve readsoptcut at the same observations withshared_series_estimator, so each fixed observation weight in its tree loses them too. When the cut dropped an observation and the solve fitted its prior, replace the prior of the result with its record, throughset_result_prior. - Any other data: solve
optwith_optimiseon it unchanged.
Arguments
opt: The estimator of the attempt, or a precomputed result.args: Additional positional arguments passed to the optimisation function.kwargs...: The keyword arguments of_optimise, passed on unchanged.
Returns
res::OptimisationResult: The result of the solve.
Related