Empirical Prior
PortfolioOptimisers.EmpiricalPrior — Type
struct EmpiricalPrior{__T_ce, __T_me, __T_horizon, __T_fill_limit, __T_max_scenarios, __T_cache, __T_x_src} <: AbstractLowOrderPriorEstimator_AEmpirical prior estimator for asset returns.
EmpiricalPrior is a low order prior estimator. It fits the mean of the asset returns with me and their covariance with ce. With a horizon, it fits both moments on the log-returns, scales them to the horizon, and converts them back to arithmetic returns.
Fields
ce: Covariance estimator.
me: Expected returns estimator.
horizon: Optional investment horizon. The prior fits the log-returns, scales their moments to the horizon, and converts them back to arithmetic moments. Ifnothing, the prior fits the arithmetic returns as they are.
fill_limit: The share of the observations of an investable column thatscenario_fillcan fill with no warning. The fill tests the worst column against it. Ifnothing,resolve_fill_limitderives it at the fit as1 - min_coverageof the binding floor of the arms, which never fires. Where no arm states a floor, no share passes with no warning, and the fit names every fill.
max_scenarios: Optional cap on the number of observations that the Prior Result carries as scenarios.nothingcarries every observation that the fit read. A cap cutsXto its lastmax_scenariosrows, andmuandsigmastay fitted over every observation. The cap limits the memory that a scenario-based measure reads, and it changes no moment. The cap is a property of the result, so it acts the same in batch and online. When the cap cuts, the result writes intoensthe number of observations that its moments read. A consumer that prices a sample size, such as an uncertainty set or a calibration rule, reads that count and not the number of carried rows. A calibration rule that prices the scenarios of the measure reads the carried rows: ACountOfin a significance slot fixes the count of scenarios in the tail, andEntropyBudgetfixes the coefficient that the model pays over its rows. A bootstrap can resample only the carried rows, so it reads the cap. An entropy pooling prior around the capped prior states probabilities for the scenarios alone. Its posterior moments are those of the scenarios, with a zero weight on every observation that the cap removed, seescenario_weights.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
x_src: The source of the returns that the prior fits on,nothingor anAbstractReturnsSource.nothingandDataSourceread the asset returnsrd.X.ExcessSourcereads the returns in excess of the benchmark,rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark.nothingmeans that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior atnothing, and not about a prior atDataSource(). A prior reads no prior, so it refusesPriorSource. A prior inside another estimator, such as thepeof aFactorPrior, keepsnothing, and the outer estimator states the source.
Constructors
EmpiricalPrior(; ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(), me::AbstractExpectedReturnsEstimator = SimpleExpectedReturns(), horizon::Option{<:Number} = nothing, fill_limit::Option{<:Real} = nothing, max_scenarios::Option{<:Integer} = nothing, cache::Option{<:AbstractPartialFitState} = nothing, x_src::Option{<:AbstractReturnsSource} = nothing) -> EmpiricalPriorKeywords correspond to the struct's fields.
Validation
ceis not anAbstractVarianceEstimator. A variance estimator has nocovorcor, soassert_covariance_estimatorraises anArgumentErrorthat namesce.- If
horizonis notnothing,horizon > 0. - If
fill_limitis notnothing,0 < fill_limit <= 1. The constructor refuses zero. Where no arm states a coverage floor,nothingalready names every fill. The fit refuses a value above1 - min_coverageof the binding floor of the arms, because such a value can never fire. - If
max_scenariosis notnothing,max_scenarios > 0. x_srcisnothing,DataSource()orExcessSource(), seeassert_returns_source.
The incremental fit
The empirical prior folds its moments and carries its rows. partial_fit! folds each observation into each member of me and ce that folds, and appends the row to a PriorCarryState of its own. The one-argument prior reads mu and sigma off the state of a folded member, and X off the carried rows. A member that does not fold runs its batch verb over the carried rows, see read_member. When both members fold, the cost of a step grows with the square of the number of assets, and not with the number of observations folded. The prior carries the rows because LowOrderPrior holds X for the scenario risk measures.
Two caps, two names
max_scenarios and the max_history of Online are two different caps.
EmpiricalPrior(; max_scenarios = w)fitsmuandsigmaover every observation, and carries the lastwrows asX. The cap is a property of the result, so it cuts the same rows in batch and online. No batch fit gives the same result, because the moments and the scenarios come from different windows.Online(EmpiricalPrior(); max_history = w)cuts the whole fit to the lastwobservations. The estimator refits from the buffer, so the result equalsprior(pe, X[end - w + 1:end, :]).
When both caps are set, the moments come from the last max_history observations, and X comes from the last max_scenarios of those. A max_scenarios at or above the number of carried observations changes nothing.
Propagated parameters
factory passes its arguments on to these @fprop fields:
ce: Recursively updated throughfactory.me: Recursively updated throughfactory.cache: Carried unchanged throughfactory.
View parameters
port_opt_view takes the view of these @vprop fields at the selected indices:
ce: Recursively viewed throughport_opt_view.me: Recursively viewed throughport_opt_view.cache: Sliced to the selected assets throughport_opt_view.
Examples
julia> EmpiricalPrior()EmpiricalPrior ce ┼ PortfolioOptimisersCovariance │ ce ┼ Covariance │ │ me ┼ SimpleExpectedReturns │ │ │ w ┴ nothing │ │ ce ┼ GeneralCovariance │ │ │ ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true) │ │ │ w ┴ nothing │ │ alg ┼ FullMoment() │ │ w ┴ nothing │ mp ┼ MatrixProcessing │ │ pdm ┼ Posdef │ │ │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ │ │ kwargs ┴ @NamedTuple{}: NamedTuple() │ │ dn ┼ nothing │ │ dt ┼ nothing │ │ alg ┼ nothing │ │ order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg) me ┼ SimpleExpectedReturns │ w ┴ nothing horizon ┼ nothing fill_limit ┼ nothing x_src ┴ nothingRelated
AbstractLowOrderPriorEstimator_AStatsBase.CovarianceEstimatorAbstractExpectedReturnsEstimatorSimpleExpectedReturnsPortfolioOptimisersCovariancepriorfactoryport_opt_viewpartial_fit!PriorCarryStateOnline
References
- [52] A. Meucci. Risk and Asset Allocation (Springer Berlin Heidelberg, 2005). Section 2.6.5, Equations 2.219 and 2.220, Section 3.2, Equation 3.64, and Section 3.3.1, Equation 3.80.
PortfolioOptimisers.prior — Function
prior(pe::EmpiricalPrior{<:Any, <:Any, Nothing, <:Any}, X::MatNum,
F::Option{<:MatNum} = nothing, pnl::Option{<:AssetPanel} = nothing;
dims::Int = 1, strict::Bool = false, kwargs...)Compute the empirical prior moments of asset returns with no horizon conversion.
pe.me fits the mean and pe.ce fits the covariance on X itself, with no log transform and no horizon scale. The method returns them in a LowOrderPrior.
Mathematical definition
The two moments are what pe.me and pe.ce return. The default pair, SimpleExpectedReturns and PortfolioOptimisersCovariance with no observation weights, gives the sample moments:
\[\begin{align} \hat{\boldsymbol{\mu}} &= \frac{1}{T} \sum_{t=1}^{T} \boldsymbol{x}_t\,, \\ \hat{\mathbf{\Sigma}} &= \frac{1}{T-1} \sum_{t=1}^{T} (\boldsymbol{x}_t - \hat{\boldsymbol{\mu}})(\boldsymbol{x}_t - \hat{\boldsymbol{\mu}})^\intercal\,. \end{align}\]
Where:
- $\hat{\boldsymbol{\mu}}$: $N \times 1$ mean vector.
- $\hat{\mathbf{\Sigma}}$: $N \times N$ covariance matrix.
- $\boldsymbol{x}_t$: $N \times 1$ vector of asset returns at time $t$.
- $T$: Number of observations.
Any other configuration of pe.me or pe.ce changes the result. For example, a shrunk mean or a denoised covariance differs from the display above.
This method takes the arithmetic moments of X. It applies no log transform, so it is not the $h = 1$ case of the horizon method. The horizon method converts the returns to $\log(1 + x_t)$ and back, and that round trip leaves the moments unchanged only in the limit of small returns.
The scenario cap
pe.max_scenarios cuts the returns matrix of the result to its last max_scenarios rows, through scenario_window. mu and sigma stay fitted over every observation. The cut runs before the fill, so the fill measures its share over the rows that a consumer reads. The default, nothing, carries every row. A cap at or above the number of observations returns X itself, and a smaller cap returns a view, so the cut copies nothing.
The counts of the moments
The result states in eoc the effective number of observations of mu and of sigma, through scenario_moment_counts. pe.me and pe.ce each state the count of their own moment with effective_observation_count. A PriorExpectedReturns in pe.me states the count that its prior states for the mean, and mean_with_count reads it from the one fit of that prior. An unweighted estimator is worth size(X, 1) observations, and a weighted one, such as an exponentially weighted covariance, is worth the count of its weights. When the cap cuts, the count of an unweighted moment is the number of observations that the moments read. A consumer that prices a sample size reads these counts and not the number of carried rows. An entry equal to the number of carried rows stays nothing.
The scenario fill
A mask-aware pe.me and pe.ce fit an asset that lists inside the window from the observations it has. The exponentially weighted family is mask-aware, and so is the plain family wherever a CoveragePolicy is set. Such an asset is investable, and its column of X still carries a NaN at every earlier row. scenario_fill writes 0 at those entries once, so that every consumer of the result reads a finite investable column. A non-investable asset keeps its whole NaN column, and the fill does not change mu or sigma.
The fill costs accuracy at four consumers, and the message names all four:
- A scenario-based risk measure understates the risk of the asset over the filled rows.
- A hierarchical optimiser can put the asset in a branch of its own and overweight it.
- An entropy pooling view on the asset is calibrated on the filled column.
- A meta-optimiser carries the fill into the outer problem.
The fill is silent while the filled share of the worst investable column stays at or below the limit that resolve_fill_limit derives. Above the limit it warns, and under strict it refuses any fill. pe.fill_limit defaults to nothing. Where an arm states a coverage floor, nothing derives 1 - min_coverage, which never fires. Where no arm states a floor, nothing names every fill. Under a plain estimator with no policy the fill never fires, because an asset that the estimator cannot cover leaves the Coverage Universe and is not investable.
Arguments
pe: Empirical prior estimator.X: Asset returns matrix (observations × assets).F: Factor returns matrix (ignored).pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.strict: Whether a zero-filled scenario raises an error instead of a warning. Understrict, any fill raises. Otherwise a column filled above the limit thatresolve_fill_limitderives warns, and every fill warns while that limit isnothing.strictalso reaches the covariance, where a peel of aCoveragePolicyraises understrictand warns otherwise.kwargs...: Additional keyword arguments, passed to the mean estimator and the covariance estimator.
Validation
dims in (1, 2).- At least one asset must be in the Coverage Universe.
pe.fill_limitis at most1 - min_coverageof the binding floor of the arms, seecoverage_floor. Otherwise the fit raises aDomainError, because a looser limit can never fire.- The zero-filled share of the worst investable column of
Xis at or below the limit thatresolve_fill_limitderives. Otherwise the fit warns and names the assets. Understrict, any fill raises anArgumentErrorthat names them. The limit isnothingwhen no arm states a floor andpe.fill_limitisnothing, and no share passes a limit ofnothing, so the fit names every fill.
Returns
pr::LowOrderPrior: The prior result, which holds the asset returns, the mean vector and the covariance matrix.
Related
PortfolioOptimisers.prior — Function
prior(pe::EmpiricalPrior{<:Any, <:Any, <:Number, <:Any}, X::MatNum,
F::Option{<:MatNum} = nothing, pnl::Option{<:AssetPanel} = nothing;
dims::Int = 1, strict::Bool = false, kwargs...)Compute the empirical prior moments of asset returns at an investment horizon.
pe.me and pe.ce fit the mean and the covariance of the log-returns. horizon_moments! scales both moments to the horizon and converts them back to arithmetic returns. The method returns them in a LowOrderPrior.
Mathematical definition
pe.me and pe.ce fit the log-returns $\log(1 + x_t)$, not X. The method scales the two log-moments by the investment horizon $h$, and then converts them back to arithmetic returns:
\[\begin{align} \tilde{\boldsymbol{\mu}} &= h \cdot \hat{\boldsymbol{\mu}}_{\log}\,, \\ \tilde{\mathbf{\Sigma}} &= h \cdot \hat{\mathbf{\Sigma}}_{\log}\,. \end{align}\]
\[\begin{align} \hat{\mu}_i &= \exp\!\left(\tilde{\mu}_i + \tfrac{1}{2}\tilde{\sigma}_{ii}\right) - 1\,, \\ \hat{\sigma}_{ij} &= (\hat{\mu}_i + 1)(\hat{\mu}_j + 1)\left(\exp(\tilde{\sigma}_{ij}) - 1\right)\,. \end{align}\]
Where:
- $\tilde{\boldsymbol{\mu}}$, $\tilde{\mathbf{\Sigma}}$: Horizon-scaled log-return mean and covariance.
- $h$: Investment horizon.
- $\hat{\boldsymbol{\mu}}_{\log}$, $\hat{\mathbf{\Sigma}}_{\log}$: Mean and covariance of the log-returns $\log(1 + x_t)$, computed by
pe.meandpe.ce. - $\hat{\mu}_i$: Arithmetic mean return for asset $i$.
- $\hat{\sigma}_{ij}$: Arithmetic covariance between assets $i$ and $j$.
The X of the returned LowOrderPrior is the arithmetic returns matrix that the caller gave. Only the moments come from the log-returns.
Algorithm
Steps 5 to 7 must run in this order. Step 6 reads the mu that step 5 left, which is $\hat{\mu}_i + 1$, because step 7 has not yet subtracted the one. The second closed form needs that factor, so the order supplies it with no second computation. If step 7 runs before step 6, each $\hat{\mu}_i + 1$ becomes $\hat{\mu}_i$. Entry $(i,\, j)$ of the covariance then falls by the factor $\hat{\mu}_i \hat{\mu}_j / ((\hat{\mu}_i + 1)(\hat{\mu}_j + 1))$, which is about six orders of magnitude on a daily return series. Steps 4 to 7 are horizon_moments!.
- Orient
Xtoobservations × assetswithdims_oriented. - Take the log-returns
X_log, aslog1p.(X). - Fit the log-return mean
muonX_logwithpe.me, and the log-return covariancesigmaonX_logwithpe.ce. - Scale both by
pe.horizonto get $\tilde{\boldsymbol{\mu}}$ and $\tilde{\mathbf{\Sigma}}$. - Overwrite
muwith the exponential term of the first closed form. This is the arithmetic mean plus one, because step 7 has not run yet. - Overwrite
sigmawith the second closed form, whose $\hat{\mu}_i + 1$ factors are themuof step 5. - Subtract one from
muto get $\hat{\boldsymbol{\mu}}$. - Cut the arithmetic
Xof step 1 withscenario_window, and fill it withscenario_fill. Return aLowOrderPriorthat holds thisX,mu,sigmaand theeocofscenario_moment_countsoverX_log.
The scenario cap
pe.max_scenarios cuts the returns matrix of the result to its last max_scenarios rows, through scenario_window. mu and sigma stay fitted over every observation. The cut runs before the fill, so the fill measures its share over the rows that a consumer reads. The default, nothing, carries every row. A cap at or above the number of observations returns X itself, and a smaller cap returns a view, so the cut copies nothing.
The counts of the moments
The result states in eoc the effective number of observations of mu and of sigma, through scenario_moment_counts. pe.me and pe.ce each state the count of their own moment with effective_observation_count. A PriorExpectedReturns in pe.me states the count that its prior states for the mean, and mean_with_count reads it from the one fit of that prior. An unweighted estimator is worth size(X, 1) observations, and a weighted one, such as an exponentially weighted covariance, is worth the count of its weights. When the cap cuts, the count of an unweighted moment is the number of observations that the moments read. A consumer that prices a sample size reads these counts and not the number of carried rows. An entry equal to the number of carried rows stays nothing.
The scenario fill
Step 8 applies the fill of the method with no horizon. It fills the arithmetic X that the caller gave, and it reads the Investable Mask off the arithmetic moments that the result carries. Under a mask-aware pe.me and pe.ce, an asset that lists inside the window is investable, and its column still carries a NaN at every earlier row. scenario_fill writes 0 at those entries once. The fill is silent while the filled share of the worst investable column stays at or below the limit that resolve_fill_limit derives. Above the limit it warns, and under strict it refuses any fill.
pe.fill_limit defaults to nothing. Where an arm states a coverage floor, nothing derives 1 - min_coverage. Where no arm states a floor, nothing names every fill. A scenario-based risk measure understates the risk of a filled asset over the filled rows. The fill does not change mu or sigma.
Arguments
pe: Empirical prior estimator.X: Asset returns matrix (observations × assets).F: Factor returns matrix (ignored).pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.strict: Whether a zero-filled scenario raises an error instead of a warning. Understrict, any fill raises. Otherwise a column filled above the limit thatresolve_fill_limitderives warns, and every fill warns while that limit isnothing.strictalso reaches the covariance, where a peel of aCoveragePolicyraises understrictand warns otherwise.kwargs...: Additional keyword arguments, passed to the mean estimator and the covariance estimator.
Validation
dims in (1, 2).- At least one asset must be in the Coverage Universe.
pe.fill_limitis at most1 - min_coverageof the binding floor of the arms, seecoverage_floor. Otherwise the fit raises aDomainError, because a looser limit can never fire.- The zero-filled share of the worst investable column of
Xis at or below the limit thatresolve_fill_limitderives. Otherwise the fit warns and names the assets. Understrict, any fill raises anArgumentErrorthat names them. The limit isnothingwhen no arm states a floor andpe.fill_limitisnothing, and no share passes a limit ofnothing, so the fit names every fill.
Returns
pr::LowOrderPrior: The prior result, which holds the asset returns, the mean vector and the covariance matrix.
Related
References
- [52]
- A. Meucci. Risk and Asset Allocation (Springer Berlin Heidelberg, 2005).