Empirical Prior

PortfolioOptimisers.EmpiricalPrior — Type
struct EmpiricalPrior{__T_ce, __T_me, __T_horizon, __T_fill_limit, __T_max_scenarios, __T_cache, __T_x_src} <: AbstractLowOrderPriorEstimator_A

Empirical prior estimator for asset returns.

EmpiricalPrior is a low order prior estimator. It fits the mean of the asset returns with me and their covariance with ce. With a horizon, it fits both moments on the log-returns, scales them to the horizon, and converts them back to arithmetic returns.

Fields

  • ce: Covariance estimator.
  • me: Expected returns estimator.
  • horizon: Optional investment horizon. The prior fits the log-returns, scales their moments to the horizon, and converts them back to arithmetic moments. If nothing, the prior fits the arithmetic returns as they are.
  • fill_limit: The share of the observations of an investable column that scenario_fill can fill with no warning. The fill tests the worst column against it. If nothing, resolve_fill_limit derives it at the fit as 1 - min_coverage of the binding floor of the arms, which never fires. Where no arm states a floor, no share passes with no warning, and the fit names every fill.
  • max_scenarios: Optional cap on the number of observations that the Prior Result carries as scenarios. nothing carries every observation that the fit read. A cap cuts X to its last max_scenarios rows, and mu and sigma stay fitted over every observation. The cap limits the memory that a scenario-based measure reads, and it changes no moment. The cap is a property of the result, so it acts the same in batch and online. When the cap cuts, the result writes into ens the number of observations that its moments read. A consumer that prices a sample size, such as an uncertainty set or a calibration rule, reads that count and not the number of carried rows. A calibration rule that prices the scenarios of the measure reads the carried rows: A CountOf in a significance slot fixes the count of scenarios in the tail, and EntropyBudget fixes the coefficient that the model pays over its rows. A bootstrap can resample only the carried rows, so it reads the cap. An entropy pooling prior around the capped prior states probabilities for the scenarios alone. Its posterior moments are those of the scenarios, with a zero weight on every observation that the cap removed, see scenario_weights.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
  • x_src: The source of the returns that the prior fits on, nothing or an AbstractReturnsSource. nothing and DataSource read the asset returns rd.X. ExcessSource reads the returns in excess of the benchmark, rd.X - rd.B, the Excess Frame, and refuses returns data with no benchmark. nothing means that the caller stated no source, so an optimiser that holds a reader of the Excess Frame warns about a prior at nothing, and not about a prior at DataSource(). A prior reads no prior, so it refuses PriorSource. A prior inside another estimator, such as the pe of a FactorPrior, keeps nothing, and the outer estimator states the source.

Constructors

EmpiricalPrior(;    ce::StatsBase.CovarianceEstimator = PortfolioOptimisersCovariance(),    me::AbstractExpectedReturnsEstimator = SimpleExpectedReturns(),    horizon::Option{<:Number} = nothing,    fill_limit::Option{<:Real} = nothing,    max_scenarios::Option{<:Integer} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing,    x_src::Option{<:AbstractReturnsSource} = nothing) -> EmpiricalPrior

Keywords correspond to the struct's fields.

Validation

  • ce is not an AbstractVarianceEstimator. A variance estimator has no cov or cor, so assert_covariance_estimator raises an ArgumentError that names ce.
  • If horizon is not nothing, horizon > 0.
  • If fill_limit is not nothing, 0 < fill_limit <= 1. The constructor refuses zero. Where no arm states a coverage floor, nothing already names every fill. The fit refuses a value above 1 - min_coverage of the binding floor of the arms, because such a value can never fire.
  • If max_scenarios is not nothing, max_scenarios > 0.
  • x_src is nothing, DataSource() or ExcessSource(), see assert_returns_source.

The incremental fit

The empirical prior folds its moments and carries its rows. partial_fit! folds each observation into each member of me and ce that folds, and appends the row to a PriorCarryState of its own. The one-argument prior reads mu and sigma off the state of a folded member, and X off the carried rows. A member that does not fold runs its batch verb over the carried rows, see read_member. When both members fold, the cost of a step grows with the square of the number of assets, and not with the number of observations folded. The prior carries the rows because LowOrderPrior holds X for the scenario risk measures.

Two caps, two names

max_scenarios and the max_history of Online are two different caps.

  • EmpiricalPrior(; max_scenarios = w) fits mu and sigma over every observation, and carries the last w rows as X. The cap is a property of the result, so it cuts the same rows in batch and online. No batch fit gives the same result, because the moments and the scenarios come from different windows.
  • Online(EmpiricalPrior(); max_history = w) cuts the whole fit to the last w observations. The estimator refits from the buffer, so the result equals prior(pe, X[end - w + 1:end, :]).

When both caps are set, the moments come from the last max_history observations, and X comes from the last max_scenarios of those. A max_scenarios at or above the number of carried observations changes nothing.

Propagated parameters

factory passes its arguments on to these @fprop fields:

  • ce: Recursively updated through factory.
  • me: Recursively updated through factory.
  • cache: Carried unchanged through factory.

View parameters

port_opt_view takes the view of these @vprop fields at the selected indices:

Examples

julia> EmpiricalPrior()EmpiricalPrior          ce ┼ PortfolioOptimisersCovariance             │   ce ┼ Covariance             │      │    me ┼ SimpleExpectedReturns             │      │       │   w ┴ nothing             │      │    ce ┼ GeneralCovariance             │      │       │   ce ┼ StatsBase.SimpleCovariance: StatsBase.SimpleCovariance(true)             │      │       │    w ┴ nothing             │      │   alg ┼ FullMoment()             │      │     w ┴ nothing             │   mp ┼ MatrixProcessing             │      │     pdm ┼ Posdef             │      │         │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton             │      │         │   kwargs ┴ @NamedTuple{}: NamedTuple()             │      │      dn ┼ nothing             │      │      dt ┼ nothing             │      │     alg ┼ nothing             │      │   order ┴ NTuple{4, Symbol}: (:pdm, :dn, :dt, :alg)          me ┼ SimpleExpectedReturns             │   w ┴ nothing     horizon ┼ nothing  fill_limit ┼ nothing       x_src ┴ nothing

Related

References

  • [52] A. Meucci. Risk and Asset Allocation (Springer Berlin Heidelberg, 2005). Section 2.6.5, Equations 2.219 and 2.220, Section 3.2, Equation 3.64, and Section 3.3.1, Equation 3.80.
source
PortfolioOptimisers.prior — Function
prior(pe::EmpiricalPrior{<:Any, <:Any, Nothing, <:Any}, X::MatNum,
      F::Option{<:MatNum} = nothing, pnl::Option{<:AssetPanel} = nothing;
      dims::Int = 1, strict::Bool = false, kwargs...)

Compute the empirical prior moments of asset returns with no horizon conversion.

pe.me fits the mean and pe.ce fits the covariance on X itself, with no log transform and no horizon scale. The method returns them in a LowOrderPrior.

Mathematical definition

The two moments are what pe.me and pe.ce return. The default pair, SimpleExpectedReturns and PortfolioOptimisersCovariance with no observation weights, gives the sample moments:

\[\begin{align} \hat{\boldsymbol{\mu}} &= \frac{1}{T} \sum_{t=1}^{T} \boldsymbol{x}_t\,, \\ \hat{\mathbf{\Sigma}} &= \frac{1}{T-1} \sum_{t=1}^{T} (\boldsymbol{x}_t - \hat{\boldsymbol{\mu}})(\boldsymbol{x}_t - \hat{\boldsymbol{\mu}})^\intercal\,. \end{align}\]

Where:

  • $\hat{\boldsymbol{\mu}}$: $N \times 1$ mean vector.
  • $\hat{\mathbf{\Sigma}}$: $N \times N$ covariance matrix.
  • $\boldsymbol{x}_t$: $N \times 1$ vector of asset returns at time $t$.
  • $T$: Number of observations.

Any other configuration of pe.me or pe.ce changes the result. For example, a shrunk mean or a denoised covariance differs from the display above.

This method takes the arithmetic moments of X. It applies no log transform, so it is not the $h = 1$ case of the horizon method. The horizon method converts the returns to $\log(1 + x_t)$ and back, and that round trip leaves the moments unchanged only in the limit of small returns.

The scenario cap

pe.max_scenarios cuts the returns matrix of the result to its last max_scenarios rows, through scenario_window. mu and sigma stay fitted over every observation. The cut runs before the fill, so the fill measures its share over the rows that a consumer reads. The default, nothing, carries every row. A cap at or above the number of observations returns X itself, and a smaller cap returns a view, so the cut copies nothing.

The counts of the moments

The result states in eoc the effective number of observations of mu and of sigma, through scenario_moment_counts. pe.me and pe.ce each state the count of their own moment with effective_observation_count. A PriorExpectedReturns in pe.me states the count that its prior states for the mean, and mean_with_count reads it from the one fit of that prior. An unweighted estimator is worth size(X, 1) observations, and a weighted one, such as an exponentially weighted covariance, is worth the count of its weights. When the cap cuts, the count of an unweighted moment is the number of observations that the moments read. A consumer that prices a sample size reads these counts and not the number of carried rows. An entry equal to the number of carried rows stays nothing.

The scenario fill

A mask-aware pe.me and pe.ce fit an asset that lists inside the window from the observations it has. The exponentially weighted family is mask-aware, and so is the plain family wherever a CoveragePolicy is set. Such an asset is investable, and its column of X still carries a NaN at every earlier row. scenario_fill writes 0 at those entries once, so that every consumer of the result reads a finite investable column. A non-investable asset keeps its whole NaN column, and the fill does not change mu or sigma.

The fill costs accuracy at four consumers, and the message names all four:

  • A scenario-based risk measure understates the risk of the asset over the filled rows.
  • A hierarchical optimiser can put the asset in a branch of its own and overweight it.
  • An entropy pooling view on the asset is calibrated on the filled column.
  • A meta-optimiser carries the fill into the outer problem.

The fill is silent while the filled share of the worst investable column stays at or below the limit that resolve_fill_limit derives. Above the limit it warns, and under strict it refuses any fill. pe.fill_limit defaults to nothing. Where an arm states a coverage floor, nothing derives 1 - min_coverage, which never fires. Where no arm states a floor, nothing names every fill. Under a plain estimator with no policy the fill never fires, because an asset that the estimator cannot cover leaves the Coverage Universe and is not investable.

Arguments

  • pe: Empirical prior estimator.
  • X: Asset returns matrix (observations × assets).
  • F: Factor returns matrix (ignored).
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • strict: Whether a zero-filled scenario raises an error instead of a warning. Under strict, any fill raises. Otherwise a column filled above the limit that resolve_fill_limit derives warns, and every fill warns while that limit is nothing. strict also reaches the covariance, where a peel of a CoveragePolicy raises under strict and warns otherwise.
  • kwargs...: Additional keyword arguments, passed to the mean estimator and the covariance estimator.

Validation

  • dims in (1, 2).
  • At least one asset must be in the Coverage Universe.
  • pe.fill_limit is at most 1 - min_coverage of the binding floor of the arms, see coverage_floor. Otherwise the fit raises a DomainError, because a looser limit can never fire.
  • The zero-filled share of the worst investable column of X is at or below the limit that resolve_fill_limit derives. Otherwise the fit warns and names the assets. Under strict, any fill raises an ArgumentError that names them. The limit is nothing when no arm states a floor and pe.fill_limit is nothing, and no share passes a limit of nothing, so the fit names every fill.

Returns

  • pr::LowOrderPrior: The prior result, which holds the asset returns, the mean vector and the covariance matrix.

Related

source
PortfolioOptimisers.prior — Function
prior(pe::EmpiricalPrior{<:Any, <:Any, <:Number, <:Any}, X::MatNum,
      F::Option{<:MatNum} = nothing, pnl::Option{<:AssetPanel} = nothing;
      dims::Int = 1, strict::Bool = false, kwargs...)

Compute the empirical prior moments of asset returns at an investment horizon.

pe.me and pe.ce fit the mean and the covariance of the log-returns. horizon_moments! scales both moments to the horizon and converts them back to arithmetic returns. The method returns them in a LowOrderPrior.

Mathematical definition

pe.me and pe.ce fit the log-returns $\log(1 + x_t)$, not X. The method scales the two log-moments by the investment horizon $h$, and then converts them back to arithmetic returns:

\[\begin{align} \tilde{\boldsymbol{\mu}} &= h \cdot \hat{\boldsymbol{\mu}}_{\log}\,, \\ \tilde{\mathbf{\Sigma}} &= h \cdot \hat{\mathbf{\Sigma}}_{\log}\,. \end{align}\]

\[\begin{align} \hat{\mu}_i &= \exp\!\left(\tilde{\mu}_i + \tfrac{1}{2}\tilde{\sigma}_{ii}\right) - 1\,, \\ \hat{\sigma}_{ij} &= (\hat{\mu}_i + 1)(\hat{\mu}_j + 1)\left(\exp(\tilde{\sigma}_{ij}) - 1\right)\,. \end{align}\]

Where:

  • $\tilde{\boldsymbol{\mu}}$, $\tilde{\mathbf{\Sigma}}$: Horizon-scaled log-return mean and covariance.
  • $h$: Investment horizon.
  • $\hat{\boldsymbol{\mu}}_{\log}$, $\hat{\mathbf{\Sigma}}_{\log}$: Mean and covariance of the log-returns $\log(1 + x_t)$, computed by pe.me and pe.ce.
  • $\hat{\mu}_i$: Arithmetic mean return for asset $i$.
  • $\hat{\sigma}_{ij}$: Arithmetic covariance between assets $i$ and $j$.

The X of the returned LowOrderPrior is the arithmetic returns matrix that the caller gave. Only the moments come from the log-returns.

Algorithm

Steps 5 to 7 must run in this order. Step 6 reads the mu that step 5 left, which is $\hat{\mu}_i + 1$, because step 7 has not yet subtracted the one. The second closed form needs that factor, so the order supplies it with no second computation. If step 7 runs before step 6, each $\hat{\mu}_i + 1$ becomes $\hat{\mu}_i$. Entry $(i,\, j)$ of the covariance then falls by the factor $\hat{\mu}_i \hat{\mu}_j / ((\hat{\mu}_i + 1)(\hat{\mu}_j + 1))$, which is about six orders of magnitude on a daily return series. Steps 4 to 7 are horizon_moments!.

  1. Orient X to observations × assets with dims_oriented.
  2. Take the log-returns X_log, as log1p.(X).
  3. Fit the log-return mean mu on X_log with pe.me, and the log-return covariance sigma on X_log with pe.ce.
  4. Scale both by pe.horizon to get $\tilde{\boldsymbol{\mu}}$ and $\tilde{\mathbf{\Sigma}}$.
  5. Overwrite mu with the exponential term of the first closed form. This is the arithmetic mean plus one, because step 7 has not run yet.
  6. Overwrite sigma with the second closed form, whose $\hat{\mu}_i + 1$ factors are the mu of step 5.
  7. Subtract one from mu to get $\hat{\boldsymbol{\mu}}$.
  8. Cut the arithmetic X of step 1 with scenario_window, and fill it with scenario_fill. Return a LowOrderPrior that holds this X, mu, sigma and the eoc of scenario_moment_counts over X_log.

The scenario cap

pe.max_scenarios cuts the returns matrix of the result to its last max_scenarios rows, through scenario_window. mu and sigma stay fitted over every observation. The cut runs before the fill, so the fill measures its share over the rows that a consumer reads. The default, nothing, carries every row. A cap at or above the number of observations returns X itself, and a smaller cap returns a view, so the cut copies nothing.

The counts of the moments

The result states in eoc the effective number of observations of mu and of sigma, through scenario_moment_counts. pe.me and pe.ce each state the count of their own moment with effective_observation_count. A PriorExpectedReturns in pe.me states the count that its prior states for the mean, and mean_with_count reads it from the one fit of that prior. An unweighted estimator is worth size(X, 1) observations, and a weighted one, such as an exponentially weighted covariance, is worth the count of its weights. When the cap cuts, the count of an unweighted moment is the number of observations that the moments read. A consumer that prices a sample size reads these counts and not the number of carried rows. An entry equal to the number of carried rows stays nothing.

The scenario fill

Step 8 applies the fill of the method with no horizon. It fills the arithmetic X that the caller gave, and it reads the Investable Mask off the arithmetic moments that the result carries. Under a mask-aware pe.me and pe.ce, an asset that lists inside the window is investable, and its column still carries a NaN at every earlier row. scenario_fill writes 0 at those entries once. The fill is silent while the filled share of the worst investable column stays at or below the limit that resolve_fill_limit derives. Above the limit it warns, and under strict it refuses any fill.

pe.fill_limit defaults to nothing. Where an arm states a coverage floor, nothing derives 1 - min_coverage. Where no arm states a floor, nothing names every fill. A scenario-based risk measure understates the risk of a filled asset over the filled rows. The fill does not change mu or sigma.

Arguments

  • pe: Empirical prior estimator.
  • X: Asset returns matrix (observations × assets).
  • F: Factor returns matrix (ignored).
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • strict: Whether a zero-filled scenario raises an error instead of a warning. Under strict, any fill raises. Otherwise a column filled above the limit that resolve_fill_limit derives warns, and every fill warns while that limit is nothing. strict also reaches the covariance, where a peel of a CoveragePolicy raises under strict and warns otherwise.
  • kwargs...: Additional keyword arguments, passed to the mean estimator and the covariance estimator.

Validation

  • dims in (1, 2).
  • At least one asset must be in the Coverage Universe.
  • pe.fill_limit is at most 1 - min_coverage of the binding floor of the arms, see coverage_floor. Otherwise the fit raises a DomainError, because a looser limit can never fire.
  • The zero-filled share of the worst investable column of X is at or below the limit that resolve_fill_limit derives. Otherwise the fit warns and names the assets. Under strict, any fill raises an ArgumentError that names them. The limit is nothing when no arm states a floor and pe.fill_limit is nothing, and no share passes a limit of nothing, so the fit names every fill.

Returns

  • pr::LowOrderPrior: The prior result, which holds the asset returns, the mean vector and the covariance matrix.

Related

source

References

[52]
A. Meucci. Risk and Asset Allocation (Springer Berlin Heidelberg, 2005).