Covariance Forecast Evaluation: private API

PortfolioOptimisers.forecast_location — Function
forecast_location(ce::Covariance, X::MatNum; dims::Int = 1, active_mask = nothing, kwargs...)
forecast_location(ce::GeneralCovariance, X::MatNum; dims::Int = 1, kwargs...)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance}, X::MatNum; dims::Int = 1, active_mask = nothing, kwargs...)
forecast_location(ce::Union{PortfolioOptimisersCovariance, CorrelationCovariance}, X::MatNum; kwargs...)
forecast_location(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)
forecast_location(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance}, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::PortfolioOptimisersCovariance, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::Covariance)
forecast_location(ce::GeneralCovariance)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance})
forecast_location(ce::PortfolioOptimisersCovariance)
forecast_location(ce::CorrelationCovariance{<:Any, Nothing})
forecast_location(ce::AbstractCovarianceEstimator)
forecast_location(pe::AbstractPriorEstimator, rd::ReturnsResult; strict::Bool = false, kwargs...)
forecast_location(pe::AbstractPriorEstimator; strict::Bool = false, kwargs...)

Read the location that a covariance forecast is about, off the estimator that formed it.

A second moment is always taken about some centre, $\mathbb{E}[(\boldsymbol{z} - \boldsymbol{c})(\boldsymbol{z} - \boldsymbol{c})^\intercal] = \hat{\mathbf{\Sigma}}$, and the centre belongs to the estimator. It is the mean that its centring estimator fitted, the weighted sample mean that it subtracted, the exponentially weighted location that its state carries, or zero for an estimator that assumes a zero mean. An evaluation against the raw outer product $\boldsymbol{z}\boldsymbol{z}^\intercal$ would judge the forecast against a proxy of $\mathbf{\Sigma} + (\boldsymbol{\mu} - \boldsymbol{c})(\boldsymbol{\mu} - \boldsymbol{c})^\intercal$. It would then read a bias of $\boldsymbol{c}^\intercal \hat{\mathbf{\Sigma}}^{-1} \boldsymbol{c} / N$ into a forecast that has none. So the evaluation centres the test rows on this location. No flag turns the centring off, because the location belongs to the estimator and not to the caller.

Each family has two arities. The data form answers for a batch fit over X. The data-less form answers for the state that the estimator carries after partial_fit!, so a refit fold and a stepped fold centre on the same quantity. For a family state, the data-less form returns the vector of the state itself, which the next fold writes in place.

  • Covariance answers the centre that weighted_centre resolves from ce.me and ce.w, or state.mu. Under a CoveragePolicy the batch fit and the fold are one computation, so the data form folds the window and reads the same state.mu. That fold runs under FullMoment whatever ce.alg is, because the centre does not depend on the moment algorithm, and the fold refuses every other algorithm. That vector is the diagonal of the per-pair centre, the available-case mean of each asset. The forecast centres each off-diagonal product on the mean of the observations that the pair shares, and one vector cannot carry that centre. The per-asset ratios read the diagonal only.
  • GeneralCovariance answers the sample mean, weighted by ce.w when it carries observation weights, or state.mu.
  • ExpWeightedCovariance and RegimeAdjustedExpWeightedCovariance answer state.location under EstimatedCentring, or zero for every asset under PreCentred, because the estimator then assumes a zero mean. The data form runs over the window the same pass that the batch fit runs, and reads the location at the end of the pass.
  • PortfolioOptimisersCovariance and CorrelationCovariance answer what the estimator that they hold answers, because a matrix transform moves no centre. A StatsBase.CovarianceEstimator that they hold is read as a GeneralCovariance, as cov reads it.
  • Any other AbstractCovarianceEstimator answers the sample mean of the window, over the finite rows of each column. Its data-less form reads the state through the state methods of this function, which refuse a state of a shape that they do not know.
  • Under Online, every family answers its data form over the rows and masks of the buffer, because a buffer means the batch verb over its rows each time the estimate is made from the state.
  • An AbstractPriorEstimator answers the centre of its sigma through prior_forecast_location. That is the mu it publishes for a family that forms both moments about one mean. An EmpiricalPrior whose me shrinks the mean publishes the shrunk mu, but its ce centres the covariance on its own estimate of the mean, so it answers the location of its ce. A factor prior and a high-order prior answer the centre of the prior that they lift or wrap. The keyword strict reaches prior alone. The location reads the returns in the frame that the prior fits on, the source that pe states through returns_source. Before the fit and before the location, shared_series_fit drops the observations of a gap in a vector benchmark under ExcessSource, and the observations at which a factor return is not finite when the tree of the prior reads the factor returns. It drops the fixed observation weights of the same observations from the tree of the prior. So the fit and the location read the same observations.

The Asset Panel form follows the Asset Panel method of the estimator's own cov. It reads the policy with coverage_policy, which answers nothing for an estimator that holds no policy. Under no policy it reduces the window to its Coverage Universe and frames the answer with NaN outside it. Under a policy, and for the two mask-aware exponentially weighted families, it hands the active mask of the panel to the estimator. A PortfolioOptimisersCovariance hands the panel to the estimator that it holds, as its cov does.

Arguments

  • ce: Covariance estimator.
  • pe: Prior estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • rd: The returns result to use.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • active_mask: The active mask of the window, observations × assets, or nothing.
  • kwargs...: Additional keyword arguments passed to the centring estimator.

Validation

  • The data-less form requires a state. An ArgumentError is thrown otherwise.
  • The data-less form requires a state of a shape that a state method reads: a CovarianceState, an exponentially weighted state or a SampleBufferState. An ArgumentError is thrown otherwise.

Returns

  • c::VecNum: The location, assets × 1.

Related

source
forecast_location(ce, state::CovarianceState)
forecast_location(ce, state::Union{ExpWeightedCovarianceState, RegimeAdjustedCovarianceState})
forecast_location(ce, state::SampleBufferState)
forecast_location(ce::PortfolioOptimisersCovariance, state::Nothing)
forecast_location(ce, state::Nothing)
forecast_location(ce, state::AbstractPartialFitState = partial_fit_cache(ce))

Read the location out of the state that a covariance estimator carries, by the type of the state.

These are the state methods of forecast_location. The data-less form calls them with the state of the estimator, one method per shape of state. An estimator that has no data-less method of its own reads its state with partial_fit_cache, the default of the last method:

  • A CovarianceState carries mu.
  • The two exponentially weighted states carry location.
  • A SampleBufferState carries the rows, the masks and the implied volatilities that it folded. The method answers the data form of the estimator over them, because a buffer means the batch verb over its rows each time the estimate is made from the state.
  • A PortfolioOptimisersCovariance whose own cache is nothing forwards to the estimator that it holds, which carries the state.

The first two methods return the vector of the state itself, and the next fold writes that vector in place. A caller that keeps the location across a fold must copy it, as covariance_forecast_evaluation does when it stores the forecasts.

Validation

  • state is not nothing. An ArgumentError is thrown otherwise, because there is nothing to read.
  • state has one of the shapes above. An ArgumentError is thrown otherwise.

Related

source
PortfolioOptimisers.prior_forecast_location — Function
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, rd::ReturnsResult; kwargs...)
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::EmpiricalPrior, pr::AbstractPriorResult, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::FactorPrior, pr::AbstractPriorResult, X::MatNum, F::MatNum, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::Union{HighOrderPriorEstimator, HighOrderFactorPriorEstimator}, pr::HighOrderPrior, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult; kwargs...)
prior_forecast_location(pe::HighOrderPriorEstimator, pr::HighOrderPrior; kwargs...)

Read the location that the sigma of a prior is about, from the estimator and the result pr that it fitted.

The rule is the same for every family. The location is the mu that the prior would publish if its mean estimator were the centring estimator of its covariance estimator. Only a family that takes its two moments from two independent estimators, or that lifts the moments of such a prior, answers something other than pr.mu:

  • EmpiricalPrior answers forecast_location of its ce over the rows that ce saw. The horizon arm fits ce on the log-returns, so it maps that log location through the map that mu takes, $c_i = \exp(h c_{\log, i} + \tfrac{1}{2} h \sigma_{\log, ii}) - 1$. It reads $h \sigma_{\log, ii} = \log(1 + \sigma_{ii} / (\mu_i + 1)^2)$ back off the published pair.
  • FactorPrior answers the lift of the location of its factor prior, $\mathbf{M} \boldsymbol{c}_f + \boldsymbol{b}$, which it writes as $\boldsymbol{\mu} + \mathbf{M} (\boldsymbol{c}_f - \boldsymbol{\mu}_f)$.
  • HighOrderPriorEstimator and HighOrderFactorPriorEstimator answer the location of the prior that they wrap, because they publish its mu and sigma unchanged.
  • Every other family answers pr.mu. The Black–Litterman and entropy pooling families and OpinionPoolingPrior form sigma about their posterior or pooled mu. CrossSectionalFactorPrior and LowDimensionEnsemblePrior publish a forecast of the next return as mu.

For the default EmpiricalPrior, the mean estimator is the centring estimator of ce, so every family answers pr.mu. The returns data form reads the asset returns in the frame that pe fits on, through benchmark_relative_returns under the source of pe, as prior reads them. The data-less form reads the state as prior does, through prior_state_location.

Arguments

  • pe: Prior estimator.
  • pr: Prior result. It is the result that pe fitted over the same rows.
  • rd: The returns result to use.
  • X: Asset returns matrix, observations × assets.
  • F: Factor returns matrix, observations × factors, or nothing.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • kwargs...: Additional keyword arguments, passed to the centring estimator as prior passes them to ce.

Returns

  • c::VecNum: The location, assets × 1.

Related

source
PortfolioOptimisers.prior_state_location — Function
prior_state_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, state; kwargs...)
prior_state_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, state::SampleBufferState; kwargs...)
prior_state_location(pe::EmpiricalPrior, pr::AbstractPriorResult, state::PriorCarryState; kwargs...)

Read the location of a prior out of the state that its estimator carries, by the type of the state.

This is the data-less arm of prior_forecast_location, and it takes the routes that the data-less prior takes. Each route reads the rows of shared_series_buffer, which drops the observations of a Shared Series gap, as the data-less prior does:

  • A SampleBufferState is a refit, so the method answers the data form over the rows, the factor rows and the masks of the buffer.
  • The PriorCarryState of an EmpiricalPrior answers the location of ce. A ce that folds answers forecast_location of its state. A ce that does not fold answers the data form over the carried rows, as read_member refits it. The horizon arm reads the log-returns of the rows, and maps the location as the data form does.
  • Every other state, and nothing, answers pr.mu.

Arguments

  • pe: Prior estimator.
  • pr: Prior result.
  • state: The state that pe carries, or nothing.
  • kwargs...: Additional keyword arguments, passed to the data form or to the refit.

Returns

  • c::VecNum: The location, assets × 1.

Related

source
PortfolioOptimisers.carried_location — Function
carried_location(ce, X::MatNum, pnl::Option{<:AssetPanel}; kwargs...)

Read the location of a covariance estimator that a fold-and-carry prior holds, out of its state or over the rows that the prior carries.

This is read_member for forecast_location. A StatsBase.CovarianceEstimator does not fold, and it is read as a GeneralCovariance over the rows, as cov reads it. The refit calls the panel form over pnl, as the data form of prior_forecast_location does over rd.pnl, so it answers a gapped sample and reads the active mask that the carried buffer records.

Arguments

  • ce: Covariance estimator.
  • X: The rows that the prior carries, observations × assets.
  • pnl: The Asset Panel that buffer_asset_panel rebuilds from the carried buffer, or nothing.
  • kwargs...: Additional keyword arguments, passed to the refit alone.

Returns

  • c::VecNum: The location, assets × 1.

Related

source
PortfolioOptimisers.horizon_location — Function
horizon_location(c::VecNum, pr::AbstractPriorResult, horizon::Number)

Map the log location of a covariance estimator to the arithmetic location at an investment horizon, as horizon_moments! maps the log mean.

Mathematical definition

\[\begin{align} h \sigma_{\log, ii} &= \log\!\left(1 + \frac{\sigma_{ii}}{(\mu_i + 1)^2}\right)\,, \\ c_i &= \exp\!\left(h c_{\log, i} + \tfrac{1}{2} h \sigma_{\log, ii}\right) - 1\,. \end{align}\]

Where:

  • $h$: Investment horizon.
  • $\mu_i$, $\sigma_{ii}$: The arithmetic mean and variance that the prior publishes.
  • $\sigma_{\log, ii}$: The log-return variance that the prior fitted. The first line inverts the second closed form of horizon_moments! on the diagonal.
  • $c_{\log, i}$: The log location of the covariance estimator.

When c is the log mean of the prior, $c_i = \mu_i$.

Arguments

  • c: The log location, assets × 1.
  • pr: Prior result.
  • horizon: The investment horizon.

Returns

  • c::VecNum: The arithmetic location, assets × 1.

Related

source
PortfolioOptimisers.resolve_forecast_weights — Function
resolve_forecast_weights(w::Nothing, a::VecInt, sa::MatNum, N::Integer)
resolve_forecast_weights(w::VecNum, a::VecInt, sa::MatNum, N::Integer)
resolve_forecast_weights(w::VecVecNum, a::VecInt, sa::MatNum, N::Integer)

Resolve the test portfolios of a step over its active subset.

nothing means inverse volatility, recomputed from the forecast of the step over the active assets, $w_i \propto 1 / \sqrt{(\hat{\mathbf{\Sigma}}_t)_{ii}}$. It keeps the most volatile asset from dominating the reading. A vector is one static portfolio, and a vector of vectors is several. The method cuts each portfolio to the active subset and rescales it to sum to one there. So the weight of a delisted asset moves to the remaining assets, in proportion to their weights. The method does not check a subset whose weights sum to zero. Such a portfolio has no scale, and its diagnostics are not finite.

Arguments

  • w: The test portfolios on the full universe, or nothing.
  • a: The active subset.
  • sa: The forecast on the active subset.
  • N: The width of the universe.

Validation

  • Every portfolio has N entries. A DimensionMismatch is thrown otherwise.

Returns

  • ws::Vector{<:VecNum}: One weights vector per portfolio, each over a.

Related

source
PortfolioOptimisers.step_implied_volatility_kwargs — Function
step_implied_volatility_kwargs(state::SampleBufferState, rd::ReturnsResult)
step_implied_volatility_kwargs(state, rd::ReturnsResult)

Give the implied volatilities of a step to the state that records them, as keywords of partial_fit!.

A SampleBufferState records iv and ivpa beside its rows, so it takes rd.iv and rd.ivpa. Any other state is an exact fold, which keeps numbers alone, and it takes nothing. The type of the state selects the method.

Arguments

  • state: The state that the estimator carries.
  • rd: The returns data of the step.

Returns

  • kwargs::NamedTuple: iv and ivpa for a buffer, or an empty NamedTuple.

Related

source
PortfolioOptimisers.online_entry_state — Method
online_entry_state(o::Online)

Name the state that a wrapped estimator carries at the entry of the online arm of the fold loop, or answer nothing.

This is the Online method of online_entry_state, for a wrapper at the root of covariance_forecast_evaluation. A wrapper holds no cache of its own, so the walk enters the estimator that it wraps and puts est. before the path that it finds there. For a covariance estimator that the caller folded before the call, the path is est.cache. The generic walk through estimator_fields would give the same answer. This method gives it without a request for a field that the wrapper does not have.

Related

source
PortfolioOptimisers.advance_previous_fold — Method
advance_previous_fold(pws, prev, fold::Tuple{CovarianceForecastStepResult, Option{<:MatNum}, Option{<:VecNum}})

Hand the next fold what the last fold held, when the result of the fold is a step of a covariance forecast evaluation and not a prediction.

The fold of covariance_forecast_evaluation answers the tuple of its CovarianceForecastStepResult, its stored forecast and its stored location. The step carries no weights, so the method threads nothing and returns prev unchanged. online_folds reaches this method under covariance_forecast_evaluation.

Related

source
PortfolioOptimisers.forecast_moments — Function
forecast_moments(ce::AbstractCovarianceEstimator, rd::ReturnsResult, train_idx::VecInt)
forecast_moments(ce::AbstractCovarianceEstimator, rd::ReturnsResult, ::Nothing)
forecast_moments(pe::AbstractPriorEstimator, rd::ReturnsResult, train_idx::VecInt)
forecast_moments(pe::AbstractPriorEstimator, rd::ReturnsResult, ::Nothing)

Read the forecast of a fold and the location that it is about, by whether the fold carries a training window.

These are the two arms of the callback of covariance_forecast_evaluation. Dispatch on the train of the fold chooses the arm, as it does for the optimiser in fit_fold_result. A training window means a refit, which every batch fold runs. nothing means that the estimator holds its window, so the fold reads the moments out of the state. The method reads a prior once for both moments.

Algorithm

  1. With a training window, take the view of rd on its rows, giving rdt. Fit a covariance estimator through the Asset Panel methods of the moment verbs, cov(ce, rdt.X, rdt.pnl) and forecast_location(ce, rdt.X, rdt.pnl). Both calls take iv = rdt.iv and ivpa = rdt.ivpa, as prior gives them to its members, so an ImpliedVolatility reads its surface and every other estimator ignores it. Fit a prior once over the view, giving pr.
  2. With nothing, read cov(ce) and forecast_location(ce) out of the state of a covariance estimator, or pr = prior(pe) out of the state of a prior.
  3. Return the forecast and the location. For a prior they are pr.sigma and prior_forecast_location of pr, which reads the centre of pr.sigma without a second fit.

Arguments

  • ce: Covariance estimator.
  • pe: Prior estimator.
  • rd: The returns data.
  • train_idx: The fold's training window, or nothing for a stepped estimator.

Returns

  • sigma::MatNum: The forecast, assets × assets.
  • c::VecNum: The location the forecast is about, assets × 1.

Related

source