Covariance Forecast Evaluation: private API
PortfolioOptimisers.forecast_location — Function
forecast_location(ce::Covariance, X::MatNum; dims::Int = 1, active_mask = nothing, kwargs...)
forecast_location(ce::GeneralCovariance, X::MatNum; dims::Int = 1, kwargs...)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance}, X::MatNum; dims::Int = 1, active_mask = nothing, kwargs...)
forecast_location(ce::Union{PortfolioOptimisersCovariance, CorrelationCovariance}, X::MatNum; kwargs...)
forecast_location(ce::AbstractCovarianceEstimator, X::MatNum; dims::Int = 1, kwargs...)
forecast_location(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance}, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::PortfolioOptimisersCovariance, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)
forecast_location(ce::Covariance)
forecast_location(ce::GeneralCovariance)
forecast_location(ce::Union{ExpWeightedCovariance, RegimeAdjustedExpWeightedCovariance})
forecast_location(ce::PortfolioOptimisersCovariance)
forecast_location(ce::CorrelationCovariance{<:Any, Nothing})
forecast_location(ce::AbstractCovarianceEstimator)
forecast_location(pe::AbstractPriorEstimator, rd::ReturnsResult; strict::Bool = false, kwargs...)
forecast_location(pe::AbstractPriorEstimator; strict::Bool = false, kwargs...)Read the location that a covariance forecast is about, off the estimator that formed it.
A second moment is always taken about some centre, $\mathbb{E}[(\boldsymbol{z} - \boldsymbol{c})(\boldsymbol{z} - \boldsymbol{c})^\intercal] = \hat{\mathbf{\Sigma}}$, and the centre belongs to the estimator. It is the mean that its centring estimator fitted, the weighted sample mean that it subtracted, the exponentially weighted location that its state carries, or zero for an estimator that assumes a zero mean. An evaluation against the raw outer product $\boldsymbol{z}\boldsymbol{z}^\intercal$ would judge the forecast against a proxy of $\mathbf{\Sigma} + (\boldsymbol{\mu} - \boldsymbol{c})(\boldsymbol{\mu} - \boldsymbol{c})^\intercal$. It would then read a bias of $\boldsymbol{c}^\intercal \hat{\mathbf{\Sigma}}^{-1} \boldsymbol{c} / N$ into a forecast that has none. So the evaluation centres the test rows on this location. No flag turns the centring off, because the location belongs to the estimator and not to the caller.
Each family has two arities. The data form answers for a batch fit over X. The data-less form answers for the state that the estimator carries after partial_fit!, so a refit fold and a stepped fold centre on the same quantity. For a family state, the data-less form returns the vector of the state itself, which the next fold writes in place.
Covarianceanswers the centre thatweighted_centreresolves fromce.meandce.w, orstate.mu. Under aCoveragePolicythe batch fit and the fold are one computation, so the data form folds the window and reads the samestate.mu. That fold runs underFullMomentwhateverce.algis, because the centre does not depend on the moment algorithm, and the fold refuses every other algorithm. That vector is the diagonal of the per-pair centre, the available-case mean of each asset. The forecast centres each off-diagonal product on the mean of the observations that the pair shares, and one vector cannot carry that centre. The per-asset ratios read the diagonal only.GeneralCovarianceanswers the sample mean, weighted byce.wwhen it carries observation weights, orstate.mu.ExpWeightedCovarianceandRegimeAdjustedExpWeightedCovarianceanswerstate.locationunderEstimatedCentring, or zero for every asset underPreCentred, because the estimator then assumes a zero mean. The data form runs over the window the same pass that the batch fit runs, and reads the location at the end of the pass.PortfolioOptimisersCovarianceandCorrelationCovarianceanswer what the estimator that they hold answers, because a matrix transform moves no centre. AStatsBase.CovarianceEstimatorthat they hold is read as aGeneralCovariance, ascovreads it.- Any other
AbstractCovarianceEstimatoranswers the sample mean of the window, over the finite rows of each column. Its data-less form reads the state through the state methods of this function, which refuse a state of a shape that they do not know. - Under
Online, every family answers its data form over the rows and masks of the buffer, because a buffer means the batch verb over its rows each time the estimate is made from the state. - An
AbstractPriorEstimatoranswers the centre of itssigmathroughprior_forecast_location. That is themuit publishes for a family that forms both moments about one mean. AnEmpiricalPriorwhosemeshrinks the mean publishes the shrunkmu, but itscecentres the covariance on its own estimate of the mean, so it answers the location of itsce. A factor prior and a high-order prior answer the centre of the prior that they lift or wrap. The keywordstrictreachesprioralone. The location reads the returns in the frame that the prior fits on, the source thatpestates throughreturns_source. Before the fit and before the location,shared_series_fitdrops the observations of a gap in a vector benchmark underExcessSource, and the observations at which a factor return is not finite when the tree of the prior reads the factor returns. It drops the fixed observation weights of the same observations from the tree of the prior. So the fit and the location read the same observations.
The Asset Panel form follows the Asset Panel method of the estimator's own cov. It reads the policy with coverage_policy, which answers nothing for an estimator that holds no policy. Under no policy it reduces the window to its Coverage Universe and frames the answer with NaN outside it. Under a policy, and for the two mask-aware exponentially weighted families, it hands the active mask of the panel to the estimator. A PortfolioOptimisersCovariance hands the panel to the estimator that it holds, as its cov does.
Arguments
ce: Covariance estimator.pe: Prior estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.rd: The returns result to use.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.active_mask: The active mask of the window,observations × assets, ornothing.kwargs...: Additional keyword arguments passed to the centring estimator.
Validation
- The data-less form requires a state. An
ArgumentErroris thrown otherwise. - The data-less form requires a state of a shape that a state method reads: a
CovarianceState, an exponentially weighted state or aSampleBufferState. AnArgumentErroris thrown otherwise.
Returns
c::VecNum: The location,assets × 1.
Related
forecast_location(ce, state::CovarianceState)
forecast_location(ce, state::Union{ExpWeightedCovarianceState, RegimeAdjustedCovarianceState})
forecast_location(ce, state::SampleBufferState)
forecast_location(ce::PortfolioOptimisersCovariance, state::Nothing)
forecast_location(ce, state::Nothing)
forecast_location(ce, state::AbstractPartialFitState = partial_fit_cache(ce))Read the location out of the state that a covariance estimator carries, by the type of the state.
These are the state methods of forecast_location. The data-less form calls them with the state of the estimator, one method per shape of state. An estimator that has no data-less method of its own reads its state with partial_fit_cache, the default of the last method:
- A
CovarianceStatecarriesmu. - The two exponentially weighted states carry
location. - A
SampleBufferStatecarries the rows, the masks and the implied volatilities that it folded. The method answers the data form of the estimator over them, because a buffer means the batch verb over its rows each time the estimate is made from the state. - A
PortfolioOptimisersCovariancewhose owncacheisnothingforwards to the estimator that it holds, which carries the state.
The first two methods return the vector of the state itself, and the next fold writes that vector in place. A caller that keeps the location across a fold must copy it, as covariance_forecast_evaluation does when it stores the forecasts.
Validation
stateis notnothing. AnArgumentErroris thrown otherwise, because there is nothing to read.statehas one of the shapes above. AnArgumentErroris thrown otherwise.
Related
PortfolioOptimisers.prior_forecast_location — Function
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, rd::ReturnsResult; kwargs...)
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::EmpiricalPrior, pr::AbstractPriorResult, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::FactorPrior, pr::AbstractPriorResult, X::MatNum, F::MatNum, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::Union{HighOrderPriorEstimator, HighOrderFactorPriorEstimator}, pr::HighOrderPrior, X::MatNum, F::Option{<:MatNum}, pnl::Option{<:AssetPanel}; kwargs...)
prior_forecast_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult; kwargs...)
prior_forecast_location(pe::HighOrderPriorEstimator, pr::HighOrderPrior; kwargs...)Read the location that the sigma of a prior is about, from the estimator and the result pr that it fitted.
The rule is the same for every family. The location is the mu that the prior would publish if its mean estimator were the centring estimator of its covariance estimator. Only a family that takes its two moments from two independent estimators, or that lifts the moments of such a prior, answers something other than pr.mu:
EmpiricalPrioranswersforecast_locationof itsceover the rows thatcesaw. The horizon arm fitsceon the log-returns, so it maps that log location through the map thatmutakes, $c_i = \exp(h c_{\log, i} + \tfrac{1}{2} h \sigma_{\log, ii}) - 1$. It reads $h \sigma_{\log, ii} = \log(1 + \sigma_{ii} / (\mu_i + 1)^2)$ back off the published pair.FactorPrioranswers the lift of the location of its factor prior, $\mathbf{M} \boldsymbol{c}_f + \boldsymbol{b}$, which it writes as $\boldsymbol{\mu} + \mathbf{M} (\boldsymbol{c}_f - \boldsymbol{\mu}_f)$.HighOrderPriorEstimatorandHighOrderFactorPriorEstimatoranswer the location of the prior that they wrap, because they publish itsmuandsigmaunchanged.- Every other family answers
pr.mu. The Black–Litterman and entropy pooling families andOpinionPoolingPriorformsigmaabout their posterior or pooledmu.CrossSectionalFactorPriorandLowDimensionEnsemblePriorpublish a forecast of the next return asmu.
For the default EmpiricalPrior, the mean estimator is the centring estimator of ce, so every family answers pr.mu. The returns data form reads the asset returns in the frame that pe fits on, through benchmark_relative_returns under the source of pe, as prior reads them. The data-less form reads the state as prior does, through prior_state_location.
Arguments
pe: Prior estimator.pr: Prior result. It is the result thatpefitted over the same rows.rd: The returns result to use.X: Asset returns matrix,observations × assets.F: Factor returns matrix,observations × factors, ornothing.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.kwargs...: Additional keyword arguments, passed to the centring estimator aspriorpasses them toce.
Returns
c::VecNum: The location,assets × 1.
Related
PortfolioOptimisers.prior_state_location — Function
prior_state_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, state; kwargs...)
prior_state_location(pe::AbstractPriorEstimator, pr::AbstractPriorResult, state::SampleBufferState; kwargs...)
prior_state_location(pe::EmpiricalPrior, pr::AbstractPriorResult, state::PriorCarryState; kwargs...)Read the location of a prior out of the state that its estimator carries, by the type of the state.
This is the data-less arm of prior_forecast_location, and it takes the routes that the data-less prior takes. Each route reads the rows of shared_series_buffer, which drops the observations of a Shared Series gap, as the data-less prior does:
- A
SampleBufferStateis a refit, so the method answers the data form over the rows, the factor rows and the masks of the buffer. - The
PriorCarryStateof anEmpiricalPrioranswers the location ofce. Acethat folds answersforecast_locationof its state. Acethat does not fold answers the data form over the carried rows, asread_memberrefits it. The horizon arm reads the log-returns of the rows, and maps the location as the data form does. - Every other state, and
nothing, answerspr.mu.
Arguments
pe: Prior estimator.pr: Prior result.state: The state thatpecarries, ornothing.kwargs...: Additional keyword arguments, passed to the data form or to the refit.
Returns
c::VecNum: The location,assets × 1.
Related
PortfolioOptimisers.carried_location — Function
carried_location(ce, X::MatNum, pnl::Option{<:AssetPanel}; kwargs...)Read the location of a covariance estimator that a fold-and-carry prior holds, out of its state or over the rows that the prior carries.
This is read_member for forecast_location. A StatsBase.CovarianceEstimator does not fold, and it is read as a GeneralCovariance over the rows, as cov reads it. The refit calls the panel form over pnl, as the data form of prior_forecast_location does over rd.pnl, so it answers a gapped sample and reads the active mask that the carried buffer records.
Arguments
ce: Covariance estimator.X: The rows that the prior carries,observations × assets.pnl: The Asset Panel thatbuffer_asset_panelrebuilds from the carried buffer, ornothing.kwargs...: Additional keyword arguments, passed to the refit alone.
Returns
c::VecNum: The location,assets × 1.
Related
PortfolioOptimisers.horizon_location — Function
horizon_location(c::VecNum, pr::AbstractPriorResult, horizon::Number)Map the log location of a covariance estimator to the arithmetic location at an investment horizon, as horizon_moments! maps the log mean.
Mathematical definition
\[\begin{align} h \sigma_{\log, ii} &= \log\!\left(1 + \frac{\sigma_{ii}}{(\mu_i + 1)^2}\right)\,, \\ c_i &= \exp\!\left(h c_{\log, i} + \tfrac{1}{2} h \sigma_{\log, ii}\right) - 1\,. \end{align}\]
Where:
- $h$: Investment horizon.
- $\mu_i$, $\sigma_{ii}$: The arithmetic mean and variance that the prior publishes.
- $\sigma_{\log, ii}$: The log-return variance that the prior fitted. The first line inverts the second closed form of
horizon_moments!on the diagonal. - $c_{\log, i}$: The log location of the covariance estimator.
When c is the log mean of the prior, $c_i = \mu_i$.
Arguments
c: The log location,assets × 1.pr: Prior result.horizon: The investment horizon.
Returns
c::VecNum: The arithmetic location,assets × 1.
Related
PortfolioOptimisers.finite_column_mean — Function
finite_column_mean(X::MatNum)Mean of each column of X over its finite entries, and NaN for a column that has none.
Related
PortfolioOptimisers.resolve_forecast_weights — Function
resolve_forecast_weights(w::Nothing, a::VecInt, sa::MatNum, N::Integer)
resolve_forecast_weights(w::VecNum, a::VecInt, sa::MatNum, N::Integer)
resolve_forecast_weights(w::VecVecNum, a::VecInt, sa::MatNum, N::Integer)Resolve the test portfolios of a step over its active subset.
nothing means inverse volatility, recomputed from the forecast of the step over the active assets, $w_i \propto 1 / \sqrt{(\hat{\mathbf{\Sigma}}_t)_{ii}}$. It keeps the most volatile asset from dominating the reading. A vector is one static portfolio, and a vector of vectors is several. The method cuts each portfolio to the active subset and rescales it to sum to one there. So the weight of a delisted asset moves to the remaining assets, in proportion to their weights. The method does not check a subset whose weights sum to zero. Such a portfolio has no scale, and its diagnostics are not finite.
Arguments
w: The test portfolios on the full universe, ornothing.a: The active subset.sa: The forecast on the active subset.N: The width of the universe.
Validation
- Every portfolio has
Nentries. ADimensionMismatchis thrown otherwise.
Returns
ws::Vector{<:VecNum}: One weights vector per portfolio, each overa.
Related
PortfolioOptimisers.step_implied_volatility_kwargs — Function
step_implied_volatility_kwargs(state::SampleBufferState, rd::ReturnsResult)
step_implied_volatility_kwargs(state, rd::ReturnsResult)Give the implied volatilities of a step to the state that records them, as keywords of partial_fit!.
A SampleBufferState records iv and ivpa beside its rows, so it takes rd.iv and rd.ivpa. Any other state is an exact fold, which keeps numbers alone, and it takes nothing. The type of the state selects the method.
Arguments
state: The state that the estimator carries.rd: The returns data of the step.
Returns
kwargs::NamedTuple:ivandivpafor a buffer, or an emptyNamedTuple.
Related
PortfolioOptimisers.online_entry_state — Method
online_entry_state(o::Online)Name the state that a wrapped estimator carries at the entry of the online arm of the fold loop, or answer nothing.
This is the Online method of online_entry_state, for a wrapper at the root of covariance_forecast_evaluation. A wrapper holds no cache of its own, so the walk enters the estimator that it wraps and puts est. before the path that it finds there. For a covariance estimator that the caller folded before the call, the path is est.cache. The generic walk through estimator_fields would give the same answer. This method gives it without a request for a field that the wrapper does not have.
Related
PortfolioOptimisers.advance_previous_fold — Method
advance_previous_fold(pws, prev, fold::Tuple{CovarianceForecastStepResult, Option{<:MatNum}, Option{<:VecNum}})Hand the next fold what the last fold held, when the result of the fold is a step of a covariance forecast evaluation and not a prediction.
The fold of covariance_forecast_evaluation answers the tuple of its CovarianceForecastStepResult, its stored forecast and its stored location. The step carries no weights, so the method threads nothing and returns prev unchanged. online_folds reaches this method under covariance_forecast_evaluation.
Related
PortfolioOptimisers.forecast_moments — Function
forecast_moments(ce::AbstractCovarianceEstimator, rd::ReturnsResult, train_idx::VecInt)
forecast_moments(ce::AbstractCovarianceEstimator, rd::ReturnsResult, ::Nothing)
forecast_moments(pe::AbstractPriorEstimator, rd::ReturnsResult, train_idx::VecInt)
forecast_moments(pe::AbstractPriorEstimator, rd::ReturnsResult, ::Nothing)Read the forecast of a fold and the location that it is about, by whether the fold carries a training window.
These are the two arms of the callback of covariance_forecast_evaluation. Dispatch on the train of the fold chooses the arm, as it does for the optimiser in fit_fold_result. A training window means a refit, which every batch fold runs. nothing means that the estimator holds its window, so the fold reads the moments out of the state. The method reads a prior once for both moments.
Algorithm
- With a training window, take the view of
rdon its rows, givingrdt. Fit a covariance estimator through the Asset Panel methods of the moment verbs,cov(ce, rdt.X, rdt.pnl)andforecast_location(ce, rdt.X, rdt.pnl). Both calls takeiv = rdt.ivandivpa = rdt.ivpa, aspriorgives them to its members, so anImpliedVolatilityreads its surface and every other estimator ignores it. Fit a prior once over the view, givingpr. - With
nothing, readcov(ce)andforecast_location(ce)out of the state of a covariance estimator, orpr = prior(pe)out of the state of a prior. - Return the forecast and the location. For a prior they are
pr.sigmaandprior_forecast_locationofpr, which reads the centre ofpr.sigmawithout a second fit.
Arguments
ce: Covariance estimator.pe: Prior estimator.rd: The returns data.train_idx: The fold's training window, ornothingfor a stepped estimator.
Returns
sigma::MatNum: The forecast,assets × assets.c::VecNum: The location the forecast is about,assets × 1.
Related