Covariance forecast summary
Three functions read a CovarianceForecastEvaluationResult. covariance_forecast_summary returns a result with one entry per evaluation in each column, so you can put two forecasts side by side. covariance_forecast_compare tests the difference between the losses of two forecasts at each step, with a Diebold-Mariano-West statistic whose p-value comes from a test of the mean, WildBootstrapMeanTest by default. The loss compares a forecast with a proxy of the true covariance, so its level alone does not say how good a forecast is, but the difference between two forecasts does. covariance_forecast_portfolio scores the stored forecasts again on a new test portfolio.
The summary and the calibration figure weight each step in the mean of a ratio by an AbstractStepWeighting. DofStepWeighting, the default, weights a step by its degrees of freedom, which gives the mean of least variance. EqualStepWeighting gives the plain mean over the steps.
PortfolioOptimisers.AbstractStepWeighting — Type
abstract type AbstractStepWeighting <: AbstractAlgorithmAbstract supertype for the step weighting of a covariance forecast evaluation, the rule that weights each scored step in the mean of a calibration ratio over the steps.
covariance_forecast_summary and plot_covariance_calibration read the rule for the Mahalanobis ratio and for the diagonal ratio. Under a calibrated Gaussian forecast every rule gives a mean whose expected value is one, so the rules differ in the variance of the mean, not in its target. DofStepWeighting gives the least variance, and it is the default. EqualStepWeighting gives the plain mean over the steps. The two rules agree when every step has the same degrees of freedom.
Interfaces
A step weighting is a marker for dispatch, and it holds no data. A new step weighting needs a method of covariance_step_weights, which returns one non-negative weight per step. The mean and the half-width of its Gaussian band follow from the weights, as covariance_step_weights states.
Related
PortfolioOptimisers.DofStepWeighting — Type
struct DofStepWeighting <: AbstractStepWeightingWeights each scored step by the degrees of freedom of its ratio under a Gaussian null. This is the default step weighting.
The ratio of step $t$ is $\chi^2_{\nu_t} / \nu_t$ under a calibrated Gaussian forecast, so its variance is $2 / \nu_t$. The weights $\nu_t$ are the inverse variances, so the weighted mean is the unbiased mean of least variance. It is itself $\chi^2_{\sum_t \nu_t} / \sum_t \nu_t$, the ratio of the pooled statistic. A date walk-forward whose folds differ in length, a listing and a delisting all move the degrees of freedom of a step, and each step then counts by the data it scored.
Constructors
DofStepWeighting() -> DofStepWeightingExamples
julia> PortfolioOptimisers.covariance_step_weights(DofStepWeighting(), [8, 8, 7])3-element Vector{Int64}: 8 8 7Related
PortfolioOptimisers.EqualStepWeighting — Type
struct EqualStepWeighting <: AbstractStepWeightingGives each scored step the same weight, so the mean of a ratio is its plain mean over the steps.
The plain mean is unbiased under a calibrated Gaussian forecast, as every mean with fixed weights is. Its variance is $2 \sum_t \nu_t^{-1} / M^2$, which is at least that of DofStepWeighting, and it has no chi-squared law. The two rules agree when every step has the same horizon and the same active count. The rule reads each step as one observation of the calibration, whatever data the step scored.
Constructors
EqualStepWeighting() -> EqualStepWeightingExamples
julia> PortfolioOptimisers.covariance_step_weights(EqualStepWeighting(), [8, 8, 7])3-element Vector{Int64}: 1 1 1Related
PortfolioOptimisers.covariance_step_weights — Function
covariance_step_weights(sw::DofStepWeighting, nu::AbstractVector{<:Integer})
covariance_step_weights(sw::EqualStepWeighting, nu::AbstractVector{<:Integer})Return the weight of each step in the mean of a calibration ratio of a covariance forecast evaluation.
covariance_forecast_summary and plot_covariance_calibration read the weights. The mean of a ratio and the half-width of its Gaussian band follow from them.
Mathematical definition
\[\begin{align} \bar{r} &= \frac{\sum_{t=1}^{M} \omega_t\, r_t}{\sum_{t=1}^{M} \omega_t}\,, \\ \bar{r} &\in 1 \pm \frac{z_{\alpha/2}}{\sum_{t=1}^{M} \omega_t} \sqrt{2 \sum_{t=1}^{M} \frac{\omega_t^2}{\nu_t}}\,. \end{align}\]
Where:
- $\bar{r}$: Mean of the ratio over the steps.
- $\omega_t$: Weight of step $t$, from this function.
- $r_t$: Ratio of step $t$.
- $\nu_t$: Degrees of freedom of the ratio of step $t$ under a Gaussian null.
- $M$: Steps of the walk-forward, the number of forecasts a run scores.
- $z_{\alpha/2}$: Upper $\alpha / 2$ quantile of the standard normal distribution.
The ratio of step $t$ has variance $2 / \nu_t$ under a calibrated Gaussian forecast, and the steps are independent, so the band is the Gaussian band of the mean with its own variance. Under DofStepWeighting the half-width is $z_{\alpha/2} \sqrt{2 / \sum_t \nu_t}$.
Algorithm
The method that Julia selects is the algorithm.
DofStepWeighting: the weight of each step is its degrees of freedom,nuitself.EqualStepWeighting: the weight of each step is one, in the element type ofnu.
Arguments
sw: The step weighting.nu: The degrees of freedom of each step, fromtarget_doffor the Mahalanobis ratio and fromtarget_step_doffor the diagonal ratio.
Returns
omega::AbstractVector: One weight per step.
Related
PortfolioOptimisers.CovarianceForecastSummaryResult — Type
struct CovarianceForecastSummaryResult{__T_names, __T_mahalanobis_mean, __T_mahalanobis_median, __T_mahalanobis_p5, __T_mahalanobis_p95, __T_mahalanobis_band_lo, __T_mahalanobis_band_hi, __T_diagonal_mean, __T_diagonal_median, __T_diagonal_p5, __T_diagonal_p95, __T_diagonal_band_lo, __T_diagonal_band_hi, __T_bias_statistic, __T_bias_p5, __T_bias_p25, __T_bias_p75, __T_bias_p95, __T_qlike_mean, __T_frobenius_mean, __T_portfolio_qlike_mean, __T_levels, __T_exceedance, __T_n_steps, __T_n_portfolios, __T_alpha, __T_step_weighting} <: AbstractResultThe headline statistics of one or more covariance forecast evaluations, one entry per evaluation.
CovarianceForecastSummaryResult is what covariance_forecast_summary returns. The shape follows the type of the input, as for PerformanceSummaryResult. The method for one evaluation gives scalar fields, equal to entry 1 of the vector method on [cfer], and exceedance is then a vector over the levels. The vector method gives one entry per evaluation, so two forecasts side by side are the length-2 case. So, like ForecastSummaryResult for the return forecasts, it needs no separate comparison type. Breaking: a summary of one evaluation held length-1 vectors before. The summary reads every column from the per-step diagnostics of a CovarianceForecastEvaluationResult. The bands are Gaussian references, and the summary does not correct them for heavy tails.
Mathematical definition
\[\begin{align} \bar{m} &= \frac{\sum_{t=1}^{M} \omega_t\, m_t}{\sum_{t=1}^{M} \omega_t}\,, \quad \nu_t = \operatorname{dof}(N_t, h_t)\,, \\ \bar{d} &= \frac{\sum_{t=1}^{M} \omega'_t\, \bar{d}_t}{\sum_{t=1}^{M} \omega'_t}\,, \quad \nu'_t = \operatorname{dof}'(h_t)\,, \\ \bar{m} &\in 1 \pm \frac{z_{\alpha/2}}{\sum_{t} \omega_t} \sqrt{2 \sum_{t=1}^{M} \frac{\omega_t^2}{\nu_t}}\,, \quad \bar{d} \in 1 \pm \frac{z_{\alpha/2}}{\sum_{t} \omega'_t} \sqrt{2 \sum_{t=1}^{M} \frac{\omega_t'^2}{\nu'_t}}\,, \\ B &= \sqrt{\frac{1}{M - 1} \sum_{t=1}^{M} \left(b_t - \bar{b}\right)^2}\,, \\ e_q &= \frac{1}{M} \sum_{t=1}^{M} \mathbb{1}\left[\nu_t\, m_t > \chi^2_{\nu_t}(q)\right]\,. \end{align}\]
Where:
- $\bar{m}$: Mahalanobis ratio over the walk-forward.
- $m_t$: Mahalanobis ratio of step $t$.
- $\omega_t$, $\omega'_t$: Weight of step $t$ in the Mahalanobis mean and in the diagonal mean, from
covariance_step_weightson $\nu_t$ and on $\nu'_t$. - $\nu_t$: Degrees of freedom of the Mahalanobis statistic of step $t$ under a Gaussian null, $N_t h_t$ for the realised covariance and $N_t$ for the horizon return.
- $N_t$: Number of active assets at step $t$.
- $h$: Horizon of a step, the number of observations the forecast is judged on.
- $M$: Steps of the walk-forward, the number of forecasts a run scores.
- $\bar{d}$: Diagonal ratio over the walk-forward.
- $\bar{d}_t$: Mean of the diagonal ratio over the active assets of step $t$.
- $\nu'_t$: Degrees of freedom of the ratio of one asset at step $t$ under a Gaussian null, $h_t$ for the realised covariance and $1$ for the horizon return.
- $z_{\alpha/2}$: Upper $\alpha / 2$ quantile of the standard normal distribution.
- $B$: Bias statistic of a test portfolio, the sample standard deviation of its standardised returns.
- $b_t$: Standardised return of the test portfolio at step $t$.
- $\bar{b}$: Mean of the standardised returns of the test portfolio.
- $e_q$: Exceedance rate at level $q$, the share of steps whose Mahalanobis statistic exceeds the chi-squared quantile of that level.
- $\chi^2_{\nu}(q)$: Quantile at level $q$ of the chi-squared distribution with $\nu$ degrees of freedom.
Under a Gaussian null, with the forecast correct and the whitened returns independent, $\mathbb{E}[\bar{m}] = 1$ under every step weighting, and the band is the Gaussian band of the mean at level $1 - \alpha$. Under DofStepWeighting, the default, $\omega_t = \nu_t$, so $\bar{m} \sum_t \nu_t \sim \chi^2_{\sum_t \nu_t}$ and the half-width is $z_{\alpha/2} \sqrt{2 / \sum_t \nu_t}$. That ratio of sums is the plain mean when every step has the same $N_t$ and $h_t$. Under EqualStepWeighting, $\omega_t = 1$, which gives the plain mean and its own wider band. The band on $\bar{d}$ is the band of the ratio of one asset. A mean over assets has at most the variance of the ratio of one asset, so the band is conservative, and it is exact only when the ratios of the assets are perfectly correlated.
Let $\kappa$ be the fourth moment of one whitened coordinate of the realised quantity, with the Gaussian value three. Under the realised covariance the coordinate is that of one return, and under the horizon return it is that of the horizon return divided by $\sqrt{h_t}$. With independent coordinates, the variance of $\bar{m}$ is $(\kappa - 1) / 2$ times its Gaussian value, so the band widens by $\sqrt{(\kappa - 1) / 2}$. A horizon return sums $h_t$ returns, so for independent and identically distributed returns its excess of $\kappa$ over three is that of one return divided by $h_t$.
The median of $\chi^2_{\nu} / \nu$ lies below one, near $(1 - 2 / (9 \nu))^3 \approx 1 - 2 / (3 \nu)$, so compare a median with that value and not with one. $B = 1$ under a calibrated forecast, and $B > 1$ when the forecast under-predicts the risk of the portfolio. $B$ needs two steps. $e_q$ is $1 - q$ under the Gaussian null, and it rises with heavy tails as well as with a misspecified forecast.
Fields
names: Name of each evaluation, one entry per evaluation.
mahalanobis_mean: Mahalanobis ratio over the walk-forward, the mean of the per-step ratios weighted bystep_weighting. One entry per evaluation. The target is one.
mahalanobis_median: Median of the per-step Mahalanobis ratio. One entry per evaluation. Under a Gaussian null its target lies below one.
mahalanobis_p5: Fifth percentile of the per-step Mahalanobis ratio. One entry per evaluation.
mahalanobis_p95: Ninety-fifth percentile of the per-step Mahalanobis ratio. One entry per evaluation.
mahalanobis_band_lo: Lower end of the Gaussian band on the Mahalanobis ratio at levelalpha. One entry per evaluation.
mahalanobis_band_hi: Upper end of the Gaussian band on the Mahalanobis ratio at levelalpha. One entry per evaluation.
diagonal_mean: Diagonal ratio over the walk-forward, the mean over the steps of the per-step mean over the active assets, weighted bystep_weighting. One entry per evaluation. The target is one.
diagonal_median: Median of the per-step diagonal ratio, which is the mean over the active assets of the step. One entry per evaluation.
diagonal_p5: Fifth percentile of the per-step diagonal ratio. One entry per evaluation.
diagonal_p95: Ninety-fifth percentile of the per-step diagonal ratio. One entry per evaluation.
diagonal_band_lo: Lower end of the Gaussian band on the diagonal ratio of one asset at levelalpha. One entry per evaluation.
diagonal_band_hi: Upper end of the Gaussian band on the diagonal ratio of one asset at levelalpha. One entry per evaluation.
bias_statistic: Bias statistic, the median over the test portfolios of the sample standard deviation of the standardised returns of each portfolio. One entry per evaluation. The target is one. It isNaNfor an evaluation with one step, and for an evaluation with aNaNstandardised return.
bias_p5: Fifth percentile of the bias statistic over the test portfolios. One entry per evaluation,NaNwherebias_statisticis.
bias_p25: Twenty-fifth percentile of the bias statistic over the test portfolios. One entry per evaluation,NaNwherebias_statisticis.
bias_p75: Seventy-fifth percentile of the bias statistic over the test portfolios. One entry per evaluation,NaNwherebias_statisticis.
bias_p95: Ninety-fifth percentile of the bias statistic over the test portfolios. One entry per evaluation,NaNwherebias_statisticis.
qlike_mean: Mean QLIKE loss over the steps. One entry per evaluation. Only a difference between two evaluations says which forecast is better.
frobenius_mean: Mean Frobenius loss over the steps. One entry per evaluation. Only a difference between two evaluations says which forecast is better.
portfolio_qlike_mean: Median over the test portfolios of the mean portfolio QLIKE loss of each portfolio. One entry per evaluation.
levels: Confidence levels of the exceedance rates, as the caller gave them.
exceedance: Exceedance rate of the per-step Mahalanobis statistic against the chi-squared quantile of each level,evaluations × levels. The target is one less the level.
n_steps: Number of scored steps of each evaluation, the steps with at least one active asset. One entry per evaluation.
n_portfolios: Number of test portfolios of each evaluation. One entry per evaluation.
alpha: Level of the Gaussian bands.
step_weighting:AbstractStepWeightingthat weights each scored step in the mean of a calibration ratio and in its Gaussian band.DofStepWeighting, the default, weights a step by the degrees of freedom of its ratio, andEqualStepWeightinggives every step the same weight.
Constructors
CovarianceForecastSummaryResult( names, mahalanobis_mean, mahalanobis_median, mahalanobis_p5, mahalanobis_p95, mahalanobis_band_lo, mahalanobis_band_hi, diagonal_mean, diagonal_median, diagonal_p5, diagonal_p95, diagonal_band_lo, diagonal_band_hi, bias_statistic, bias_p5, bias_p25, bias_p75, bias_p95, qlike_mean, frobenius_mean, portfolio_qlike_mean, levels, exceedance, n_steps, n_portfolios, alpha, step_weighting) -> CovarianceForecastSummaryResultCovarianceForecastSummaryResult(; names, mahalanobis_mean, mahalanobis_median, mahalanobis_p5, mahalanobis_p95, mahalanobis_band_lo, mahalanobis_band_hi, diagonal_mean, diagonal_median, diagonal_p5, diagonal_p95, diagonal_band_lo, diagonal_band_hi, bias_statistic, bias_p5, bias_p25, bias_p75, bias_p95, qlike_mean, frobenius_mean, portfolio_qlike_mean, levels::Union{Tuple{Vararg{Real}}, AbstractVector{<:Real}}, exceedance, n_steps::Union{Integer, VecInt}, n_portfolios::Union{Integer, VecInt}, alpha::Number, step_weighting::AbstractStepWeighting) -> CovarianceForecastSummaryResultArguments correspond to the struct's fields, in the order they are declared. covariance_forecast_summary builds the type, and a caller reads it. The positional constructor checks the values with assert_covariance_summary_fields. The keyword constructor takes each field by name, with no default, and calls it.
Validation
- The rules of
assert_covariance_summary_fields: the levels are valid, every field of an entry has the shape ofn_steps, each evaluation scored a step, each exceedance rate is a rate, and each band is ordered. A breach raises aDomainErroror aDimensionMismatch. - The keyword constructor refuses a value of another type than its signature states with a
TypeError. The positional constructor checks only the types of its signature: the generic positional method that@concreteadds takes values of any other type, such as aFloat64n_steps, with no check.
Related
PortfolioOptimisers.CovarianceForecastComparisonResult — Type
struct CovarianceForecastComparisonResult{__T_names, __T_mean_difference, __T_t_stat, __T_p_value, __T_mean_ci_lower, __T_mean_ci_upper, __T_n_steps} <: AbstractResultThe Diebold–Mariano–West comparison of two covariance forecasts, one row per loss.
CovarianceForecastComparisonResult is what covariance_forecast_compare returns. Each column has one entry per loss compared. The losses are the whole-matrix QLIKE, the Frobenius loss, and the portfolio QLIKE of each test portfolio. A positive mean difference says that the first forecast lost more. The t-statistic, the p-value and the interval come from the Mean Test of the comparison, so read the p-value or the interval to judge the evidence: the t-statistic of WildBootstrapMeanTest is not on the scale of the normal law.
Fields
names: Name of each loss compared:"qlike","frobenius", then"portfolio_qlike_k"for each test portfoliok.
mean_difference: Mean over the steps of the loss of the first forecast less that of the second. One entry per loss.
t_stat: t-statistic of the mean difference, the mean over its long-run standard error. One entry per loss.
p_value: Two-sided p-value of the t-statistic under equal expected loss, against the law that the Mean Test reads. One entry per loss.
mean_ci_lower: Lower bound of the confidence interval of the mean difference, at the level that the Mean Test sets. One entry per loss.
mean_ci_upper: Upper bound of the confidence interval of the mean difference, at the level that the Mean Test sets. One entry per loss.
n_steps: Number of steps compared, the steps that both evaluations scored.
Constructors
CovarianceForecastComparisonResult( names, mean_difference, t_stat, p_value, mean_ci_lower, mean_ci_upper, n_steps) -> CovarianceForecastComparisonResultArguments correspond to the struct's fields, in the order they are declared. covariance_forecast_compare builds the type, and a caller reads it. The type has no keyword constructor, and it validates nothing.
Related
PortfolioOptimisers.covariance_forecast_summary — Function
covariance_forecast_summary(cfers::AbstractVector{<:CovarianceForecastEvaluationResult};
names::Option{<:AbstractVector} = nothing,
alpha::Real = 0.05,
levels::Union{Tuple{Vararg{Real}}, AbstractVector{<:Real}} = (0.95, 0.99),
step_weighting::AbstractStepWeighting = DofStepWeighting())
-> CovarianceForecastSummaryResult
covariance_forecast_summary(cfer::CovarianceForecastEvaluationResult; kwargs...)Summarise one or more covariance forecast evaluations, one entry per evaluation.
For each evaluation, the summary gives the mean, the median and the tail percentiles of the two calibration ratios, the Gaussian band on each mean, the bias statistic of the test portfolios with its percentiles over the portfolios, the mean of each loss, and the exceedance rate of the Mahalanobis statistic at each level. The method for one Result gives scalar fields, through covariance_summary_entry, and the vector method gives one entry per evaluation, also for a vector of one. Breaking: the method for one Result gave length-1 vectors before.
The mean of a ratio weights each step by step_weighting. The default, DofStepWeighting, gives a ratio of sums, which weights each step by its degrees of freedom under the target of the evaluation (target_dof, target_step_dof). A date walk-forward whose folds differ in length then weights each step by its length. A listing or a delisting also changes the weight of a step of the Mahalanobis ratio, because its degrees of freedom count the active assets. So the Mahalanobis mean is the plain mean only when every step has the same horizon and the same active count, and the diagonal mean is the plain mean when every step has the same horizon. EqualStepWeighting gives the plain mean in every case, with the band of the plain mean.
Algorithm
- For each evaluation, keep the scored steps, the steps with
n_valid > 0. An unscored step has no diagnostic, so no statistic reads it. Find the degrees of freedom of each step for the Mahalanobis ratio throughtarget_dof, and for the ratio of one asset throughtarget_step_dof. Find the weight of each step from them throughcovariance_step_weights. - Reduce the diagonal ratio of each step to its mean over the active assets.
- For each ratio, take the weighted mean, the median, and the fifth and ninety-fifth percentiles through
summary_quantile. - Find the half-width of the Gaussian band of each mean at
alpha, from the weights and the degrees of freedom, throughcovariance_ratio_band. - Take the sample standard deviation of the standardised returns of each test portfolio. Take the median and the percentiles of these over the portfolios, through
summary_quantile. - Take the mean of each loss over the steps, and the median over the portfolios of the mean portfolio QLIKE.
- For each level, find the share of the steps whose Mahalanobis statistic exceeds the chi-squared quantile of that level.
Arguments
cfers: The evaluations to summarise.names: A name per evaluation, ornothingfor"forecast_1","forecast_2", ….alpha: Level of the Gaussian bands,1 - alphacoverage.levels: Confidence levels of the exceedance rates, a tuple or a vector of real numbers.step_weighting:AbstractStepWeightingthat weights each scored step in the mean of a calibration ratio and in its Gaussian band.DofStepWeighting, the default, weights a step by the degrees of freedom of its ratio, andEqualStepWeightinggives every step the same weight.
Validation
cfersis not empty. AnIsEmptyErroris thrown otherwise.- Every evaluation scored at least one step. An
IsEmptyErroris thrown otherwise. nameshas one entry per evaluation when given. ADimensionMismatchis thrown otherwise.0 < alpha < 1, and every level lies in(0, 1). ADomainErroris thrown otherwise.
Returns
summary::CovarianceForecastSummaryResult: The columnar summary. The bias columns of an evaluation with one step areNaN, because a sample standard deviation needs two steps.
Related
PortfolioOptimisers.covariance_forecast_compare — Function
covariance_forecast_compare(a::CovarianceForecastEvaluationResult, b::CovarianceForecastEvaluationResult;
mt::AbstractMeanTestEstimator = WildBootstrapMeanTest())
-> CovarianceForecastComparisonResultTest whether two covariance forecasts differ in expected loss, one test per loss.
The level of a loss on a proxy is not a calibration reading, so two mean losses side by side do not say which forecast is better. The comparison tests the difference of the losses instead. For each loss the per-step difference is a series, and the Mean Test mt compares its mean with zero through a long-run standard error. A step that either evaluation did not score has no loss. It is missing from the series, and the lag between two steps is their distance in the walk-forward, as loss_difference_test states.
Mathematical definition
\[\begin{align} \delta_t &= L^{A}_t - L^{B}_t\,, \quad t \in \mathcal{T}\,, \\ \bar{\delta} &= \frac{1}{M} \sum_{t \in \mathcal{T}} \delta_t\,, \\ t_{\delta} &= \frac{\bar{\delta}}{s_{\delta}}\,. \end{align}\]
Where:
- $\delta_t$: Difference of the losses of forecasts $A$ and $B$ at step $t$.
- $L^{A}_t$, $L^{B}_t$: Loss of forecast $A$ and of forecast $B$ at step $t$.
- $\mathcal{T}$: The steps that both evaluations scored.
- $M$: Steps of the walk-forward, the number of forecasts a run scores. Here it counts the steps in $\mathcal{T}$.
- $\bar{\delta}$: Mean loss difference over the walk-forward.
- $s_{\delta}$: Long-run standard error of $\bar{\delta}$, which the Mean Test
mtdefines. - $t_{\delta}$: t-statistic of the difference. The p-value and the interval of
mtread it.
Section 1.1 of [227] states the test of equal expected loss as a test of the mean of the loss difference against zero. Its rule takes $k - 1$ lags under the uniform window for a forecast $k$ entries of the loss series ahead. One step of a walk-forward is one entry, and the test windows do not overlap, so the rule takes zero lags. OverlapMeanTest gives that statistic, with the divisor and the Student law that its law sets.
The rule of zero lags holds only when the differences of two steps are uncorrelated. Two steps can share training rows, and the losses of a covariance forecast cluster with the volatility, so the differences can be correlated over many steps. A fixed number of Bartlett lags does not repair that: the Bartlett weight of a lag is below one, so at a fixed number of lags the estimate does not converge to the long-run variance. The default WildBootstrapMeanTest reads the dependence at every lag up to a bandwidth that is a fixed fraction of the span of the steps. It reads the p-value off a wild bootstrap of the statistic, so a change of the volatility over time does not move the level of the test.
Under equal expected loss, the law that mt reads is the null law of $t_{\delta}$ when the estimates that the forecasts read do not move its variance. Theorem 4.1(a) of [228] states two such cases: the steps are few next to the rows of the first training window, or the expected gradient of the loss difference in the estimates is zero. Otherwise Theorem 4.1(b) of [228] adds terms to the variance, and $s_{\delta}$ omits them. [228] states the theorem for a training window that grows, and Footnote 3 sends the rolling window to an earlier version of that work. A positive value says that forecast $A$ lost more than forecast $B$.
Two degenerate differences have a stated answer:
- Two identical forecasts have $\bar{\delta} = 0$ and no long-run variance, so their t-statistic, p-value and interval are
NaN. - A step that both evaluations scored and whose loss is not finite makes the mean difference of that loss not finite, and its test
NaN.
Arguments
a: The first evaluation.b: The second evaluation.mt: The Mean Test of the loss difference. Each step is one entry of the series, so the test reads a window of one.
Validation
a.dates == b.datesanda.horizon == b.horizon. AnArgumentErroris thrown otherwise, because a per-step difference needs the same steps on both sides.a.x_src == b.x_src. AnArgumentErroris thrown otherwise, because a forecast ofX - Band a forecast ofXare scored on different returns, so their losses do not compare.aandbcarry the same number of test portfolios. ADimensionMismatchis thrown otherwise.- At least one step is scored by both evaluations. An
IsEmptyErroris thrown otherwise.
Returns
cmp::CovarianceForecastComparisonResult: One row per loss.
Related
CovarianceForecastComparisonResultCovarianceForecastEvaluationResultcovariance_forecast_evaluationcovariance_forecast_summaryWildBootstrapMeanTestOverlapMeanTestloss_difference_testlog_wealth_regret
References
- [227] F. X. Diebold and R. S. Mariano. Comparing predictive accuracy. Journal of Business & Economic Statistics 13, 253–263 (1995). Section 1.1.
- [228] K. D. West. Asymptotic inference about predictive ability. Econometrica 64, 1067–1084 (1996). Section 2, Theorem 4.1.
- [17] M. Demetrescu, C. Hanck and R. Kruse-Becher. Robust fixed-b inference in the presence of time-varying volatility. Econometrics and Statistics 37, 154–173 (2026). Section 3.1 and Section 4.
PortfolioOptimisers.CovarianceForecastPortfolioResult — Type
struct CovarianceForecastPortfolioResult{__T_standardised_return, __T_portfolio_qlike} <: AbstractResultThe portfolio diagnostics of the stored forecasts of an evaluation, scored again on a new test portfolio by covariance_forecast_portfolio.
Each matrix holds one row for each step of the evaluation and one column for each test portfolio, as the two portfolio fields of a CovarianceForecastEvaluationResult do.
Fields
standardised_return: Standardised return of each test portfolio at each step,steps × portfolios.
portfolio_qlike: QLIKE loss of the variance of each test portfolio at each step,steps × portfolios.
Constructors
CovarianceForecastPortfolioResult(standardised_return::MatNum, portfolio_qlike::MatNum)CovarianceForecastPortfolioResult(; standardised_return::MatNum, portfolio_qlike::MatNum)The positional constructor checks the values. The keyword constructor takes each field by name, with no default, and calls it.
Validation
standardised_returnandportfolio_qlikehave the same size,steps × portfolios. Otherwise, the constructor raises aDimensionMismatch.- The keyword constructor refuses a value that is not a matrix of numbers with a
TypeError. The positional constructor checks only two matrices: the generic positional method that@concreteadds takes values of any other type, such as two vectors, with no check.
Related
PortfolioOptimisers.covariance_forecast_portfolio — Function
covariance_forecast_portfolio(cfer::CovarianceForecastEvaluationResult, rd::ReturnsResult,
w::Option{<:VecNum_VecVecNum}) -> CovarianceForecastPortfolioResultScore the stored forecasts of an evaluation again, on a new test portfolio.
With store_forecasts = true the Result holds the forecast of every step and the location that the forecast was centred on, so this verb scores a new portfolio without a new run of the loop. The own w of the evaluation reproduces its columns.
The test rows are read in the frame of the evaluation, cfer.x_src, because a forecast of X - B is scored on the rows of X - B. The verb takes no x_src of its own, so it cannot score the forecasts of one frame on the rows of another.
Algorithm
- Drop the observations with a gap in a Shared Series with
shared_series_viewundercfer.x_src, as the evaluation dropped them before it split the folds. - Read the returns in the frame of the evaluation with
benchmark_relative_returnsundercfer.x_src, givingX. - For each step, read the stored forecast and location, and the test rows
X[cfer.test_idx[i], :]. - Score them on
wwith the target of the evaluation, throughcovariance_forecast_step. - Put the standardised returns and the portfolio QLIKE losses of the step in its row of each matrix.
Arguments
cfer: The evaluation, run withstore_forecasts = true.rd: The returns result to use. It must be the returns data that the evaluation ran on, because each step reads its rows throughcfer.test_idx, which counts the observations that the evaluation kept.w: The test portfolios on the full universe, ascovariance_forecast_evaluationtakes them.
Validation
cfer.sigmais notnothing. AnArgumentErroris thrown otherwise, which namesstore_forecasts.rd.Xis notnothing. AnIsNothingErroris thrown otherwise.- The rules of
shared_series_viewand ofbenchmark_relative_returnsundercfer.x_src.
Returns
proj::CovarianceForecastPortfolioResult:standardised_returnandportfolio_qlike, eachsteps × portfolios.
Related
References
- [17]
- M. Demetrescu, C. Hanck and R. Kruse-Becher. Robust Fixed-b Inference in the Presence of Time-Varying Volatility. Econometrics and Statistics 37, 154–173 (2026).
- [227]
- F. X. Diebold and R. S. Mariano. Comparing predictive accuracy. Journal of Business & Economic Statistics 13, 253–263 (1995).
- [228]
- K. D. West. Asymptotic inference about predictive ability. Econometrica 64, 1067–1084 (1996).