Regime adjusted exp weighted covariance (b): private API
Functions
PortfolioOptimisers.regime_target_statistic — Function
regime_target_statistic(
target::RegimeAdjustedTarget,
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Computes the regime statistic of one observation for a target, from the covariance block that stands before it.
A target that has no method of its own reads its regime_statistic of the block that regime_covariance_block returns, and takes no bias correction.
Arguments
target::RegimeAdjustedTarget: Regime-adjustment target.cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Current centred returns vector of every asset.idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.
Returns
stats::Option{<:VecNum}: One statistic per calibration direction, ornothingwhere the observation takes no regime update.
Related
regime_target_statistic(
_::DiagonalTarget,
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Computes the regime statistic of the diagonal target, with each term divided by the bias of its own estimated variance, and the sum divided by the factor of its law.
Each term $u_{i}^{2} / \hat{C}_{ii}$ reads one estimated variance of $K_{i}$ observations, so its mean is $\mathbb{E}[Q^{-1}]$ at $K_{i}$, from regime_bias_table, and not one. Each term is divided by that factor whatever the method, so the sum has the mean $n$ at every correlation of the assets.
The mean of the sum is $n$ at every correlation, but its root and its log are not: they read the law of the sum, which the correlation of the assets and the noise of each estimate set. So the sum is then divided by diagonal_law_factor of the method, which reads the moments of each term from the table of RegimeTermMoments.
Arguments
::DiagonalTarget: Diagonal regime-adjustment target.cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Current centred returns vector of every asset.idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.
Returns
stats::VecNum: One statistic.
Related
regime_target_statistic(
target::PortfolioTarget,
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Computes the regime statistic of the portfolio target, divided by the bias of the estimated variance of each direction.
A direction reads one estimated variance $p^{\top} \hat{C} p$, so its statistic is biased by the factor of regime_bias_table that the method reads, at $K$, the smallest count of observations among the contributing assets. That factor is exact for a fixed direction on one shared history. On the separate correlation path the variance of a direction mixes the variances at decay with the correlations at cor_decay, so no one table holds it, and portfolio_regime_bias gives each direction its own factor from the two tables. A HAC estimate reads the table of its banded weight matrix, which has about twice the excess of the plain weights: at two lags the squared multiplier of an equal-weight direction is 1.015 over 8 seeds, within its standard error of 0.013, from 1.165 raw.
The default inverse-volatility direction (w = nothing) is built from the same estimate, so the direction and the error of the estimate are correlated. Its statistic is also divided by $1 + \Delta$, the second-order excess of inverse_volatility_bias, on the estimated correlation of the block. At 12 assets and a half-life of 10 the squared multiplier on iid Normal returns is 1.134 raw, 1.058 after the factor alone and 1.001 after both. The excess is the same for the three methods to first order. What remains is of the next order, about $9 s^{2}$ with $s$ the sum of the squared weights, so it is largest in the warm-up rows: at 12 assets, a half-life of 10 and $R = I$, 1.34 at $K = 5$, 1.08 at $K = 10$, 1.02 at $K = 20$ and 1.004 in the steady state. At the defaults the first row enters at $K$ equal to the half-life, where the rest is 1.006 at a half-life of 40.
The block passes through indefinite_block with the rule of target.indefinite first, and a row whose block the rule refuses is not scored. The block is young while regime_bias_open is false for the count of contributing assets, at $K$, on the decay of the correlation structure and with the HAC lags of the estimator: the gate of MahalanobisTarget for the same assets.
Arguments
target::PortfolioTarget: Portfolio regime-adjustment target.cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Current centred returns vector of every asset.idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.
Returns
stats::Option{<:VecNum}: One statistic per direction, ornothingwhere no direction keeps a positive weight, or where the rule oftarget.indefiniterefuses the block.
Related
regime_target_statistic(
target::MahalanobisTarget,
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Computes the squared Mahalanobis distance of one observation, divided by the bias of the estimated block that the regime method reads, where ce.debias is ExactDebias() or LawDebias().
Algorithm
- Where
ce.debiasisnothing, the factor is one. Else take $K$, the smallest count of observations among the contributing assets, and returnnothingwhereregime_bias_openrefuses it. Else read the decay of the correlation structure:cor_decayon the separate path, elsedecay. Find the factor ofce.regime_methodwithmahalanobis_regime_bias!, on the plain weights or, with a HAC adjustment, on the banded weight matrix of the weights ofcorrelation_hac_lags!, times the factor ofvariance_noise_bias!, which carries the noise of the variance atdecayon the separate path and is one elsewhere. - Compute the squared distance with
regime_statistic, and returnnothingwhere the block does not factorise. - Divide the statistic by the factor.
Arguments
target::MahalanobisTarget: Mahalanobis regime-adjustment target.cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.X::VecNum: Current centred returns vector of every asset.idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.
Returns
stats::Option{<:VecNum}: One statistic, ornothingwhere the observation takes no regime update.
Related
PortfolioOptimisers.diagonal_law_factor — Function
diagonal_law_factor(
_::RegimeAdjustedMethod,
_::RegimeAdjustedCovarianceState,
_::RegimeAdjustedExpWeightedCovariance,
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
_::AbstractVector{<:Integer},
_::AbstractVector
) -> Any
Returns one, the law factor of a regime method that reads the mean of the diagonal statistic.
The per-term factor of regime_target_statistic makes the mean of the sum $n$ at every correlation, which is the constant that RootMeanSquaredAdjusted divides by. A regime method that the library does not define takes no law factor either.
Arguments
::RegimeAdjustedMethod: Regime adjustment method.::RegimeAdjustedCovarianceState: Online covariance computation cache (unused).::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration (unused).C::MatNum: Bias-corrected covariance block of the contributing assets.::AbstractVector{<:Integer}: Index of those assets (unused).::AbstractVector: Moments of each term (unused).
Returns
factor::Number:one(eltype(C)).
Related
diagonal_law_factor(
method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted},
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer},
m::AbstractVector
) -> Any
Computes the factor by which the diagonal statistic is divided, so that its root or its log has the expectation that the method's constant assumes.
Each term of the diagonal statistic $S = \sum_{i} a_{i} z_{i}^{2}$ is the square of a return with the correlation $R$, times the noise $a_{i}$ of its estimated variance. Without the noise, the law of the sum is $\sum_{k} \mu_{k} \chi^{2}_{k}(1)$ on the eigenvalues $\mu_{k}$ of $R$, which the constants of regime_denom and regime_kappa do not read. At 12 assets without the noise, the squared multiplier on Normal returns is 0.97 for the first moment and 0.95 for the log at a correlation of 0.3, and 0.76 and 0.55 at 0.9.
The noise has the mean one, but the root and the log see it through their own moment, and the correlation of the assets correlates the noise of their estimates, so the mean's factor of each term leaves 0.986 and 0.971 at a correlation of 0.9. The method reads the noise in its own variable $g_{i}$, $a_{i}^{1/2}$ for the first moment and $\ln a_{i}$ for the log, in which the statistic is linear along the direction where every term carries the same noise. The factor is the expectation at the mean of each $g_{i}$, which is the exact law of $D \tilde{R} D$, with $D = \mathrm{diag}(b_{i}^{1/2})$, plus the second-order term in the deviations of $g$, whose covariance is $(v_{i} v_{j})^{1/2} \tilde{r}_{ij}^{2}$ (the covariance of two estimated variances is $r_{ij}^{2}$ times their variance). The second-order term vanishes along that direction, so the factor is exact at one asset and at a correlation of one, and its error elsewhere is of the third order in the noise: on simulated iid Normal returns it is within 0.05 % at a half-life of 5, where the mean's factor leaves 3.4 % and 6.6 %.
The true $R$ is not known, and the eigenvalues of its estimate $\hat{R}$ are too dispersed: the law of $\hat{R}$ over-corrects by up to 1.3 % and 2.7 %. So diagonal_law_correlation shrinks $\hat{R}$ towards the identity until its dispersion is the unbiased one. The factor reads the correlation alone, so a scaled return leaves it unchanged. Under debias = nothing the factor is one, which is the raw statistic.
Arguments
method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted}: Regime adjustment method.cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.C::MatNum: Bias-corrected covariance block of the contributing assets.idx::AbstractVector{<:Integer}: Index of those assets.m::AbstractVector: Moments $(f_{i}, b_{i}, v_{i})$ of each term, fromRegimeTermMoments.
Returns
factor::Number: The factor of the method's law, or one wherece.debiasisnothing.
Related
PortfolioOptimisers.diagonal_law_correlation — Function
diagonal_law_correlation(
cache::RegimeAdjustedCovarianceState,
ce::RegimeAdjustedExpWeightedCovariance,
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
idx::AbstractVector{<:Integer}
) -> Any
Computes the correlation of the contributing assets, shrunk towards the identity until the dispersion of its eigenvalues is unbiased.
A sample correlation $\hat{r}$ of $K$ effective observations has the variance $(1 - r^{2})^{2} / K$, so $\hat{r}^{2}$ is too large by that amount on average, and the eigenvalues of $\hat{R}$ are too dispersed. The dispersion of the eigenvalues is $\sum_{k} (\mu_{k} - 1)^{2} = \sum_{i \neq j} r_{ij}^{2}$. The shrunk correlation $\tilde{R} = I + \alpha (\hat{R} - I)$ is a linear shrinkage towards the identity, with the intensity at which the dispersion of $\tilde{R}$ is the unbiased estimate of the dispersion of $R$. It is exact at the identity and at a correlation of one, where $\hat{r}^{2}$ has no variance.
On the separate path under HAC, where each row divides by the volatility before its update, the damped rows carry less noise than the Bartlett weights say: at two lags and half-lives of 10 and 20 the Bartlett weights put the mean of $\hat{r}^{2}$ at $R = I$ 14 % too high, and the kernel of hac_row_kernel 4 % too high. With the kernel the unbiased dispersion reads 4.9 % low on a three-factor correlation of 12 assets, where the Bartlett weights read 6.7 % low, and 6.9 % low at four lags, where they read 12.7 % low. The factor of diagonal_law_factor then moves towards its truth by up to 0.3 % for the first moment and 0.7 % for the log, measured without return noise. At $R = I$ the Bartlett weights over-shrink to the identity, which is the truth there, and the kernel moves the factor up by 0.05 % to 0.26 %, within 0.22 % of it.
Mathematical definition
\[\begin{align} \frac{1}{K_{ij}} &= \frac{1 - \lambda}{1 + \lambda} \cdot \frac{2 - W_{ij}}{W_{ij}}\,, \\ \alpha^{2} &= \frac{\sum_{i \neq j} \left(\hat{r}_{ij}^{2} - (1 - \hat{r}_{ij}^{2})^{2} / K_{ij}\right)} {\sum_{i \neq j} \hat{r}_{ij}^{2}}\,, \\ \tilde{R} &= I + \alpha (\hat{R} - I)\,. \end{align}\]
Where:
- $K_{ij}$: Effective count of the observations of the pair, the inverse of the sum of its squared normalised weights, from
pair_weight_square_sum. A HAC estimate has the law of a plain one on the eigenvalues of its weight matrix, so its count is the inverse of $\operatorname{tr}(A^{2})$, about half as large at two lags. On the separate path under HAC, where each row divides by the volatility before its update, the lagged terms of a row are damped, and the weights are the kernel ofhac_row_kernel, fromcorrelation_hac_lags!. - $\lambda$: Decay of the correlation,
cor_decayon the separate path anddecayelse. - $W_{ij}$: Weight that the pair holds, $1 - \lambda^{k}$ after $k$ joint observations.
- $\hat{r}_{ij}$: Entry of the estimated correlation.
- $\alpha$: Shrinkage intensity, clamped to $[0, 1]$.
A pair that holds no weight has no estimate, and is left out of both sums.
Arguments
cache::RegimeAdjustedCovarianceState: Online covariance computation cache.ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.C::MatNum: Bias-corrected covariance block of the contributing assets.idx::AbstractVector{<:Integer}: Index of those assets.
Returns
R::MatNum: The shrunk correlation $\tilde{R}$.
Related
PortfolioOptimisers.pair_weight_square_sum — Function
pair_weight_square_sum(
lambda::Number,
W::Number,
hac_lags::Union{Nothing, Integer, AbstractVector{<:Union{Number, AbstractJuMPScalar}}}
) -> Any
Computes the sum of the squared normalised weights of a pair's estimate from the weight the pair holds, the variance of a sample correlation of the pair relative to $(1 - r^{2})^{2}$.
A pair of $k$ joint observations holds $W = 1 - \lambda^{k}$, so the sum is a function of $W$ alone. With HAC lags the weights are the eigenvalues of the banded weight matrix $A$ of regime_bias_table, and the sum is $\operatorname{tr}(A^{2})$, which adds the squared weight of each lag that reaches a joint observation.
Mathematical definition
\[\begin{align} \operatorname{tr}(A^{2}) &= \frac{v}{W^{2}} \left(W (2 - W) + 2 \sum_{i=1}^{L} k_{i}^{2} \max\left(1 - \frac{(1 - W)^{2}}{\lambda^{2 i}}, 0\right)\right)\,, \quad v = \frac{1 - \lambda}{1 + \lambda}\,. \end{align}\]
Where:
- $k_{i}$: Weight of lag $i$, the Bartlett weight $1 - i / (L + 1)$.
- $L$:
hac_lags, and the sum is empty where it isnothing. Wherehac_lagsis a vector, $L$ is its length and $k_{i}$ its entries, such as the kernel ofhac_row_kernel.
Arguments
lambda::Number: Decay of the pair's estimate.W::Number: Weight that the pair holds.hac_lags::Option{<:Union{Integer, VecNum}}: Count of HAC lags, the weight of each lag, ornothing.
Returns
s::Number: The sum of the squared normalised weights.
Related
PortfolioOptimisers.regime_law_log_laplace — Function
regime_law_log_laplace(
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
a::Number,
y::Number,
t::Number
) -> Any
Computes the log of the Laplace transform of a weighted sum of gamma variates, divided by its count.
The law of $S / n = \sum_{k} \mu_{k} G_{k} / n$ with $G_{k}$ independent $\mathrm{Gamma}(a, y)$ variates has the Laplace transform $\prod_{k} (1 + y \mu_{k} t / n)^{-a}$. At $a = 1/2$ and $y = 2$ each $G_{k}$ is a $\chi^{2}(1)$ variate.
Arguments
mu::VecNum: Weights of the sum, the eigenvalues of the law.a::Number: Shape of each gamma variate.y::Number: Scale of each gamma variate.t::Number: Argument of the transform.
Returns
lL::Number: $-a \sum_{k} \ln(1 + y \mu_{k} t / n)$.
Related
PortfolioOptimisers.regime_law_grid — Function
regime_law_grid(T::Type) -> Any
Returns the grid of the integration variable $x = \ln t$ on which regime_law_factor and regime_law_correction take the trapezoid rule.
Each integrand is smooth and analytic in a strip about the real line, so the trapezoid rule converges faster than any power of the step. The transform of a wide block grows fast inside the strip, so the step must be small: at a step of $1/4$ the factors agree with their closed forms at $\mu_{k} = 1$ and at one eigenvalue $n$ to $10^{-14}$, from $n = 1$ to $20\,000$, whereas a step of $1/2$ is $3 \times 10^{-9}$ wrong at $n = 200$. The slowest integrand decays as $e^{-\lvert x \rvert / 2}$ at both ends, so the grid stops at $\lvert x \rvert = 75$.
Arguments
T::Type: Number type of the grid.
Returns
x::AbstractRange: The grid, of step $1/4$ on $[-75, 75]$.
Related
PortfolioOptimisers.regime_law_factor — Function
regime_law_factor(
method::FirstMomentRegimeAdjusted,
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Computes the factor of the first-moment method, the squared ratio of the expected root of a sum of $\chi^{2}(1)$ variates to the constant regime_denom.
Mathematical definition
\[\mathbb{E}[\sqrt{S}] = \frac{\sqrt{n}}{2 \sqrt{\pi}} \int_{0}^{\infty} \left(1 - L(t)\right) t^{-3/2}\, \mathrm{d}t\,, \quad L(t) = \prod_{k} \left(1 + 2 \mu_{k} t / n\right)^{-1/2}\,.\]
The integral is taken on $t = e^{x}$ over regime_law_grid. At $\mu_{k} = 1$ it is the root of a $\chi^{2}(n)$ variate, and at one eigenvalue $n$ it is $\sqrt{2 n / \pi}$.
Arguments
method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.mu::VecNum: Eigenvalues of the law, the weight of each $\chi^{2}(1)$ variate.
Returns
factor::Number: $(\mathbb{E}[\sqrt{S}] / d_{n})^{2}$, with $d_{n}$ fromregime_denom.
Related
regime_law_factor(
method::LogRegimeAdjusted,
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Computes the factor of the log method, the exponential of the difference between the expected log of a sum of gamma variates and the constant regime_kappa.
The method takes each square as a $\mathrm{Gamma}(x, y)$ variate, which is the law that its constant assumes, so the factor is one at $\mu_{k} = 1$ whatever the parameters.
Mathematical definition
\[\mathbb{E}[\ln S] = \ln n + \int_{0}^{\infty} \left(\frac{1}{1 + t} - L(t)\right) t^{-1}\, \mathrm{d}t - \gamma\,, \quad L(t) = \prod_{k} \left(1 + y \mu_{k} t / n\right)^{-x}\,.\]
Where $\gamma$ is the Euler-Mascheroni constant. The integral is taken on $t = e^{x}$ over regime_law_grid.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.mu::VecNum: Eigenvalues of the law, the weight of each gamma variate.
Returns
factor::Number: $\exp(\mathbb{E}[\ln S] - \kappa_{n})$, with $\kappa_{n}$ fromregime_kappa.
Related
regime_law_factor(
method::FirstMomentRegimeAdjusted,
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Computes the factor of the first-moment method for a sum whose terms carry the noise of their estimated variances.
The expected root is that of the law of $\mu$, from the two-argument method, plus the second-order term of regime_law_correction.
Arguments
method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.mu::VecNum: Eigenvalues of $D \tilde{R} D$.V::MatNum: Their eigenvectors.Cm::MatNum: Covariance of the deviations of the variable of the method.
Returns
factor::Number: $((\mathbb{E}[\sqrt{S_{b}}] + \Delta) / d_{n})^{2}$.
Related
regime_law_factor(
method::LogRegimeAdjusted,
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Computes the factor of the log method for a sum whose terms carry the noise of their estimated variances.
The expected log is that of the law of $\mu$, from the two-argument method, plus the second-order term of regime_law_correction.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.mu::VecNum: Eigenvalues of $D \tilde{R} D$.V::MatNum: Their eigenvectors.Cm::MatNum: Covariance of the deviations of the variable of the method.
Returns
factor::Number: $\exp(\mathbb{E}[\ln S_{b}] + \Delta - \kappa_{n})$.
Related
PortfolioOptimisers.regime_law_shape — Function
regime_law_shape(
_::FirstMomentRegimeAdjusted
) -> Tuple{Rational{Int64}, Int64}
Returns the shape of each square and the degrees of freedom of its Wishart form, for the first-moment method: a $\chi^{2}(1)$ variate.
Arguments
::FirstMomentRegimeAdjusted: First-moment regime adjustment method.
Returns
(a, nu)::Tuple: $(1/2, 1)$.
Related
regime_law_shape(
method::LogRegimeAdjusted
) -> Tuple{Any, Any}
Returns the shape of each square and the degrees of freedom of its Wishart form, for the log method, which takes each square as a $\mathrm{Gamma}(x, y)$ variate: the diagonal of a Wishart matrix of $2 x$ degrees of freedom.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.
Returns
(a, nu)::Tuple: $(x, 2 x)$.
Related
PortfolioOptimisers.regime_law_weights — Function
regime_law_weights(
_::FirstMomentRegimeAdjusted,
ltau::Number
) -> Tuple{Any, Any}
Returns the logs of the weights of the linear and the quadratic integrands of the first-moment correction at one node, the measure $\mathrm{d}\tau = \tau\, \mathrm{d}x$ included.
$\mathbb{E}[y_{i}^{2} S^{-1/2}]$ and $\mathbb{E}[y_{i}^{2} y_{j}^{2} S^{-3/2}]$ are $\Gamma(p)^{-1} \int \tau^{p - 1} \mathbb{E}[\cdot\, e^{-\tau S}]\, \mathrm{d}\tau$ at $p = 1/2$ and $p = 3/2$, so the weights are $\tau^{1/2} / \sqrt{\pi}$ and $2 \tau^{3/2} / \sqrt{\pi}$. regime_law_correction reads the vector $2 \tau d$ in place of $d$, so each weight here carries the factor $1 / (2 \tau)$ of each $d$ that it multiplies, and both weights are $\tau^{-1/2} / (2 \sqrt{\pi})$.
Arguments
::FirstMomentRegimeAdjusted: First-moment regime adjustment method.ltau::Number: Log of the node $\tau$ of the integration variable.
Returns
(lwl, lwq)::Tuple: $-(\ln \tau + \ln \pi) / 2 - \ln 2$ in both entries.
Related
regime_law_weights(
method::LogRegimeAdjusted,
ltau::Number
) -> Tuple{Any, Any}
Returns the logs of the weights of the linear and the quadratic integrands of the log correction at one node, the measure $\mathrm{d}\tau = \tau\, \mathrm{d}x$ included.
$\mathbb{E}[W_{ii} / S]$ and $\mathbb{E}[W_{ii} W_{jj} / S^{2}]$ are $\int \tau^{p - 1} \mathbb{E}[\cdot\, e^{-\tau S}]\, \mathrm{d}\tau$ at $p = 1$ and $p = 2$, and the linear one carries the degrees of freedom $\nu = 2 x$ of the tilted mean, so the weights are $2 x \tau$ and $\tau^{2}$. regime_law_correction reads the vector $2 \tau d$ in place of $d$, so each weight here carries the factor $1 / (2 \tau)$ of each $d$ that it multiplies, and the weights are $x$ and $1 / 4$ at every node.
Arguments
method::LogRegimeAdjusted: Log regime adjustment method.ltau::Number: Log of the node $\tau$ of the integration variable, read for its number type alone, so that the log of $x$ is not rounded to a narrower type than that of the data.
Returns
(lwl, lwq)::Tuple: $(\ln x, -2 \ln 2)$.
Related
PortfolioOptimisers.regime_law_correction — Function
regime_law_correction(
method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted},
mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any
Computes the second-order term of the expected root or log of a diagonal sum whose terms carry the noise of their estimated variances.
Write each term as $e^{g_{i}} y_{i}^{2}$ (the log method) or $(1 + g_{i})^{2} y_{i}^{2}$ (the first moment), with $y \sim \mathcal{N}(0, D \tilde{R} D)$, the deviation $g$ of mean zero and covariance $C_{m}$, and $g$ independent of $y$. A Taylor expansion of the root or the log of the sum in $g$ gives, to the second order,
\[\Delta = \frac{1}{2} \left(\sum_{i} C_{m, ii}\, \mathbb{E}[y_{i}^{2} h_{1}(S)] - \sum_{i, j} C_{m, ij}\, \mathbb{E}[y_{i}^{2} y_{j}^{2} h_{2}(S)]\right)\,,\]
with $h_{1} = S^{-1/2}$ and $h_{2} = S^{-3/2}$ for the root, and $S^{-1}$ and $S^{-2}$ for the log. Along the direction where every $g_{i}$ is equal, the root and the log are linear in $g$, so the two sums cancel there. Each expectation is an integral over $\tau$ of the tilted law $e^{-\tau S}$, under which $y$ has the covariance $M(\tau) = D \tilde{R} D (I + 2 \tau D \tilde{R} D)^{-1} = V \mathrm{diag}(d(\tau)) V'$, with $d_{k} = \mu_{k} / (1 + 2 \tau \mu_{k})$, and the Wishart moments $\mathbb{E}[W_{ii}] = \nu M_{ii}$ and $\mathbb{E}[W_{ii} W_{jj}] = \nu^{2} M_{ii} M_{jj} + 2 \nu M_{ij}^{2}$.
Algorithm
- On each node of
regime_law_grid, with $\tau = e^{x} / n$, accumulate the vector $j = \sum w_{l} L d$ and the matrix $J = \sum w_{q} L d d'$, with $L = \prod_{k} (1 + 2 \tau \mu_{k})^{-a}$ and the weights ofregime_law_weights. Each term reads $e = 2 \tau d$, whose entries lie in $[0, 1)$, and $\exp(\ln w + \ln L)$, with the factor $1 / (2 \tau)$ of each $d$ in $w$. Alone, $w_{q}$ and $d d'$ overflow and underflow the number type at the two ends of the grid, and their product is bounded there. - The linear sum is $(W' \mathrm{diag}(C_{m}))' j$, with $W = V \circ V$, and the part of the quadratic sum in $M_{ii} M_{jj}$ is $\langle W' C_{m} W, J \rangle$.
- The part in $M_{ij}^{2}$ is $\sum_{kl} J_{kl} \sum_{ij} C_{m, ij} V_{ik} V_{jk} V_{il} V_{jl}$. $J$ is positive semi-definite and its eigenvalues fall fast, so with $J = \sum_{q} \sigma_{q} g_{q} g_{q}'$ it is $\sum_{q} \sigma_{q} \sum_{ij} C_{m, ij} (V \mathrm{diag}(g_{q}) V')_{ij}^{2}$ over the $\sigma_{q}$ above the machine epsilon of the largest, a few products of order $n^{3}$ in place of one of order $n^{4}$. At 12 and 40 assets it keeps 2 to 8 terms and agrees with the sum over every node to $10^{-15}$.
Arguments
method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted}: Regime adjustment method.mu::VecNum: Eigenvalues of $D \tilde{R} D$.V::MatNum: Their eigenvectors.Cm::MatNum: Covariance of the deviations of the variable of the method.
Returns
Delta::Number: The second-order term of $\mathbb{E}[\sqrt{S}]$ or of $\mathbb{E}[\ln S]$.
Related
PortfolioOptimisers.exp_weight_cross_sum — Function
exp_weight_cross_sum(
a::Number,
b::Number,
K::Integer
) -> Any
exp_weight_cross_sum(
a::Number,
b::Number,
K::Integer,
hac_lags::Union{Nothing, Integer, AbstractVector{<:Union{Number, AbstractJuMPScalar}}}
) -> Any
Computes the sum of the products of two sets of normalised exponential weights over $K$ observations.
Mathematical definition
\[\sum_{j=0}^{K-1} w_{j}(a)\, w_{j}(b) = \frac{(1 - a)(1 - b)\left(1 - (ab)^{K}\right)}{(1 - ab)\left(1 - a^{K}\right)\left(1 - b^{K}\right)}\,, \quad w_{j}(a) = \frac{(1 - a)\, a^{j}}{1 - a^{K}}\,.\]
At $a = b$ it is the sum of the squared weights, the variance of an estimate of $K$ observations relative to the variance of one. With HAC lags the sum is $\operatorname{tr}(A_{a} A_{b})$ over the banded weight matrices of regime_bias_table, whose law has the eigenvalues of $A$ as its weights, so each lag $i$ that reaches an observation adds $2 k_{i}^{2} (1 - (ab)^{K - i})$ to the factor $1 - (ab)^{K}$ of the numerator. At $a = b$ its inverse is the effective count of observations that regime_bias_open reads.
Arguments
a::Number: Decay of the first set of weights.b::Number: Decay of the second set of weights.K::Integer: Count of observations that the weights span.hac_lags::Option{<:Union{Integer, VecNum}}: Count of HAC lags, the weight $k_{i}$ of each lag, ornothing.
Returns
s::Number: The sum of the products of the weights.
Related
PortfolioOptimisers.hac_ldl_row! — Function
hac_ldl_row!(
buf::NamedTuple,
s::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
decay::Number,
lam::Number,
m::Integer
) -> NamedTuple
Adds the row of observation m to the banded LDLᵀ factorisation of $I + s B$ at every point of the grid, and adds the log of its pivot to $\ln G$.
The matrix is banded with L lags, so a new row reads only the last L rows of the factor and their pivots. The buffers hold them, the last row first.
Arguments
buf::NamedTuple: Buffers of the factorisation (mutated):l[g, r, i], the entry of row $m - r$ at column $m - r - i$;d[g, r], the pivot of row $m - r$;rowandnum, scratch;lG, $\ln G$ at each point, $-\infty$ at a point whose pivot was not positive; andk, the weight $k_{i}$ of each lag, fromhac_lag_weights.s::VecNum: Grid of $s$.decay::Number: Decay of the weights.lam::Number: $\lambda^{m}$.m::Integer: Index of the new row, from zero for the newest observation.
Returns
buf::NamedTuple: The buffers after the row.
Related
PortfolioOptimisers.inverse_volatility_bias — Function
inverse_volatility_bias(
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
sv::Number,
svc::Number,
min_val::Number
) -> Any
Computes the second-order excess bias of the regime statistic of the inverse-volatility direction, which is built from the estimate it divides by.
The direction $p_{i} \propto 1 / \hat{\sigma}_{i}$ reads the estimated volatilities, so its statistic is $q^{\top} R q / 1^{\top} \hat{R} 1$ in units of the true volatilities, with $q_{i} = \sigma_{i} / \hat{\sigma}_{i}$. An asset whose volatility is under-estimated takes more weight, and its error enters the numerator but not the denominator. A second-order expansion in the error of the estimate gives the mean of the statistic as the factor of a fixed direction times $1 + \Delta$. The excess is the same for the three regime methods to first order. On one decay $s_{vc} = s_{v}$ and $\Delta = 2 s_{v} \kappa$, with $\kappa = 1 - \sum_{i} c_{i}^{3} / A^{2}$: $1 - 1/n$ at $R = I$ and zero at a correlation of one, where the estimated direction is proportional to the true one.
Mathematical definition
\[\begin{align} \Delta &= s_{v} \left(1 + \frac{\Sigma_{3}}{A} - \frac{2 B}{A^{2}}\right) + s_{vc} \left(1 - \frac{\Sigma_{3}}{A} - \frac{2 T}{A^{2}} + \frac{2 B}{A^{2}}\right)\,, \\ c &= R 1\,, \quad A = 1^{\top} c\,, \quad T = \sum_{i} c_{i}^{3}\,, \quad B = \sum_{ij} c_{i} R_{ij}^{2} c_{j}\,, \quad \Sigma_{3} = \sum_{ij} R_{ij}^{3}\,. \end{align}\]
Where:
- $R$: Correlation of the block, the plug-in for the true correlation.
- $s_{v}$: Sum of the squared weights of the variance estimate.
- $s_{vc}$: Sum of the products of the weights of the variance and the correlation estimates.
The plug-in and the next order leave about $9 s_{v}^{2}$. At 12 assets and a half-life of 10 the formula at the true $R$ matches the simulated excess to $5 \times 10^{-4}$, on both paths. A block whose $A$ is not positive has no inverse-volatility portfolio with a variance, and takes no excess.
Arguments
C::MatNum: Bias-corrected covariance block of the contributing assets.sv::Number: Sum of the squared weights of the variance estimate.svc::Number: Sum of the products of the weights of the variance and the correlation estimates.min_val::Number: Floor applied to each variance, as inregime_statistic.
Returns
delta::Number: The excess $\Delta$.
Related
PortfolioOptimisers.regime_bias_state — Function
regime_bias_state(
ce::RegimeAdjustedExpWeightedVariance,
_::Type{T}
) -> Any
Makes the empty table of bias factors for a new state of a regime-adjusted variance estimator.
Arguments
ce::RegimeAdjustedExpWeightedVariance: Estimator configuration.::Type{T}: Element type of the state.
Returns
bias::Option{<:VecNum}: An empty vector wheredebiasesholds force.debiasand a regime method is set, elsenothing.
Related
regime_bias_state(
ce::RegimeAdjustedExpWeightedCovariance,
_::Type{T}
) -> Any
Makes the empty store of bias factors for a new state of a regime-adjusted covariance estimator.
The regime target names the store with regime_bias_store: the store of a DiagonalTarget holds the moments of one term of RegimeTermMoments, a triple for each count, and the HAC kernel of its correlation; the store of a MahalanobisTarget holds the interpolation nodes of its factor for each count of assets; the table of any other target holds one factor for each count.
Arguments
ce::RegimeAdjustedExpWeightedCovariance: Estimator configuration.::Type{T}: Element type of the state.
Returns
bias::Option{<:Union{AbstractVector, DiagonalBiasStore, PortfolioBiasStore, MahalanobisBiasStore}}: The empty store wheredebiasesholds force.debiasand a regime method is set, elsenothing.
Related
PortfolioOptimisers.safe_regime_cholesky — Function
safe_regime_cholesky(
C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
min_val::Number
) -> Any
Factorises a covariance block for the Mahalanobis regime statistic, and refuses rather than throws when no ridge makes it factorise.
A block with fewer observations than assets is singular. Where the estimator has debias = ExactDebias(), regime_target_statistic skips such a block before it reaches this function, so the ridge serves a block that is singular in the data, and the raw statistic of debias = nothing. The regime statistic is one observation of a smoother rather than a result a caller reads, so a refusal skips that observation's regime update, and the fit continues.
Algorithm
- Build a
RidgeCholeskySquareRootwithridge = max(min_val, 0)and three tries. A negativemin_valgives the ridge of a zeromin_val, the flooreps * scale. - Return
ridge_choleskyofCunder it: the plain factorisation, or the first ridge that makesCfactorise, ornothing.
Arguments
C::MatNum: Covariance block of the assets that contribute to the statistic.min_val::Number: Scale of the first ridge.
Returns
chol::Option{<:LinearAlgebra.Cholesky}: The factorisation, ornothingwhere no ridge makes the block factorise.
Related
PortfolioOptimisers.mahalanobis_bias — Function
mahalanobis_bias(
decay::Number,
K::Integer,
n::Integer
) -> Any
Computes the deterministic equivalent of the mean bias factor of the squared Mahalanobis distance against an exponentially weighted covariance estimate.
The factor is $b = \mathbb{E}[\operatorname{tr}(W^{-1})] / n$ of $W = \sum_{j} w_{j} z_{j} z_{j}^{\top}$, $z_{j} \sim N(0, I_{n})$, which is the mean of $u^{\top} \hat{C}^{-1} u / n$ for a correctly calibrated return $u$ that is independent of the estimate $\hat{C}$. It depends on the weights and on n alone. It solves $1 / b = \sum_{j} w_{j} / (1 + (n + 1) w_{j} b)$, which exists where K > n + 1 and is exact at equal weights. On exponential weights it is 0.55 % above the mean at 12 assets and a half-life of 10, so inverse_wishart_bias takes it only to name the pole of the factor at K = n + 1, and mahalanobis_regime_bias! corrects the result.
Algorithm
- Return
nothingwhereK <= n + 1. - Start at $b = 1$. The map $h(b) = b\, g(b) - 1$, with $g$ the right-hand side of the fixed point, is concave and increasing, and $h(1) < 0$, so Newton's method from $b = 1$ rises to the root without overshoot.
- Evaluate $g$ and $h'$ with
mahalanobis_bias_sums, and stop when the step is below four units in the last place of $b$.
Arguments
decay::Number: Decay of the weights.K::Integer: Count of observations in the estimate.n::Integer: Count of assets that contribute to the statistic.
Returns
b::Option{<:Number}: The bias factor, ornothingwhereK <= n + 1.
Related
PortfolioOptimisers.mahalanobis_bias_sums — Function
mahalanobis_bias_sums(
decay::Number,
K::Integer,
c::Integer,
w0::Number,
b::Number
) -> Tuple{Any, Any}
Computes the right-hand side of the fixed point of mahalanobis_bias and the derivative of $b\, g(b)$.
The weights fall geometrically, and a weight $w_{j}$ with $c\, w_{j}\, b$ below the machine epsilon of decay enters both sums as itself, so the loop stops there and adds the remaining mass $1 - \sum_{i < j} w_{i}$ to each sum. That keeps the cost bounded on a long history.
Arguments
decay::Number: Decay of the weights.K::Integer: Count of observations in the estimate.c::Integer:n + 1.w0::Number: Normalised weight of the newest observation,(1 - decay) / (1 - decay^K).b::Number: Current value of the bias factor.
Returns
(g, dh)::Tuple{Number, Number}: $\sum_{j} w_{j} / (1 + c\, w_{j}\, b)$ and $\sum_{j} w_{j} / (1 + c\, w_{j}\, b)^{2}$.
Related
PortfolioOptimisers.taylor_mul — Function
taylor_mul(
x::Tuple{var"#s2444", var"#s2444"} where var"#s2444"<:Number,
y::Tuple{var"#s2443", var"#s2443"} where var"#s2443"<:Number
) -> Tuple{Any, Any}
Multiplies two values carried with their first derivatives.
Arguments
x::NTuple{2, <:Number}: $(x, x')$.y::NTuple{2, <:Number}: $(y, y')$.
Returns
xy::NTuple{2, <:Number}: $(x y, x' y + x y')$.
Related
PortfolioOptimisers.taylor_div — Function
taylor_div(
x::Tuple{var"#s2444", var"#s2444"} where var"#s2444"<:Number,
y::Tuple{var"#s2443", var"#s2443"} where var"#s2443"<:Number
) -> Tuple{Any, Any}
Divides two values carried with their first derivatives.
Arguments
x::NTuple{2, <:Number}: $(x, x')$.y::NTuple{2, <:Number}: $(y, y')$, with $y \neq 0$.
Returns
q::NTuple{2, <:Number}: $(q, q')$ of $q = x / y$, with $q' = (x' - q y') / y$.
Related
PortfolioOptimisers.gap_fill_value — Method
gap_fill_value(ce::RegimeAdjustedExpWeightedCovariance) -> Float64Answer NaN, so a gapped sample reaches the recursion with its gaps intact.
The recursion updates only the sub-block of the assets that are valid at each observation, and the regime weight is taken from that sub-block alone, so a fill decays a frozen block and moves the regime that the block weighs. The consumer therefore hands the sample as it stands, together with the active mask that explains the gap.
Arguments
ce: Covariance estimator.
Returns
fv::Float64:NaN.
Related
Base.copy — Method
copy(
x::RegimeAdjustedCovarianceState
) -> Union{RegimeAdjustedCovarianceState{Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows<:Base.RefValue}, RegimeAdjustedCovarianceState{__T_ret_buffer, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows} where {__T_ret_buffer<:DataStructures.CircularBuffer, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows<:Base.RefValue}}
Copies a RegimeAdjustedCovarianceState, so the copy shares no array with the original.
The copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. Every array field is copied, and the two scalar fields pass through. The row count goes into a new Ref, so a fold of the copy does not count in the original. The circular buffer of recent centred returns is rebuilt at the same capacity, and each observation it holds is copied into it, so a fold on the copy pushes into a buffer of its own. The two fields of the separate correlation recursion pass through as nothing where they are nothing. The store of bias factors is nothing, an array or a dictionary, and deepcopy copies an array or a dictionary and returns nothing unchanged.
Arguments
x: The cache to copy.
Returns
state::RegimeAdjustedCovarianceState: A fresh cache, equal tox, whose arrays are fresh.
Related
PortfolioOptimisers.observation_count — Method
observation_count(
state::RegimeAdjustedCovarianceState
) -> Any
RegimeAdjustedCovarianceState method of observation_count.
The state counts the observations of each asset in obs_count, and a reset sets such a count to zero. n_regime_obs counts only the observations that the regime statistic scored. So the method reads rows, the number of folded rows, which is the size(X, 1) of a batch fit over the same rows.
Arguments
state: The state that holds the count.
Returns
n::Integer: The number of observations folded so far.
Related
PortfolioOptimisers.variance_series — Method
variance_series(
ce::RegimeAdjustedExpWeightedCovariance,
X::MatNum;
dims::Int = 1,
estimation_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
kwargs...
) -> Matrix{<:Number}Compute the point-in-time regime-adjusted exponentially weighted variance series.
Row t holds the diagonal of what cov returns for the first t observations of X, so no row reads an observation after its own. The update is a recursion over one observation, so this method overrides the expanding-window fallback with a single forward pass: it reads the cache after each observation instead of refitting.
The fallback cannot answer this estimator. It slices X once per row and passes every keyword unsliced, so a mask of the whole window meets a window of t observations and the size check refuses the call.
Arguments
ce: Regime-adjusted exponentially weighted covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.dims: Dimension along which the computation runs.estimation_mask: Optional boolean matrix with the same size asX. When provided, only assets whereestimation_mask[i, :](or[:, i]) istruecontribute to the regime state update for observationi.active_mask: Optional boolean matrix with the same size asX. When provided, assets that become inactive have their covariance and observation count reset.kwargs: More keyword arguments, which the method ignores.
Validation
dims in (1, 2).- If
estimation_maskis notnothing,size(X) == size(estimation_mask). - If
active_maskis notnothing,size(X) == size(active_mask).
Returns
val::Matrix{<:Number}: Variance series, shaped as(T, N)ifdims == 1or(N, T)ifdims == 2. An asset with fewer thance.min_obsobservations at rowtisNaNthere.
Related
PortfolioOptimisers.variance_series — Method
variance_series(
ce::RegimeAdjustedExpWeightedCovariance,
X::MatNum,
pnl::Option{<:AssetPanel};
dims::Int = 1,
kwargs...
) -> Matrix{<:Number}Compute the point-in-time regime-adjusted exponentially weighted variance series from a window of an Asset Panel.
This is the diagonal of the covariance of the same call, read after each observation, and it reads the panel's two masks through the same override.
Arguments
ce: Regime-adjusted exponentially weighted covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.pnl: OptionalAssetPanel. The Coverage Universe of the fit comes from its active mask. If it isnothing, the rule reads only the finiteness of the returns.dims: Dimension along which the computation runs.kwargs: More keyword arguments, which the method ignores.
Returns
val::Matrix{<:Number}: Variance series on the full asset universe, shaped as(T, N)ifdims == 1or(N, T)ifdims == 2.
Related
RegimeAdjustedExpWeightedCovariancevariance_series(ce::RegimeAdjustedExpWeightedCovariance, X::MatNum; dims::Int = 1, estimation_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing, kwargs...)variance_series(ce::AbstractCovarianceEstimator, X::MatNum, pnl::Option{<:AssetPanel}; dims::Int = 1, kwargs...)