Regime adjusted exp weighted covariance (b): private API

Functions

PortfolioOptimisers.regime_target_statistic — Function
regime_target_statistic(
    target::RegimeAdjustedTarget,
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Computes the regime statistic of one observation for a target, from the covariance block that stands before it.

A target that has no method of its own reads its regime_statistic of the block that regime_covariance_block returns, and takes no bias correction.

Arguments

  • target::RegimeAdjustedTarget: Regime-adjustment target.
  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Current centred returns vector of every asset.
  • idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.

Returns

  • stats::Option{<:VecNum}: One statistic per calibration direction, or nothing where the observation takes no regime update.

Related

source
regime_target_statistic(
    _::DiagonalTarget,
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Computes the regime statistic of the diagonal target, with each term divided by the bias of its own estimated variance, and the sum divided by the factor of its law.

Each term $u_{i}^{2} / \hat{C}_{ii}$ reads one estimated variance of $K_{i}$ observations, so its mean is $\mathbb{E}[Q^{-1}]$ at $K_{i}$, from regime_bias_table, and not one. Each term is divided by that factor whatever the method, so the sum has the mean $n$ at every correlation of the assets.

The mean of the sum is $n$ at every correlation, but its root and its log are not: they read the law of the sum, which the correlation of the assets and the noise of each estimate set. So the sum is then divided by diagonal_law_factor of the method, which reads the moments of each term from the table of RegimeTermMoments.

Arguments

  • ::DiagonalTarget: Diagonal regime-adjustment target.
  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Current centred returns vector of every asset.
  • idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.

Returns

  • stats::VecNum: One statistic.

Related

source
regime_target_statistic(
    target::PortfolioTarget,
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Computes the regime statistic of the portfolio target, divided by the bias of the estimated variance of each direction.

A direction reads one estimated variance $p^{\top} \hat{C} p$, so its statistic is biased by the factor of regime_bias_table that the method reads, at $K$, the smallest count of observations among the contributing assets. That factor is exact for a fixed direction on one shared history. On the separate correlation path the variance of a direction mixes the variances at decay with the correlations at cor_decay, so no one table holds it, and portfolio_regime_bias gives each direction its own factor from the two tables. A HAC estimate reads the table of its banded weight matrix, which has about twice the excess of the plain weights: at two lags the squared multiplier of an equal-weight direction is 1.015 over 8 seeds, within its standard error of 0.013, from 1.165 raw.

The default inverse-volatility direction (w = nothing) is built from the same estimate, so the direction and the error of the estimate are correlated. Its statistic is also divided by $1 + \Delta$, the second-order excess of inverse_volatility_bias, on the estimated correlation of the block. At 12 assets and a half-life of 10 the squared multiplier on iid Normal returns is 1.134 raw, 1.058 after the factor alone and 1.001 after both. The excess is the same for the three methods to first order. What remains is of the next order, about $9 s^{2}$ with $s$ the sum of the squared weights, so it is largest in the warm-up rows: at 12 assets, a half-life of 10 and $R = I$, 1.34 at $K = 5$, 1.08 at $K = 10$, 1.02 at $K = 20$ and 1.004 in the steady state. At the defaults the first row enters at $K$ equal to the half-life, where the rest is 1.006 at a half-life of 40.

The block passes through indefinite_block with the rule of target.indefinite first, and a row whose block the rule refuses is not scored. The block is young while regime_bias_open is false for the count of contributing assets, at $K$, on the decay of the correlation structure and with the HAC lags of the estimator: the gate of MahalanobisTarget for the same assets.

Arguments

  • target::PortfolioTarget: Portfolio regime-adjustment target.
  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Current centred returns vector of every asset.
  • idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.

Returns

  • stats::Option{<:VecNum}: One statistic per direction, or nothing where no direction keeps a positive weight, or where the rule of target.indefinite refuses the block.

Related

source
regime_target_statistic(
    target::MahalanobisTarget,
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    X::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Computes the squared Mahalanobis distance of one observation, divided by the bias of the estimated block that the regime method reads, where ce.debias is ExactDebias() or LawDebias().

Algorithm

  1. Where ce.debias is nothing, the factor is one. Else take $K$, the smallest count of observations among the contributing assets, and return nothing where regime_bias_open refuses it. Else read the decay of the correlation structure: cor_decay on the separate path, else decay. Find the factor of ce.regime_method with mahalanobis_regime_bias!, on the plain weights or, with a HAC adjustment, on the banded weight matrix of the weights of correlation_hac_lags!, times the factor of variance_noise_bias!, which carries the noise of the variance at decay on the separate path and is one elsewhere.
  2. Compute the squared distance with regime_statistic, and return nothing where the block does not factorise.
  3. Divide the statistic by the factor.

Arguments

  • target::MahalanobisTarget: Mahalanobis regime-adjustment target.
  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • X::VecNum: Current centred returns vector of every asset.
  • idx::AbstractVector{<:Integer}: Index of the assets that contribute to the statistic.

Returns

  • stats::Option{<:VecNum}: One statistic, or nothing where the observation takes no regime update.

Related

source
PortfolioOptimisers.diagonal_law_factor — Function
diagonal_law_factor(
    _::RegimeAdjustedMethod,
    _::RegimeAdjustedCovarianceState,
    _::RegimeAdjustedExpWeightedCovariance,
    C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    _::AbstractVector{<:Integer},
    _::AbstractVector
) -> Any

Returns one, the law factor of a regime method that reads the mean of the diagonal statistic.

The per-term factor of regime_target_statistic makes the mean of the sum $n$ at every correlation, which is the constant that RootMeanSquaredAdjusted divides by. A regime method that the library does not define takes no law factor either.

Arguments

  • ::RegimeAdjustedMethod: Regime adjustment method.
  • ::RegimeAdjustedCovarianceState: Online covariance computation cache (unused).
  • ::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration (unused).
  • C::MatNum: Bias-corrected covariance block of the contributing assets.
  • ::AbstractVector{<:Integer}: Index of those assets (unused).
  • ::AbstractVector: Moments of each term (unused).

Returns

  • factor::Number: one(eltype(C)).

Related

source
diagonal_law_factor(
    method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted},
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer},
    m::AbstractVector
) -> Any

Computes the factor by which the diagonal statistic is divided, so that its root or its log has the expectation that the method's constant assumes.

Each term of the diagonal statistic $S = \sum_{i} a_{i} z_{i}^{2}$ is the square of a return with the correlation $R$, times the noise $a_{i}$ of its estimated variance. Without the noise, the law of the sum is $\sum_{k} \mu_{k} \chi^{2}_{k}(1)$ on the eigenvalues $\mu_{k}$ of $R$, which the constants of regime_denom and regime_kappa do not read. At 12 assets without the noise, the squared multiplier on Normal returns is 0.97 for the first moment and 0.95 for the log at a correlation of 0.3, and 0.76 and 0.55 at 0.9.

The noise has the mean one, but the root and the log see it through their own moment, and the correlation of the assets correlates the noise of their estimates, so the mean's factor of each term leaves 0.986 and 0.971 at a correlation of 0.9. The method reads the noise in its own variable $g_{i}$, $a_{i}^{1/2}$ for the first moment and $\ln a_{i}$ for the log, in which the statistic is linear along the direction where every term carries the same noise. The factor is the expectation at the mean of each $g_{i}$, which is the exact law of $D \tilde{R} D$, with $D = \mathrm{diag}(b_{i}^{1/2})$, plus the second-order term in the deviations of $g$, whose covariance is $(v_{i} v_{j})^{1/2} \tilde{r}_{ij}^{2}$ (the covariance of two estimated variances is $r_{ij}^{2}$ times their variance). The second-order term vanishes along that direction, so the factor is exact at one asset and at a correlation of one, and its error elsewhere is of the third order in the noise: on simulated iid Normal returns it is within 0.05 % at a half-life of 5, where the mean's factor leaves 3.4 % and 6.6 %.

The true $R$ is not known, and the eigenvalues of its estimate $\hat{R}$ are too dispersed: the law of $\hat{R}$ over-corrects by up to 1.3 % and 2.7 %. So diagonal_law_correlation shrinks $\hat{R}$ towards the identity until its dispersion is the unbiased one. The factor reads the correlation alone, so a scaled return leaves it unchanged. Under debias = nothing the factor is one, which is the raw statistic.

Arguments

  • method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted}: Regime adjustment method.
  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • C::MatNum: Bias-corrected covariance block of the contributing assets.
  • idx::AbstractVector{<:Integer}: Index of those assets.
  • m::AbstractVector: Moments $(f_{i}, b_{i}, v_{i})$ of each term, from RegimeTermMoments.

Returns

  • factor::Number: The factor of the method's law, or one where ce.debias is nothing.

Related

source
PortfolioOptimisers.diagonal_law_correlation — Function
diagonal_law_correlation(
    cache::RegimeAdjustedCovarianceState,
    ce::RegimeAdjustedExpWeightedCovariance,
    C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    idx::AbstractVector{<:Integer}
) -> Any

Computes the correlation of the contributing assets, shrunk towards the identity until the dispersion of its eigenvalues is unbiased.

A sample correlation $\hat{r}$ of $K$ effective observations has the variance $(1 - r^{2})^{2} / K$, so $\hat{r}^{2}$ is too large by that amount on average, and the eigenvalues of $\hat{R}$ are too dispersed. The dispersion of the eigenvalues is $\sum_{k} (\mu_{k} - 1)^{2} = \sum_{i \neq j} r_{ij}^{2}$. The shrunk correlation $\tilde{R} = I + \alpha (\hat{R} - I)$ is a linear shrinkage towards the identity, with the intensity at which the dispersion of $\tilde{R}$ is the unbiased estimate of the dispersion of $R$. It is exact at the identity and at a correlation of one, where $\hat{r}^{2}$ has no variance.

On the separate path under HAC, where each row divides by the volatility before its update, the damped rows carry less noise than the Bartlett weights say: at two lags and half-lives of 10 and 20 the Bartlett weights put the mean of $\hat{r}^{2}$ at $R = I$ 14 % too high, and the kernel of hac_row_kernel 4 % too high. With the kernel the unbiased dispersion reads 4.9 % low on a three-factor correlation of 12 assets, where the Bartlett weights read 6.7 % low, and 6.9 % low at four lags, where they read 12.7 % low. The factor of diagonal_law_factor then moves towards its truth by up to 0.3 % for the first moment and 0.7 % for the log, measured without return noise. At $R = I$ the Bartlett weights over-shrink to the identity, which is the truth there, and the kernel moves the factor up by 0.05 % to 0.26 %, within 0.22 % of it.

Mathematical definition

\[\begin{align} \frac{1}{K_{ij}} &= \frac{1 - \lambda}{1 + \lambda} \cdot \frac{2 - W_{ij}}{W_{ij}}\,, \\ \alpha^{2} &= \frac{\sum_{i \neq j} \left(\hat{r}_{ij}^{2} - (1 - \hat{r}_{ij}^{2})^{2} / K_{ij}\right)} {\sum_{i \neq j} \hat{r}_{ij}^{2}}\,, \\ \tilde{R} &= I + \alpha (\hat{R} - I)\,. \end{align}\]

Where:

  • $K_{ij}$: Effective count of the observations of the pair, the inverse of the sum of its squared normalised weights, from pair_weight_square_sum. A HAC estimate has the law of a plain one on the eigenvalues of its weight matrix, so its count is the inverse of $\operatorname{tr}(A^{2})$, about half as large at two lags. On the separate path under HAC, where each row divides by the volatility before its update, the lagged terms of a row are damped, and the weights are the kernel of hac_row_kernel, from correlation_hac_lags!.
  • $\lambda$: Decay of the correlation, cor_decay on the separate path and decay else.
  • $W_{ij}$: Weight that the pair holds, $1 - \lambda^{k}$ after $k$ joint observations.
  • $\hat{r}_{ij}$: Entry of the estimated correlation.
  • $\alpha$: Shrinkage intensity, clamped to $[0, 1]$.

A pair that holds no weight has no estimate, and is left out of both sums.

Arguments

  • cache::RegimeAdjustedCovarianceState: Online covariance computation cache.
  • ce::RegimeAdjustedExpWeightedCovariance: Covariance estimator configuration.
  • C::MatNum: Bias-corrected covariance block of the contributing assets.
  • idx::AbstractVector{<:Integer}: Index of those assets.

Returns

  • R::MatNum: The shrunk correlation $\tilde{R}$.

Related

source
PortfolioOptimisers.pair_weight_square_sum — Function
pair_weight_square_sum(
    lambda::Number,
    W::Number,
    hac_lags::Union{Nothing, Integer, AbstractVector{<:Union{Number, AbstractJuMPScalar}}}
) -> Any

Computes the sum of the squared normalised weights of a pair's estimate from the weight the pair holds, the variance of a sample correlation of the pair relative to $(1 - r^{2})^{2}$.

A pair of $k$ joint observations holds $W = 1 - \lambda^{k}$, so the sum is a function of $W$ alone. With HAC lags the weights are the eigenvalues of the banded weight matrix $A$ of regime_bias_table, and the sum is $\operatorname{tr}(A^{2})$, which adds the squared weight of each lag that reaches a joint observation.

Mathematical definition

\[\begin{align} \operatorname{tr}(A^{2}) &= \frac{v}{W^{2}} \left(W (2 - W) + 2 \sum_{i=1}^{L} k_{i}^{2} \max\left(1 - \frac{(1 - W)^{2}}{\lambda^{2 i}}, 0\right)\right)\,, \quad v = \frac{1 - \lambda}{1 + \lambda}\,. \end{align}\]

Where:

  • $k_{i}$: Weight of lag $i$, the Bartlett weight $1 - i / (L + 1)$.
  • $L$: hac_lags, and the sum is empty where it is nothing. Where hac_lags is a vector, $L$ is its length and $k_{i}$ its entries, such as the kernel of hac_row_kernel.

Arguments

  • lambda::Number: Decay of the pair's estimate.
  • W::Number: Weight that the pair holds.
  • hac_lags::Option{<:Union{Integer, VecNum}}: Count of HAC lags, the weight of each lag, or nothing.

Returns

  • s::Number: The sum of the squared normalised weights.

Related

source
PortfolioOptimisers.regime_law_log_laplace — Function
regime_law_log_laplace(
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    a::Number,
    y::Number,
    t::Number
) -> Any

Computes the log of the Laplace transform of a weighted sum of gamma variates, divided by its count.

The law of $S / n = \sum_{k} \mu_{k} G_{k} / n$ with $G_{k}$ independent $\mathrm{Gamma}(a, y)$ variates has the Laplace transform $\prod_{k} (1 + y \mu_{k} t / n)^{-a}$. At $a = 1/2$ and $y = 2$ each $G_{k}$ is a $\chi^{2}(1)$ variate.

Arguments

  • mu::VecNum: Weights of the sum, the eigenvalues of the law.
  • a::Number: Shape of each gamma variate.
  • y::Number: Scale of each gamma variate.
  • t::Number: Argument of the transform.

Returns

  • lL::Number: $-a \sum_{k} \ln(1 + y \mu_{k} t / n)$.

Related

source
PortfolioOptimisers.regime_law_grid — Function
regime_law_grid(T::Type) -> Any

Returns the grid of the integration variable $x = \ln t$ on which regime_law_factor and regime_law_correction take the trapezoid rule.

Each integrand is smooth and analytic in a strip about the real line, so the trapezoid rule converges faster than any power of the step. The transform of a wide block grows fast inside the strip, so the step must be small: at a step of $1/4$ the factors agree with their closed forms at $\mu_{k} = 1$ and at one eigenvalue $n$ to $10^{-14}$, from $n = 1$ to $20\,000$, whereas a step of $1/2$ is $3 \times 10^{-9}$ wrong at $n = 200$. The slowest integrand decays as $e^{-\lvert x \rvert / 2}$ at both ends, so the grid stops at $\lvert x \rvert = 75$.

Arguments

  • T::Type: Number type of the grid.

Returns

  • x::AbstractRange: The grid, of step $1/4$ on $[-75, 75]$.

Related

source
PortfolioOptimisers.regime_law_factor — Function
regime_law_factor(
    method::FirstMomentRegimeAdjusted,
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Computes the factor of the first-moment method, the squared ratio of the expected root of a sum of $\chi^{2}(1)$ variates to the constant regime_denom.

Mathematical definition

\[\mathbb{E}[\sqrt{S}] = \frac{\sqrt{n}}{2 \sqrt{\pi}} \int_{0}^{\infty} \left(1 - L(t)\right) t^{-3/2}\, \mathrm{d}t\,, \quad L(t) = \prod_{k} \left(1 + 2 \mu_{k} t / n\right)^{-1/2}\,.\]

The integral is taken on $t = e^{x}$ over regime_law_grid. At $\mu_{k} = 1$ it is the root of a $\chi^{2}(n)$ variate, and at one eigenvalue $n$ it is $\sqrt{2 n / \pi}$.

Arguments

  • method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.
  • mu::VecNum: Eigenvalues of the law, the weight of each $\chi^{2}(1)$ variate.

Returns

  • factor::Number: $(\mathbb{E}[\sqrt{S}] / d_{n})^{2}$, with $d_{n}$ from regime_denom.

Related

source
regime_law_factor(
    method::LogRegimeAdjusted,
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Computes the factor of the log method, the exponential of the difference between the expected log of a sum of gamma variates and the constant regime_kappa.

The method takes each square as a $\mathrm{Gamma}(x, y)$ variate, which is the law that its constant assumes, so the factor is one at $\mu_{k} = 1$ whatever the parameters.

Mathematical definition

\[\mathbb{E}[\ln S] = \ln n + \int_{0}^{\infty} \left(\frac{1}{1 + t} - L(t)\right) t^{-1}\, \mathrm{d}t - \gamma\,, \quad L(t) = \prod_{k} \left(1 + y \mu_{k} t / n\right)^{-x}\,.\]

Where $\gamma$ is the Euler-Mascheroni constant. The integral is taken on $t = e^{x}$ over regime_law_grid.

Arguments

  • method::LogRegimeAdjusted: Log regime adjustment method.
  • mu::VecNum: Eigenvalues of the law, the weight of each gamma variate.

Returns

  • factor::Number: $\exp(\mathbb{E}[\ln S] - \kappa_{n})$, with $\kappa_{n}$ from regime_kappa.

Related

source
regime_law_factor(
    method::FirstMomentRegimeAdjusted,
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Computes the factor of the first-moment method for a sum whose terms carry the noise of their estimated variances.

The expected root is that of the law of $\mu$, from the two-argument method, plus the second-order term of regime_law_correction.

Arguments

  • method::FirstMomentRegimeAdjusted: First-moment regime adjustment method.
  • mu::VecNum: Eigenvalues of $D \tilde{R} D$.
  • V::MatNum: Their eigenvectors.
  • Cm::MatNum: Covariance of the deviations of the variable of the method.

Returns

  • factor::Number: $((\mathbb{E}[\sqrt{S_{b}}] + \Delta) / d_{n})^{2}$.

Related

source
regime_law_factor(
    method::LogRegimeAdjusted,
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Computes the factor of the log method for a sum whose terms carry the noise of their estimated variances.

The expected log is that of the law of $\mu$, from the two-argument method, plus the second-order term of regime_law_correction.

Arguments

  • method::LogRegimeAdjusted: Log regime adjustment method.
  • mu::VecNum: Eigenvalues of $D \tilde{R} D$.
  • V::MatNum: Their eigenvectors.
  • Cm::MatNum: Covariance of the deviations of the variable of the method.

Returns

  • factor::Number: $\exp(\mathbb{E}[\ln S_{b}] + \Delta - \kappa_{n})$.

Related

source
PortfolioOptimisers.regime_law_shape — Function
regime_law_shape(
    _::FirstMomentRegimeAdjusted
) -> Tuple{Rational{Int64}, Int64}

Returns the shape of each square and the degrees of freedom of its Wishart form, for the first-moment method: a $\chi^{2}(1)$ variate.

Arguments

  • ::FirstMomentRegimeAdjusted: First-moment regime adjustment method.

Returns

  • (a, nu)::Tuple: $(1/2, 1)$.

Related

source
regime_law_shape(
    method::LogRegimeAdjusted
) -> Tuple{Any, Any}

Returns the shape of each square and the degrees of freedom of its Wishart form, for the log method, which takes each square as a $\mathrm{Gamma}(x, y)$ variate: the diagonal of a Wishart matrix of $2 x$ degrees of freedom.

Arguments

  • method::LogRegimeAdjusted: Log regime adjustment method.

Returns

  • (a, nu)::Tuple: $(x, 2 x)$.

Related

source
PortfolioOptimisers.regime_law_weights — Function
regime_law_weights(
    _::FirstMomentRegimeAdjusted,
    ltau::Number
) -> Tuple{Any, Any}

Returns the logs of the weights of the linear and the quadratic integrands of the first-moment correction at one node, the measure $\mathrm{d}\tau = \tau\, \mathrm{d}x$ included.

$\mathbb{E}[y_{i}^{2} S^{-1/2}]$ and $\mathbb{E}[y_{i}^{2} y_{j}^{2} S^{-3/2}]$ are $\Gamma(p)^{-1} \int \tau^{p - 1} \mathbb{E}[\cdot\, e^{-\tau S}]\, \mathrm{d}\tau$ at $p = 1/2$ and $p = 3/2$, so the weights are $\tau^{1/2} / \sqrt{\pi}$ and $2 \tau^{3/2} / \sqrt{\pi}$. regime_law_correction reads the vector $2 \tau d$ in place of $d$, so each weight here carries the factor $1 / (2 \tau)$ of each $d$ that it multiplies, and both weights are $\tau^{-1/2} / (2 \sqrt{\pi})$.

Arguments

  • ::FirstMomentRegimeAdjusted: First-moment regime adjustment method.
  • ltau::Number: Log of the node $\tau$ of the integration variable.

Returns

  • (lwl, lwq)::Tuple: $-(\ln \tau + \ln \pi) / 2 - \ln 2$ in both entries.

Related

source
regime_law_weights(
    method::LogRegimeAdjusted,
    ltau::Number
) -> Tuple{Any, Any}

Returns the logs of the weights of the linear and the quadratic integrands of the log correction at one node, the measure $\mathrm{d}\tau = \tau\, \mathrm{d}x$ included.

$\mathbb{E}[W_{ii} / S]$ and $\mathbb{E}[W_{ii} W_{jj} / S^{2}]$ are $\int \tau^{p - 1} \mathbb{E}[\cdot\, e^{-\tau S}]\, \mathrm{d}\tau$ at $p = 1$ and $p = 2$, and the linear one carries the degrees of freedom $\nu = 2 x$ of the tilted mean, so the weights are $2 x \tau$ and $\tau^{2}$. regime_law_correction reads the vector $2 \tau d$ in place of $d$, so each weight here carries the factor $1 / (2 \tau)$ of each $d$ that it multiplies, and the weights are $x$ and $1 / 4$ at every node.

Arguments

  • method::LogRegimeAdjusted: Log regime adjustment method.
  • ltau::Number: Log of the node $\tau$ of the integration variable, read for its number type alone, so that the log of $x$ is not rounded to a narrower type than that of the data.

Returns

  • (lwl, lwq)::Tuple: $(\ln x, -2 \ln 2)$.

Related

source
PortfolioOptimisers.regime_law_correction — Function
regime_law_correction(
    method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted},
    mu::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    V::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    Cm::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> Any

Computes the second-order term of the expected root or log of a diagonal sum whose terms carry the noise of their estimated variances.

Write each term as $e^{g_{i}} y_{i}^{2}$ (the log method) or $(1 + g_{i})^{2} y_{i}^{2}$ (the first moment), with $y \sim \mathcal{N}(0, D \tilde{R} D)$, the deviation $g$ of mean zero and covariance $C_{m}$, and $g$ independent of $y$. A Taylor expansion of the root or the log of the sum in $g$ gives, to the second order,

\[\Delta = \frac{1}{2} \left(\sum_{i} C_{m, ii}\, \mathbb{E}[y_{i}^{2} h_{1}(S)] - \sum_{i, j} C_{m, ij}\, \mathbb{E}[y_{i}^{2} y_{j}^{2} h_{2}(S)]\right)\,,\]

with $h_{1} = S^{-1/2}$ and $h_{2} = S^{-3/2}$ for the root, and $S^{-1}$ and $S^{-2}$ for the log. Along the direction where every $g_{i}$ is equal, the root and the log are linear in $g$, so the two sums cancel there. Each expectation is an integral over $\tau$ of the tilted law $e^{-\tau S}$, under which $y$ has the covariance $M(\tau) = D \tilde{R} D (I + 2 \tau D \tilde{R} D)^{-1} = V \mathrm{diag}(d(\tau)) V'$, with $d_{k} = \mu_{k} / (1 + 2 \tau \mu_{k})$, and the Wishart moments $\mathbb{E}[W_{ii}] = \nu M_{ii}$ and $\mathbb{E}[W_{ii} W_{jj}] = \nu^{2} M_{ii} M_{jj} + 2 \nu M_{ij}^{2}$.

Algorithm

  1. On each node of regime_law_grid, with $\tau = e^{x} / n$, accumulate the vector $j = \sum w_{l} L d$ and the matrix $J = \sum w_{q} L d d'$, with $L = \prod_{k} (1 + 2 \tau \mu_{k})^{-a}$ and the weights of regime_law_weights. Each term reads $e = 2 \tau d$, whose entries lie in $[0, 1)$, and $\exp(\ln w + \ln L)$, with the factor $1 / (2 \tau)$ of each $d$ in $w$. Alone, $w_{q}$ and $d d'$ overflow and underflow the number type at the two ends of the grid, and their product is bounded there.
  2. The linear sum is $(W' \mathrm{diag}(C_{m}))' j$, with $W = V \circ V$, and the part of the quadratic sum in $M_{ii} M_{jj}$ is $\langle W' C_{m} W, J \rangle$.
  3. The part in $M_{ij}^{2}$ is $\sum_{kl} J_{kl} \sum_{ij} C_{m, ij} V_{ik} V_{jk} V_{il} V_{jl}$. $J$ is positive semi-definite and its eigenvalues fall fast, so with $J = \sum_{q} \sigma_{q} g_{q} g_{q}'$ it is $\sum_{q} \sigma_{q} \sum_{ij} C_{m, ij} (V \mathrm{diag}(g_{q}) V')_{ij}^{2}$ over the $\sigma_{q}$ above the machine epsilon of the largest, a few products of order $n^{3}$ in place of one of order $n^{4}$. At 12 and 40 assets it keeps 2 to 8 terms and agrees with the sum over every node to $10^{-15}$.

Arguments

  • method::Union{FirstMomentRegimeAdjusted, LogRegimeAdjusted}: Regime adjustment method.
  • mu::VecNum: Eigenvalues of $D \tilde{R} D$.
  • V::MatNum: Their eigenvectors.
  • Cm::MatNum: Covariance of the deviations of the variable of the method.

Returns

  • Delta::Number: The second-order term of $\mathbb{E}[\sqrt{S}]$ or of $\mathbb{E}[\ln S]$.

Related

source
PortfolioOptimisers.exp_weight_cross_sum — Function
exp_weight_cross_sum(
    a::Number,
    b::Number,
    K::Integer
) -> Any
exp_weight_cross_sum(
    a::Number,
    b::Number,
    K::Integer,
    hac_lags::Union{Nothing, Integer, AbstractVector{<:Union{Number, AbstractJuMPScalar}}}
) -> Any

Computes the sum of the products of two sets of normalised exponential weights over $K$ observations.

Mathematical definition

\[\sum_{j=0}^{K-1} w_{j}(a)\, w_{j}(b) = \frac{(1 - a)(1 - b)\left(1 - (ab)^{K}\right)}{(1 - ab)\left(1 - a^{K}\right)\left(1 - b^{K}\right)}\,, \quad w_{j}(a) = \frac{(1 - a)\, a^{j}}{1 - a^{K}}\,.\]

At $a = b$ it is the sum of the squared weights, the variance of an estimate of $K$ observations relative to the variance of one. With HAC lags the sum is $\operatorname{tr}(A_{a} A_{b})$ over the banded weight matrices of regime_bias_table, whose law has the eigenvalues of $A$ as its weights, so each lag $i$ that reaches an observation adds $2 k_{i}^{2} (1 - (ab)^{K - i})$ to the factor $1 - (ab)^{K}$ of the numerator. At $a = b$ its inverse is the effective count of observations that regime_bias_open reads.

Arguments

  • a::Number: Decay of the first set of weights.
  • b::Number: Decay of the second set of weights.
  • K::Integer: Count of observations that the weights span.
  • hac_lags::Option{<:Union{Integer, VecNum}}: Count of HAC lags, the weight $k_{i}$ of each lag, or nothing.

Returns

  • s::Number: The sum of the products of the weights.

Related

source
PortfolioOptimisers.hac_ldl_row! — Function
hac_ldl_row!(
    buf::NamedTuple,
    s::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    decay::Number,
    lam::Number,
    m::Integer
) -> NamedTuple

Adds the row of observation m to the banded LDLᵀ factorisation of $I + s B$ at every point of the grid, and adds the log of its pivot to $\ln G$.

The matrix is banded with L lags, so a new row reads only the last L rows of the factor and their pivots. The buffers hold them, the last row first.

Arguments

  • buf::NamedTuple: Buffers of the factorisation (mutated): l[g, r, i], the entry of row $m - r$ at column $m - r - i$; d[g, r], the pivot of row $m - r$; row and num, scratch; lG, $\ln G$ at each point, $-\infty$ at a point whose pivot was not positive; and k, the weight $k_{i}$ of each lag, from hac_lag_weights.
  • s::VecNum: Grid of $s$.
  • decay::Number: Decay of the weights.
  • lam::Number: $\lambda^{m}$.
  • m::Integer: Index of the new row, from zero for the newest observation.

Returns

  • buf::NamedTuple: The buffers after the row.

Related

source
PortfolioOptimisers.inverse_volatility_bias — Function
inverse_volatility_bias(
    C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    sv::Number,
    svc::Number,
    min_val::Number
) -> Any

Computes the second-order excess bias of the regime statistic of the inverse-volatility direction, which is built from the estimate it divides by.

The direction $p_{i} \propto 1 / \hat{\sigma}_{i}$ reads the estimated volatilities, so its statistic is $q^{\top} R q / 1^{\top} \hat{R} 1$ in units of the true volatilities, with $q_{i} = \sigma_{i} / \hat{\sigma}_{i}$. An asset whose volatility is under-estimated takes more weight, and its error enters the numerator but not the denominator. A second-order expansion in the error of the estimate gives the mean of the statistic as the factor of a fixed direction times $1 + \Delta$. The excess is the same for the three regime methods to first order. On one decay $s_{vc} = s_{v}$ and $\Delta = 2 s_{v} \kappa$, with $\kappa = 1 - \sum_{i} c_{i}^{3} / A^{2}$: $1 - 1/n$ at $R = I$ and zero at a correlation of one, where the estimated direction is proportional to the true one.

Mathematical definition

\[\begin{align} \Delta &= s_{v} \left(1 + \frac{\Sigma_{3}}{A} - \frac{2 B}{A^{2}}\right) + s_{vc} \left(1 - \frac{\Sigma_{3}}{A} - \frac{2 T}{A^{2}} + \frac{2 B}{A^{2}}\right)\,, \\ c &= R 1\,, \quad A = 1^{\top} c\,, \quad T = \sum_{i} c_{i}^{3}\,, \quad B = \sum_{ij} c_{i} R_{ij}^{2} c_{j}\,, \quad \Sigma_{3} = \sum_{ij} R_{ij}^{3}\,. \end{align}\]

Where:

  • $R$: Correlation of the block, the plug-in for the true correlation.
  • $s_{v}$: Sum of the squared weights of the variance estimate.
  • $s_{vc}$: Sum of the products of the weights of the variance and the correlation estimates.

The plug-in and the next order leave about $9 s_{v}^{2}$. At 12 assets and a half-life of 10 the formula at the true $R$ matches the simulated excess to $5 \times 10^{-4}$, on both paths. A block whose $A$ is not positive has no inverse-volatility portfolio with a variance, and takes no excess.

Arguments

  • C::MatNum: Bias-corrected covariance block of the contributing assets.
  • sv::Number: Sum of the squared weights of the variance estimate.
  • svc::Number: Sum of the products of the weights of the variance and the correlation estimates.
  • min_val::Number: Floor applied to each variance, as in regime_statistic.

Returns

  • delta::Number: The excess $\Delta$.

Related

source
PortfolioOptimisers.regime_bias_state — Function
regime_bias_state(
    ce::RegimeAdjustedExpWeightedVariance,
    _::Type{T}
) -> Any

Makes the empty table of bias factors for a new state of a regime-adjusted variance estimator.

Arguments

  • ce::RegimeAdjustedExpWeightedVariance: Estimator configuration.
  • ::Type{T}: Element type of the state.

Returns

  • bias::Option{<:VecNum}: An empty vector where debiases holds for ce.debias and a regime method is set, else nothing.

Related

source
regime_bias_state(
    ce::RegimeAdjustedExpWeightedCovariance,
    _::Type{T}
) -> Any

Makes the empty store of bias factors for a new state of a regime-adjusted covariance estimator.

The regime target names the store with regime_bias_store: the store of a DiagonalTarget holds the moments of one term of RegimeTermMoments, a triple for each count, and the HAC kernel of its correlation; the store of a MahalanobisTarget holds the interpolation nodes of its factor for each count of assets; the table of any other target holds one factor for each count.

Arguments

  • ce::RegimeAdjustedExpWeightedCovariance: Estimator configuration.
  • ::Type{T}: Element type of the state.

Returns

  • bias::Option{<:Union{AbstractVector, DiagonalBiasStore, PortfolioBiasStore, MahalanobisBiasStore}}: The empty store where debiases holds for ce.debias and a regime method is set, else nothing.

Related

source
PortfolioOptimisers.safe_regime_cholesky — Function
safe_regime_cholesky(
    C::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    min_val::Number
) -> Any

Factorises a covariance block for the Mahalanobis regime statistic, and refuses rather than throws when no ridge makes it factorise.

A block with fewer observations than assets is singular. Where the estimator has debias = ExactDebias(), regime_target_statistic skips such a block before it reaches this function, so the ridge serves a block that is singular in the data, and the raw statistic of debias = nothing. The regime statistic is one observation of a smoother rather than a result a caller reads, so a refusal skips that observation's regime update, and the fit continues.

Algorithm

  1. Build a RidgeCholeskySquareRoot with ridge = max(min_val, 0) and three tries. A negative min_val gives the ridge of a zero min_val, the floor eps * scale.
  2. Return ridge_cholesky of C under it: the plain factorisation, or the first ridge that makes C factorise, or nothing.

Arguments

  • C::MatNum: Covariance block of the assets that contribute to the statistic.
  • min_val::Number: Scale of the first ridge.

Returns

  • chol::Option{<:LinearAlgebra.Cholesky}: The factorisation, or nothing where no ridge makes the block factorise.

Related

source
PortfolioOptimisers.mahalanobis_bias — Function
mahalanobis_bias(
    decay::Number,
    K::Integer,
    n::Integer
) -> Any

Computes the deterministic equivalent of the mean bias factor of the squared Mahalanobis distance against an exponentially weighted covariance estimate.

The factor is $b = \mathbb{E}[\operatorname{tr}(W^{-1})] / n$ of $W = \sum_{j} w_{j} z_{j} z_{j}^{\top}$, $z_{j} \sim N(0, I_{n})$, which is the mean of $u^{\top} \hat{C}^{-1} u / n$ for a correctly calibrated return $u$ that is independent of the estimate $\hat{C}$. It depends on the weights and on n alone. It solves $1 / b = \sum_{j} w_{j} / (1 + (n + 1) w_{j} b)$, which exists where K > n + 1 and is exact at equal weights. On exponential weights it is 0.55 % above the mean at 12 assets and a half-life of 10, so inverse_wishart_bias takes it only to name the pole of the factor at K = n + 1, and mahalanobis_regime_bias! corrects the result.

Algorithm

  1. Return nothing where K <= n + 1.
  2. Start at $b = 1$. The map $h(b) = b\, g(b) - 1$, with $g$ the right-hand side of the fixed point, is concave and increasing, and $h(1) < 0$, so Newton's method from $b = 1$ rises to the root without overshoot.
  3. Evaluate $g$ and $h'$ with mahalanobis_bias_sums, and stop when the step is below four units in the last place of $b$.

Arguments

  • decay::Number: Decay of the weights.
  • K::Integer: Count of observations in the estimate.
  • n::Integer: Count of assets that contribute to the statistic.

Returns

  • b::Option{<:Number}: The bias factor, or nothing where K <= n + 1.

Related

source
PortfolioOptimisers.mahalanobis_bias_sums — Function
mahalanobis_bias_sums(
    decay::Number,
    K::Integer,
    c::Integer,
    w0::Number,
    b::Number
) -> Tuple{Any, Any}

Computes the right-hand side of the fixed point of mahalanobis_bias and the derivative of $b\, g(b)$.

The weights fall geometrically, and a weight $w_{j}$ with $c\, w_{j}\, b$ below the machine epsilon of decay enters both sums as itself, so the loop stops there and adds the remaining mass $1 - \sum_{i < j} w_{i}$ to each sum. That keeps the cost bounded on a long history.

Arguments

  • decay::Number: Decay of the weights.
  • K::Integer: Count of observations in the estimate.
  • c::Integer: n + 1.
  • w0::Number: Normalised weight of the newest observation, (1 - decay) / (1 - decay^K).
  • b::Number: Current value of the bias factor.

Returns

  • (g, dh)::Tuple{Number, Number}: $\sum_{j} w_{j} / (1 + c\, w_{j}\, b)$ and $\sum_{j} w_{j} / (1 + c\, w_{j}\, b)^{2}$.

Related

source
PortfolioOptimisers.taylor_mul — Function
taylor_mul(
    x::Tuple{var"#s2444", var"#s2444"} where var"#s2444"<:Number,
    y::Tuple{var"#s2443", var"#s2443"} where var"#s2443"<:Number
) -> Tuple{Any, Any}

Multiplies two values carried with their first derivatives.

Arguments

  • x::NTuple{2, <:Number}: $(x, x')$.
  • y::NTuple{2, <:Number}: $(y, y')$.

Returns

  • xy::NTuple{2, <:Number}: $(x y, x' y + x y')$.

Related

source
PortfolioOptimisers.taylor_div — Function
taylor_div(
    x::Tuple{var"#s2444", var"#s2444"} where var"#s2444"<:Number,
    y::Tuple{var"#s2443", var"#s2443"} where var"#s2443"<:Number
) -> Tuple{Any, Any}

Divides two values carried with their first derivatives.

Arguments

  • x::NTuple{2, <:Number}: $(x, x')$.
  • y::NTuple{2, <:Number}: $(y, y')$, with $y \neq 0$.

Returns

  • q::NTuple{2, <:Number}: $(q, q')$ of $q = x / y$, with $q' = (x' - q y') / y$.

Related

source
PortfolioOptimisers.gap_fill_value — Method
gap_fill_value(ce::RegimeAdjustedExpWeightedCovariance) -> Float64

Answer NaN, so a gapped sample reaches the recursion with its gaps intact.

The recursion updates only the sub-block of the assets that are valid at each observation, and the regime weight is taken from that sub-block alone, so a fill decays a frozen block and moves the regime that the block weighs. The consumer therefore hands the sample as it stands, together with the active mask that explains the gap.

Arguments

  • ce: Covariance estimator.

Returns

  • fv::Float64: NaN.

Related

source
Base.copy — Method
copy(
    x::RegimeAdjustedCovarianceState
) -> Union{RegimeAdjustedCovarianceState{Nothing, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows<:Base.RefValue}, RegimeAdjustedCovarianceState{__T_ret_buffer, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows} where {__T_ret_buffer<:DataStructures.CircularBuffer, _A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, _L, _M, _N, _O, _P, __T_rows<:Base.RefValue}}

Copies a RegimeAdjustedCovarianceState, so the copy shares no array with the original.

The copy method of the AbstractPartialFitState interface, which partial_fit calls before it folds. Every array field is copied, and the two scalar fields pass through. The row count goes into a new Ref, so a fold of the copy does not count in the original. The circular buffer of recent centred returns is rebuilt at the same capacity, and each observation it holds is copied into it, so a fold on the copy pushes into a buffer of its own. The two fields of the separate correlation recursion pass through as nothing where they are nothing. The store of bias factors is nothing, an array or a dictionary, and deepcopy copies an array or a dictionary and returns nothing unchanged.

Arguments

  • x: The cache to copy.

Returns

  • state::RegimeAdjustedCovarianceState: A fresh cache, equal to x, whose arrays are fresh.

Related

source
PortfolioOptimisers.observation_count — Method
observation_count(
    state::RegimeAdjustedCovarianceState
) -> Any

RegimeAdjustedCovarianceState method of observation_count.

The state counts the observations of each asset in obs_count, and a reset sets such a count to zero. n_regime_obs counts only the observations that the regime statistic scored. So the method reads rows, the number of folded rows, which is the size(X, 1) of a batch fit over the same rows.

Arguments

  • state: The state that holds the count.

Returns

  • n::Integer: The number of observations folded so far.

Related

source
PortfolioOptimisers.variance_series — Method
variance_series(
    ce::RegimeAdjustedExpWeightedCovariance,
    X::MatNum;
    dims::Int = 1,
    estimation_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    active_mask::Option{<:AbstractMatrix{<:Bool}} = nothing,
    kwargs...
) -> Matrix{<:Number}

Compute the point-in-time regime-adjusted exponentially weighted variance series.

Row t holds the diagonal of what cov returns for the first t observations of X, so no row reads an observation after its own. The update is a recursion over one observation, so this method overrides the expanding-window fallback with a single forward pass: it reads the cache after each observation instead of refitting.

The fallback cannot answer this estimator. It slices X once per row and passes every keyword unsliced, so a mask of the whole window meets a window of t observations and the size check refuses the call.

Arguments

  • ce: Regime-adjusted exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • dims: Dimension along which the computation runs.
  • estimation_mask: Optional boolean matrix with the same size as X. When provided, only assets where estimation_mask[i, :] (or [:, i]) is true contribute to the regime state update for observation i.
  • active_mask: Optional boolean matrix with the same size as X. When provided, assets that become inactive have their covariance and observation count reset.
  • kwargs: More keyword arguments, which the method ignores.

Validation

  • dims in (1, 2).
  • If estimation_mask is not nothing, size(X) == size(estimation_mask).
  • If active_mask is not nothing, size(X) == size(active_mask).

Returns

  • val::Matrix{<:Number}: Variance series, shaped as (T, N) if dims == 1 or (N, T) if dims == 2. An asset with fewer than ce.min_obs observations at row t is NaN there.

Related

source
PortfolioOptimisers.variance_series — Method
variance_series(
    ce::RegimeAdjustedExpWeightedCovariance,
    X::MatNum,
    pnl::Option{<:AssetPanel};
    dims::Int = 1,
    kwargs...
) -> Matrix{<:Number}

Compute the point-in-time regime-adjusted exponentially weighted variance series from a window of an Asset Panel.

This is the diagonal of the covariance of the same call, read after each observation, and it reads the panel's two masks through the same override.

Arguments

  • ce: Regime-adjusted exponentially weighted covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • pnl: Optional AssetPanel. The Coverage Universe of the fit comes from its active mask. If it is nothing, the rule reads only the finiteness of the returns.
  • dims: Dimension along which the computation runs.
  • kwargs: More keyword arguments, which the method ignores.

Returns

  • val::Matrix{<:Number}: Variance series on the full asset universe, shaped as (T, N) if dims == 1 or (N, T) if dims == 2.

Related

source