Gerber covariance: private API

The Gerber statistic measures how often two assets move together. A movement smaller than a threshold does not count. The statistic counts the observations at which both assets move beyond their thresholds in the same direction, and those at which they move beyond them in opposite directions. It uses the counts and not the sizes of the movements, so an extreme movement weighs as much as any other. It extends Kendall's tau [33].

The library implements the three published variants, Gerber0, Gerber1 and Gerber2, which normalise the counts in different ways [34].

The three variants

The functions below count the joint movements and compute the statistic of each variant.

PortfolioOptimisers.gerber_updown — Function
gerber_updown(
    ce::GerberCovariance,
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    sd::AbstractArray{<:Union{Number, AbstractJuMPScalar}}
) -> Tuple{Matrix{Bool}, Matrix{Bool}}

Build the up and down indicator matrices that every Gerber correlation algorithm uses.

Mathematical definition

\[\begin{align} U_{t,\,i} &= \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]\,, \\ D_{t,\,i} &= \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]\,. \end{align}\]

Where:

  • $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
  • $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
  • $x_{t,\,i}$: Return of asset $i$ at observation $t$.
  • $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
  • $\sigma_i$: Standard deviation of asset $i$.
  • $T$: Number of observations.
  • $N$: Number of assets.

The two bands never overlap. No observation has a mark in both matrices, so their sum is the crossing indicator. At a zero band edge, the sign test keeps the two bands apart. The threshold ce.t = 0 gives a zero band edge, and a zero entry of sd gives one too. The Statistics.cor method prevents the second case, because it raises each entry of sd to at least eps. Without the sign test, both closed comparisons hold on a return of exactly zero, and the sum is two, not one.

For a positive band edge, the sign test changes nothing, because $x_{t,\,i} \geq t \, \sigma_i > 0$ already implies $x_{t,\,i} > 0$. At a zero band edge, $\mathbf{U} - \mathbf{D}$ is the sign of the return, and the Gerber statistic measures how often the signs of two assets agree.

Algorithm

  1. Multiply the standard deviation vector by the threshold to get the band edge of each asset, ts = sd * ce.t.
  2. Mark U[t, i] when X[t, i] >= ts[i] and X[t, i] is positive.
  3. Mark D[t, i] when X[t, i] <= -ts[i] and X[t, i] is negative.

Arguments

  • ce: Gerber covariance estimator.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • sd: Standard deviation vector of X, shaped to be consistent with X.

Returns

  • (U, D)::Tuple{Matrix{Bool}, Matrix{Bool}}: The up and the down indicator matrices.

Related

source
PortfolioOptimisers.concordance_counts — Function
concordance_counts(
    pmn::AbstractMatrix,
    ppn::AbstractMatrix
) -> Tuple{Any, Any}

Recover the concordant and the discordant counts from their difference and their sum.

The gerber methods get the difference and the sum from matrix products. This function gets the two counts from them with no more matrix products. The arithmetic is exact, so comovement_ratio receives the same numerator and denominator as the matrix formula.

Mathematical definition

\[\begin{align} n_{c} &= \frac{(n_{c} + n_{d}) + (n_{c} - n_{d})}{2}\,, \\ n_{d} &= \frac{(n_{c} + n_{d}) - (n_{c} - n_{d})}{2}\,. \end{align}\]

Where:

  • $n_{c}$: Concordant count of a pair, the observations on which both assets crossed their thresholds in the same direction.
  • $n_{d}$: Discordant count of a pair, the observations on which both assets crossed their thresholds in opposite directions.

Arguments

  • pmn::AbstractMatrix: The difference nconc - ndisc.
  • ppn::AbstractMatrix: The sum nconc + ndisc.

Returns

  • (nconc, ndisc)::Tuple{AbstractMatrix, AbstractMatrix}: The concordant and the discordant counts.

Related

source
PortfolioOptimisers.gerber — Method
gerber(
    ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber0},
    X::MatNum,
    sd::ArrNum
) -> MatNum

Compute the Gerber correlation matrix with the denominator of Gerber0.

Mathematical definition

\[\begin{align} \hat{\boldsymbol{\rho}} &= \left(\mathbf{H}^\intercal \mathbf{H}\right) \oslash \left(\mathbf{V}^\intercal \mathbf{V}\right)\,. \end{align}\]

Where:

  • $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
  • $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
  • $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
  • $\mathbf{V} = \mathbf{U} + \mathbf{D}$: Crossing matrix. Its entry is $1$ when the asset crossed its threshold in either direction, and $0$ when it did not.
  • $x_{t,\,i}$: Return of asset $i$ at observation $t$.
  • $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
  • $\sigma_i$: Standard deviation of asset $i$.
  • $T$: Number of observations.
  • $N$: Number of assets.
  • $\oslash$: Element-wise division.

The entry of $\mathbf{H}^\intercal \mathbf{H}$ is $n_{c} - n_{d}$ and the entry of $\mathbf{V}^\intercal \mathbf{V}$ is $n_{c} + n_{d}$, so this is the pairwise statistic of Gerber0 in matrix form. Equations 5 to 8 of [33] give the same matrix form.

Algorithm

  1. Build the indicator matrices U and D with gerber_updown.
  2. Form the signed crossing matrix UmD = U - D and the crossing matrix UpD = U + D.
  3. Recover the concordant count nconc and the discordant count ndisc from transpose(UmD) * UmD and transpose(UpD) * UpD with concordance_counts.
  4. Reduce every pair to its statistic with comovement_ratio. The result is rho.
  5. Write one onto a zero diagonal entry of rho with comovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry, and that entry is one by definition.
  6. Repair rho with posdef!. It does nothing when ce.pdm is nothing.

Arguments

  • ce: Gerber covariance estimator. Its alg field is a Gerber0.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • sd: Standard deviation vector of X, shaped to be consistent with X.

Returns

  • rho::MatNum: Correlation matrix assets x assets.

Related

References

  • [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, Equations 5 to 8.
source
PortfolioOptimisers.gerber — Method
gerber(
    ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber1},
    X::MatNum,
    sd::ArrNum
) -> MatNum

Compute the Gerber correlation matrix with the denominator of Gerber1.

Mathematical definition

\[\begin{align} \hat{\boldsymbol{\rho}} &= \left(\mathbf{H}^\intercal \mathbf{H}\right) \oslash \left(T \boldsymbol{1}\boldsymbol{1}^\intercal - \mathbf{N}^\intercal \mathbf{N}\right)\,. \end{align}\]

Where:

  • $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
  • $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
  • $\mathbf{N} \in \{0,1\}^{T \times N}$: Neutral indicator matrix, $N_{t,\,i} = \mathbf{1}[\lvert x_{t,\,i} \rvert < t \, \sigma_i \lor x_{t,\,i} = 0]$. It is the complement of $\mathbf{U} + \mathbf{D}$.
  • $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
  • $x_{t,\,i}$: Return of asset $i$ at observation $t$.
  • $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
  • $\sigma_i$: Standard deviation of asset $i$.
  • $T$: Number of observations.
  • $N$: Number of assets.
  • $\oslash$: Element-wise division.
  • $\boldsymbol{1}$: Vector of ones.

The entry of $\mathbf{N}^\intercal \mathbf{N}$ counts the observations on which neither asset crosses. The denominator therefore counts the observations on which at least one of them crosses. That is $n_{c} + n_{d} + n_{n}$, the pairwise denominator of Gerber1, and $T - n^{NN}$ of Equation 11 of [33]. The source states no matrix form of this denominator, so the product $\mathbf{N}^\intercal \mathbf{N}$ is the form of the library.

Algorithm

  1. Build the indicator matrices U and D with gerber_updown.
  2. Form the neutral matrix Nt, which marks the observations on which the asset crosses in neither direction.
  3. Form NtN = transpose(Nt) * Nt, the count of observations on which both assets of a pair are neutral. Form nneutral, the count of neutral observations of each asset.
  4. Form the signed crossing matrix UmD = U - D.
  5. Recover the concordant count nconc and the discordant count ndisc with concordance_counts. The difference is transpose(UmD) * UmD. The sum is T .- nneutral .- transpose(nneutral) .+ NtN, the count of observations on which both assets cross.
  6. Form nneut = nneutral .+ transpose(nneutral) .- 2 .* NtN, the count of observations on which exactly one asset of the pair crosses.
  7. Reduce every pair to its statistic with comovement_ratio. The result is rho.
  8. Write one onto a zero diagonal entry of rho with comovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry, and that entry is one by definition.
  9. Repair rho with posdef!. It does nothing when ce.pdm is nothing.

Arguments

  • ce: Gerber covariance estimator. Its alg field is a Gerber1.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • sd: Standard deviation vector of X, shaped to be consistent with X.

Returns

  • rho::MatNum: Correlation matrix assets x assets.

Related

References

  • [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, Equation 11.
source
PortfolioOptimisers.gerber — Method
gerber(
    ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber2},
    X::MatNum,
    sd::ArrNum
) -> MatNum

Compute the Gerber correlation matrix with the denominator of Gerber2.

Mathematical definition

\[\begin{align} \mathbf{G} &= \mathbf{H}^\intercal \mathbf{H}\,, \\ \boldsymbol{g} &= \sqrt{\mathrm{diag}(\mathbf{G})}\,, \\ \hat{\boldsymbol{\rho}} &= \mathbf{G} \oslash (\boldsymbol{g} \boldsymbol{g}^\intercal)\,. \end{align}\]

Where:

  • $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
  • $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
  • $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
  • $\mathbf{G}$: Net co-movement matrix before the division. Its entry is $n_{c} - n_{d}$.
  • $\boldsymbol{g}$: Vector of the square roots of the diagonal of $\mathbf{G}$.
  • $x_{t,\,i}$: Return of asset $i$ at observation $t$.
  • $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
  • $\sigma_i$: Standard deviation of asset $i$.
  • $\oslash$: Element-wise division.

The diagonal of $\mathbf{G}$ counts the crossings of each asset, so the diagonal of $\hat{\boldsymbol{\rho}}$ is one by construction. Entry $(i, j)$ is the net co-movement count divided by the geometric mean of the crossing counts of the two assets. This is Equation 1 of [34] and footnote 3 of [33]. The matrix form is the form of the library.

Algorithm

  1. Build the indicator matrices U and D with gerber_updown.
  2. Form the signed crossing matrix UmD = U - D.
  3. Form the net co-movement matrix rho = transpose(UmD) * UmD.
  4. Normalise rho in place with standardise_comovement!.
  5. Write one onto a zero diagonal entry of rho with comovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry of $\mathbf{G}$. The clamp of step 4 keeps that entry at zero, and the entry is one by definition.
  6. Repair rho with posdef!. It does nothing when ce.pdm is nothing.

Arguments

  • ce: Gerber covariance estimator. Its alg field is a Gerber2.
  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • sd: Standard deviation vector of X, shaped to be consistent with X.

Returns

  • rho::MatNum: Correlation matrix assets x assets.

Related

References

  • [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, footnote 3.
  • [34] E. Flint and D. Polakow. Deconstructing the Gerber statistic. Finance Research Letters 56, 104144 (2023). Section 2, Equation 1.
source
PortfolioOptimisers.show_fields — Method
show_fields(_::GerberCovariance) -> NTuple{5, Symbol}

Renders every field of a GerberCovariance except cache.

A cache holds the state of an incremental fit, not a part of the configuration. Without this method, that state prints under the estimator at every site that renders one. Set set_show_nothing_fields!(:GerberCovariance, true) to render it.

Arguments

  • ::GerberCovariance: Gerber covariance estimator, read for its type alone.

Returns

  • fields::Tuple: The field names to render, which is (:ve, :me, :pdm, :t, :alg).

Related

source

References

[33]
S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021).
[34]
E. Flint and D. Polakow. Deconstructing the Gerber statistic. Finance Research Letters 56, 104144 (2023).