Gerber covariance: private API
The Gerber statistic measures how often two assets move together. A movement smaller than a threshold does not count. The statistic counts the observations at which both assets move beyond their thresholds in the same direction, and those at which they move beyond them in opposite directions. It uses the counts and not the sizes of the movements, so an extreme movement weighs as much as any other. It extends Kendall's tau [33].
The library implements the three published variants, Gerber0, Gerber1 and Gerber2, which normalise the counts in different ways [34].
The three variants
The functions below count the joint movements and compute the statistic of each variant.
PortfolioOptimisers.gerber_updown — Function
gerber_updown(
ce::GerberCovariance,
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
sd::AbstractArray{<:Union{Number, AbstractJuMPScalar}}
) -> Tuple{Matrix{Bool}, Matrix{Bool}}
Build the up and down indicator matrices that every Gerber correlation algorithm uses.
Mathematical definition
\[\begin{align} U_{t,\,i} &= \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]\,, \\ D_{t,\,i} &= \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]\,. \end{align}\]
Where:
- $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
- $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
- $x_{t,\,i}$: Return of asset $i$ at observation $t$.
- $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
- $\sigma_i$: Standard deviation of asset $i$.
- $T$: Number of observations.
- $N$: Number of assets.
The two bands never overlap. No observation has a mark in both matrices, so their sum is the crossing indicator. At a zero band edge, the sign test keeps the two bands apart. The threshold ce.t = 0 gives a zero band edge, and a zero entry of sd gives one too. The Statistics.cor method prevents the second case, because it raises each entry of sd to at least eps. Without the sign test, both closed comparisons hold on a return of exactly zero, and the sum is two, not one.
For a positive band edge, the sign test changes nothing, because $x_{t,\,i} \geq t \, \sigma_i > 0$ already implies $x_{t,\,i} > 0$. At a zero band edge, $\mathbf{U} - \mathbf{D}$ is the sign of the return, and the Gerber statistic measures how often the signs of two assets agree.
Algorithm
- Multiply the standard deviation vector by the threshold to get the band edge of each asset,
ts = sd * ce.t. - Mark
U[t, i]whenX[t, i] >= ts[i]andX[t, i]is positive. - Mark
D[t, i]whenX[t, i] <= -ts[i]andX[t, i]is negative.
Arguments
ce: Gerber covariance estimator.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.sd: Standard deviation vector ofX, shaped to be consistent withX.
Returns
(U, D)::Tuple{Matrix{Bool}, Matrix{Bool}}: The up and the down indicator matrices.
Related
PortfolioOptimisers.concordance_counts — Function
concordance_counts(
pmn::AbstractMatrix,
ppn::AbstractMatrix
) -> Tuple{Any, Any}
Recover the concordant and the discordant counts from their difference and their sum.
The gerber methods get the difference and the sum from matrix products. This function gets the two counts from them with no more matrix products. The arithmetic is exact, so comovement_ratio receives the same numerator and denominator as the matrix formula.
Mathematical definition
\[\begin{align} n_{c} &= \frac{(n_{c} + n_{d}) + (n_{c} - n_{d})}{2}\,, \\ n_{d} &= \frac{(n_{c} + n_{d}) - (n_{c} - n_{d})}{2}\,. \end{align}\]
Where:
- $n_{c}$: Concordant count of a pair, the observations on which both assets crossed their thresholds in the same direction.
- $n_{d}$: Discordant count of a pair, the observations on which both assets crossed their thresholds in opposite directions.
Arguments
pmn::AbstractMatrix: The differencenconc - ndisc.ppn::AbstractMatrix: The sumnconc + ndisc.
Returns
(nconc, ndisc)::Tuple{AbstractMatrix, AbstractMatrix}: The concordant and the discordant counts.
Related
PortfolioOptimisers.gerber — Method
gerber(
ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber0},
X::MatNum,
sd::ArrNum
) -> MatNumCompute the Gerber correlation matrix with the denominator of Gerber0.
Mathematical definition
\[\begin{align} \hat{\boldsymbol{\rho}} &= \left(\mathbf{H}^\intercal \mathbf{H}\right) \oslash \left(\mathbf{V}^\intercal \mathbf{V}\right)\,. \end{align}\]
Where:
- $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
- $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
- $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
- $\mathbf{V} = \mathbf{U} + \mathbf{D}$: Crossing matrix. Its entry is $1$ when the asset crossed its threshold in either direction, and $0$ when it did not.
- $x_{t,\,i}$: Return of asset $i$ at observation $t$.
- $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
- $\sigma_i$: Standard deviation of asset $i$.
- $T$: Number of observations.
- $N$: Number of assets.
- $\oslash$: Element-wise division.
The entry of $\mathbf{H}^\intercal \mathbf{H}$ is $n_{c} - n_{d}$ and the entry of $\mathbf{V}^\intercal \mathbf{V}$ is $n_{c} + n_{d}$, so this is the pairwise statistic of Gerber0 in matrix form. Equations 5 to 8 of [33] give the same matrix form.
Algorithm
- Build the indicator matrices
UandDwithgerber_updown. - Form the signed crossing matrix
UmD = U - Dand the crossing matrixUpD = U + D. - Recover the concordant count
nconcand the discordant countndiscfromtranspose(UmD) * UmDandtranspose(UpD) * UpDwithconcordance_counts. - Reduce every pair to its statistic with
comovement_ratio. The result isrho. - Write one onto a zero diagonal entry of
rhowithcomovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry, and that entry is one by definition. - Repair
rhowithposdef!. It does nothing whence.pdmisnothing.
Arguments
ce: Gerber covariance estimator. Itsalgfield is aGerber0.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.sd: Standard deviation vector ofX, shaped to be consistent withX.
Returns
rho::MatNum: Correlation matrixassets x assets.
Related
GerberCovarianceGerber0gerber_updownconcordance_countscomovement_ratiocomovement_unit_diagonal!posdef!
References
- [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, Equations 5 to 8.
PortfolioOptimisers.gerber — Method
gerber(
ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber1},
X::MatNum,
sd::ArrNum
) -> MatNumCompute the Gerber correlation matrix with the denominator of Gerber1.
Mathematical definition
\[\begin{align} \hat{\boldsymbol{\rho}} &= \left(\mathbf{H}^\intercal \mathbf{H}\right) \oslash \left(T \boldsymbol{1}\boldsymbol{1}^\intercal - \mathbf{N}^\intercal \mathbf{N}\right)\,. \end{align}\]
Where:
- $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
- $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
- $\mathbf{N} \in \{0,1\}^{T \times N}$: Neutral indicator matrix, $N_{t,\,i} = \mathbf{1}[\lvert x_{t,\,i} \rvert < t \, \sigma_i \lor x_{t,\,i} = 0]$. It is the complement of $\mathbf{U} + \mathbf{D}$.
- $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
- $x_{t,\,i}$: Return of asset $i$ at observation $t$.
- $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
- $\sigma_i$: Standard deviation of asset $i$.
- $T$: Number of observations.
- $N$: Number of assets.
- $\oslash$: Element-wise division.
- $\boldsymbol{1}$: Vector of ones.
The entry of $\mathbf{N}^\intercal \mathbf{N}$ counts the observations on which neither asset crosses. The denominator therefore counts the observations on which at least one of them crosses. That is $n_{c} + n_{d} + n_{n}$, the pairwise denominator of Gerber1, and $T - n^{NN}$ of Equation 11 of [33]. The source states no matrix form of this denominator, so the product $\mathbf{N}^\intercal \mathbf{N}$ is the form of the library.
Algorithm
- Build the indicator matrices
UandDwithgerber_updown. - Form the neutral matrix
Nt, which marks the observations on which the asset crosses in neither direction. - Form
NtN = transpose(Nt) * Nt, the count of observations on which both assets of a pair are neutral. Formnneutral, the count of neutral observations of each asset. - Form the signed crossing matrix
UmD = U - D. - Recover the concordant count
nconcand the discordant countndiscwithconcordance_counts. The difference istranspose(UmD) * UmD. The sum isT .- nneutral .- transpose(nneutral) .+ NtN, the count of observations on which both assets cross. - Form
nneut = nneutral .+ transpose(nneutral) .- 2 .* NtN, the count of observations on which exactly one asset of the pair crosses. - Reduce every pair to its statistic with
comovement_ratio. The result isrho. - Write one onto a zero diagonal entry of
rhowithcomovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry, and that entry is one by definition. - Repair
rhowithposdef!. It does nothing whence.pdmisnothing.
Arguments
ce: Gerber covariance estimator. Itsalgfield is aGerber1.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.sd: Standard deviation vector ofX, shaped to be consistent withX.
Returns
rho::MatNum: Correlation matrixassets x assets.
Related
GerberCovarianceGerber1gerber_updownconcordance_countscomovement_ratiocomovement_unit_diagonal!posdef!
References
- [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, Equation 11.
PortfolioOptimisers.gerber — Method
gerber(
ce::GerberCovariance{<:Any, <:Any, <:Any, <:Any, <:Gerber2},
X::MatNum,
sd::ArrNum
) -> MatNumCompute the Gerber correlation matrix with the denominator of Gerber2.
Mathematical definition
\[\begin{align} \mathbf{G} &= \mathbf{H}^\intercal \mathbf{H}\,, \\ \boldsymbol{g} &= \sqrt{\mathrm{diag}(\mathbf{G})}\,, \\ \hat{\boldsymbol{\rho}} &= \mathbf{G} \oslash (\boldsymbol{g} \boldsymbol{g}^\intercal)\,. \end{align}\]
Where:
- $\mathbf{U} \in \{0,1\}^{T \times N}$: Up indicator matrix, $U_{t,\,i} = \mathbf{1}[x_{t,\,i} \geq t \, \sigma_i \land x_{t,\,i} > 0]$.
- $\mathbf{D} \in \{0,1\}^{T \times N}$: Down indicator matrix, $D_{t,\,i} = \mathbf{1}[x_{t,\,i} \leq -t \, \sigma_i \land x_{t,\,i} < 0]$.
- $\mathbf{H} = \mathbf{U} - \mathbf{D}$: Signed crossing matrix. Its entry is $1$ when the asset crossed upwards, $-1$ when it crossed downwards, and $0$ when it did not cross.
- $\mathbf{G}$: Net co-movement matrix before the division. Its entry is $n_{c} - n_{d}$.
- $\boldsymbol{g}$: Vector of the square roots of the diagonal of $\mathbf{G}$.
- $x_{t,\,i}$: Return of asset $i$ at observation $t$.
- $t$: Threshold parameter, when $t$ is a symbol that stands alone. A subscript $t$ is the observation index. An asset crosses at an observation when its return is at least $t$ of its own standard deviations away from zero. A return of exactly zero never crosses.
- $\sigma_i$: Standard deviation of asset $i$.
- $\oslash$: Element-wise division.
The diagonal of $\mathbf{G}$ counts the crossings of each asset, so the diagonal of $\hat{\boldsymbol{\rho}}$ is one by construction. Entry $(i, j)$ is the net co-movement count divided by the geometric mean of the crossing counts of the two assets. This is Equation 1 of [34] and footnote 3 of [33]. The matrix form is the form of the library.
Algorithm
- Build the indicator matrices
UandDwithgerber_updown. - Form the signed crossing matrix
UmD = U - D. - Form the net co-movement matrix
rho = transpose(UmD) * UmD. - Normalise
rhoin place withstandardise_comovement!. - Write one onto a zero diagonal entry of
rhowithcomovement_unit_diagonal!. An asset that never crosses its threshold has a zero diagonal entry of $\mathbf{G}$. The clamp of step 4 keeps that entry at zero, and the entry is one by definition. - Repair
rhowithposdef!. It does nothing whence.pdmisnothing.
Arguments
ce: Gerber covariance estimator. Itsalgfield is aGerber2.X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.sd: Standard deviation vector ofX, shaped to be consistent withX.
Returns
rho::MatNum: Correlation matrixassets x assets.
Related
References
- [33] S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021). Section 2.2, footnote 3.
- [34] E. Flint and D. Polakow. Deconstructing the Gerber statistic. Finance Research Letters 56, 104144 (2023). Section 2, Equation 1.
PortfolioOptimisers.show_fields — Method
show_fields(_::GerberCovariance) -> NTuple{5, Symbol}
Renders every field of a GerberCovariance except cache.
A cache holds the state of an incremental fit, not a part of the configuration. Without this method, that state prints under the estimator at every site that renders one. Set set_show_nothing_fields!(:GerberCovariance, true) to render it.
Arguments
::GerberCovariance: Gerber covariance estimator, read for its type alone.
Returns
fields::Tuple: The field names to render, which is(:ve, :me, :pdm, :t, :alg).
Related
References
- [33]
- S. Gerber, H. Markowitz, P. Ernst, Y. Miao, B. Javid and P. Sargen. The Gerber statistic: A robust co-movement measure for portfolio optimization. Available at SSRN 3880054 (2021).
- [34]
- E. Flint and D. Polakow. Deconstructing the Gerber statistic. Finance Research Letters 56, 104144 (2023).