Detone

Most assets move with the market, and that common movement dominates their correlations. Detoning removes the n largest eigenvalues, which carry the market movement, so the correlations that remain show how the assets relate to each other apart from the market [26].

A detoned matrix can have zero or negative eigenvalues, so it does not suit an optimisation that needs a positive definite covariance. Use it where the matrix only groups the assets, as in a clustering optimisation.

PortfolioOptimisers.AbstractDetoneEstimator — Type
abstract type AbstractDetoneEstimator <: AbstractEstimator

Abstract supertype for all detoning estimators.

A type that detones a matrix is a subtype of AbstractDetoneEstimator.

Interfaces

To add a detoning estimator, subtype AbstractDetoneEstimator, keep its parameters in the fields of the struct, and add these methods:

  • detone!(dt::AbstractDetoneEstimator, X::MatNum) -> MatNum: Detones X in place.
  • detone(dt::AbstractDetoneEstimator, X::MatNum) -> MatNum: Optional. Returns the detoned X in a new matrix. A fallback method copies X and calls detone!, so add this method only when it avoids the copy.

Arguments

  • dt: Matrix detoning estimator.
  • X: Covariance-like or correlation-like matrix assets × assets.

Returns

  • X::MatNum: The detoned input matrix X.

Examples

The example below defines a minimal detoning estimator.

julia> struct MyDetoneEstimator <: PortfolioOptimisers.AbstractDetoneEstimator endjulia> function PortfolioOptimisers.detone!(dt::MyDetoneEstimator, X::PortfolioOptimisers.MatNum)           # Implement your in-place detoning estimator here.           println("Detoning matrix in-place...")           return X       endjulia> function PortfolioOptimisers.detone(dt::MyDetoneEstimator, X::PortfolioOptimisers.MatNum)           X = copy(X)           println("Copy X...")           detone!(dt, X)           return X       endjulia> detone!(MyDetoneEstimator(), [1.0 2.0; 2.0 1.0])Detoning matrix in-place...2×2 Matrix{Float64}: 1.0  2.0 2.0  1.0julia> detone(MyDetoneEstimator(), [1.0 2.0; 2.0 1.0])Copy X...Detoning matrix in-place...2×2 Matrix{Float64}: 1.0  2.0 2.0  1.0

Related

source
PortfolioOptimisers.Detone — Type
struct Detone{__T_pdm, __T_n} <: AbstractDetoneEstimator

Removes the n largest principal components, the market modes, from a covariance or correlation matrix, or fewer when the n-th of them ties with the next one. detone! and detone apply it.

In financial data, the largest principal components often show movements of the whole market, which can hide the signals of single assets. The field n sets the largest number of these components that Detone removes, so the result shows the relations between assets that the market modes do not explain [26].

A detoned matrix is singular before posdef! runs under pdm, because the subtraction sets the removed eigenvalues to zero. Section 2.6 of [26] states that this is not a problem for a clustering method, which does not invert the matrix. That section also states that a mean-variance optimisation cannot use the detoned matrix directly.

Mathematical definition

The algorithm subtracts the $n$ largest eigenmodes, and rescales the remainder to a unit diagonal:

\[\begin{align} \mathbf{C} &= \mathbf{X} - \sum_{k=N-n+1}^{N} \lambda_k \boldsymbol{v}_k \boldsymbol{v}_k^\intercal\,, \\ \tilde{X}_{ij} &= \frac{C_{ij}}{\sqrt{C_{ii} C_{jj}}}\,. \end{align}\]

Where:

  • $\mathbf{C}$: The remainder after the subtraction of the market modes.
  • $\tilde{\mathbf{X}}$: The detoned matrix.
  • $\mathbf{X}$: The original correlation or covariance matrix.
  • $\lambda_k$: The $k$-th eigenvalue of $\mathbf{X}$ in ascending order, so the sum takes the $n$ largest.
  • $\boldsymbol{v}_k$: The eigenvector of $\lambda_k$.
  • $n$: The number of market modes to remove. After a tie, the algorithm removes $m < n$ modes, as the paragraph below states.
  • $N$: Number of assets.

When the algorithm subtracts a set of eigenmodes, the diagonal of $\mathbf{C}$ falls below one. So the rescaling changes every entry of the matrix. The subtraction sets the removed eigenvalues to zero and keeps the others, so $\mathbf{C}$ and $\tilde{\mathbf{X}}$ are positive semidefinite and singular. posdef! then runs on $\tilde{\mathbf{X}}$ under pdm.

The $n$ largest eigenmodes are one subspace only when $\lambda_{N-n+1} > \lambda_{N-n}$. When the two eigenvalues are equal, the eigenvectors in the subtraction are an arbitrary choice from a larger subspace, and the result depends on that choice. The identity is such a matrix, and Denoise gives the identity when it finds no signal. So the algorithm removes the largest number $m \leq n$ of leading modes that is one subspace, and it gives the matrix back when no such $m$ exists. A covariance then comes back to within the rounding of its conversion to a correlation and back. A tie is a difference of not more than $N \varepsilon \max_k \lvert \lambda_k \rvert$, the rounding bound of the decomposition. An asset whose remainder $C_{ii}$ is not more than that bound has all of its variance in the removed modes, so it has no correlation after the subtraction. Its row and column of $\tilde{\mathbf{X}}$ are zero, with a one on the diagonal. So the diagonal of a covariance returns without change, and $n = N$ gives the identity.

Equation 5 of [31] splits a correlation matrix into the market mode, the group modes and the noise modes, and the subtraction at $n = 1$ removes the first of those parts. That work does not rescale. Section 2.6 of [26] states detoning. It subtracts the market components, usually one but possibly more, and rescales the remainder to a unit diagonal. A search of the literature found no earlier work that states the rescale. That section detones a denoised matrix, and this estimator detones the matrix that it gets.

Algorithm

detone! runs these steps under this estimator.

  1. Read dt.n into n, and check that 0 < n <= size(X, 2).
  2. When X has a zero variance, set its row and column to zero and detone the block of the positive variances with a recursive call, through zero_variance_block!, and return X. A zero variance has no correlation, so step 4 cannot standardise it. The recursive call checks dt.n against the size of the block.
  3. Read the diagonal of X into s. When any entry of s is not one, X is a covariance matrix. Then replace s with its square roots and convert X to a correlation matrix with StatsBase.cov2cor!. The test is any(!isone, s), so it is the value of the diagonal that decides, never the type of X.
  4. Eigendecompose X into the ascending eigenvalues vals and the eigenvectors vecs. Set the tolerance tol = N * eps(T) * maximum(abs, vals).
  5. Find m, the largest number in 1:n for which m == N, or vals[N - m + 1] - vals[N - m] > tol. When no such m exists, skip to step 10. So dt.n = 1 removes the single largest component, which is the market mode, when it stands above the next one.
  6. Subtract the m largest eigenmodes from X. The result is the remainder $\mathbf{C}$.
  7. Set the row and the column of each asset whose remainder on the diagonal is at most tol to zero, and its diagonal entry to one.
  8. Rescale the remainder to unit diagonal with StatsBase.cov2cor.
  9. Repair the rescaled matrix with posdef!, under dt.pdm.
  10. When step 3 converted a covariance matrix, convert X back with StatsBase.cor2cov!. The standard deviations are the ones read in step 3, so the original diagonal returns exactly.

Fields

  • pdm: Optional positive definite matrix estimator.
  • n: Largest number of leading principal components to remove. The detoner removes fewer when the last of them ties with the next eigenvalue, as Detone states.

Constructors

Detone(;    pdm::Option{<:AbstractPosdefEstimator} = Posdef(),    n::Integer = 1,) -> Detone

Keywords correspond to the struct's fields.

Validation

  • n > 0.

Examples

julia> Detone(; n = 2)Detone  pdm ┼ Posdef      │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton      │   kwargs ┴ @NamedTuple{}: NamedTuple()    n ┴ Int64: 2

Related

References

  • [31] D.-H. Kim and H. Jeong. Systematic analysis of group identification in stock markets. Phys. Rev. E 72, 046133 (2005). Equation 5.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 2.6.
source
PortfolioOptimisers.detone — Function
detone(dt::Option{<:AbstractDetoneEstimator}, X::MatNum) -> MatNum

Out-of-place version of detone!.

Algorithm

  1. Copy X.
  2. Apply detone! to the copy, and return it. The function never changes the input.

Arguments

  • dt: Optional matrix detoning estimator.
    • ::Detone: Removes at most the dt.n largest principal components from a copy of X.
    • ::Nothing: Returns X unchanged.
  • X: Covariance-like or correlation-like matrix assets × assets.

Returns

  • X::MatNum: A new matrix equal to the detoned version of the input.

Examples

julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5);       X = X' * X;julia> Xd = detone(Detone(), X);julia> size(Xd)(5, 5)

Related

References

  • [31] D.-H. Kim and H. Jeong. Systematic analysis of group identification in stock markets. Phys. Rev. E 72, 046133 (2005). Equation 5.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 2.6.
source
PortfolioOptimisers.detone! — Function
detone!(dt::Option{<:AbstractDetoneEstimator}, X::MatNum) -> MatNum

Removes the n largest principal components, the market modes, from a covariance or correlation matrix in place, or fewer when the n-th of them ties with the next one.

When the diagonal of X is not all ones, the function converts X to a correlation matrix, applies the algorithm, and scales the result back to a covariance.

A matrix with no leading mode that stands above the next one, such as the identity, comes back, to within the rounding of the conversion of a covariance, as Detone states.

Arguments

  • dt: Optional matrix detoning estimator.

    • ::Detone: Removes at most the dt.n largest principal components from X in place.
    • ::Nothing: Returns X unchanged.
  • X: Covariance-like or correlation-like matrix assets × assets.

Validation

  • 0 < dt.n <= size(X, 2). When X has a zero variance, dt.n is also at most the number of positive variances, which the call on their block checks.

Returns

  • X::MatNum: The input matrix X, modified in place.

Examples

julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5);julia> X = X' * X5×5 Matrix{Float64}: 3.29494  2.0765   1.73334  2.01524  1.77493 2.0765   2.46967  1.39953  1.97242  2.07886 1.73334  1.39953  1.90712  1.17071  1.30459 2.01524  1.97242  1.17071  2.24818  1.87091 1.77493  2.07886  1.30459  1.87091  2.44414julia> detone!(Detone(), X)5×5 Matrix{Float64}:  3.29494    -1.14673     0.0868439  -0.502106   -1.71581 -1.14673     2.46967    -0.876289   -0.0864304   0.274663  0.0868439  -0.876289    1.90712    -1.18851    -0.750345 -0.502106   -0.0864304  -1.18851     2.24818    -0.0774753 -1.71581     0.274663   -0.750345   -0.0774753   2.44414

Related

References

  • [31] D.-H. Kim and H. Jeong. Systematic analysis of group identification in stock markets. Phys. Rev. E 72, 046133 (2005). Equation 5.
  • [26] M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020). Section 2.6.
source

References

[26]
M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
[31]
D.-H. Kim and H. Jeong. Systematic analysis of group identification in stock markets. Phys. Rev. E 72, 046133 (2005).