Detone
Most assets move with the market, and that common movement dominates their correlations. Detoning removes the n largest eigenvalues, which carry the market movement, so the correlations that remain show how the assets relate to each other apart from the market [26].
A detoned matrix can have zero or negative eigenvalues, so it does not suit an optimisation that needs a positive definite covariance. Use it where the matrix only groups the assets, as in a clustering optimisation.
PortfolioOptimisers.AbstractDetoneEstimator — Type
abstract type AbstractDetoneEstimator <: AbstractEstimatorAbstract supertype for all detoning estimators.
A type that detones a matrix is a subtype of AbstractDetoneEstimator.
Interfaces
To add a detoning estimator, subtype AbstractDetoneEstimator, keep its parameters in the fields of the struct, and add these methods:
detone!(dt::AbstractDetoneEstimator, X::MatNum) -> MatNum: DetonesXin place.detone(dt::AbstractDetoneEstimator, X::MatNum) -> MatNum: Optional. Returns the detonedXin a new matrix. A fallback method copiesXand callsdetone!, so add this method only when it avoids the copy.
Arguments
dt: Matrix detoning estimator.X: Covariance-like or correlation-like matrixassets × assets.
Returns
X::MatNum: The detoned input matrixX.
Examples
The example below defines a minimal detoning estimator.
julia> struct MyDetoneEstimator <: PortfolioOptimisers.AbstractDetoneEstimator endjulia> function PortfolioOptimisers.detone!(dt::MyDetoneEstimator, X::PortfolioOptimisers.MatNum) # Implement your in-place detoning estimator here. println("Detoning matrix in-place...") return X endjulia> function PortfolioOptimisers.detone(dt::MyDetoneEstimator, X::PortfolioOptimisers.MatNum) X = copy(X) println("Copy X...") detone!(dt, X) return X endjulia> detone!(MyDetoneEstimator(), [1.0 2.0; 2.0 1.0])Detoning matrix in-place...2×2 Matrix{Float64}: 1.0 2.0 2.0 1.0julia> detone(MyDetoneEstimator(), [1.0 2.0; 2.0 1.0])Copy X...Detoning matrix in-place...2×2 Matrix{Float64}: 1.0 2.0 2.0 1.0Related
PortfolioOptimisers.Detone — Type
struct Detone{__T_pdm, __T_n} <: AbstractDetoneEstimatorRemoves the n largest principal components, the market modes, from a covariance or correlation matrix, or fewer when the n-th of them ties with the next one. detone! and detone apply it.
In financial data, the largest principal components often show movements of the whole market, which can hide the signals of single assets. The field n sets the largest number of these components that Detone removes, so the result shows the relations between assets that the market modes do not explain [26].
A detoned matrix is singular before posdef! runs under pdm, because the subtraction sets the removed eigenvalues to zero. Section 2.6 of [26] states that this is not a problem for a clustering method, which does not invert the matrix. That section also states that a mean-variance optimisation cannot use the detoned matrix directly.
Mathematical definition
The algorithm subtracts the $n$ largest eigenmodes, and rescales the remainder to a unit diagonal:
\[\begin{align} \mathbf{C} &= \mathbf{X} - \sum_{k=N-n+1}^{N} \lambda_k \boldsymbol{v}_k \boldsymbol{v}_k^\intercal\,, \\ \tilde{X}_{ij} &= \frac{C_{ij}}{\sqrt{C_{ii} C_{jj}}}\,. \end{align}\]
Where:
- $\mathbf{C}$: The remainder after the subtraction of the market modes.
- $\tilde{\mathbf{X}}$: The detoned matrix.
- $\mathbf{X}$: The original correlation or covariance matrix.
- $\lambda_k$: The $k$-th eigenvalue of $\mathbf{X}$ in ascending order, so the sum takes the $n$ largest.
- $\boldsymbol{v}_k$: The eigenvector of $\lambda_k$.
- $n$: The number of market modes to remove. After a tie, the algorithm removes $m < n$ modes, as the paragraph below states.
- $N$: Number of assets.
When the algorithm subtracts a set of eigenmodes, the diagonal of $\mathbf{C}$ falls below one. So the rescaling changes every entry of the matrix. The subtraction sets the removed eigenvalues to zero and keeps the others, so $\mathbf{C}$ and $\tilde{\mathbf{X}}$ are positive semidefinite and singular. posdef! then runs on $\tilde{\mathbf{X}}$ under pdm.
The $n$ largest eigenmodes are one subspace only when $\lambda_{N-n+1} > \lambda_{N-n}$. When the two eigenvalues are equal, the eigenvectors in the subtraction are an arbitrary choice from a larger subspace, and the result depends on that choice. The identity is such a matrix, and Denoise gives the identity when it finds no signal. So the algorithm removes the largest number $m \leq n$ of leading modes that is one subspace, and it gives the matrix back when no such $m$ exists. A covariance then comes back to within the rounding of its conversion to a correlation and back. A tie is a difference of not more than $N \varepsilon \max_k \lvert \lambda_k \rvert$, the rounding bound of the decomposition. An asset whose remainder $C_{ii}$ is not more than that bound has all of its variance in the removed modes, so it has no correlation after the subtraction. Its row and column of $\tilde{\mathbf{X}}$ are zero, with a one on the diagonal. So the diagonal of a covariance returns without change, and $n = N$ gives the identity.
Equation 5 of [31] splits a correlation matrix into the market mode, the group modes and the noise modes, and the subtraction at $n = 1$ removes the first of those parts. That work does not rescale. Section 2.6 of [26] states detoning. It subtracts the market components, usually one but possibly more, and rescales the remainder to a unit diagonal. A search of the literature found no earlier work that states the rescale. That section detones a denoised matrix, and this estimator detones the matrix that it gets.
Algorithm
detone! runs these steps under this estimator.
- Read
dt.ninton, and check that0 < n <= size(X, 2). - When
Xhas a zero variance, set its row and column to zero and detone the block of the positive variances with a recursive call, throughzero_variance_block!, and returnX. A zero variance has no correlation, so step 4 cannot standardise it. The recursive call checksdt.nagainst the size of the block. - Read the diagonal of
Xintos. When any entry ofsis not one,Xis a covariance matrix. Then replaceswith its square roots and convertXto a correlation matrix withStatsBase.cov2cor!. The test isany(!isone, s), so it is the value of the diagonal that decides, never the type ofX. - Eigendecompose
Xinto the ascending eigenvaluesvalsand the eigenvectorsvecs. Set the tolerancetol = N * eps(T) * maximum(abs, vals). - Find
m, the largest number in1:nfor whichm == N, orvals[N - m + 1] - vals[N - m] > tol. When no suchmexists, skip to step 10. Sodt.n = 1removes the single largest component, which is the market mode, when it stands above the next one. - Subtract the
mlargest eigenmodes fromX. The result is the remainder $\mathbf{C}$. - Set the row and the column of each asset whose remainder on the diagonal is at most
tolto zero, and its diagonal entry to one. - Rescale the remainder to unit diagonal with
StatsBase.cov2cor. - Repair the rescaled matrix with
posdef!, underdt.pdm. - When step 3 converted a covariance matrix, convert
Xback withStatsBase.cor2cov!. The standard deviations are the ones read in step 3, so the original diagonal returns exactly.
Fields
pdm: Optional positive definite matrix estimator.
n: Largest number of leading principal components to remove. The detoner removes fewer when the last of them ties with the next eigenvalue, asDetonestates.
Constructors
Detone(; pdm::Option{<:AbstractPosdefEstimator} = Posdef(), n::Integer = 1,) -> DetoneKeywords correspond to the struct's fields.
Validation
n > 0.
Examples
julia> Detone(; n = 2)Detone pdm ┼ Posdef │ alg ┼ UnionAll: NearestCorrelationMatrix.Newton │ kwargs ┴ @NamedTuple{}: NamedTuple() n ┴ Int64: 2Related
References
PortfolioOptimisers.detone — Function
detone(dt::Option{<:AbstractDetoneEstimator}, X::MatNum) -> MatNumOut-of-place version of detone!.
Algorithm
- Copy
X. - Apply
detone!to the copy, and return it. The function never changes the input.
Arguments
dt: Optional matrix detoning estimator.::Detone: Removes at most thedt.nlargest principal components from a copy ofX.::Nothing: ReturnsXunchanged.
X: Covariance-like or correlation-like matrixassets × assets.
Returns
X::MatNum: A new matrix equal to the detoned version of the input.
Examples
julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5); X = X' * X;julia> Xd = detone(Detone(), X);julia> size(Xd)(5, 5)Related
References
PortfolioOptimisers.detone! — Function
detone!(dt::Option{<:AbstractDetoneEstimator}, X::MatNum) -> MatNumRemoves the n largest principal components, the market modes, from a covariance or correlation matrix in place, or fewer when the n-th of them ties with the next one.
When the diagonal of X is not all ones, the function converts X to a correlation matrix, applies the algorithm, and scales the result back to a covariance.
A matrix with no leading mode that stands above the next one, such as the identity, comes back, to within the rounding of the conversion of a covariance, as Detone states.
Arguments
dt: Optional matrix detoning estimator.::Detone: Removes at most thedt.nlargest principal components fromXin place.::Nothing: ReturnsXunchanged.
X: Covariance-like or correlation-like matrixassets × assets.
Validation
0 < dt.n <= size(X, 2). WhenXhas a zero variance,dt.nis also at most the number of positive variances, which the call on their block checks.
Returns
X::MatNum: The input matrixX, modified in place.
Examples
julia> using StableRNGsjulia> rng = StableRNG(123456789);julia> X = rand(rng, 10, 5);julia> X = X' * X5×5 Matrix{Float64}: 3.29494 2.0765 1.73334 2.01524 1.77493 2.0765 2.46967 1.39953 1.97242 2.07886 1.73334 1.39953 1.90712 1.17071 1.30459 2.01524 1.97242 1.17071 2.24818 1.87091 1.77493 2.07886 1.30459 1.87091 2.44414julia> detone!(Detone(), X)5×5 Matrix{Float64}: 3.29494 -1.14673 0.0868439 -0.502106 -1.71581 -1.14673 2.46967 -0.876289 -0.0864304 0.274663 0.0868439 -0.876289 1.90712 -1.18851 -0.750345 -0.502106 -0.0864304 -1.18851 2.24818 -0.0774753 -1.71581 0.274663 -0.750345 -0.0774753 2.44414Related
detoneDetone: the closed form this function applies, and the steps it runs.zero_variance_block!MatNumOptionPosdef
References
References
- [26]
- M. M. De Prado. Machine learning for asset managers (Cambridge University Press, 2020).
- [31]
- D.-H. Kim and H. Jeong. Systematic analysis of group identification in stock markets. Phys. Rev. E 72, 046133 (2005).