Distance Covariance: private API
PortfolioOptimisers.calc_pairwise_dists — Method
calc_pairwise_dists(ce::DistanceCovariance, v1::VecNum, v2::VecNum) -> (MatNum, MatNum)Compute the pairwise distance matrix of each of two vectors with the metric of the estimator.
cor_distance and cov_distance call this helper. It takes no observation weights. The weights apply to the statistic that calc_centred_dists and calc_dcov2 compute, and not to the data that the metric measures.
Algorithm
- Call
Distances.pairwisewithce.metric,ce.argsandce.kwargsonv1and onv2. The method does not scale or weight the two vectors. - Return the two matrices as
(D1, D2). Each matrix is $T \times T$ for series of $T$ observations.D1[t, s]is the distance between observations $t$ and $s$ ofv1.
Arguments
ce: TheDistanceCovarianceestimator that holds the metric.v1,v2: The two data vectors.
Returns
- The tuple
(D1, D2)of the pairwise distance matrices ofv1and ofv2.
Related
PortfolioOptimisers.calc_centred_dists — Method
calc_centred_dists(a::MatNum, w::Option{<:StatsBase.AbstractWeights}) -> MatNumDoubly centre a pairwise distance matrix, with the weight w on each observation.
cor_distance and cov_distance call this helper. The weights apply to the three means. A constant weight vector cancels from each mean, so it gives the unweighted matrix.
Algorithm
- Without weights, take the row means, the column means and the grand mean of
a. - With weights, take the same three means with the weights
w. Each mean divides by the sum of the weights that it uses, so the scale ofwcancels. - Subtract the row means and the column means from
a. Add the grand mean to getA.
Arguments
a: The pairwise distance matrix.w: The observation weights, ornothingfor the unweighted statistic.
Returns
A::MatNum: The doubly centred distance matrix.
Related
PortfolioOptimisers.calc_dcov2 — Method
calc_dcov2(A::MatNum, B::MatNum, w::Option{<:StatsBase.AbstractWeights}) -> NumberContract two doubly centred distance matrices into a squared distance covariance.
cor_distance and cov_distance call this helper. The weights apply to the two indices of the contraction, so a constant weight vector cancels and gives the unweighted value.
Algorithm
- Without weights, divide the Frobenius inner product of
AandBby the number of entries ofA. - With weights, contract the element-wise product of
AandBwithwon its two indices. Divide the result by the square of the sum of the weights.
Arguments
A,B: The doubly centred distance matrices thatcalc_centred_distsreturns.w: The observation weights, ornothingfor the unweighted statistic.
Returns
dcov2::Number: The squared distance covariance of the two matrices.
Related
PortfolioOptimisers.cor_distance — Function
cor_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum,
w::Option{<:StatsBase.AbstractWeights} = nothing)Compute the distance correlation of two vectors with a DistanceCovariance estimator.
The metric, the args and the kwargs of ce set the pairwise distances, and w sets the weights of the observations. A constant series has a distance variance of zero, so its distance correlation with each series is NaN.
Mathematical definition
Let $a_{ts} = d(v_{1t}, v_{1s})$ and $b_{ts} = d(v_{2t}, v_{2s})$ be the entries of the two pairwise distance matrices. The doubly centred matrices subtract and add three means, and each mean uses the observation weights:
\[\begin{align} \bar{a}_{t\cdot} &= \frac{\sum\limits_{s=1}^{T} w_{s} a_{ts}}{\sum\limits_{s=1}^{T} w_{s}}\,, \\ \bar{a}_{\cdot s} &= \frac{\sum\limits_{t=1}^{T} w_{t} a_{ts}}{\sum\limits_{t=1}^{T} w_{t}}\,, \\ \bar{a}_{\cdot\cdot} &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} a_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ A_{ts} &= a_{ts} - \bar{a}_{t\cdot} - \bar{a}_{\cdot s} + \bar{a}_{\cdot\cdot}\,, \\ B_{ts} &= b_{ts} - \bar{b}_{t\cdot} - \bar{b}_{\cdot s} + \bar{b}_{\cdot\cdot}\,. \end{align}\]
Where:
- $a_{ts}$, $b_{ts}$: The pairwise distances between observations $t$ and $s$.
- $\bar{a}_{t\cdot}$: The $t$-th weighted row mean of $\mathbf{a}$.
- $\bar{a}_{\cdot s}$: The $s$-th weighted column mean of $\mathbf{a}$.
- $\bar{a}_{\cdot\cdot}$: The weighted grand mean of $\mathbf{a}$.
- $A_{ts}$, $B_{ts}$: The doubly centred pairwise distances.
- $w_{t}$: Observation weight of observation $t$.
- $T$: Number of observations.
The three means of $\mathbf{b}$ have the same form as the three means of $\mathbf{a}$.
The squared distance covariances and the distance correlation are:
\[\begin{align} \widehat{\mathrm{dCov}}^2(X,X) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} A_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ \widehat{\mathrm{dCov}}^2(X,Y) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ \widehat{\mathrm{dCov}}^2(Y,Y) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} B_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,. \end{align}\]
Where:
- $\widehat{\mathrm{dCov}}^2(X,Y)$: The squared distance covariance of $X$ and $Y$.
\[\begin{align} \hat{R}_{\mathrm{dist}}(X, Y) &= \frac{\sqrt{\widehat{\mathrm{dCov}}^2(X,Y)}}{\sqrt{\sqrt{\widehat{\mathrm{dCov}}^2(X,X)} \cdot \sqrt{\widehat{\mathrm{dCov}}^2(Y,Y)}}}\,. \end{align}\]
Where:
- $\hat{R}_{\mathrm{dist}}(X, Y)$: The distance correlation of $X$ and $Y$.
In each quotient above, the numerator and the denominator have the same degree in the weights, so a constant weight cancels. With $w_{t} = 1$, the squared distance covariance of $X$ and $Y$ is $\mathbf{A}:\mathbf{B} / T^{2}$, the Frobenius inner product $\sum_{t,\,s} A_{ts} B_{ts}$ divided by $T^{2}$. This is the unweighted statistic, and the other two squared distance covariances have the same form.
Algorithm
- Compute the two pairwise distance matrices $\mathbf{a}$ and $\mathbf{b}$ with
calc_pairwise_dists. It uses the metric, theargsand thekwargsofce. - Doubly centre each matrix with
calc_centred_diststo get $\mathbf{A}$ and $\mathbf{B}$. The three means that it subtracts and adds use the weights. - Contract the three pairs $(\mathbf{A}, \mathbf{A})$, $(\mathbf{A}, \mathbf{B})$ and $(\mathbf{B}, \mathbf{B})$ with
calc_dcov2to get the three squared distance covariances. - Return $\sqrt{\widehat{\mathrm{dCov}}^2(X,Y)}$ divided by the square root of the product of the other two square roots.
Arguments
ce: The distance covariance estimator.v1: The first data vector.v2: The second data vector.w: The observation weights, ornothingfor the unweighted statistic.
Validation
length(v1) == length(v2).length(v1) > 1.
Returns
rho::Number: The distance correlation ofv1andv2. A series that is not constant gives exactly1.0against itself. A constant series givesNaN.
Related
PortfolioOptimisers.cov_distance — Function
cov_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum,
w::Option{<:StatsBase.AbstractWeights} = nothing)Compute the distance covariance of two vectors with a DistanceCovariance estimator.
The metric, the args and the kwargs of ce set the pairwise distances, and w sets the weights of the observations.
Mathematical definition
The doubly centred matrices $\mathbf{A}$ and $\mathbf{B}$ are those of cor_distance:
\[\begin{align} \widehat{\mathrm{dCov}}(X, Y) &= \sqrt{\frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}}\,. \end{align}\]
Where:
- $\widehat{\mathrm{dCov}}(X, Y)$: The distance covariance of $X$ and $Y$.
- $A_{ts}$, $B_{ts}$: The doubly centred pairwise distances.
- $w_{t}$: Observation weight of observation $t$.
- $T$: Number of observations.
With $w_{t} = 1$, the quotient is $\mathbf{A}:\mathbf{B} / T^{2}$, the Frobenius inner product $\sum_{t,\,s} A_{ts} B_{ts}$ divided by $T^{2}$. This is the unweighted statistic.
The square root takes no absolute value, and it needs none. The doubly centred V-statistic is non-negative for a metric of strong negative type, and the Euclidean default is such a metric.
Algorithm
- Compute the two pairwise distance matrices $\mathbf{a}$ and $\mathbf{b}$ with
calc_pairwise_dists. It uses the metric, theargsand thekwargsofce. - Doubly centre each matrix with
calc_centred_diststo get $\mathbf{A}$ and $\mathbf{B}$. The three means that it subtracts and adds use the weights. - Contract the pair $(\mathbf{A}, \mathbf{B})$ with
calc_dcov2to get the squared distance covariance. - Return the square root of that value. Steps 1 and 2 are the steps of
cor_distance. This method computes one of the three contractions, and it computes no ratio.
Arguments
ce: The distance covariance estimator.v1: The first data vector.v2: The second data vector.w: The observation weights, ornothingfor the unweighted statistic.
Validation
length(v1) == length(v2).length(v1) > 1.
Returns
rho::Number: The distance covariance ofv1andv2. A series against itself gives its distance standard deviation $\widehat{\mathrm{dVar}}^{1/2}(X)$, which is not the sample standard deviation. A constant series gives0.
Related
PortfolioOptimisers.cor_distance — Function
cor_distance(ce::DistanceCovariance, X::MatNum,
w::Option{<:StatsBase.AbstractWeights} = nothing)Compute the distance correlation matrix of the columns of a data matrix with a DistanceCovariance estimator.
Entry $(i, j)$ of the symmetric result is the distance correlation of assets $i$ and $j$. The metric of ce sets the distances, w sets the observation weights, and ce.ex runs the loop over the columns.
Algorithm
- Allocate an $N \times N$ matrix, where $N$ is the number of columns of
X. The indices of the result are assets and not observations, so a non-squareXcannot hide a transposed index. - For each column
j, and for each columniwithi <= j, callcor_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum)on the two columns. Write the value intorho[i, j]and intorho[j, i]. The executorce.exruns the outer loop. - Return the symmetric matrix. Step 2 computes the pair
(j, j)too, so the diagonal entry of a column that is not constant is exactly1.0. The diagonal entry of a constant column isNaN.
Arguments
ce: The distance covariance estimator.X: The data matrix,observations × assets.w: The observation weights, ornothingfor the unweighted statistic.
Returns
rho::Matrix{<:Number}: The distance correlation matrix.
Related
PortfolioOptimisers.cov_distance — Function
cov_distance(ce::DistanceCovariance, X::MatNum,
w::Option{<:StatsBase.AbstractWeights} = nothing)Compute the distance covariance matrix of the columns of a data matrix with a DistanceCovariance estimator.
Entry $(i, j)$ of the symmetric result is the distance covariance of assets $i$ and $j$. The metric of ce sets the distances, w sets the observation weights, and ce.ex runs the loop over the columns.
Algorithm
- Allocate an $N \times N$ matrix, where $N$ is the number of columns of
X. The indices of the result are assets and not observations, so a non-squareXcannot hide a transposed index. - For each column
j, and for each columniwithi <= j, callcov_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum)on the two columns. Write the value intosigma[i, j]and intosigma[j, i]. The executorce.exruns the outer loop. - Return the symmetric matrix. Step 2 computes the pair
(j, j)too, so the diagonal holds the distance standard deviation of each asset.
Arguments
ce: The distance covariance estimator.X: The data matrix,observations × assets.w: The observation weights, ornothingfor the unweighted statistic.
Returns
sigma::Matrix{<:Number}: The symmetric matrix of pairwise distance covariances.
Related
PortfolioOptimisers.show_fields — Method
show_fields(_::DistanceCovariance) -> NTuple{5, Symbol}
Renders every field of a DistanceCovariance except cache.
A cache holds the state of an incremental fit. That state is not part of the configuration, and it prints under the estimator at each site that renders the estimator. Set set_show_nothing_fields!(:DistanceCovariance, true) to render it.
Arguments
::DistanceCovariance: The distance covariance estimator. The method reads only its type.
Returns
fields::Tuple: The names of the fields to render,(:metric, :args, :kwargs, :w, :ex).
Related