Distance Covariance: private API

PortfolioOptimisers.calc_pairwise_dists — Method
calc_pairwise_dists(ce::DistanceCovariance, v1::VecNum, v2::VecNum) -> (MatNum, MatNum)

Compute the pairwise distance matrix of each of two vectors with the metric of the estimator.

cor_distance and cov_distance call this helper. It takes no observation weights. The weights apply to the statistic that calc_centred_dists and calc_dcov2 compute, and not to the data that the metric measures.

Algorithm

  1. Call Distances.pairwise with ce.metric, ce.args and ce.kwargs on v1 and on v2. The method does not scale or weight the two vectors.
  2. Return the two matrices as (D1, D2). Each matrix is $T \times T$ for series of $T$ observations. D1[t, s] is the distance between observations $t$ and $s$ of v1.

Arguments

Returns

  • The tuple (D1, D2) of the pairwise distance matrices of v1 and of v2.

Related

source
PortfolioOptimisers.calc_centred_dists — Method
calc_centred_dists(a::MatNum, w::Option{<:StatsBase.AbstractWeights}) -> MatNum

Doubly centre a pairwise distance matrix, with the weight w on each observation.

cor_distance and cov_distance call this helper. The weights apply to the three means. A constant weight vector cancels from each mean, so it gives the unweighted matrix.

Algorithm

  1. Without weights, take the row means, the column means and the grand mean of a.
  2. With weights, take the same three means with the weights w. Each mean divides by the sum of the weights that it uses, so the scale of w cancels.
  3. Subtract the row means and the column means from a. Add the grand mean to get A.

Arguments

  • a: The pairwise distance matrix.
  • w: The observation weights, or nothing for the unweighted statistic.

Returns

  • A::MatNum: The doubly centred distance matrix.

Related

source
PortfolioOptimisers.calc_dcov2 — Method
calc_dcov2(A::MatNum, B::MatNum, w::Option{<:StatsBase.AbstractWeights}) -> Number

Contract two doubly centred distance matrices into a squared distance covariance.

cor_distance and cov_distance call this helper. The weights apply to the two indices of the contraction, so a constant weight vector cancels and gives the unweighted value.

Algorithm

  1. Without weights, divide the Frobenius inner product of A and B by the number of entries of A.
  2. With weights, contract the element-wise product of A and B with w on its two indices. Divide the result by the square of the sum of the weights.

Arguments

  • A, B: The doubly centred distance matrices that calc_centred_dists returns.
  • w: The observation weights, or nothing for the unweighted statistic.

Returns

  • dcov2::Number: The squared distance covariance of the two matrices.

Related

source
PortfolioOptimisers.cor_distance — Function
cor_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum,
             w::Option{<:StatsBase.AbstractWeights} = nothing)

Compute the distance correlation of two vectors with a DistanceCovariance estimator.

The metric, the args and the kwargs of ce set the pairwise distances, and w sets the weights of the observations. A constant series has a distance variance of zero, so its distance correlation with each series is NaN.

Mathematical definition

Let $a_{ts} = d(v_{1t}, v_{1s})$ and $b_{ts} = d(v_{2t}, v_{2s})$ be the entries of the two pairwise distance matrices. The doubly centred matrices subtract and add three means, and each mean uses the observation weights:

\[\begin{align} \bar{a}_{t\cdot} &= \frac{\sum\limits_{s=1}^{T} w_{s} a_{ts}}{\sum\limits_{s=1}^{T} w_{s}}\,, \\ \bar{a}_{\cdot s} &= \frac{\sum\limits_{t=1}^{T} w_{t} a_{ts}}{\sum\limits_{t=1}^{T} w_{t}}\,, \\ \bar{a}_{\cdot\cdot} &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} a_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ A_{ts} &= a_{ts} - \bar{a}_{t\cdot} - \bar{a}_{\cdot s} + \bar{a}_{\cdot\cdot}\,, \\ B_{ts} &= b_{ts} - \bar{b}_{t\cdot} - \bar{b}_{\cdot s} + \bar{b}_{\cdot\cdot}\,. \end{align}\]

Where:

  • $a_{ts}$, $b_{ts}$: The pairwise distances between observations $t$ and $s$.
  • $\bar{a}_{t\cdot}$: The $t$-th weighted row mean of $\mathbf{a}$.
  • $\bar{a}_{\cdot s}$: The $s$-th weighted column mean of $\mathbf{a}$.
  • $\bar{a}_{\cdot\cdot}$: The weighted grand mean of $\mathbf{a}$.
  • $A_{ts}$, $B_{ts}$: The doubly centred pairwise distances.
  • $w_{t}$: Observation weight of observation $t$.
  • $T$: Number of observations.

The three means of $\mathbf{b}$ have the same form as the three means of $\mathbf{a}$.

The squared distance covariances and the distance correlation are:

\[\begin{align} \widehat{\mathrm{dCov}}^2(X,X) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} A_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ \widehat{\mathrm{dCov}}^2(X,Y) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,, \\ \widehat{\mathrm{dCov}}^2(Y,Y) &= \frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} B_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}\,. \end{align}\]

Where:

  • $\widehat{\mathrm{dCov}}^2(X,Y)$: The squared distance covariance of $X$ and $Y$.

\[\begin{align} \hat{R}_{\mathrm{dist}}(X, Y) &= \frac{\sqrt{\widehat{\mathrm{dCov}}^2(X,Y)}}{\sqrt{\sqrt{\widehat{\mathrm{dCov}}^2(X,X)} \cdot \sqrt{\widehat{\mathrm{dCov}}^2(Y,Y)}}}\,. \end{align}\]

Where:

  • $\hat{R}_{\mathrm{dist}}(X, Y)$: The distance correlation of $X$ and $Y$.

In each quotient above, the numerator and the denominator have the same degree in the weights, so a constant weight cancels. With $w_{t} = 1$, the squared distance covariance of $X$ and $Y$ is $\mathbf{A}:\mathbf{B} / T^{2}$, the Frobenius inner product $\sum_{t,\,s} A_{ts} B_{ts}$ divided by $T^{2}$. This is the unweighted statistic, and the other two squared distance covariances have the same form.

Algorithm

  1. Compute the two pairwise distance matrices $\mathbf{a}$ and $\mathbf{b}$ with calc_pairwise_dists. It uses the metric, the args and the kwargs of ce.
  2. Doubly centre each matrix with calc_centred_dists to get $\mathbf{A}$ and $\mathbf{B}$. The three means that it subtracts and adds use the weights.
  3. Contract the three pairs $(\mathbf{A}, \mathbf{A})$, $(\mathbf{A}, \mathbf{B})$ and $(\mathbf{B}, \mathbf{B})$ with calc_dcov2 to get the three squared distance covariances.
  4. Return $\sqrt{\widehat{\mathrm{dCov}}^2(X,Y)}$ divided by the square root of the product of the other two square roots.

Arguments

  • ce: The distance covariance estimator.
  • v1: The first data vector.
  • v2: The second data vector.
  • w: The observation weights, or nothing for the unweighted statistic.

Validation

  • length(v1) == length(v2).
  • length(v1) > 1.

Returns

  • rho::Number: The distance correlation of v1 and v2. A series that is not constant gives exactly 1.0 against itself. A constant series gives NaN.

Related

source
PortfolioOptimisers.cov_distance — Function
cov_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum,
             w::Option{<:StatsBase.AbstractWeights} = nothing)

Compute the distance covariance of two vectors with a DistanceCovariance estimator.

The metric, the args and the kwargs of ce set the pairwise distances, and w sets the weights of the observations.

Mathematical definition

The doubly centred matrices $\mathbf{A}$ and $\mathbf{B}$ are those of cor_distance:

\[\begin{align} \widehat{\mathrm{dCov}}(X, Y) &= \sqrt{\frac{\sum\limits_{t=1}^{T} \sum\limits_{s=1}^{T} w_{t} w_{s} A_{ts} B_{ts}}{\left(\sum\limits_{t=1}^{T} w_{t}\right)^{2}}}\,. \end{align}\]

Where:

  • $\widehat{\mathrm{dCov}}(X, Y)$: The distance covariance of $X$ and $Y$.
  • $A_{ts}$, $B_{ts}$: The doubly centred pairwise distances.
  • $w_{t}$: Observation weight of observation $t$.
  • $T$: Number of observations.

With $w_{t} = 1$, the quotient is $\mathbf{A}:\mathbf{B} / T^{2}$, the Frobenius inner product $\sum_{t,\,s} A_{ts} B_{ts}$ divided by $T^{2}$. This is the unweighted statistic.

The square root takes no absolute value, and it needs none. The doubly centred V-statistic is non-negative for a metric of strong negative type, and the Euclidean default is such a metric.

Algorithm

  1. Compute the two pairwise distance matrices $\mathbf{a}$ and $\mathbf{b}$ with calc_pairwise_dists. It uses the metric, the args and the kwargs of ce.
  2. Doubly centre each matrix with calc_centred_dists to get $\mathbf{A}$ and $\mathbf{B}$. The three means that it subtracts and adds use the weights.
  3. Contract the pair $(\mathbf{A}, \mathbf{B})$ with calc_dcov2 to get the squared distance covariance.
  4. Return the square root of that value. Steps 1 and 2 are the steps of cor_distance. This method computes one of the three contractions, and it computes no ratio.

Arguments

  • ce: The distance covariance estimator.
  • v1: The first data vector.
  • v2: The second data vector.
  • w: The observation weights, or nothing for the unweighted statistic.

Validation

  • length(v1) == length(v2).
  • length(v1) > 1.

Returns

  • rho::Number: The distance covariance of v1 and v2. A series against itself gives its distance standard deviation $\widehat{\mathrm{dVar}}^{1/2}(X)$, which is not the sample standard deviation. A constant series gives 0.

Related

source
PortfolioOptimisers.cor_distance — Function
cor_distance(ce::DistanceCovariance, X::MatNum,
             w::Option{<:StatsBase.AbstractWeights} = nothing)

Compute the distance correlation matrix of the columns of a data matrix with a DistanceCovariance estimator.

Entry $(i, j)$ of the symmetric result is the distance correlation of assets $i$ and $j$. The metric of ce sets the distances, w sets the observation weights, and ce.ex runs the loop over the columns.

Algorithm

  1. Allocate an $N \times N$ matrix, where $N$ is the number of columns of X. The indices of the result are assets and not observations, so a non-square X cannot hide a transposed index.
  2. For each column j, and for each column i with i <= j, call cor_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum) on the two columns. Write the value into rho[i, j] and into rho[j, i]. The executor ce.ex runs the outer loop.
  3. Return the symmetric matrix. Step 2 computes the pair (j, j) too, so the diagonal entry of a column that is not constant is exactly 1.0. The diagonal entry of a constant column is NaN.

Arguments

  • ce: The distance covariance estimator.
  • X: The data matrix, observations × assets.
  • w: The observation weights, or nothing for the unweighted statistic.

Returns

  • rho::Matrix{<:Number}: The distance correlation matrix.

Related

source
PortfolioOptimisers.cov_distance — Function
cov_distance(ce::DistanceCovariance, X::MatNum,
             w::Option{<:StatsBase.AbstractWeights} = nothing)

Compute the distance covariance matrix of the columns of a data matrix with a DistanceCovariance estimator.

Entry $(i, j)$ of the symmetric result is the distance covariance of assets $i$ and $j$. The metric of ce sets the distances, w sets the observation weights, and ce.ex runs the loop over the columns.

Algorithm

  1. Allocate an $N \times N$ matrix, where $N$ is the number of columns of X. The indices of the result are assets and not observations, so a non-square X cannot hide a transposed index.
  2. For each column j, and for each column i with i <= j, call cov_distance(ce::DistanceCovariance, v1::VecNum, v2::VecNum) on the two columns. Write the value into sigma[i, j] and into sigma[j, i]. The executor ce.ex runs the outer loop.
  3. Return the symmetric matrix. Step 2 computes the pair (j, j) too, so the diagonal holds the distance standard deviation of each asset.

Arguments

  • ce: The distance covariance estimator.
  • X: The data matrix, observations × assets.
  • w: The observation weights, or nothing for the unweighted statistic.

Returns

  • sigma::Matrix{<:Number}: The symmetric matrix of pairwise distance covariances.

Related

source
PortfolioOptimisers.show_fields — Method
show_fields(_::DistanceCovariance) -> NTuple{5, Symbol}

Renders every field of a DistanceCovariance except cache.

A cache holds the state of an incremental fit. That state is not part of the configuration, and it prints under the estimator at each site that renders the estimator. Set set_show_nothing_fields!(:DistanceCovariance, true) to render it.

Arguments

  • ::DistanceCovariance: The distance covariance estimator. The method reads only its type.

Returns

  • fields::Tuple: The names of the fields to render, (:metric, :args, :kwargs, :w, :ex).

Related

source