Histogram: private API

PortfolioOptimisers.calc_hist_data — Function
calc_hist_data(xj::VecNum, xi::VecNum, bins::Integer)

Compute histogram-based marginal and joint distributions for two variables.

The function bins xj and xi into bins bins each. It returns the Shannon entropy of each marginal histogram and the joint histogram of counts. mutual_info, variation_info and mutual_variation_info read these three values for each pair.

A bin is closed on the left, so the lower edge is the minimum itself and needs no widening. The upper edge is exclusive, so it is nextfloat of the maximum, one unit in the last place above it at every magnitude. The largest observation therefore falls strictly inside the last bin whatever the magnitude of the data, and a constant column gives an entropy of zero rather than NaN.

Algorithm

  1. Set bp1 to bins + 1, the number of bin edges.
  2. Build the edge range of xj from minimum(xj) to nextfloat(maximum(xj)), with bp1 points. Repeat for xi.
  3. Bin xj over its own edges into the counts hx, and divide hx by its own sum to make it a probability mass function.
  4. Repeat step 3 for xi to get hy.
  5. Take the Shannon entropy of hx into ex, and of hy into ey.
  6. Bin the pair over both edge ranges into the joint counts hxy. The function does not normalise hxy, because intrinsic_mutual_info normalises it.

Arguments

  • xj: Data vector for variable j.
  • xi: Data vector for variable i.
  • bins: Number of bins to use for the histograms.

Returns

  • ex::Number: Shannon entropy of the marginal probability mass function of xj, in nats.
  • ey::Number: Shannon entropy of the marginal probability mass function of xi, in nats.
  • hxy::Matrix{<:Number}: Joint histogram of counts, not normalised to probabilities.

Related

source
PortfolioOptimisers.intrinsic_mutual_info — Function
intrinsic_mutual_info(X::MatNum)

Compute the intrinsic mutual information from a joint histogram.

The function computes the mutual information of two variables from their joint histogram X. mutual_info, variation_info and mutual_variation_info call it once for each pair.

The word "intrinsic" names the raw estimate from the joint histogram. mutual_info can divide that estimate by a marginal entropy, and mutual_info and mutual_variation_info clamp it at zero, but this function does neither. The estimate is in nats, and it can be larger than one. It is the value that mutual_info returns when normalise is false, before the clamp. It is not the intrinsic conditional information of secret-key agreement, which is a different quantity with the same name.

Mathematical definition

Let $\mathbf{X}$ be the joint histogram of counts, with the marginal probabilities $p_i = \sum_j X_{ij} / n$ and $p_j = \sum_i X_{ij} / n$. The estimate is:

\[\begin{align} \hat{I}(X; Y) &= \sum_{i,j:\, X_{ij} > 0} \frac{X_{ij}}{n} \log\!\left(\frac{X_{ij} / n}{p_i \, p_j}\right)\,. \end{align}\]

Where:

  • $\hat{I}(X; Y)$: Estimated mutual information between $X$ and $Y$.
  • $X_{ij}$: Joint histogram count at bin $(i, j)$.
  • $n = \sum_{i,j} X_{ij}$: Total count.
  • $p_i = \sum_j X_{ij} / n$, $p_j = \sum_i X_{ij} / n$: Marginal probabilities.

The sum equals $H(X) + H(Y) - H(X, Y)$, the third form of the rate of transmission in Section 12 of [50], with the entropies of Section 6 computed from the histogram frequencies. This is a plug-in estimate, and the library applies no bias correction.

An empty bin adds nothing to the sum, because $p \log p$ tends to zero as $p$ does. The sum therefore runs over the non-empty bins alone. When either axis has a single bin, that variable has one value under the binning. It then shares no information with the other variable, and the estimate is zero.

Algorithm

  1. Sum X over its columns into p_i, and over its rows into p_j. Both are unnormalised marginal counts.
  2. When either marginal has length one, return zero.
  3. Find the indices of the non-zero entries of X into mask, and read those entries into the vector nz.
  4. Take the logarithm of nz into log_nz. Sum nz into nz_sum, the total count, and divide nz by it into the joint probabilities nz_nm.
  5. Take the outer product of the two marginal counts at the masked indices into outer. Negate the logarithm of outer and add the logarithm of each marginal total. The result is log_outer, the negated logarithm of the product of the marginal probabilities.
  6. Make the contribution mi of each bin from nz_nm, log_nz, nz_sum and log_outer, and set to zero every contribution whose magnitude is below the machine epsilon.
  7. Return the sum of mi.

Arguments

  • X: Joint histogram matrix.

Returns

  • mi::Number: The intrinsic mutual information between the two variables, in nats.

Related

References

  • [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
source
PortfolioOptimisers.variation_info — Function
variation_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
               normalise::Bool = true)

Compute the variation of information (VI) matrix for a set of variables.

The function computes the variation of information of each pair of columns of X from histogram estimates of the entropies and the mutual information. The variation of information is the information that one variable has and the other does not, summed over the two directions. It is a metric on discrete distributions. It is non-negative, it is zero exactly when the two variables agree, it is symmetric, and it satisfies the triangle inequality. mutual_info is not a metric, because its diagonal is the entropy and a metric is zero on its diagonal.

Mathematical definition

Let $H(X)$ and $H(Y)$ be the marginal Shannon entropies, and $I(X;Y)$ the mutual information. The variation of information is:

\[\begin{align} \mathrm{VI}(X, Y) &= H(X) + H(Y) - 2\,I(X;Y)\,. \end{align}\]

Where:

  • $\mathrm{VI}(X, Y)$: Variation of information between $X$ and $Y$.
  • $H(X)$, $H(Y)$: Marginal Shannon entropies.
  • $I(X;Y)$: Mutual information.

When normalise = true, the function divides it by the joint entropy $H(X,Y) = H(X) + H(Y) - I(X;Y)$:

\[\begin{align} \widetilde{\mathrm{VI}}(X, Y) &= \frac{H(X) + H(Y) - 2\,I(X;Y)}{H(X) + H(Y) - I(X;Y)}\,. \end{align}\]

Where:

  • $\widetilde{\mathrm{VI}}(X, Y)$: Normalised variation of information.
  • $H(X,Y) = H(X) + H(Y) - I(X;Y)$: Joint entropy.

Equation 19 of Section 3 of [51] states the unnormalised form and proves that it is a metric. The divisor of the normalised form is the joint entropy, and Equation 1 of [113] states that quotient as a distance on discrete distributions. Equation 6.25 of [12] is $1 - I(X;Y) / \max(H(X), H(Y))$ instead. That quotient lies in the same interval and is also a metric, the second metric of Theorem 2 of [114]. This function does not compute it.

Algorithm

  1. Read the shape of X into T, the number of observations, and N, the number of variables.
  2. Allocate the N × N result var_mtx.
  3. For each variable j, write an exact zero at var_mtx[j, j]. $\mathrm{VI}(X, X)$ is zero by definition. The histogram estimate of $I(X; X)$ does not reproduce the estimate of $H(X)$ bit for bit, so an estimate of the self pair leaves about 1e-16 on the diagonal. A matrix with a non-zero diagonal is not a distance matrix. The exact zero also saves N histogram computations.
  4. For each pair (j, i) below the diagonal, take the bin count from calc_num_bins into nbins.
  5. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
  6. Take the mutual information of hxy from intrinsic_mutual_info into mut_ixy, and apply the definition above to get var_ixy.
  7. When normalise is true, divide var_ixy by the joint entropy vxy.
  8. Clamp var_ixy below at zero, and write it into both var_mtx[j, i] and var_mtx[i, j].

Arguments

  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • bins: Binning algorithm or fixed number of bins.
  • normalise: If true, the result is the normalised mutual information or the normalised variation of information.

Returns

  • var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal. In nats when normalise is false, and dimensionless on [0, 1] when it is true.

Related

References

  • [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equation 19.
  • [113] C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961). Theorem, Equation 1.
  • [114] A. Kraskov, H. Stögbauer, R. G. Andrzejak and P. Grassberger. Hierarchical clustering using mutual information. Europhysics Letters 70, 278–284 (2005). Section 2.3, Theorem 2.
  • [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.2.2, Equations 6.24 and 6.25.
source
PortfolioOptimisers.mutual_info — Function
mutual_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
            normalise::Bool = true)

Compute the mutual information matrix for a set of variables.

The function computes the mutual information of each pair of columns of X from histogram estimates. The mutual information is the information that two variables share, and it is zero when the two variables are independent.

Mutual information is not a metric, because its diagonal is the marginal entropy and a metric is zero on its diagonal. It is larger for a more dependent pair, and a distance is smaller. variation_info is the metric of the two measures. The function estimates the diagonal and does not set it, so the diagonal is the marginal entropy in nats when normalise is false, and one when it is true.

Mathematical definition

Mutual information between assets $i$ and $j$:

\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j) = \sum_{x,y} p(x,y) \log\frac{p(x,y)}{p(x)\,p(y)}\,. \end{align}\]

Where:

  • $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
  • $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
  • $H(X_i, X_j)$: Joint entropy.
  • $p(x,y)$: Joint probability mass function.

When normalise = true, the function divides the mutual information by the smaller marginal entropy:

\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,. \end{align}\]

Where:

  • $\tilde{I}(X_i; X_j)$: Normalised mutual information.

Equation 18 of [51] shows that the smaller marginal entropy is the largest value that the mutual information of the pair can take. The normalised form is therefore in $[0, 1]$, and it is one exactly when one variable determines the other. Equation 6.19 of [12] states this quotient, and no work that first states it was found.

Algorithm

  1. Read the shape of X into T, the number of observations, and N, the number of variables.
  2. Allocate the N × N result mut_mtx.
  3. For each pair (j, i) on or below the diagonal, take the bin count from calc_num_bins into nbins.
  4. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
  5. Take the mutual information of hxy from intrinsic_mutual_info into mut_ixy.
  6. When normalise is true, divide mut_ixy by the smaller of ex and ey with normalised_mutual_info. A constant variable has an entropy of zero, so its normalised mutual information is NaN.
  7. Clamp mut_ixy below at zero, and write it into both mut_mtx[j, i] and mut_mtx[i, j].

Arguments

  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • bins: Binning algorithm or fixed number of bins.
  • normalise: If true, the result is the normalised mutual information or the normalised variation of information.

Returns

  • mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values. In nats when normalise is false, and dimensionless on [0, 1] when it is true.

Related

References

  • [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
  • [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equation 18.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.1.6, Equation 6.19. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source
PortfolioOptimisers.normalised_mutual_info — Function
normalised_mutual_info(mut_ixy::Number, ex::Number, ey::Number) -> Number

Divide the mutual information of a pair by the smaller of its two marginal entropies.

The quotient is the normalised mutual information of mutual_info. A constant variable has an entropy of zero, so the quotient is 0 / 0, and the function returns NaN. The mutual information of such a pair is zero only up to the rounding of the histogram, so a plain division can give Inf in place of NaN.

Algorithm

  1. Take the smaller entropy h of ex and ey.
  2. When h is zero, return h / h, the NaN of the number type of h.
  3. Otherwise return mut_ixy / h.

Arguments

  • mut_ixy: The mutual information of the pair, in nats.
  • ex, ey: The marginal entropies of the two variables, in nats.

Returns

  • nmi::Number: The normalised mutual information, or NaN when an entropy is zero.

Related

source
PortfolioOptimisers.mutual_variation_info — Function
mutual_variation_info(X::MatNum, bins::Int_Bin = Knuth(), normalise::Bool = true)

Compute the pairwise mutual information and variation of information matrices from a data matrix.

The function computes both matrices in one pass over the pairs, so the two matrices use one bin count and one joint histogram for each pair. With the same bins, the two results equal what mutual_info and variation_info return. The two normalisers differ. The function divides the mutual information by the smaller marginal entropy, and the variation of information by the joint entropy. The default bins is Knuth here and HacineGharbiRavier in the other two functions, so a caller who wants equal results must give the same bins to each function.

Mathematical definition

\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j)\,, \\ \mathrm{VI}(X_i, X_j) &= H(X_i) + H(X_j) - 2\,I(X_i; X_j)\,. \end{align}\]

When normalise = true, the function divides each by its own normaliser:

\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,, \\ \widetilde{\mathrm{VI}}(X_i, X_j) &= \frac{\mathrm{VI}(X_i, X_j)}{H(X_i) + H(X_j) - I(X_i; X_j)}\,. \end{align}\]

Where:

  • $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
  • $\mathrm{VI}(X_i, X_j)$: Variation of information between assets $i$ and $j$.
  • $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
  • $H(X_i, X_j) = H(X_i) + H(X_j) - I(X_i; X_j)$: Joint entropy.
  • $\tilde{I}(X_i; X_j)$: Normalised mutual information.
  • $\widetilde{\mathrm{VI}}(X_i, X_j)$: Normalised variation of information.

The two unnormalised forms are the rate of transmission of Section 12 of [50] and Equation 19 of [51]. The quotient by the joint entropy is Equation 1 of [113]. The quotient of the mutual information by the smaller marginal entropy is Equation 6.19 of [12], and no work that first states it was found.

Algorithm

  1. Read the shape of X into T, the number of observations, and N, the number of variables.
  2. Allocate the two N × N results mut_mtx and var_mtx.
  3. For each pair (j, i) on or below the diagonal, take the bin count from calc_num_bins into nbins.
  4. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
  5. Take the mutual information of hxy from intrinsic_mutual_info into mut_ixy, and make var_ixy from it and the two entropies.
  6. When normalise is true, divide var_ixy by the joint entropy vxy, and divide mut_ixy by the smaller of ex and ey with normalised_mutual_info, which gives NaN for a zero entropy.
  7. Clamp both below at zero, and write each into both of its symmetric positions.
  8. After the inner loop, write an exact zero at var_mtx[j, j], for the reason that variation_info gives. mut_mtx[j, j] keeps its estimate, because $I(X; X) = H(X)$ is not zero.

Arguments

  • X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
  • bins: Binning algorithm or fixed number of bins.
  • normalise: If true, the result is the normalised mutual information or the normalised variation of information.

Returns

  • mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values, whose diagonal is the marginal entropy when normalise is false and one when it is true.
  • var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal.

Related

References

  • [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
  • [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equations 18 and 19.
  • [113] C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961). Theorem, Equation 1.
  • [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.1.6, Equation 6.19. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
source

References

[12]
D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
[50]
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).
[51]
M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007).
[113]
C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961).
[114]
A. Kraskov, H. Stögbauer, R. G. Andrzejak and P. Grassberger. Hierarchical clustering using mutual information. Europhysics Letters 70, 278–284 (2005).