Histogram: private API

const Int_Bin = Union{<:AbstractBins, <:Integer}

Alias for a histogram binning algorithm or an integer number of bins.

Matches either an AbstractBins algorithm (auto-selecting bin counts) or a plain Integer (fixed number of bins). Used in histogram-based mutual information and variation of information calculations.

Related

• AbstractBins
• calc_num_bins
• mutual_info
• variation_info
• mutual_variation_info

calc_hist_data(xj::VecNum, xi::VecNum, bins::Integer)

Compute histogram-based marginal and joint distributions for two variables.

This function computes the normalised histograms (probability mass functions) for two variables xj and xi using the specified number of bins, as well as their joint histogram. It returns the marginal entropies and the joint histogram, which are used in mutual information and variation of information calculations.

A bin is closed on the left, so the lower edge is the minimum itself and needs no widening. The upper edge is exclusive, so it is nextfloat of the maximum, one unit in the last place above it at every magnitude. The largest observation therefore falls strictly inside the last bin whatever the magnitude of the data, and a constant column gives an entropy of zero rather than NaN.

Algorithm

1. Add one to bins, giving bp1, the number of bin edges.
2. Build the edge range of xj from minimum(xj) to nextfloat(maximum(xj)), with bp1 points. Repeat for xi.
3. Bin xj over its own edges, giving the counts hx, and divide hx by its own sum to make it a probability mass function.
4. Repeat step 3 for xi, giving hy.
5. Take the Shannon entropy of hx into ex, and of hy into ey.
6. Bin the pair over both edge ranges, giving the joint counts hxy. It is left unnormalised, because intrinsic_mutual_info normalises it itself.

Arguments

• xj: Data vector for variable j.
• xi: Data vector for variable i.
• bins: Number of bins to use for the histograms.

Returns

• ex::Number: Shannon entropy of the marginal probability mass function of xj, in nats.
• ey::Number: Shannon entropy of the marginal probability mass function of xi, in nats.
• hxy::Matrix{<:Number}: Joint histogram (counts, not normalised to probability).

Related

• variation_info
• mutual_info
• mutual_variation_info
• intrinsic_mutual_info
• calc_num_bins

intrinsic_mutual_info(X::MatNum)

Compute the intrinsic mutual information from a joint histogram.

This function computes the mutual information between two variables given their joint histogram matrix X. It is used as a core step in information-theoretic measures such as mutual information and variation of information.

Intrinsic names the estimate itself: the quantity the joint histogram carries, before mutual_info divides it by a marginal entropy and before either matrix-valued measure clamps it. It is therefore in nats, it is not bounded above by one, and it is the value that mutual_info returns when normalise is false. It is not the intrinsic conditional information of secret-key agreement, which is a different quantity under the same word.

Mathematical definition

Given the joint histogram $\mathbf{X}$ (unnormalised counts), with marginals $p_i = \sum_j X_{ij} / n$ and $p_j = \sum_i X_{ij} / n$:

\[\begin{align} \hat{I}(X; Y) &= \sum_{i,j:\, X_{ij} > 0} \frac{X_{ij}}{n} \log\!\left(\frac{X_{ij} / n}{p_i \, p_j}\right)\,. \end{align}\]

Where:

• $\hat{I}(X; Y)$: Estimated mutual information between $X$ and $Y$.
• $X_{ij}$: Joint histogram count at bin $(i, j)$.
• $n = \sum_{i,j} X_{ij}$: Total count.
• $p_i = \sum_j X_{ij} / n$, $p_j = \sum_i X_{ij} / n$: Marginal probabilities.

A bin the pair never visits contributes nothing, because $p \log p$ tends to zero as $p$ does. The sum therefore runs over the non-empty bins alone. When either axis has a single bin the two variables are indistinguishable under the binning, and the estimate is zero.

Algorithm

1. Sum X over its columns into p_i, and over its rows into p_j. Both are unnormalised marginal counts.
2. When either marginal has length one, return zero.
3. Find the indices of the non-zero entries of X into mask, and read those entries into the vector nz.
4. Sum nz into nz_sum, the total count, and divide nz by it, giving the joint probabilities nz_nm.
5. Take the outer product of the two marginal counts at the masked indices into outer, and turn it into the logarithm of the product of the marginal probabilities, log_outer, by subtracting the logarithm of each marginal total.
6. Form the per-bin contributions mi from nz_nm and the two logarithms, and set to zero every contribution whose magnitude is below the machine epsilon.
7. Return the sum of mi.

Arguments

• X: Joint histogram matrix.

Returns

• mi::Number: The intrinsic mutual information between the two variables, in nats.

Related

• calc_hist_data
• variation_info
• mutual_info
• mutual_variation_info

References

• [24] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).

variation_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
               normalise::Bool = true)

Compute the variation of information (VI) matrix for a set of variables.

This function computes the pairwise variation of information between all columns of the data matrix X, using histogram-based entropy and mutual information estimates. VI quantifies the amount of information lost and gained when moving from one variable to another, and is a true metric on the space of discrete distributions: it is non-negative, it is zero exactly when the two variables agree, it is symmetric, and it satisfies the triangle inequality. mutual_info is the complementary measure and is not a metric — it grows with agreement rather than with disagreement, and its diagonal is the entropy rather than zero.

Mathematical definition

Let $H(X)$, $H(Y)$ denote the marginal Shannon entropies and $I(X;Y)$ the mutual information. The variation of information is:

\[\begin{align} \mathrm{VI}(X, Y) &= H(X) + H(Y) - 2\,I(X;Y)\,. \end{align}\]

Where:

• $\mathrm{VI}(X, Y)$: Variation of information between $X$ and $Y$.
• $H(X)$, $H(Y)$: Marginal Shannon entropies.
• $I(X;Y)$: Mutual information.

When normalise = true, it is divided by the joint entropy $H(X,Y) = H(X) + H(Y) - I(X;Y)$:

\[\begin{align} \widetilde{\mathrm{VI}}(X, Y) &= \frac{H(X) + H(Y) - 2\,I(X;Y)}{H(X) + H(Y) - I(X;Y)}\,. \end{align}\]

Where:

• $\widetilde{\mathrm{VI}}(X, Y)$: Normalised variation of information.
• $H(X,Y) = H(X) + H(Y) - I(X;Y)$: Joint entropy.

The divisor is the joint entropy, which keeps the normalised form a metric on $[0, 1]$. Equation 6.25 of the source divides by $\max(H(X), H(Y))$ instead, which is bounded by the same interval but is not a metric.

Algorithm

1. Read the shape of X into T, the number of observations, and N, the number of variables.
2. Allocate the N × N result var_mtx.
3. For each variable j, write an exact zero at var_mtx[j, j]. $\mathrm{VI}(X, X)$ is zero by definition, and the histogram estimate of $I(X; X)$ does not reproduce the estimate of $H(X)$ bit for bit, so estimating the self pair leaves roughly 1e-16 on the diagonal. That is enough to stop the result being a distance matrix, and skipping the self pair also saves N histogram computations.
4. For each pair (j, i) below the diagonal, take the bin count from calc_num_bins, giving nbins.
5. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
6. Take the mutual information of hxy from intrinsic_mutual_info, giving mut_ixy, and apply the definition above, giving var_ixy.
7. When normalise is true, divide var_ixy by the joint entropy vxy.
8. Clamp var_ixy below at zero, and write it into both var_mtx[j, i] and var_mtx[i, j].

Arguments

• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• bins: Binning algorithm or fixed number of bins.
• normalise: Whether to normalise the mutual and/or variation of information calculation.

Returns

• var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal. In nats when normalise is false, and dimensionless on [0, 1] when it is true.

Related

• mutual_info
• mutual_variation_info
• calc_hist_data
• calc_num_bins
• intrinsic_mutual_info
• Int_Bin

References

• [24] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.2.2, equation 6.24.

mutual_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
            normalise::Bool = true)

Compute the mutual information (MI) matrix for a set of variables.

This function computes the pairwise mutual information between all columns of the data matrix X, using histogram-based entropy and mutual information estimates. MI quantifies the amount of shared information between pairs of variables, and is widely used in information-theoretic analysis of dependencies.

Mutual information is a measure of agreement and is not a metric: it grows with dependence rather than with distance, and its diagonal carries the marginal entropy rather than zero. variation_info is the metric of the pair. The diagonal is estimated here, not pinned, so it is the marginal entropy in nats when normalise is false and one when it is true.

Mathematical definition

Mutual information between assets $i$ and $j$:

\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j) = \sum_{x,y} p(x,y) \log\frac{p(x,y)}{p(x)\,p(y)}\,. \end{align}\]

Where:

• $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
• $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
• $H(X_i, X_j)$: Joint entropy.
• $p(x,y)$: Joint probability mass function.

When normalise = true, the MI is normalised by the minimum marginal entropy:

\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,. \end{align}\]

Where:

• $\tilde{I}(X_i; X_j)$: Normalised mutual information.

The smaller marginal entropy is the largest value the mutual information of the pair can take, so the normalised form is bounded by $[0, 1]$ and reaches one exactly when one variable determines the other.

Algorithm

1. Read the shape of X into T, the number of observations, and N, the number of variables.
2. Allocate the N × N result mut_mtx.
3. For each pair (j, i) on or below the diagonal, take the bin count from calc_num_bins, giving nbins.
4. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
5. Take the mutual information of hxy from intrinsic_mutual_info, giving mut_ixy.
6. When normalise is true, divide mut_ixy by the smaller of ex and ey.
7. Clamp mut_ixy below at zero, and write it into both mut_mtx[j, i] and mut_mtx[i, j].

Arguments

• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• bins: Binning algorithm or fixed number of bins.
• normalise: Whether to normalise the mutual and/or variation of information calculation.

Returns

• mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values. In nats when normalise is false, and dimensionless on [0, 1] when it is true.

Related

• variation_info
• mutual_variation_info
• calc_hist_data
• calc_num_bins
• intrinsic_mutual_info
• Int_Bin
• MutualInfoCovariance

References

• [24] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.1.6, equations 6.18 and 6.19.

mutual_variation_info(X::MatNum, bins::Int_Bin = Knuth(), normalise::Bool = true)

Compute the pairwise mutual information and variation of information matrices from a data matrix.

Both matrices come from one pass over the pairs, so the two share a bin count and a joint histogram per pair. The result is what mutual_info and variation_info return, and the two normalisers differ: mutual information is divided by the smaller marginal entropy, variation of information by the joint entropy. The default bins is Knuth here and HacineGharbiRavier in the other two, so the three agree only when bins is given.

Mathematical definition

\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j)\,, \\ \mathrm{VI}(X_i, X_j) &= H(X_i) + H(X_j) - 2\,I(X_i; X_j)\,. \end{align}\]

When normalise = true, each is divided by its own normaliser:

\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,, \\ \widetilde{\mathrm{VI}}(X_i, X_j) &= \frac{\mathrm{VI}(X_i, X_j)}{H(X_i) + H(X_j) - I(X_i; X_j)}\,. \end{align}\]

Where:

• $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
• $\mathrm{VI}(X_i, X_j)$: Variation of information between assets $i$ and $j$.
• $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
• $H(X_i, X_j) = H(X_i) + H(X_j) - I(X_i; X_j)$: Joint entropy.
• $\tilde{I}(X_i; X_j)$: Normalised mutual information.
• $\widetilde{\mathrm{VI}}(X_i, X_j)$: Normalised variation of information.

Algorithm

1. Read the shape of X into T, the number of observations, and N, the number of variables.
2. Allocate the two N × N results mut_mtx and var_mtx.
3. For each pair (j, i) on or below the diagonal, take the bin count from calc_num_bins, giving nbins.
4. Take the two marginal entropies ex and ey and the joint histogram hxy from calc_hist_data.
5. Take the mutual information of hxy from intrinsic_mutual_info, giving mut_ixy, and form var_ixy from it and the two entropies.
6. When normalise is true, divide var_ixy by the joint entropy vxy and mut_ixy by the smaller of ex and ey.
7. Clamp both below at zero, and write each into both of its symmetric positions.
8. After the inner loop, write an exact zero at var_mtx[j, j], for the reason variation_info gives. mut_mtx[j, j] keeps its estimate, because $I(X; X) = H(X)$ is a real value there rather than a zero.

Arguments

• X: Data matrix observations × assets if the dims keyword does not exist or dims = 1, assets × observations when dims = 2.
• bins: Binning algorithm or fixed number of bins.
• normalise: Whether to normalise the mutual and/or variation of information calculation.

Returns

• mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values, whose diagonal is the marginal entropy when normalise is false and one when it is true.
• var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal.

Related

• Int_Bin
• AbstractBins
• mutual_info
• variation_info
• calc_hist_data
• calc_num_bins
• intrinsic_mutual_info

References

• [24] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Sections 6.1.6 and 6.2.2, equations 6.18, 6.19 and 6.24.