Histogram: private API
PortfolioOptimisers.Int_Bin — Type
const Int_Bin = Union{AbstractBins, Integer}Alias for a histogram binning algorithm or an integer number of bins.
Matches an AbstractBins algorithm, which selects the bin count from the data, or an Integer, which fixes the bin count. The bins argument of the histogram estimates of the mutual information and the variation of information is an Int_Bin.
Related
PortfolioOptimisers.calc_hist_data — Function
calc_hist_data(xj::VecNum, xi::VecNum, bins::Integer)Compute histogram-based marginal and joint distributions for two variables.
The function bins xj and xi into bins bins each. It returns the Shannon entropy of each marginal histogram and the joint histogram of counts. mutual_info, variation_info and mutual_variation_info read these three values for each pair.
A bin is closed on the left, so the lower edge is the minimum itself and needs no widening. The upper edge is exclusive, so it is nextfloat of the maximum, one unit in the last place above it at every magnitude. The largest observation therefore falls strictly inside the last bin whatever the magnitude of the data, and a constant column gives an entropy of zero rather than NaN.
Algorithm
- Set
bp1tobins + 1, the number of bin edges. - Build the edge range of
xjfromminimum(xj)tonextfloat(maximum(xj)), withbp1points. Repeat forxi. - Bin
xjover its own edges into the countshx, and dividehxby its own sum to make it a probability mass function. - Repeat step 3 for
xito gethy. - Take the Shannon entropy of
hxintoex, and ofhyintoey. - Bin the pair over both edge ranges into the joint counts
hxy. The function does not normalisehxy, becauseintrinsic_mutual_infonormalises it.
Arguments
xj: Data vector for variablej.xi: Data vector for variablei.bins: Number of bins to use for the histograms.
Returns
ex::Number: Shannon entropy of the marginal probability mass function ofxj, in nats.ey::Number: Shannon entropy of the marginal probability mass function ofxi, in nats.hxy::Matrix{<:Number}: Joint histogram of counts, not normalised to probabilities.
Related
PortfolioOptimisers.intrinsic_mutual_info — Function
intrinsic_mutual_info(X::MatNum)Compute the intrinsic mutual information from a joint histogram.
The function computes the mutual information of two variables from their joint histogram X. mutual_info, variation_info and mutual_variation_info call it once for each pair.
The word "intrinsic" names the raw estimate from the joint histogram. mutual_info can divide that estimate by a marginal entropy, and mutual_info and mutual_variation_info clamp it at zero, but this function does neither. The estimate is in nats, and it can be larger than one. It is the value that mutual_info returns when normalise is false, before the clamp. It is not the intrinsic conditional information of secret-key agreement, which is a different quantity with the same name.
Mathematical definition
Let $\mathbf{X}$ be the joint histogram of counts, with the marginal probabilities $p_i = \sum_j X_{ij} / n$ and $p_j = \sum_i X_{ij} / n$. The estimate is:
\[\begin{align} \hat{I}(X; Y) &= \sum_{i,j:\, X_{ij} > 0} \frac{X_{ij}}{n} \log\!\left(\frac{X_{ij} / n}{p_i \, p_j}\right)\,. \end{align}\]
Where:
- $\hat{I}(X; Y)$: Estimated mutual information between $X$ and $Y$.
- $X_{ij}$: Joint histogram count at bin $(i, j)$.
- $n = \sum_{i,j} X_{ij}$: Total count.
- $p_i = \sum_j X_{ij} / n$, $p_j = \sum_i X_{ij} / n$: Marginal probabilities.
The sum equals $H(X) + H(Y) - H(X, Y)$, the third form of the rate of transmission in Section 12 of [50], with the entropies of Section 6 computed from the histogram frequencies. This is a plug-in estimate, and the library applies no bias correction.
An empty bin adds nothing to the sum, because $p \log p$ tends to zero as $p$ does. The sum therefore runs over the non-empty bins alone. When either axis has a single bin, that variable has one value under the binning. It then shares no information with the other variable, and the estimate is zero.
Algorithm
- Sum
Xover its columns intop_i, and over its rows intop_j. Both are unnormalised marginal counts. - When either marginal has length one, return zero.
- Find the indices of the non-zero entries of
Xintomask, and read those entries into the vectornz. - Take the logarithm of
nzintolog_nz. Sumnzintonz_sum, the total count, and dividenzby it into the joint probabilitiesnz_nm. - Take the outer product of the two marginal counts at the masked indices into
outer. Negate the logarithm ofouterand add the logarithm of each marginal total. The result islog_outer, the negated logarithm of the product of the marginal probabilities. - Make the contribution
miof each bin fromnz_nm,log_nz,nz_sumandlog_outer, and set to zero every contribution whose magnitude is below the machine epsilon. - Return the sum of
mi.
Arguments
X: Joint histogram matrix.
Returns
mi::Number: The intrinsic mutual information between the two variables, in nats.
Related
References
- [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
PortfolioOptimisers.variation_info — Function
variation_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
normalise::Bool = true)Compute the variation of information (VI) matrix for a set of variables.
The function computes the variation of information of each pair of columns of X from histogram estimates of the entropies and the mutual information. The variation of information is the information that one variable has and the other does not, summed over the two directions. It is a metric on discrete distributions. It is non-negative, it is zero exactly when the two variables agree, it is symmetric, and it satisfies the triangle inequality. mutual_info is not a metric, because its diagonal is the entropy and a metric is zero on its diagonal.
Mathematical definition
Let $H(X)$ and $H(Y)$ be the marginal Shannon entropies, and $I(X;Y)$ the mutual information. The variation of information is:
\[\begin{align} \mathrm{VI}(X, Y) &= H(X) + H(Y) - 2\,I(X;Y)\,. \end{align}\]
Where:
- $\mathrm{VI}(X, Y)$: Variation of information between $X$ and $Y$.
- $H(X)$, $H(Y)$: Marginal Shannon entropies.
- $I(X;Y)$: Mutual information.
When normalise = true, the function divides it by the joint entropy $H(X,Y) = H(X) + H(Y) - I(X;Y)$:
\[\begin{align} \widetilde{\mathrm{VI}}(X, Y) &= \frac{H(X) + H(Y) - 2\,I(X;Y)}{H(X) + H(Y) - I(X;Y)}\,. \end{align}\]
Where:
- $\widetilde{\mathrm{VI}}(X, Y)$: Normalised variation of information.
- $H(X,Y) = H(X) + H(Y) - I(X;Y)$: Joint entropy.
Equation 19 of Section 3 of [51] states the unnormalised form and proves that it is a metric. The divisor of the normalised form is the joint entropy, and Equation 1 of [113] states that quotient as a distance on discrete distributions. Equation 6.25 of [12] is $1 - I(X;Y) / \max(H(X), H(Y))$ instead. That quotient lies in the same interval and is also a metric, the second metric of Theorem 2 of [114]. This function does not compute it.
Algorithm
- Read the shape of
XintoT, the number of observations, andN, the number of variables. - Allocate the
N × Nresultvar_mtx. - For each variable
j, write an exact zero atvar_mtx[j, j]. $\mathrm{VI}(X, X)$ is zero by definition. The histogram estimate of $I(X; X)$ does not reproduce the estimate of $H(X)$ bit for bit, so an estimate of the self pair leaves about1e-16on the diagonal. A matrix with a non-zero diagonal is not a distance matrix. The exact zero also savesNhistogram computations. - For each pair
(j, i)below the diagonal, take the bin count fromcalc_num_binsintonbins. - Take the two marginal entropies
exandeyand the joint histogramhxyfromcalc_hist_data. - Take the mutual information of
hxyfromintrinsic_mutual_infointomut_ixy, and apply the definition above to getvar_ixy. - When
normaliseis true, dividevar_ixyby the joint entropyvxy. - Clamp
var_ixybelow at zero, and write it into bothvar_mtx[j, i]andvar_mtx[i, j].
Arguments
X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.bins: Binning algorithm or fixed number of bins.normalise: Iftrue, the result is the normalised mutual information or the normalised variation of information.
Returns
var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal. In nats whennormaliseisfalse, and dimensionless on[0, 1]when it istrue.
Related
References
- [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equation 19.
- [113] C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961). Theorem, Equation 1.
- [114] A. Kraskov, H. Stögbauer, R. G. Andrzejak and P. Grassberger. Hierarchical clustering using mutual information. Europhysics Letters 70, 278–284 (2005). Section 2.3, Theorem 2.
- [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.2.2, Equations 6.24 and 6.25.
PortfolioOptimisers.mutual_info — Function
mutual_info(X::MatNum, bins::Int_Bin = HacineGharbiRavier(),
normalise::Bool = true)Compute the mutual information matrix for a set of variables.
The function computes the mutual information of each pair of columns of X from histogram estimates. The mutual information is the information that two variables share, and it is zero when the two variables are independent.
Mutual information is not a metric, because its diagonal is the marginal entropy and a metric is zero on its diagonal. It is larger for a more dependent pair, and a distance is smaller. variation_info is the metric of the two measures. The function estimates the diagonal and does not set it, so the diagonal is the marginal entropy in nats when normalise is false, and one when it is true.
Mathematical definition
Mutual information between assets $i$ and $j$:
\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j) = \sum_{x,y} p(x,y) \log\frac{p(x,y)}{p(x)\,p(y)}\,. \end{align}\]
Where:
- $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
- $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
- $H(X_i, X_j)$: Joint entropy.
- $p(x,y)$: Joint probability mass function.
When normalise = true, the function divides the mutual information by the smaller marginal entropy:
\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,. \end{align}\]
Where:
- $\tilde{I}(X_i; X_j)$: Normalised mutual information.
Equation 18 of [51] shows that the smaller marginal entropy is the largest value that the mutual information of the pair can take. The normalised form is therefore in $[0, 1]$, and it is one exactly when one variable determines the other. Equation 6.19 of [12] states this quotient, and no work that first states it was found.
Algorithm
- Read the shape of
XintoT, the number of observations, andN, the number of variables. - Allocate the
N × Nresultmut_mtx. - For each pair
(j, i)on or below the diagonal, take the bin count fromcalc_num_binsintonbins. - Take the two marginal entropies
exandeyand the joint histogramhxyfromcalc_hist_data. - Take the mutual information of
hxyfromintrinsic_mutual_infointomut_ixy. - When
normaliseis true, dividemut_ixyby the smaller ofexandeywithnormalised_mutual_info. A constant variable has an entropy of zero, so its normalised mutual information isNaN. - Clamp
mut_ixybelow at zero, and write it into bothmut_mtx[j, i]andmut_mtx[i, j].
Arguments
X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.bins: Binning algorithm or fixed number of bins.normalise: Iftrue, the result is the normalised mutual information or the normalised variation of information.
Returns
mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values. In nats whennormaliseisfalse, and dimensionless on[0, 1]when it istrue.
Related
variation_infomutual_variation_infocalc_hist_datacalc_num_binsintrinsic_mutual_infoInt_BinMutualInfoCovariance
References
- [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
- [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equation 18.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.1.6, Equation 6.19. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
PortfolioOptimisers.normalised_mutual_info — Function
normalised_mutual_info(mut_ixy::Number, ex::Number, ey::Number) -> NumberDivide the mutual information of a pair by the smaller of its two marginal entropies.
The quotient is the normalised mutual information of mutual_info. A constant variable has an entropy of zero, so the quotient is 0 / 0, and the function returns NaN. The mutual information of such a pair is zero only up to the rounding of the histogram, so a plain division can give Inf in place of NaN.
Algorithm
- Take the smaller entropy
hofexandey. - When
his zero, returnh / h, theNaNof the number type ofh. - Otherwise return
mut_ixy / h.
Arguments
mut_ixy: The mutual information of the pair, in nats.ex,ey: The marginal entropies of the two variables, in nats.
Returns
nmi::Number: The normalised mutual information, orNaNwhen an entropy is zero.
Related
PortfolioOptimisers.mutual_variation_info — Function
mutual_variation_info(X::MatNum, bins::Int_Bin = Knuth(), normalise::Bool = true)Compute the pairwise mutual information and variation of information matrices from a data matrix.
The function computes both matrices in one pass over the pairs, so the two matrices use one bin count and one joint histogram for each pair. With the same bins, the two results equal what mutual_info and variation_info return. The two normalisers differ. The function divides the mutual information by the smaller marginal entropy, and the variation of information by the joint entropy. The default bins is Knuth here and HacineGharbiRavier in the other two functions, so a caller who wants equal results must give the same bins to each function.
Mathematical definition
\[\begin{align} I(X_i; X_j) &= H(X_i) + H(X_j) - H(X_i, X_j)\,, \\ \mathrm{VI}(X_i, X_j) &= H(X_i) + H(X_j) - 2\,I(X_i; X_j)\,. \end{align}\]
When normalise = true, the function divides each by its own normaliser:
\[\begin{align} \tilde{I}(X_i; X_j) &= \frac{I(X_i; X_j)}{\min\bigl(H(X_i),\, H(X_j)\bigr)}\,, \\ \widetilde{\mathrm{VI}}(X_i, X_j) &= \frac{\mathrm{VI}(X_i, X_j)}{H(X_i) + H(X_j) - I(X_i; X_j)}\,. \end{align}\]
Where:
- $I(X_i; X_j)$: Mutual information between assets $i$ and $j$.
- $\mathrm{VI}(X_i, X_j)$: Variation of information between assets $i$ and $j$.
- $H(X_i)$, $H(X_j)$: Marginal Shannon entropies.
- $H(X_i, X_j) = H(X_i) + H(X_j) - I(X_i; X_j)$: Joint entropy.
- $\tilde{I}(X_i; X_j)$: Normalised mutual information.
- $\widetilde{\mathrm{VI}}(X_i, X_j)$: Normalised variation of information.
The two unnormalised forms are the rate of transmission of Section 12 of [50] and Equation 19 of [51]. The quotient by the joint entropy is Equation 1 of [113]. The quotient of the mutual information by the smaller marginal entropy is Equation 6.19 of [12], and no work that first states it was found.
Algorithm
- Read the shape of
XintoT, the number of observations, andN, the number of variables. - Allocate the two
N × Nresultsmut_mtxandvar_mtx. - For each pair
(j, i)on or below the diagonal, take the bin count fromcalc_num_binsintonbins. - Take the two marginal entropies
exandeyand the joint histogramhxyfromcalc_hist_data. - Take the mutual information of
hxyfromintrinsic_mutual_infointomut_ixy, and makevar_ixyfrom it and the two entropies. - When
normaliseis true, dividevar_ixyby the joint entropyvxy, and dividemut_ixyby the smaller ofexandeywithnormalised_mutual_info, which givesNaNfor a zero entropy. - Clamp both below at zero, and write each into both of its symmetric positions.
- After the inner loop, write an exact zero at
var_mtx[j, j], for the reason thatvariation_infogives.mut_mtx[j, j]keeps its estimate, because $I(X; X) = H(X)$ is not zero.
Arguments
X: Data matrixobservations × assetsif thedimskeyword does not exist ordims = 1,assets × observationswhendims = 2.bins: Binning algorithm or fixed number of bins.normalise: Iftrue, the result is the normalised mutual information or the normalised variation of information.
Returns
mut_mtx::Matrix{<:Number}: Symmetric matrix of pairwise mutual information values, whose diagonal is the marginal entropy whennormaliseisfalseand one when it istrue.var_mtx::Matrix{<:Number}: Symmetric matrix of pairwise variation of information values, with an exactly zero diagonal.
Related
References
- [50] C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948). Section 6 and Section 12.
- [51] M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007). Section 3, Equations 18 and 19.
- [113] C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961). Theorem, Equation 1.
- [12] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 6.1.6, Equation 6.19. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
References
- [12]
- D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025).
- [50]
- C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal 27, 379–423 (1948).
- [51]
- M. Meilă. Comparing clusterings—an information based distance. Journal of Multivariate Analysis 98, 873–895 (2007).
- [113]
- C. Rajski. A metric space of discrete probability distributions. Information and Control 4, 371–377 (1961).
- [114]
- A. Kraskov, H. Stögbauer, R. G. Andrzejak and P. Grassberger. Hierarchical clustering using mutual information. Europhysics Letters 70, 278–284 (2005).