Combinatorial

struct CombinatorialCrossValidation{__T_n_folds, __T_n_test_folds, __T_purged_size, __T_embargo_size, __T_wd, __T_fa, __T_store_weight_path, __T_strict} <: NonSequentialCrossValidationEstimator

Implements combinatorial non-sequential cross-validation with purging and embargoing, allowing for all possible combinations of test folds.

The observations are cut into n_folds consecutive folds, and every combination of n_test_folds of them is one split's test set. The remaining folds form that split's training set, less the purged and embargoed rows on each side of every test block. Each fold is a test fold in the same number of splits, so the test blocks recombine into n_test_paths full backtest paths, each covering every observation exactly once.

Mathematical definition

\[\begin{align} S(n,\, k) &= \binom{n}{k} = \frac{n!}{k!\,(n-k)!}\,, \\ \varphi(n,\, k) &= \frac{k}{n} \binom{n}{k} = \frac{(n-1)!}{(k-1)!\,(n-k)!}\,, \\ \bar{T}(n,\, k) &= \frac{T}{n} (n - k)\,. \end{align}\]

Where:

• $S(n,\, k)$: Number of splits, one per combination of test folds.
• $\varphi(n,\, k)$: Number of recombined test paths.
• $\bar{T}(n,\, k)$: Average number of training observations per split, before purging and embargoing.
• $n$: Number of folds, n_folds.
• $k$: Number of test folds per split, n_test_folds.
• $T$: Number of observations.

Fields

• n_folds: Number of folds.

• n_test_folds: Number of folds held out for testing in each combination. The remaining n_folds - n_test_folds folds train.

• purged_size: Number of observations to purge between train and test sets.

• embargo_size: Number of observations to embargo after the test set.

• wd: Weight drift the fold's return series is read under, or nothing to read it at the target weights of the fold.

• fa: Fee amortisation algorithm the fold's realised series charges the two fixed fee terms on, or nothing to inherit the clock the fee itself states. It overrides Fees.fa for that series alone, and it reaches the fit not at all.

• store_weight_path: If true, the fold stores the weight path it computed; if false, a reader rebuilds it on demand.

• strict: If true, a Held Gap raises an ArgumentError; if false, it warns and the pair contributes zero. A Held Gap is an (observation, asset) pair at which the fold's weight is non-zero and the asset's return is missing, which is what a delisting inside a test window makes.

Constructors

CombinatorialCrossValidation(;    n_folds::Integer = 10,    n_test_folds::Integer = 2,    purged_size::Integer = 0,    embargo_size::Integer = 0,    wd::Option{<:AbstractWeightDrift} = nothing,    fa::Option{<:AbstractFeeAmortisation} = nothing,    store_weight_path::Bool = false,    strict::Bool = false,    max_comb::Integer = 100_000,) -> CombinatorialCrossValidation

Keyword arguments correspond to the struct's fields.

Weight drift

wd is the Weight Drift of the scheme, and nothing is the library's original behaviour: a fold's return series is X * w net of fees, read at the target weights of that fold. A SelfFinancingDrift reads the series as the wealth ratio of the drifted holdings instead, and the fold carries a HeldWeightsResult. store_weight_path makes the fold store the weight path it computed, which a reader otherwise rebuilds on demand. strict decides what a Held Gap does: an asset that delists inside a test window carries a non-zero weight and a missing return, and the fold zeroes that pair and warns, or refuses with an ArgumentError under strict.

A combinatorial enumeration is not a timeline, so this scheme carries no Previous-Weights Source. Its splits recombine into several paths, and a split's folds are independent of the others.

Fee clock

fa is the clock the fold's realised series charges the two fixed fee terms on, and it overrides the fa of the fee itself. nothing inherits that fee's clock, which is the library's original behaviour. A FirstObservationFees charges the two terms on the first observation of the fold, and an AmortisedFees spreads them over the fold. The field reaches the fit not at all, so the optimiser keeps pricing the fee the way its own objective must.

The default holds out 2 of 10 folds for testing. This gives binomial(10, 2) = 45 splits, 9 recombined test paths, and an average training set of 80% of the observations. Because binomial(n, k) == binomial(n, n - k), a transposed pair such as n_test_folds = 8 produces the same number of splits on a training set of only 20% of the observations. Choose n_test_folds well below div(n_folds, 2), or let optimal_number_folds choose the pair.

Validation

• n_folds must be non-empty, greater than zero, and finite.
• n_test_folds must be non-empty, greater than zero, and finite.
• purged_size and embargo_size must be non-empty and finite.
• binomial(n_folds, n_test_folds) <= max_comb. max_comb is a constructor argument, not a field, so it bounds the split count at construction and is not carried on the estimator.
• Base.split additionally checks purged_size + embargo_size < div(T, n_folds), because a gap as wide as the smallest fold would empty a training fold.

Examples

julia> CombinatorialCrossValidation(; n_folds = 10, n_test_folds = 2, purged_size = 2,                                    embargo_size = 1)CombinatorialCrossValidation            n_folds ┼ Int64: 10       n_test_folds ┼ Int64: 2        purged_size ┼ Int64: 2       embargo_size ┼ Int64: 1                 wd ┼ nothing                 fa ┼ nothing  store_weight_path ┼ Bool: false             strict ┴ Bool: false

Related

• cross_val_predict
• search_cross_validation
• NonSequentialCrossValidationEstimator
• CombinatorialCrossValidationResult
• n_splits
• n_test_paths
• average_train_size
• optimal_number_folds

References

• [118] M. López de Prado. Advances in Financial Machine Learning (John Wiley & Sons, Hoboken, NJ, 2018). Chapter 12.
• [5] D. Cajas. Advanced Portfolio Optimization: A Cutting-edge Quantitative Approach (Springer Nature Switzerland, 2025). Section 15.3, Equations 15.1 and 15.2.

struct CombinatorialCrossValidationResult{__T_train_idx, __T_test_idx, __T_path_ids} <: NonSequentialCrossValidationResult

Result type produced by CombinatorialCrossValidation after splitting data into combinatorial training and testing folds.

Stores the train index vectors, the nested test index vectors, and a matrix of path identifiers.

Every field is indexed by split. train_idx[j] holds the training rows of split j, purged and embargoed. test_idx[j] holds that split's n_test_folds test blocks, sorted by first observation. Column j of path_ids names the path each of those blocks belongs to, in the same order, which is what recombines the blocks of all splits into full backtest paths.

Fields

• train_idx: Training set indices.

• test_idx: Test set indices.

• path_ids: Path identifiers for cross-validation splits.

Constructors

CombinatorialCrossValidationResult(;    train_idx::VecVecInt,    test_idx::VecVecVecInt,    path_ids::AbstractMatrix{<:Integer}) -> CombinatorialCrossValidationResult

Keywords correspond to the struct's fields.

Validation

• !isempty(train_idx).
• !isempty(test_idx).
• !isempty(path_ids).
• length(train_idx) == length(test_idx) == size(path_ids, 2).

Related

• CombinatorialCrossValidation
• NonSequentialCrossValidationResult
• n_splits
• get_path_ids
• sort_predictions!

Base.split(ccv::CombinatorialCrossValidation, rd::Prices_RR) -> CombinatorialCrossValidationResult

Split the data rd into all possible combinations of training and test folds using combinatorial cross-validation with optional purging and embargoing.

Arguments

• ccv::CombinatorialCrossValidation: Combinatorial cross-validation estimator.
• rd: Returns-level or price-level data to split (Prices_RR).

Validation

• purged_size + embargo_size < div(T, n_folds), where T is the number of observations. A gap as wide as the smallest fold would empty a training fold.

Returns

• CombinatorialCrossValidationResult: Result containing train indices, nested test index vectors (one per path), and a matrix of path IDs mapping folds to paths.

Details

• The folds are consecutive and cover every observation exactly once. mod(T, n_folds) extra rows go to the last fold, so the fold sizes differ by at most n_folds - 1 rows.
• Every training row within purged_size rows before a test block, or within purged_size + embargo_size rows after one, is dropped from that split's training set.
• Column j of path_ids names the path of each test block of split j, in the same order as test_idx[j], which is sorted by first observation. Collecting the blocks of one path over all splits therefore reproduces the whole timeline in order.

Related

• CombinatorialCrossValidation
• CombinatorialCrossValidationResult
• n_splits
• get_path_ids
• cv_nobs

optimal_number_folds(T::Integer, target_train_size::Integer,
                     target_n_test_paths::Integer; train_size_w::Number = 1,
                     n_test_paths_w::Number = 1, maxval::Number = 1e5) -> Tuple{Int, Int}

Find the optimal (n_folds, n_test_folds) pair for combinatorial cross-validation by minimising a weighted cost that balances the average training size against the number of test paths.

Mathematical definition

The cost function for a candidate (n_folds, n_test_folds) pair is:

\[\begin{align} \text{cost} &= w_{\text{ntp}} \frac{|P(n,k) - P^*|}{P^*} + w_{\text{tr}} \frac{|\bar{T}(n,k) - T^*|}{T^*}\,. \end{align}\]

Where:

• $\text{cost}$: Weighted cost for the candidate fold configuration.
• $w_{\text{ntp}}$: Weight on the test-paths component.
• $w_{\text{tr}}$: Weight on the training-size component.
• $P(n,k)$: Number of test paths for $n$ folds and $k$ test folds.
• $\bar{T}(n,k)$: Average training size for $n$ folds and $k$ test folds.
• $P^*$: Target number of test paths (target_n_test_paths).
• $T^*$: Target training size (target_train_size).

Arguments

• T: Total number of observations in the dataset.
• target_train_size: Desired average number of observations in each training set.
• target_n_test_paths: Desired number of recombined test paths.
• train_size_w: Weight applied to the training-size component of the cost (default 1).
• n_test_paths_w: Weight applied to the test-paths component of the cost (default 1).
• maxval: Pruning threshold (default 1e5). Within one n_folds, let m be the first n_test_folds whose cost exceeds maxval. The search then skips every n_test_folds in (m, n_folds - m) and resumes at the mirror value n_folds - m, because binomial(n, k) == binomial(n, n - k) makes the two ends of the range the cheap ones and the middle the expensive one.

Returns

• Tuple{Int, Int}: The optimal (n_folds, n_test_folds) pair minimising the weighted cost. Returns (0, 0) when no valid configuration is found.

Details

• The search walks n_folds from 3 to T + 1 and n_test_folds from 2 to n_folds.
• A candidate whose split count overflows Int is treated as infinitely expensive rather than raising. Such a configuration is unusable: CombinatorialCrossValidation refuses it through max_comb long before the count reaches that size. Raising maxval widens the search into that region, so the guard is what keeps a large maxval from failing.

Related

• CombinatorialCrossValidation
• n_test_paths
• average_train_size