Preprocessing

Preprocessing estimators

Preprocessing estimators transform price- or returns-level data under a fit/apply contract: fit_preprocessing learns whatever state the transformation needs from a training window — the surviving asset universe, imputation parameters, thresholds — and apply_preprocessing replays that state on unseen windows, so no information flows from test data back into the transformation.

They are ordinary estimators and know nothing about pipelines. A Pipeline drives them through these two verbs, exactly as it drives prior estimators through prior or optimisers through optimise.

PortfolioOptimisers.fit_preprocessing — Function

fit_preprocessing(est::AbstractPreprocessingEstimator, data) -> fitted

Fit a preprocessing estimator on a data window and return the fitted object consumed by apply_preprocessing.

The fitted object carries whatever state the transformation needs to be replayed consistently on unseen data — a fill's seed, thresholds, and the selected asset universe. Stateless preprocessing estimators return themselves.

Interfaces

Concrete preprocessing estimators must implement:

fit_preprocessing(est::MyPreprocessing, data) -> fitted: Compute the fitted state from the training window.
apply_preprocessing(fitted, data) -> data′: Transform a data window with the fitted state.

Arguments

est: The preprocessing estimator.
data: The training data window (PricesResult or ReturnsResult depending on the estimator's level).

Returns

fitted: The fitted object, typically an AbstractPreprocessingResult or the estimator itself when stateless.

Related

source

fit_preprocessing(
    tts::TrainTestSplit,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult

Fit a TrainTestSplit by cutting the data into its two windows.

Unlike the other preprocessing estimators, the fitted result is not replayed on unseen data: a holdout's rows are a fact about the fitting window alone, so apply_preprocessing on a TrainTestSplitResult passes the window through unchanged.

Algorithm

fit_preprocessing calls train_test_split on the data, and returns the TrainTestSplitResult that holds both windows.
apply_preprocessing on a TrainTestSplitResult returns its data argument unchanged.
apply_preprocessing on a TrainTestSplit returns its data argument unchanged, so an unfitted step is a pass-through as well.

Related

source

fit_preprocessing(
    sel::AbstractAssetSelector,
    rd::AbstractReturnsResult
) -> AssetSelectorResult

Fit any AbstractAssetSelector by recording the asset universe select_assets keeps.

This is the one funnel of the family, and it is where the Coverage Universe is applied. The window is reduced first, so every selector ranks among the assets that are live throughout the training window: a score, a redundancy and a rule all read live columns alone, and a new selector cannot forget the rule. An all-dead window throws an IsEmptyError where the mask is derived, so the refusal is coverage_mask's.

Algorithm

Reduce the training window to its Coverage Universe with coverage_reduction(rd::AbstractReturnsResult).
Call select_assets on the reduced window, giving the keep-mask keep.
Check that keep holds one entry per asset column of the reduced window.
Check that keep keeps at least one asset.
Return an AssetSelectorResult holding the names of the kept assets, in their original column order.

The result records names, so the expansion is free: apply_preprocessing finds each name in the window it replays on, and a name the reduction dropped is simply absent from the fitted universe.

Arguments

sel: The asset selector.
rd: The training-window returns data.

Validation

The carrier must hold an observations × assets returns matrix; one that collapsed the asset axis matches no method of the reduction and is named by a MethodError.
At least one asset must be in the Coverage Universe of the training window.
select_assets must return a mask whose length matches the number of asset columns of the reduced window.
The selection must keep at least one asset; a selector that empties the universe throws rather than passing a zero-asset problem downstream (the MissingDataFilter precedent).

Returns

res::AssetSelectorResult: The fitted asset universe.

Related

source

fit_preprocessing(ptr::PricesToReturns)

Reads a stepped PricesToReturns out: the conversion is stateless to a reader, so its fitted object is the estimator with the state dropped, exactly what the batch fit returns.

Related

source

fit_preprocessing(est::PriceGapFill)

Reads a stepped PriceGapFill out as the PriceGapFillResult the batch fit over the same rows gives: the assets observed so far, each with its last observed price, and the last timestamp folded as the end of the training window.

Related

source

fit_preprocessing(mdf::MissingDataFilter)

Reads a stepped MissingDataFilter out as the MissingDataFilterResult the batch fit over the same rows gives.

Related

source

PortfolioOptimisers.apply_preprocessing — Function

apply_preprocessing(fitted, data) -> data′

Transform a data window with a fitted preprocessing object.

Applying the fitted object produced by fit_preprocessing on the training window to an unseen (test) window replays the same transformation — the same asset universe, the same fill seed — so train and test data stay consistent and no information flows from test to train.

Arguments

fitted: The fitted object returned by fit_preprocessing (an AbstractPreprocessingResult, or a stateless estimator).
data: The data window to transform.

Returns

data′: The transformed data window.

Related

source

apply_preprocessing(
    res::AssetSelectorResult,
    rd::AbstractReturnsResult
) -> Union{ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, Nothing} where {_A, _B, _C, _D, _E, _F, _G, _H, _I}, ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, AssetPanel{__T_pf, __T_amsk, __T_emsk}} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, __T_pf, __T_amsk, __T_emsk}}

Replay a fitted asset universe on a data window.

The surviving columns are emitted in fitted order, not in the window's own column order, because the terminal weights are indexed by the training universe and assert_universe_aligned compares the two name vectors elementwise.

Algorithm

For each fitted asset name, in fitted order, find the column of the window that carries it, and record that position in idx.
Check that the name is present. A name the window does not carry throws.
Return a port_opt_view of the window at idx. The positions are in fitted order, so the view reorders the window's columns when the two orders differ.

Arguments

res: The fitted asset universe.
rd: The data window to transform.

Validation

Every fitted asset name must be present in the window; a missing one throws rather than silently shrinking the universe. The message names the missing asset and the window's size through unknown_variable_msg, never a whole universe.

Returns

rd′::AbstractReturnsResult: The window restricted to the fitted universe, in fitted order.

Related

source