Preprocessing

Preprocessing estimators

A preprocessing estimator transforms prices or returns in two steps. fit_preprocessing learns what the transform needs from a training window, such as the assets to keep, the values to impute or a threshold. apply_preprocessing applies that fitted result to another window, and learns nothing from it. The data of a test window never changes the transform.

A preprocessing estimator does not depend on a pipeline. A Pipeline calls these two functions on it, as it calls prior on a prior estimator and optimise on an optimiser.

PortfolioOptimisers.fit_preprocessing — Function
fit_preprocessing(est::AbstractPreprocessingEstimator, data) -> fitted

Fit a preprocessing estimator on a training window, and return the fitted object that apply_preprocessing reads.

The fitted object holds the state that the transform needs to apply to another window, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. A stateless estimator, such as PricesToReturns, returns itself.

The method for an AbstractPreprocessingEstimator is a fallback that throws. Each concrete estimator implements its own method, and AbstractPreprocessingEstimator states what that method must do.

Arguments

  • est: The preprocessing estimator.
  • data: The training window, a PricesResult or a ReturnsResult as the data level of est needs.

Validation

  • An estimator whose type does not implement this method throws an ArgumentError that names the type.

Returns

Related

source
fit_preprocessing(
    tts::TrainTestSplit,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult

Fit a TrainTestSplit by cutting the data into its two windows.

The fitted result does not replay on new data, unlike the fitted results of the other preprocessing estimators. The rows of a holdout belong to the fitting data alone, so apply_preprocessing on a TrainTestSplitResult returns its data unchanged. apply_preprocessing on an unfitted TrainTestSplit also returns its data unchanged.

Algorithm

  1. Call train_test_split with tts and data, which returns the TrainTestSplitResult of both windows.

Validation

  • The sizes of tts pass the checks of safe_index for the rows of data.

Related

source
fit_preprocessing(
    sel::AbstractAssetSelector,
    rd::AbstractReturnsResult
) -> AssetSelectorResult

Fit any AbstractAssetSelector by recording the asset universe select_assets keeps.

This one method fits every asset selector, and it applies the Coverage Universe. It reduces the window before it calls select_assets, so every selector ranks only the assets that are live at every row of the training window. A new selector gets the reduction with no code of its own. A window with no live asset throws an IsEmptyError from coverage_mask, before any selector runs.

Algorithm

  1. Reduce the training window to its Coverage Universe with coverage_reduction(rd::AbstractReturnsResult).
  2. Call select_assets on the reduced window, giving the keep-mask keep.
  3. Check that keep holds one entry per asset column of the reduced window.
  4. Check that keep keeps at least one asset.
  5. Return an AssetSelectorResult holding the names of the kept assets, in the column order of the window.

Arguments

  • sel: The asset selector.
  • rd: The training-window returns data.

Validation

  • rd must hold an observations × assets returns matrix. A returns result that collapsed the asset axis, such as PredictionReturnsResult, matches no method of the reduction and throws a MethodError.
  • At least one asset must be in the Coverage Universe of the training window, else IsEmptyError.
  • select_assets must return one entry per asset column of the reduced window, else DimensionMismatch.
  • The mask must keep at least one asset, else IsEmptyError, so no later step receives a problem with no asset.

Returns

  • res::AssetSelectorResult: The fitted asset universe, as names. A name that the reduction dropped is absent from it. apply_preprocessing finds each name in the window that it replays on.

Related

source
fit_preprocessing(ptr::PricesToReturns)

Reads a stepped PricesToReturns out.

The conversion keeps no fitted value. So its fitted object is the estimator with cache set to nothing, which is the object that the batch fit returns.

Returns

  • ptr′::PricesToReturns: The estimator with no state.

Related

source
fit_preprocessing(est::PriceGapFill)

Reads a stepped PriceGapFill out as the PriceGapFillResult that the batch fit over the same rows gives.

The result holds each asset of the folded blocks, with its last observed price, and the last folded timestamp as the end of the training window. An asset whose carried price an absence retired has a missing seed. An asset that no block priced also has a missing seed.

Algorithm

  1. Read the state with partial_fit_cache.
  2. Read into v the carried price of each asset whose flag in held is set, and missing for the others. An asset that no block priced has no carried price, so its entry is missing too. The element type of v drops Missing when no seed is missing.
  3. Build the PriceGapFillResult from a copy of the names of the state, v, the end te of the state, and the fill and strict of the step.

Validation

Returns

  • res::PriceGapFillResult: The fitted gap fill of the folded observations.

Related

source
fit_preprocessing(mdf::MissingDataFilter)

Reads a stepped MissingDataFilter out as the MissingDataFilterResult that the batch fit over the same rows gives.

Algorithm

  1. Read the state with partial_fit_cache.
  2. Keep the assets whose share of missing observations miss / n does not exceed col_thr, giving keep, and check that one asset at least survives. share_at_most computes the share in the type of col_thr, as the batch fit does.
  3. Build the MissingDataFilterResult from the surviving asset names and row_thr.

Validation

  • The step keeps a state. partial_fit_cache throws an ArgumentError otherwise.
  • One asset at least survives the column filter. An IsEmptyError is thrown otherwise.

Returns

  • res::MissingDataFilterResult: The fitted filter of the folded observations.

Related

source
PortfolioOptimisers.apply_preprocessing — Function
apply_preprocessing(fitted, data) -> data′

Transform a data window with a fitted preprocessing object.

fitted is the object that fit_preprocessing returned for a training window. On another window it does the same transform, with the same asset universe and the same per-asset values, and it fits nothing on that window. So no data of a test window reaches the fit.

The method for an AbstractPreprocessingEstimator or an AbstractPreprocessingResult is a fallback that throws. Each concrete fitted object implements its own method.

Arguments

Validation

  • A fitted object whose type does not implement this method throws an ArgumentError that names the type.

Returns

Related

source
apply_preprocessing(
    res::AssetSelectorResult,
    rd::AbstractReturnsResult
) -> ReturnsResult

Replay a fitted asset universe on a data window.

The result holds the kept columns in the fitted order, not in the column order of the window. The last step computes its weights over the training universe, and assert_universe_aligned compares the two name vectors entry by entry.

Algorithm

  1. For each fitted asset name, in fitted order, find the column of the window that carries it, and record that position in idx.
  2. Check that the name is present. A name the window does not carry throws.
  3. Return a port_opt_view of the window at idx. The positions are in fitted order, so the view reorders the window's columns when the two orders differ.

Arguments

  • res: The fitted asset universe.
  • rd: The data window to transform.

Validation

  • Every fitted asset name must be in the window, else ArgumentError, so the universe never shrinks without an error. The message names the missing asset and the size of the window through unknown_variable_msg, and it does not list the whole universe.

Returns

  • rd′::AbstractReturnsResult: The window restricted to the fitted universe, in fitted order.

Related

source
PortfolioOptimisers.keeps_observations — Method
keeps_observations(fitted) -> Bool

Return true when a fitted preprocessing object keeps every observation of a window that it transforms.

A transform keeps the observations when the window it returns holds the observations of its input, in the same order. It can change the assets and the values. A Pipeline reads this answer when it predicts on a fold of returns data. When every fitted step keeps the observations, the fold records its rows on the PredictionResult, and a realised factor_attribution of the cross-validation finds the block rows of each fold by position.

The fallback returns false, which makes no promise. A fold whose steps make no promise records no rows, so the attribution matches it by timestamp, or lines it up at the tail. A wrong true makes the attribution read the wrong rows, so a fitted object returns true only when no window loses, adds or moves an observation. The asset selectors return true, because they select columns alone. PricesToReturns keeps the fallback, because it drops the first price row.

Arguments

Returns

  • flag::Bool: true when the transform keeps every observation of the window in order, and false when it makes no such promise.

Related

source

Types

A new estimator subtypes AbstractPricesPreprocessingEstimator or AbstractReturnsPreprocessingEstimator. Its fitted result subtypes AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult.

PortfolioOptimisers.AbstractPreprocessingEstimator — Type
abstract type AbstractPreprocessingEstimator <: AbstractEstimator

Abstract supertype for all preprocessing estimator types.

A preprocessing estimator transforms price or returns data in two steps. fit_preprocessing fits it on a training window and returns a fitted object. The fitted object holds the state of the transform, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. apply_preprocessing applies the fitted object to a window and fits nothing on that window, so the data of a test window never changes the transform. A stateless estimator is its own fitted object.

A preprocessing estimator does not depend on a pipeline. A Pipeline calls the same two functions that any other caller calls.

The data level of an estimator decides which slot of a pipeline the estimator reads and writes. A concrete estimator that reads and writes one level subtypes one of the two data-level subtypes:

An estimator that changes the level, or that works at both levels, subtypes AbstractPreprocessingEstimator directly. PricesToReturns reads prices and writes returns, and TrainTestSplit splits a window of either level. Each has its own methods of run_step and apply_fitted_step, because a pipeline refuses a direct subtype that has no run_step method.

Interfaces

To implement a new preprocessing estimator, subtype AbstractPricesPreprocessingEstimator or AbstractReturnsPreprocessingEstimator, and implement the following methods:

fit_preprocessing

  • fit_preprocessing(est::MyPreprocessing, data) -> fitted: Fit the state of the transform on the training window data. A stateless estimator returns est.

apply_preprocessing

  • apply_preprocessing(fitted, data) -> data′: Transform the window data with fitted, the object that fit_preprocessing returned.

keeps_observations

  • keeps_observations(fitted) -> Bool: Optional. Return true when the transform keeps every observation of the window, in order. The fallback returns false.

A subtype that does not implement fit_preprocessing or apply_preprocessing gets the fallback of that method, which throws an ArgumentError that names the subtype.

Related

source
PortfolioOptimisers.AbstractPricesPreprocessingEstimator — Type
abstract type AbstractPricesPreprocessingEstimator <: AbstractPreprocessingEstimator

Abstract supertype for preprocessing estimators that consume and produce price-level data.

A concrete subtype transforms a PricesResult into another PricesResult. A Pipeline fits the estimator on the prices in its prices slot, and writes the transformed prices back to the same slot.

Interfaces

To implement a new price-level estimator, subtype AbstractPricesPreprocessingEstimator and implement the following methods:

fit_preprocessing

  • fit_preprocessing(est::MyPricesPreprocessing, pr::PricesResult) -> MyPricesPreprocessingResult: Fit the state of the transform on the training prices pr. A stateless estimator returns est.

apply_preprocessing

  • apply_preprocessing(res::MyPricesPreprocessingResult, pr::PricesResult) -> PricesResult: Transform the price window pr with the fitted state of res.

Related

source
PortfolioOptimisers.AbstractReturnsPreprocessingEstimator — Type
abstract type AbstractReturnsPreprocessingEstimator <: AbstractPreprocessingEstimator

Abstract supertype for preprocessing estimators that consume and produce returns-level data.

A concrete subtype transforms a ReturnsResult into another ReturnsResult. A Pipeline fits the estimator on the returns in its returns slot, and writes the transformed returns back to the same slot.

Interfaces

To implement a new returns-level estimator, subtype AbstractReturnsPreprocessingEstimator and implement the following methods:

fit_preprocessing

  • fit_preprocessing(est::MyReturnsPreprocessing, rd::AbstractReturnsResult) -> MyReturnsPreprocessingResult: Fit the state of the transform on the training returns rd. A stateless estimator returns est.

apply_preprocessing

  • apply_preprocessing(res::MyReturnsPreprocessingResult, rd::AbstractReturnsResult) -> AbstractReturnsResult: Transform the returns window rd with the fitted state of res.

keeps_observations

  • keeps_observations(fitted) -> Bool: Optional. Return true when the transform keeps every observation of the window, in order. The fallback returns false.

Related

source
PortfolioOptimisers.AbstractPreprocessingResult — Type
abstract type AbstractPreprocessingResult <: AbstractResult

Abstract supertype for all preprocessing result types.

fit_preprocessing returns a preprocessing result from a training window. The result holds the fitted state that apply_preprocessing needs to do the same transform on another window, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. A stateless estimator returns itself and makes no result. A TrainTestSplitResult is not a preprocessing result, because it holds the two windows of one split and applies to no other window.

A concrete result subtypes the data-level subtype that matches its estimator, AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult. A Pipeline reads that subtype to find the slot that the result transforms.

Interfaces

To implement a new preprocessing result, subtype AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult, and implement the following method:

  • apply_preprocessing(res::MyPreprocessingResult, data) -> data′: Transform the window data with the fitted state of res.
  • keeps_observations(res::MyPreprocessingResult) -> Bool: Optional. Return true when the transform keeps every observation of the window, in order. The fallback returns false.

Related

source
PortfolioOptimisers.AbstractPricesPreprocessingResult — Type
abstract type AbstractPricesPreprocessingResult <: AbstractPreprocessingResult

Abstract supertype for preprocessing results that apply to price-level data (PricesResult).

A Pipeline applies a result of this type to the prices in its prices slot, and passes a returns window through with no change.

Interfaces

To implement a new price-level result, subtype AbstractPricesPreprocessingResult and implement the following method:

  • apply_preprocessing(res::MyPricesPreprocessingResult, pr::PricesResult) -> PricesResult: Transform the price window pr with the fitted state of res.

Related

source
PortfolioOptimisers.AbstractReturnsPreprocessingResult — Type
abstract type AbstractReturnsPreprocessingResult <: AbstractPreprocessingResult

Abstract supertype for preprocessing results that apply to returns-level data (ReturnsResult).

A Pipeline applies a result of this type to the returns in its returns slot, and passes a price window through with no change.

Interfaces

To implement a new returns-level result, subtype AbstractReturnsPreprocessingResult and implement the following method:

  • apply_preprocessing(res::MyReturnsPreprocessingResult, rd::AbstractReturnsResult) -> AbstractReturnsResult: Transform the returns window rd with the fitted state of res.
  • keeps_observations(res::MyReturnsPreprocessingResult) -> Bool: Optional. Return true when the transform keeps every observation of the window, in order. The fallback returns false.

Related

source