Preprocessing
Preprocessing estimators
A preprocessing estimator transforms prices or returns in two steps. fit_preprocessing learns what the transform needs from a training window, such as the assets to keep, the values to impute or a threshold. apply_preprocessing applies that fitted result to another window, and learns nothing from it. The data of a test window never changes the transform.
A preprocessing estimator does not depend on a pipeline. A Pipeline calls these two functions on it, as it calls prior on a prior estimator and optimise on an optimiser.
PortfolioOptimisers.fit_preprocessing — Function
fit_preprocessing(est::AbstractPreprocessingEstimator, data) -> fittedFit a preprocessing estimator on a training window, and return the fitted object that apply_preprocessing reads.
The fitted object holds the state that the transform needs to apply to another window, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. A stateless estimator, such as PricesToReturns, returns itself.
The method for an AbstractPreprocessingEstimator is a fallback that throws. Each concrete estimator implements its own method, and AbstractPreprocessingEstimator states what that method must do.
Arguments
est: The preprocessing estimator.data: The training window, aPricesResultor aReturnsResultas the data level ofestneeds.
Validation
- An estimator whose type does not implement this method throws an
ArgumentErrorthat names the type.
Returns
fitted: The fitted object. It is anAbstractPreprocessingResult, or the estimator itself when the estimator is stateless. ATrainTestSplitreturns aTrainTestSplitResult.
Related
AbstractPreprocessingEstimatorapply_preprocessingAbstractPreprocessingResultpartial_fit_transform: the online form, whose data-lessfit_preprocessing(est)reads the fitted object out of the estimator's state.
fit_preprocessing(
tts::TrainTestSplit,
data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult
Fit a TrainTestSplit by cutting the data into its two windows.
The fitted result does not replay on new data, unlike the fitted results of the other preprocessing estimators. The rows of a holdout belong to the fitting data alone, so apply_preprocessing on a TrainTestSplitResult returns its data unchanged. apply_preprocessing on an unfitted TrainTestSplit also returns its data unchanged.
Algorithm
- Call
train_test_splitwithttsanddata, which returns theTrainTestSplitResultof both windows.
Validation
- The sizes of
ttspass the checks ofsafe_indexfor the rows ofdata.
Related
fit_preprocessing(
sel::AbstractAssetSelector,
rd::AbstractReturnsResult
) -> AssetSelectorResult
Fit any AbstractAssetSelector by recording the asset universe select_assets keeps.
This one method fits every asset selector, and it applies the Coverage Universe. It reduces the window before it calls select_assets, so every selector ranks only the assets that are live at every row of the training window. A new selector gets the reduction with no code of its own. A window with no live asset throws an IsEmptyError from coverage_mask, before any selector runs.
Algorithm
- Reduce the training window to its Coverage Universe with
coverage_reduction(rd::AbstractReturnsResult). - Call
select_assetson the reduced window, giving the keep-maskkeep. - Check that
keepholds one entry per asset column of the reduced window. - Check that
keepkeeps at least one asset. - Return an
AssetSelectorResultholding the names of the kept assets, in the column order of the window.
Arguments
sel: The asset selector.rd: The training-window returns data.
Validation
rdmust hold anobservations × assetsreturns matrix. A returns result that collapsed the asset axis, such asPredictionReturnsResult, matches no method of the reduction and throws aMethodError.- At least one asset must be in the Coverage Universe of the training window, else
IsEmptyError. select_assetsmust return one entry per asset column of the reduced window, elseDimensionMismatch.- The mask must keep at least one asset, else
IsEmptyError, so no later step receives a problem with no asset.
Returns
res::AssetSelectorResult: The fitted asset universe, as names. A name that the reduction dropped is absent from it.apply_preprocessingfinds each name in the window that it replays on.
Related
fit_preprocessing(ptr::PricesToReturns)Reads a stepped PricesToReturns out.
The conversion keeps no fitted value. So its fitted object is the estimator with cache set to nothing, which is the object that the batch fit returns.
Returns
ptr′::PricesToReturns: The estimator with no state.
Related
fit_preprocessing(est::PriceGapFill)Reads a stepped PriceGapFill out as the PriceGapFillResult that the batch fit over the same rows gives.
The result holds each asset of the folded blocks, with its last observed price, and the last folded timestamp as the end of the training window. An asset whose carried price an absence retired has a missing seed. An asset that no block priced also has a missing seed.
Algorithm
- Read the state with
partial_fit_cache. - Read into
vthe carried price of each asset whose flag inheldis set, andmissingfor the others. An asset that no block priced has no carried price, so its entry ismissingtoo. The element type ofvdropsMissingwhen no seed ismissing. - Build the
PriceGapFillResultfrom a copy of the names of the state,v, the endteof the state, and thefillandstrictof the step.
Validation
- The step keeps a state.
partial_fit_cachethrows anArgumentErrorotherwise.
Returns
res::PriceGapFillResult: The fitted gap fill of the folded observations.
Related
fit_preprocessing(mdf::MissingDataFilter)Reads a stepped MissingDataFilter out as the MissingDataFilterResult that the batch fit over the same rows gives.
Algorithm
- Read the state with
partial_fit_cache. - Keep the assets whose share of missing observations
miss / ndoes not exceedcol_thr, givingkeep, and check that one asset at least survives.share_at_mostcomputes the share in the type ofcol_thr, as the batch fit does. - Build the
MissingDataFilterResultfrom the surviving asset names androw_thr.
Validation
- The step keeps a state.
partial_fit_cachethrows anArgumentErrorotherwise. - One asset at least survives the column filter. An
IsEmptyErroris thrown otherwise.
Returns
res::MissingDataFilterResult: The fitted filter of the folded observations.
Related
PortfolioOptimisers.apply_preprocessing — Function
apply_preprocessing(fitted, data) -> data′Transform a data window with a fitted preprocessing object.
fitted is the object that fit_preprocessing returned for a training window. On another window it does the same transform, with the same asset universe and the same per-asset values, and it fits nothing on that window. So no data of a test window reaches the fit.
The method for an AbstractPreprocessingEstimator or an AbstractPreprocessingResult is a fallback that throws. Each concrete fitted object implements its own method.
Arguments
fitted: The object thatfit_preprocessingreturned, anAbstractPreprocessingResultor a stateless estimator.data: The data window to transform.
Validation
- A fitted object whose type does not implement this method throws an
ArgumentErrorthat names the type.
Returns
data′: The transformed window, at the data level ofdata.PricesToReturnschanges the level. It takes aPricesResultand returns aReturnsResult.
Related
apply_preprocessing(
res::AssetSelectorResult,
rd::AbstractReturnsResult
) -> ReturnsResult
Replay a fitted asset universe on a data window.
The result holds the kept columns in the fitted order, not in the column order of the window. The last step computes its weights over the training universe, and assert_universe_aligned compares the two name vectors entry by entry.
Algorithm
- For each fitted asset name, in fitted order, find the column of the window that carries it, and record that position in
idx. - Check that the name is present. A name the window does not carry throws.
- Return a
port_opt_viewof the window atidx. The positions are in fitted order, so the view reorders the window's columns when the two orders differ.
Arguments
res: The fitted asset universe.rd: The data window to transform.
Validation
- Every fitted asset name must be in the window, else
ArgumentError, so the universe never shrinks without an error. The message names the missing asset and the size of the window throughunknown_variable_msg, and it does not list the whole universe.
Returns
rd′::AbstractReturnsResult: The window restricted to the fitted universe, in fitted order.
Related
PortfolioOptimisers.keeps_observations — Method
keeps_observations(fitted) -> BoolReturn true when a fitted preprocessing object keeps every observation of a window that it transforms.
A transform keeps the observations when the window it returns holds the observations of its input, in the same order. It can change the assets and the values. A Pipeline reads this answer when it predicts on a fold of returns data. When every fitted step keeps the observations, the fold records its rows on the PredictionResult, and a realised factor_attribution of the cross-validation finds the block rows of each fold by position.
The fallback returns false, which makes no promise. A fold whose steps make no promise records no rows, so the attribution matches it by timestamp, or lines it up at the tail. A wrong true makes the attribution read the wrong rows, so a fitted object returns true only when no window loses, adds or moves an observation. The asset selectors return true, because they select columns alone. PricesToReturns keeps the fallback, because it drops the first price row.
Arguments
fitted: The object thatfit_preprocessingreturned, anAbstractPreprocessingResultor a stateless estimator.
Returns
flag::Bool:truewhen the transform keeps every observation of the window in order, andfalsewhen it makes no such promise.
Related
Types
A new estimator subtypes AbstractPricesPreprocessingEstimator or AbstractReturnsPreprocessingEstimator. Its fitted result subtypes AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult.
PortfolioOptimisers.AbstractPreprocessingEstimator — Type
abstract type AbstractPreprocessingEstimator <: AbstractEstimatorAbstract supertype for all preprocessing estimator types.
A preprocessing estimator transforms price or returns data in two steps. fit_preprocessing fits it on a training window and returns a fitted object. The fitted object holds the state of the transform, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. apply_preprocessing applies the fitted object to a window and fits nothing on that window, so the data of a test window never changes the transform. A stateless estimator is its own fitted object.
A preprocessing estimator does not depend on a pipeline. A Pipeline calls the same two functions that any other caller calls.
The data level of an estimator decides which slot of a pipeline the estimator reads and writes. A concrete estimator that reads and writes one level subtypes one of the two data-level subtypes:
AbstractPricesPreprocessingEstimator: reads and writes aPricesResult.AbstractReturnsPreprocessingEstimator: reads and writes aReturnsResult.
An estimator that changes the level, or that works at both levels, subtypes AbstractPreprocessingEstimator directly. PricesToReturns reads prices and writes returns, and TrainTestSplit splits a window of either level. Each has its own methods of run_step and apply_fitted_step, because a pipeline refuses a direct subtype that has no run_step method.
Interfaces
To implement a new preprocessing estimator, subtype AbstractPricesPreprocessingEstimator or AbstractReturnsPreprocessingEstimator, and implement the following methods:
fit_preprocessing
fit_preprocessing(est::MyPreprocessing, data) -> fitted: Fit the state of the transform on the training windowdata. A stateless estimator returnsest.
apply_preprocessing
apply_preprocessing(fitted, data) -> data′: Transform the windowdatawithfitted, the object thatfit_preprocessingreturned.
keeps_observations
keeps_observations(fitted) -> Bool: Optional. Returntruewhen the transform keeps every observation of the window, in order. The fallback returnsfalse.
A subtype that does not implement fit_preprocessing or apply_preprocessing gets the fallback of that method, which throws an ArgumentError that names the subtype.
Related
PortfolioOptimisers.AbstractPricesPreprocessingEstimator — Type
abstract type AbstractPricesPreprocessingEstimator <: AbstractPreprocessingEstimatorAbstract supertype for preprocessing estimators that consume and produce price-level data.
A concrete subtype transforms a PricesResult into another PricesResult. A Pipeline fits the estimator on the prices in its prices slot, and writes the transformed prices back to the same slot.
Interfaces
To implement a new price-level estimator, subtype AbstractPricesPreprocessingEstimator and implement the following methods:
fit_preprocessing
fit_preprocessing(est::MyPricesPreprocessing, pr::PricesResult) -> MyPricesPreprocessingResult: Fit the state of the transform on the training pricespr. A stateless estimator returnsest.
apply_preprocessing
apply_preprocessing(res::MyPricesPreprocessingResult, pr::PricesResult) -> PricesResult: Transform the price windowprwith the fitted state ofres.
Related
PortfolioOptimisers.AbstractReturnsPreprocessingEstimator — Type
abstract type AbstractReturnsPreprocessingEstimator <: AbstractPreprocessingEstimatorAbstract supertype for preprocessing estimators that consume and produce returns-level data.
A concrete subtype transforms a ReturnsResult into another ReturnsResult. A Pipeline fits the estimator on the returns in its returns slot, and writes the transformed returns back to the same slot.
Interfaces
To implement a new returns-level estimator, subtype AbstractReturnsPreprocessingEstimator and implement the following methods:
fit_preprocessing
fit_preprocessing(est::MyReturnsPreprocessing, rd::AbstractReturnsResult) -> MyReturnsPreprocessingResult: Fit the state of the transform on the training returnsrd. A stateless estimator returnsest.
apply_preprocessing
apply_preprocessing(res::MyReturnsPreprocessingResult, rd::AbstractReturnsResult) -> AbstractReturnsResult: Transform the returns windowrdwith the fitted state ofres.
keeps_observations
keeps_observations(fitted) -> Bool: Optional. Returntruewhen the transform keeps every observation of the window, in order. The fallback returnsfalse.
Related
PortfolioOptimisers.AbstractPreprocessingResult — Type
abstract type AbstractPreprocessingResult <: AbstractResultAbstract supertype for all preprocessing result types.
fit_preprocessing returns a preprocessing result from a training window. The result holds the fitted state that apply_preprocessing needs to do the same transform on another window, such as the per-asset values of a gap fill, a row threshold or the selected asset universe. A stateless estimator returns itself and makes no result. A TrainTestSplitResult is not a preprocessing result, because it holds the two windows of one split and applies to no other window.
A concrete result subtypes the data-level subtype that matches its estimator, AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult. A Pipeline reads that subtype to find the slot that the result transforms.
Interfaces
To implement a new preprocessing result, subtype AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult, and implement the following method:
apply_preprocessing(res::MyPreprocessingResult, data) -> data′: Transform the windowdatawith the fitted state ofres.keeps_observations(res::MyPreprocessingResult) -> Bool: Optional. Returntruewhen the transform keeps every observation of the window, in order. The fallback returnsfalse.
Related
PortfolioOptimisers.AbstractPricesPreprocessingResult — Type
abstract type AbstractPricesPreprocessingResult <: AbstractPreprocessingResultAbstract supertype for preprocessing results that apply to price-level data (PricesResult).
A Pipeline applies a result of this type to the prices in its prices slot, and passes a returns window through with no change.
Interfaces
To implement a new price-level result, subtype AbstractPricesPreprocessingResult and implement the following method:
apply_preprocessing(res::MyPricesPreprocessingResult, pr::PricesResult) -> PricesResult: Transform the price windowprwith the fitted state ofres.
Related
PortfolioOptimisers.AbstractReturnsPreprocessingResult — Type
abstract type AbstractReturnsPreprocessingResult <: AbstractPreprocessingResultAbstract supertype for preprocessing results that apply to returns-level data (ReturnsResult).
A Pipeline applies a result of this type to the returns in its returns slot, and passes a price window through with no change.
Interfaces
To implement a new returns-level result, subtype AbstractReturnsPreprocessingResult and implement the following method:
apply_preprocessing(res::MyReturnsPreprocessingResult, rd::AbstractReturnsResult) -> AbstractReturnsResult: Transform the returns windowrdwith the fitted state ofres.keeps_observations(res::MyReturnsPreprocessingResult) -> Bool: Optional. Returntruewhen the transform keeps every observation of the window, in order. The fallback returnsfalse.
Related