Preprocessing partial fit

Incremental updates of the data steps

A Pipeline accepts partial_fit!, and so do the data steps before its prior step, or before its optimiser step when it has no prior. A step that transforms each row on its own updates with each block. It takes a block of observations and returns the transformed block, through PortfolioOptimisers.partial_fit_transform. To do so, PricesToReturns stores the last row of prices it saw, PriceGapFill with a CarriedPrice stores the carried prices, and MissingDataFilter stores its counts of missing values. Call fit_preprocessing on such a step with no data to get its fitted result. A step whose configuration needs a whole window returns false from PortfolioOptimisers.supports_partial_fit, and the pipeline throws an error that names the step before the first update.

PortfolioOptimisers.fit_preprocessing — Method
fit_preprocessing(est::PriceGapFill)

Reads a stepped PriceGapFill out as the PriceGapFillResult that the batch fit over the same rows gives.

The result holds each asset of the folded blocks, with its last observed price, and the last folded timestamp as the end of the training window. An asset whose carried price an absence retired has a missing seed. An asset that no block priced also has a missing seed.

Algorithm

  1. Read the state with partial_fit_cache.
  2. Read into v the carried price of each asset whose flag in held is set, and missing for the others. An asset that no block priced has no carried price, so its entry is missing too. The element type of v drops Missing when no seed is missing.
  3. Build the PriceGapFillResult from a copy of the names of the state, v, the end te of the state, and the fill and strict of the step.

Validation

Returns

  • res::PriceGapFillResult: The fitted gap fill of the folded observations.

Related

source
PortfolioOptimisers.fit_preprocessing — Method
fit_preprocessing(mdf::MissingDataFilter)

Reads a stepped MissingDataFilter out as the MissingDataFilterResult that the batch fit over the same rows gives.

Algorithm

  1. Read the state with partial_fit_cache.
  2. Keep the assets whose share of missing observations miss / n does not exceed col_thr, giving keep, and check that one asset at least survives. share_at_most computes the share in the type of col_thr, as the batch fit does.
  3. Build the MissingDataFilterResult from the surviving asset names and row_thr.

Validation

  • The step keeps a state. partial_fit_cache throws an ArgumentError otherwise.
  • One asset at least survives the column filter. An IsEmptyError is thrown otherwise.

Returns

  • res::MissingDataFilterResult: The fitted filter of the folded observations.

Related

source