Preprocessing partial fit: private API

Types

PortfolioOptimisers.PricesToReturnsState — Type
struct PricesToReturnsState{__T_tail, __T_anchor} <: AbstractPartialFitState

Keeps the prices that an online conversion to returns needs from the blocks before the current one.

PricesToReturns keeps this state in cache. A return reads two consecutive prices. So the conversion of a new block needs the last price row of the block before it, and nothing more of that block. tail is that row. It is a one-row PricesResult, so that the span, the panel and the implied volatilities of the row stay with its prices.

A CatchUpGapReturn reads further back. The observation that ends a gap books the move against the last observed price, and that price can be any number of blocks before. So anchor keeps that price for each series column. Under the default rule, which reads no earlier price, anchor is nothing.

Fields

  • anchor: The last observed price of each series column, in the order of series_values, or NaN where no price was observed. nothing under the default gap rule.

Constructors

PricesToReturnsState(; tail::PricesResult, anchor::Option{<:AbstractVector} = nothing)

Validation

  • tail holds one observation. A DimensionMismatch is thrown otherwise.

Related

source
PortfolioOptimisers.PriceGapFillState — Type
struct PriceGapFillState{__T_nx, __T_v, __T_held, __T_te} <: AbstractPartialFitState

Keeps the carried price of each asset and the end of the last block for an online gap fill with a carried price.

PriceGapFill with a CarriedPrice keeps this state in cache. The carried price is the price that the gaps of the next block take, and the seed that the fitted result replays. The fitted result records the end of the last block as the end of its training window.

An absence from the listing after the last observed price retires the carried price, because the next block cannot see that absence. The state keeps the price and clears its flag in held. fit_preprocessing with no data then gives the asset a missing seed, as the batch fit over the same rows does.

Fields

  • nx: The names of the asset columns, which the first block pins.
  • v: The last observed price of each asset, or missing where no price was observed.
  • held: Whether the listing of each asset holds from its last observed price to the end of the last block, so that the price seeds the next block. For an asset with no observed price, whether the folded blocks hold no absence.
  • te: The last timestamp of the last block folded. The carried prices precede each observation after it, and the fitted result records it as the end of its training window.

Constructors

PriceGapFillState(; nx::AbstractVector{Symbol}, v::AbstractVector,                  held::AbstractVector{Bool}, te)

Validation

  • length(nx) == length(v) == length(held). A DimensionMismatch is thrown otherwise.

Related

source
PortfolioOptimisers.MissingDataFilterState — Type
struct MissingDataFilterState{__T_nx, __T_miss, __T_n} <: AbstractPartialFitState

Keeps the number of observations and the missing count of each asset for an online missing-data filter.

MissingDataFilter keeps this state in cache, and fit_preprocessing with no data computes the column filter from the counts.

Fields

  • nx: The names of the asset columns, which the first block pins.
  • miss: The number of missing observations of each asset, over each block folded.
  • n: The number of observations folded.

Constructors

MissingDataFilterState(; nx::AbstractVector{Symbol}, miss::AbstractVector{<:Integer}, n::Integer)

Validation

  • length(nx) == length(miss). A DimensionMismatch is thrown otherwise.
  • n >= 0. A DomainError is thrown otherwise.

Related

source

Functions

PortfolioOptimisers.vcat_observations — Function
vcat_observations(a::PricesResult, b::PricesResult) -> PricesResult
vcat_observations(a::ReturnsResult, b::ReturnsResult) -> ReturnsResult

Concatenates the observations of two PricesResult values or two ReturnsResult values of one universe, a first.

Each online form of a data step reads this concatenation. PricesToReturns puts the last price row it kept in front of a new block, so that the conversion reads consecutive prices. The input-data buffer of Online(pipe) appends each block that it receives. a pins the universe: the names of the assets, the factors and the benchmark, the implied-volatility adjustment, and a static AssetPanel. The function refuses a block b with a different universe, and the message names the field, as the step of ReturnsBufferState does. It also refuses a column that one of a and b holds and the other does not, because a factor series that some blocks hold and others do not is not one series.

The function concatenates the AssetPanels with Base.vcat(a::AssetPanel, bs::AssetPanel...), which joins the Panel Fields and the masks of a time-varying panel. When the two blocks hold equal static panels, the result keeps the panel of a.

Algorithm

The method that Julia selects is the algorithm.

  1. PricesResult: check that the implied-volatility adjustment ivpa of b equals the one of a. Concatenate the prices X, the factors F, the benchmark B, the Exogenous Series E and the implied volatilities iv with vcat_optional, which also checks the column names. Concatenate the panel pnl and the Listing Span span with vcat_optional.
  2. ReturnsResult: check that the names nx, nf, nb and ne and the adjustment ivpa of b equal the ones of a. Concatenate X, F, B, E, the timestamps ts and iv with vcat_optional, and the panel pnl with vcat_returns_panel, which lets one block hold no panel.

Arguments

  • a: The earlier observations.
  • b: The later observations.

Validation

  • The names and the implied-volatility adjustment of b equal the ones of a. An ArgumentError is thrown otherwise.
  • Each optional column is held by both a and b or by neither. An ArgumentError is thrown otherwise. The panel of a ReturnsResult is the exception, see vcat_returns_panel.
  • The panels of a and b concatenate. Base.vcat(a::AssetPanel, bs::AssetPanel...) throws otherwise.

Returns

  • c: A value of the type of a, with the observations of a followed by the observations of b.

Related

source
PortfolioOptimisers.vcat_returns_panel — Function
vcat_returns_panel(a::ReturnsResult, b::ReturnsResult) -> Option{<:AssetPanel}

Concatenates the Asset Panels of two blocks of returns data, where one block can hold no panel.

The ingestion layer emits no panel for a block with no gap and no Panel Field, and the online conversion decides it again for each block. So a gapped stream gives blocks with a mask-only panel and blocks with none. A block with no panel states the universe of the finiteness of its returns, so it stands in as the panel finite_universe_panel makes, which gives the same Coverage Universe. The concatenation is then the panel that the conversion of the whole history gives.

Algorithm

  1. Neither block holds a panel: return nothing.
  2. Otherwise: replace a missing panel with finite_universe_panel of the returns of its block, and concatenate the two panels with Base.vcat(a::AssetPanel, bs::AssetPanel...).

Arguments

  • a: The earlier block.
  • b: The later block.

Validation

Returns

  • pnl::Option{<:AssetPanel}: The panel of the observations of a followed by the observations of b, or nothing.

Related

source
PortfolioOptimisers.finite_universe_panel — Function
finite_universe_panel(X::MatNum) -> AssetPanel

Make the mask-only Asset Panel that states the universe of the finiteness of a block of returns.

Every asset is active, and an asset is estimable where its return is finite, which is the convention emsk = amsk .& isfinite.(X) of the ingestion layer. The Coverage Universe of this panel equals the Coverage Universe of no panel, so the panel stands in for a block that holds none.

Algorithm

  1. Write the active mask as an all-true mask of the shape of X, and the estimation mask as isfinite.(X).
  2. Compress the pair with compress_all_true, and make the AssetPanel with pf = nothing.

Arguments

  • X: The asset returns of the block, observations × assets.

Returns

  • pnl::AssetPanel: The mask-only panel.

Related

source
PortfolioOptimisers.returns_with_universe — Function
returns_with_universe(rd::ReturnsResult) -> ReturnsResult

Give a ReturnsResult that holds an Asset Panel, for a reader that needs the universe masks and no Panel Field.

The ingestion layer emits no panel for a window with no gap and no Panel Field, because its all-true masks state nothing that the finiteness of the returns does not state. A reader that takes its shape or its activity from the masks, such as a rolling Descriptor, a constant Factor Exposure or the fit of a CrossSectionalFactorPrior on returns-derived exposures, then reads the mask-only panel that finite_universe_panel makes. It is the panel that the layer would have emitted, with the same Coverage Universe.

Algorithm

  1. rd holds a panel, or holds no asset returns: return rd.
  2. Otherwise: return rd with its pnl set to finite_universe_panel of its asset returns.

Arguments

  • rd: The returns result to use.

Returns

  • rd′::ReturnsResult: rd itself, or rd with a mask-only panel.

Related

source
PortfolioOptimisers.assert_pinned_value — Function
assert_pinned_value(pinned, given, name::Symbol)

Refuses a block whose pinned value differs from the value that the first block pinned, and names the field.

Arguments

  • pinned: The value that the first block pinned.
  • given: The value that the new block carries.
  • name: The name of the field, which the message quotes.

Validation

  • isequal(pinned, given). An ArgumentError is thrown otherwise.

Returns

  • nothing.

Related

source
PortfolioOptimisers.vcat_optional — Function
vcat_optional(a::Nothing, b::Nothing, name::Symbol) -> Nothing
vcat_optional(a::TimeSeries.TimeArray, b::TimeSeries.TimeArray, name::Symbol) -> TimeSeries.TimeArray
vcat_optional(a, b, name::Symbol)

Concatenates one optional column of two blocks of observations along the observation axis.

Algorithm

The method that Julia selects is the algorithm.

  1. a and b are nothing: return nothing.
  2. a and b are TimeArrays: check that the column names of b equal the ones of a, in the same order, and concatenate the two.
  3. Otherwise check that neither is nothing, and concatenate the two with vcat.

Arguments

  • a: The column of the earlier block, or nothing.
  • b: The column of the later block, or nothing.
  • name: The name of the field, which the message of a refusal quotes.

Validation

  • Both blocks hold the column, or neither does. An ArgumentError is thrown otherwise.
  • Two TimeArrays have the same column names in the same order. An ArgumentError is thrown otherwise.

Returns

  • c: The rows of a followed by the rows of b, or nothing.

Related

source
PortfolioOptimisers.partial_fit_transform — Function
partial_fit_transform(est, data) -> (est′, data′)

Folds a block of observations into a data step, and returns the block that the transform of the step gives.

This verb is the online form of a preprocessing step that changes the rows it receives. A Pipeline gives each step the block that the step before it returned, and takes back the block that the step returns. So the fold and the transform of a step are one call. The transform of the new rows reads the state that the earlier rows left, and the fold moves the state past the new rows. The two halves cannot be separated, because the first return of a new block reads the last price row that the step kept. So the verb returns both.

For PricesToReturns, PriceGapFill and MissingDataFilter, partial_fit! returns the first element of this verb. A step makes its estimate from the state with fit_preprocessing and no data. A caller's own preprocessing estimator joins the online route of a Pipeline with three methods: this verb, a fit_preprocessing method that takes no data, and a supports_partial_fit method that returns true. It needs no partial_fit! method.

Arguments

  • est: The data step. A step of the library keeps its state in cache, which is nothing before the first block.
  • data: The block of observations, as the batch verb of the step takes it.

Validation

  • A method of this verb exists for the type of est. The fallback throws an ArgumentError that names the type.

Returns

  • (est′, data′): The step with the block folded, and the block as the step returns it.

Related

source
partial_fit_transform(ptr::PricesToReturns, pr::PricesResult) -> (ptr′, rd)

Converts a block of prices to returns as the whole history does, and keeps what the next block needs.

Each return of the rows that the verb returns equals the return that the batch conversion of the whole history writes at that row. A return reads two consecutive prices, a Gap Return reads the last observed price, and the state keeps both.

Algorithm

  1. Check that the gap rule has an online form, and that the block holds one observation at least.
  2. On the first block, check that the block holds two observations when padding is false. Convert the block alone with prices_to_returns, giving rd. The block is the whole history.
  3. On a later block, put the kept row tail in front of the block with vcat_observations, giving window. Convert window under the ret_method and the padding of the step and the default gap rule, giving rw. Keep the last n rows of rw as rd, where n is the number of observations of the block. The kept row gives the return of the first new observation, and the step drops the row that padding adds at the front of window.
  4. Under a CatchUpGapReturn, on a later block, write the resolved cells of the new rows into R with block_gap_return!, and rebuild rd from R with returns_with_series.
  5. Keep the last price row of the block as the new tail. Under a CatchUpGapReturn, move the anchor past the block with advance_anchor, from an anchor of NaN on the first block.

Arguments

  • ptr: The conversion, with its state in cache, or nothing before the first block.
  • pr: The block of prices.

Validation

  • The gap rule is nothing or a CatchUpGapReturn. A caller's own rule can read the whole price column, so its online form can differ from the batch conversion. An ArgumentError is thrown otherwise.
  • The block holds one observation at least. An IsEmptyError is thrown otherwise.
  • Without padding, the first block holds two observations at least, because a conversion of one price row gives no return. An IsEmptyError is thrown otherwise.
  • A later block has the universe of the first block. vcat_observations throws an ArgumentError otherwise.

Returns

  • (ptr′, rd): The conversion with the block folded, and the returns of the block as a ReturnsResult.

Related

source
partial_fit_transform(est::PriceGapFill, pr::PricesResult) -> (est′, pr′)

Fills the gaps of a block of prices as the fit over the whole history does, and carries the last observed prices forward.

A column that no earlier block priced has no carried price. A gap that opens such a column stays a gap until the first observed price of the column, because the batch replay of PriceGapFill leaves it so on the training window. A price later in the block is not a seed for a gap before it. A carried price that an absence from the listing follows is not a seed either, because the next return would be a move across the absence.

Algorithm

  1. Check that the convention is a CarriedPrice.
  2. Read the asset names names and a copy vals of the prices, with each gap as NaN. When the step keeps a state, check that names equals the names that the state pinned, and read the carried prices v and their flags held from the state. Otherwise v is missing and held is true for each asset.
  3. Resolve the Listing Span span that bounds the fill with gap_fill_span, as the batch replay does.
  4. Find t0, the first row of the block after the end te of the state. With no state, t0 is one row past the end of the block.
  5. For each column j, find the last observed row t.
  6. Fill the gaps of column j of vals with gap_fill_column!. A column with a carried price and a set flag takes the price as the seed from row t0. Any other column gets a missing seed, so its gaps fill from the earlier prices of the block only.
  7. When t exists, the new carried price vnew[j] is vals[t, j], and the new flag tells with gap_fill_open whether the Listing Span of pr holds from t to the end of the block. Otherwise the new flag is the old flag, cleared when the span breaks inside the block.
  8. Build the filled PricesResult from vals, and the new state from names, vnew, the new flags and the last timestamp of the block.

Arguments

  • est: The gap fill, with its state in cache, or nothing before the first block.
  • pr: The block of prices.

Validation

  • The convention is a CarriedPrice. A fill by a statistic of the window changes each earlier gap when the window grows, so it has no online form. supports_partial_fit answers false for it, and a Pipeline refuses it at warm-up. An ArgumentError is thrown otherwise.
  • A later block has the asset names of the first block. An ArgumentError is thrown otherwise.
  • Under strict, a block with a gap comes from a PricesResult with a Listing Span. gap_fill_span throws an ArgumentError otherwise.

Returns

  • (est′, pr′): The gap fill with the block folded, and the block with its gaps filled.

Related

source
partial_fit_transform(mdf::MissingDataFilter, pr::PricesResult) -> (mdf′, pr)

Counts the gaps of a block of prices, and returns the block unchanged.

The column filter changes only which assets survive, and a longer window can change that set. So the verb leaves the column filter to the call with no data. There, fit_preprocessing with no data gives the MissingDataFilterResult of the whole history, and a Pipeline applies it as a view. At row_thr = 1 the row filter drops no row, so every row passes.

Algorithm

  1. Check that row_thr is one.
  2. Count the missing observations of each asset in the block, the cells of asset_gap_cells, giving miss.
  3. With no state, make a state from the asset names, miss and the number of observations of the block. With a state, check that the asset names equal the names that it pinned, and add miss and the number of observations to its counts.
  4. Return the step with the new state, and pr unchanged.

Arguments

  • mdf: The filter, with its state in cache, or nothing before the first block.
  • pr: The block of prices.

Validation

  • row_thr == 1. A lower threshold drops a row that an earlier step kept when a column leaves the universe, and the returns on each side of that row change. supports_partial_fit answers false for it, and a Pipeline refuses it at warm-up. An ArgumentError is thrown otherwise.
  • A later block has the asset names of the first block. An ArgumentError is thrown otherwise.

Returns

  • (mdf′, pr): The filter with the block folded, and the block itself.

Related

source
PortfolioOptimisers.series_values — Function
series_values(pr::PricesResult) -> Any

Puts the series columns of the price data pr side by side: the assets, then the factors, then the benchmark, then the Exogenous Series.

series_values_returns lays the columns of a ReturnsResult out in the same order, so a column index of one matrix names the same series in the other.

Algorithm

  1. Read the asset prices X with each gap written as NaN, giving the first block of cols.
  2. When pr holds factor prices F, append them to cols in the same way.
  3. When pr holds benchmark prices B, append them to cols in the same way.
  4. When pr holds an Exogenous Series E, append it to cols in the same way.
  5. Concatenate cols horizontally.

Returns

  • P::AbstractMatrix: The prices, observations × series, with each gap as NaN.

Related

source
PortfolioOptimisers.advance_anchor — Function
advance_anchor(
    anchor::AbstractVector,
    P::AbstractMatrix
) -> Any

Moves the anchor of each series column of a PricesToReturnsState past a block.

The new anchor of a column is its last observed price in the block. A column with no observed price in the block keeps its anchor.

Algorithm

  1. Copy anchor into out.
  2. For each column j of P, find the last row t whose price is observed. When there is one, write P[t, j] into out[j].

Arguments

  • anchor: The last observed price of each series column before the block, or NaN.
  • P: The prices of the block, observations × series.

Returns

  • out::AbstractVector: A new vector. anchor does not change.

Related

source
PortfolioOptimisers.block_gap_return! — Function
block_gap_return!(
    alg::AbstractGapReturnAlgorithm,
    R::AbstractMatrix,
    P::AbstractMatrix,
    anchor::AbstractVector,
    last::AbstractMatrix,
    ret_method::Symbol
) -> AbstractMatrix

Writes the cells that a Gap Return algorithm resolves in the returns of a new block, and reads the anchor of the state for the prices before the block.

The batch conversion gives gap_return a whole column. This function gives it the column of the block with two rows in front: the anchor and the last price row kept. So the algorithm meets the observed price that a gap books its move against, and the price just before the first row of the block, as it does in the whole history. The function copies back only the cells of the new rows, and only the cells that gap_return_writable admits over the extended column. The batch conversion keeps the same rule. An anchor is an earlier observed price, and a column with no observed price admits no cell before its first observation.

Algorithm

  1. Make p, a column of prices two rows longer than P, and r, a column of returns one row longer than R.
  2. For each series column j, write anchor[j] and last[1, j] into the first two rows of p, and the prices of the block after them. Write NaN into the first row of r, and the returns of the block after it.
  3. Find the writable cells w of the extended column with gap_return_writable. Go to the next column when no cell is writable.
  4. Apply alg to the extended column with gap_return, giving out.
  5. Copy out into R at each writable cell of the new rows.

Arguments

  • alg: The Gap Return algorithm.
  • R: The returns of the new block, observations × series, changed in place.
  • P: The prices of the new block, observations × series.
  • anchor: The last observed price of each series column before the block.
  • last: The last price row before the block, 1 × series.
  • ret_method: :simple or :log.

Validation

  • alg returns one value for each return cell of the extended column. A DimensionMismatch is thrown otherwise.

Returns

  • R: The same matrix, with the writable cells of the new rows written.

Related

source
PortfolioOptimisers.series_values_returns — Function
series_values_returns(rd::ReturnsResult) -> Any

Puts the series columns of the returns data rd side by side, in the order of series_values: the assets, then the factors, then the benchmark, then the Exogenous Series.

Algorithm

  1. Copy the asset returns X into the first block of cols.
  2. When rd holds factor returns F, append a copy to cols.
  3. When rd holds benchmark returns B, append a copy to cols, as one column when B is a vector.
  4. Concatenate cols horizontally.

Returns

  • R::Matrix: The returns, observations × series. The matrix is a copy, so a change to it does not reach rd.

Related

source
PortfolioOptimisers.returns_with_series — Function
returns_with_series(
    rd::ReturnsResult,
    R::AbstractMatrix,
    _::AbstractGapReturnAlgorithm
) -> Union{ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, Nothing} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K}, ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, AssetPanel{__T_pf, __T_amsk, __T_emsk}} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, __T_pf, __T_amsk, __T_emsk}}

Rebuilds a ReturnsResult from the series matrix that series_values_returns lays out.

A Gap Return writes a finite return where the default conversion left none. So the function derives the estimation mask again, as the conversion does.

Algorithm

  1. Split R into the asset returns X, the factor returns F, the benchmark returns B and the Exogenous Series E, in the layout of series_values_returns. B is one column when the benchmark of rd is a vector.
  2. When rd holds a time-varying panel, keep its active mask amsk, and derive the estimation mask emsk as the cells of amsk where X is finite. When emsk is true everywhere, compress_all_true replaces the two masks with one all-true mask.
  3. Build the ReturnsResult from X, F, B, E, the panel, and the other fields of rd.

Returns

  • rd′::ReturnsResult: The returns data of R.

Related

source