Preprocessing partial fit: private API
Types
PortfolioOptimisers.PricesToReturnsState — Type
struct PricesToReturnsState{__T_tail, __T_anchor} <: AbstractPartialFitStateKeeps the prices that an online conversion to returns needs from the blocks before the current one.
PricesToReturns keeps this state in cache. A return reads two consecutive prices. So the conversion of a new block needs the last price row of the block before it, and nothing more of that block. tail is that row. It is a one-row PricesResult, so that the span, the panel and the implied volatilities of the row stay with its prices.
A CatchUpGapReturn reads further back. The observation that ends a gap books the move against the last observed price, and that price can be any number of blocks before. So anchor keeps that price for each series column. Under the default rule, which reads no earlier price, anchor is nothing.
Fields
tail: The last price row folded, as a one-rowPricesResult.
anchor: The last observed price of each series column, in the order ofseries_values, orNaNwhere no price was observed.nothingunder the default gap rule.
Constructors
PricesToReturnsState(; tail::PricesResult, anchor::Option{<:AbstractVector} = nothing)Validation
tailholds one observation. ADimensionMismatchis thrown otherwise.
Related
PortfolioOptimisers.PriceGapFillState — Type
struct PriceGapFillState{__T_nx, __T_v, __T_held, __T_te} <: AbstractPartialFitStateKeeps the carried price of each asset and the end of the last block for an online gap fill with a carried price.
PriceGapFill with a CarriedPrice keeps this state in cache. The carried price is the price that the gaps of the next block take, and the seed that the fitted result replays. The fitted result records the end of the last block as the end of its training window.
An absence from the listing after the last observed price retires the carried price, because the next block cannot see that absence. The state keeps the price and clears its flag in held. fit_preprocessing with no data then gives the asset a missing seed, as the batch fit over the same rows does.
Fields
nx: The names of the asset columns, which the first block pins.
v: The last observed price of each asset, ormissingwhere no price was observed.
held: Whether the listing of each asset holds from its last observed price to the end of the last block, so that the price seeds the next block. For an asset with no observed price, whether the folded blocks hold no absence.
te: The last timestamp of the last block folded. The carried prices precede each observation after it, and the fitted result records it as the end of its training window.
Constructors
PriceGapFillState(; nx::AbstractVector{Symbol}, v::AbstractVector, held::AbstractVector{Bool}, te)Validation
length(nx) == length(v) == length(held). ADimensionMismatchis thrown otherwise.
Related
PortfolioOptimisers.MissingDataFilterState — Type
struct MissingDataFilterState{__T_nx, __T_miss, __T_n} <: AbstractPartialFitStateKeeps the number of observations and the missing count of each asset for an online missing-data filter.
MissingDataFilter keeps this state in cache, and fit_preprocessing with no data computes the column filter from the counts.
Fields
nx: The names of the asset columns, which the first block pins.
miss: The number of missing observations of each asset, over each block folded.
n: The number of observations folded.
Constructors
MissingDataFilterState(; nx::AbstractVector{Symbol}, miss::AbstractVector{<:Integer}, n::Integer)Validation
length(nx) == length(miss). ADimensionMismatchis thrown otherwise.n >= 0. ADomainErroris thrown otherwise.
Related
Functions
PortfolioOptimisers.vcat_observations — Function
vcat_observations(a::PricesResult, b::PricesResult) -> PricesResult
vcat_observations(a::ReturnsResult, b::ReturnsResult) -> ReturnsResultConcatenates the observations of two PricesResult values or two ReturnsResult values of one universe, a first.
Each online form of a data step reads this concatenation. PricesToReturns puts the last price row it kept in front of a new block, so that the conversion reads consecutive prices. The input-data buffer of Online(pipe) appends each block that it receives. a pins the universe: the names of the assets, the factors and the benchmark, the implied-volatility adjustment, and a static AssetPanel. The function refuses a block b with a different universe, and the message names the field, as the step of ReturnsBufferState does. It also refuses a column that one of a and b holds and the other does not, because a factor series that some blocks hold and others do not is not one series.
The function concatenates the AssetPanels with Base.vcat(a::AssetPanel, bs::AssetPanel...), which joins the Panel Fields and the masks of a time-varying panel. When the two blocks hold equal static panels, the result keeps the panel of a.
Algorithm
The method that Julia selects is the algorithm.
PricesResult: check that the implied-volatility adjustmentivpaofbequals the one ofa. Concatenate the pricesX, the factorsF, the benchmarkB, the Exogenous SeriesEand the implied volatilitiesivwithvcat_optional, which also checks the column names. Concatenate the panelpnland the Listing Spanspanwithvcat_optional.ReturnsResult: check that the namesnx,nf,nbandneand the adjustmentivpaofbequal the ones ofa. ConcatenateX,F,B,E, the timestampstsandivwithvcat_optional, and the panelpnlwithvcat_returns_panel, which lets one block hold no panel.
Arguments
a: The earlier observations.b: The later observations.
Validation
- The names and the implied-volatility adjustment of
bequal the ones ofa. AnArgumentErroris thrown otherwise. - Each optional column is held by both
aandbor by neither. AnArgumentErroris thrown otherwise. The panel of aReturnsResultis the exception, seevcat_returns_panel. - The panels of
aandbconcatenate.Base.vcat(a::AssetPanel, bs::AssetPanel...)throws otherwise.
Returns
c: A value of the type ofa, with the observations ofafollowed by the observations ofb.
Related
PortfolioOptimisers.vcat_returns_panel — Function
vcat_returns_panel(a::ReturnsResult, b::ReturnsResult) -> Option{<:AssetPanel}Concatenates the Asset Panels of two blocks of returns data, where one block can hold no panel.
The ingestion layer emits no panel for a block with no gap and no Panel Field, and the online conversion decides it again for each block. So a gapped stream gives blocks with a mask-only panel and blocks with none. A block with no panel states the universe of the finiteness of its returns, so it stands in as the panel finite_universe_panel makes, which gives the same Coverage Universe. The concatenation is then the panel that the conversion of the whole history gives.
Algorithm
- Neither block holds a panel: return
nothing. - Otherwise: replace a missing panel with
finite_universe_panelof the returns of its block, and concatenate the two panels withBase.vcat(a::AssetPanel, bs::AssetPanel...).
Arguments
a: The earlier block.b: The later block.
Validation
- The panels concatenate.
Base.vcat(a::AssetPanel, bs::AssetPanel...)throws otherwise. So a block with no panel and a block whose panel holds Panel Fields, or is static, are refused.
Returns
pnl::Option{<:AssetPanel}: The panel of the observations ofafollowed by the observations ofb, ornothing.
Related
PortfolioOptimisers.finite_universe_panel — Function
finite_universe_panel(X::MatNum) -> AssetPanelMake the mask-only Asset Panel that states the universe of the finiteness of a block of returns.
Every asset is active, and an asset is estimable where its return is finite, which is the convention emsk = amsk .& isfinite.(X) of the ingestion layer. The Coverage Universe of this panel equals the Coverage Universe of no panel, so the panel stands in for a block that holds none.
Algorithm
- Write the active mask as an all-true mask of the shape of
X, and the estimation mask asisfinite.(X). - Compress the pair with
compress_all_true, and make theAssetPanelwithpf = nothing.
Arguments
X: The asset returns of the block,observations × assets.
Returns
pnl::AssetPanel: The mask-only panel.
Related
PortfolioOptimisers.returns_with_universe — Function
returns_with_universe(rd::ReturnsResult) -> ReturnsResultGive a ReturnsResult that holds an Asset Panel, for a reader that needs the universe masks and no Panel Field.
The ingestion layer emits no panel for a window with no gap and no Panel Field, because its all-true masks state nothing that the finiteness of the returns does not state. A reader that takes its shape or its activity from the masks, such as a rolling Descriptor, a constant Factor Exposure or the fit of a CrossSectionalFactorPrior on returns-derived exposures, then reads the mask-only panel that finite_universe_panel makes. It is the panel that the layer would have emitted, with the same Coverage Universe.
Algorithm
rdholds a panel, or holds no asset returns: returnrd.- Otherwise: return
rdwith itspnlset tofinite_universe_panelof its asset returns.
Arguments
rd: The returns result to use.
Returns
rd′::ReturnsResult:rditself, orrdwith a mask-only panel.
Related
PortfolioOptimisers.assert_pinned_value — Function
assert_pinned_value(pinned, given, name::Symbol)
Refuses a block whose pinned value differs from the value that the first block pinned, and names the field.
Arguments
pinned: The value that the first block pinned.given: The value that the new block carries.name: The name of the field, which the message quotes.
Validation
isequal(pinned, given). AnArgumentErroris thrown otherwise.
Returns
nothing.
Related
PortfolioOptimisers.vcat_optional — Function
vcat_optional(a::Nothing, b::Nothing, name::Symbol) -> Nothing
vcat_optional(a::TimeSeries.TimeArray, b::TimeSeries.TimeArray, name::Symbol) -> TimeSeries.TimeArray
vcat_optional(a, b, name::Symbol)Concatenates one optional column of two blocks of observations along the observation axis.
Algorithm
The method that Julia selects is the algorithm.
aandbarenothing: returnnothing.aandbareTimeArrays: check that the column names ofbequal the ones ofa, in the same order, and concatenate the two.- Otherwise check that neither is
nothing, and concatenate the two withvcat.
Arguments
a: The column of the earlier block, ornothing.b: The column of the later block, ornothing.name: The name of the field, which the message of a refusal quotes.
Validation
- Both blocks hold the column, or neither does. An
ArgumentErroris thrown otherwise. - Two
TimeArrays have the same column names in the same order. AnArgumentErroris thrown otherwise.
Returns
c: The rows ofafollowed by the rows ofb, ornothing.
Related
PortfolioOptimisers.data_row_count — Function
data_row_count(pr::PricesResult) -> Int64
Counts the observations of a PricesResult or a ReturnsResult.
Returns
n::Int: The number of rows ofX.
Related
PortfolioOptimisers.partial_fit_transform — Function
partial_fit_transform(est, data) -> (est′, data′)Folds a block of observations into a data step, and returns the block that the transform of the step gives.
This verb is the online form of a preprocessing step that changes the rows it receives. A Pipeline gives each step the block that the step before it returned, and takes back the block that the step returns. So the fold and the transform of a step are one call. The transform of the new rows reads the state that the earlier rows left, and the fold moves the state past the new rows. The two halves cannot be separated, because the first return of a new block reads the last price row that the step kept. So the verb returns both.
For PricesToReturns, PriceGapFill and MissingDataFilter, partial_fit! returns the first element of this verb. A step makes its estimate from the state with fit_preprocessing and no data. A caller's own preprocessing estimator joins the online route of a Pipeline with three methods: this verb, a fit_preprocessing method that takes no data, and a supports_partial_fit method that returns true. It needs no partial_fit! method.
Arguments
est: The data step. A step of the library keeps its state incache, which isnothingbefore the first block.data: The block of observations, as the batch verb of the step takes it.
Validation
- A method of this verb exists for the type of
est. The fallback throws anArgumentErrorthat names the type.
Returns
(est′, data′): The step with the block folded, and the block as the step returns it.
Related
partial_fit_transform(ptr::PricesToReturns, pr::PricesResult) -> (ptr′, rd)Converts a block of prices to returns as the whole history does, and keeps what the next block needs.
Each return of the rows that the verb returns equals the return that the batch conversion of the whole history writes at that row. A return reads two consecutive prices, a Gap Return reads the last observed price, and the state keeps both.
Algorithm
- Check that the gap rule has an online form, and that the block holds one observation at least.
- On the first block, check that the block holds two observations when
paddingisfalse. Convert the block alone withprices_to_returns, givingrd. The block is the whole history. - On a later block, put the kept row
tailin front of the block withvcat_observations, givingwindow. Convertwindowunder theret_methodand thepaddingof the step and the default gap rule, givingrw. Keep the lastnrows ofrwasrd, wherenis the number of observations of the block. The kept row gives the return of the first new observation, and the step drops the row thatpaddingadds at the front ofwindow. - Under a
CatchUpGapReturn, on a later block, write the resolved cells of the new rows intoRwithblock_gap_return!, and rebuildrdfromRwithreturns_with_series. - Keep the last price row of the block as the new
tail. Under aCatchUpGapReturn, move the anchor past the block withadvance_anchor, from an anchor ofNaNon the first block.
Arguments
ptr: The conversion, with its state incache, ornothingbefore the first block.pr: The block of prices.
Validation
- The gap rule is
nothingor aCatchUpGapReturn. A caller's own rule can read the whole price column, so its online form can differ from the batch conversion. AnArgumentErroris thrown otherwise. - The block holds one observation at least. An
IsEmptyErroris thrown otherwise. - Without
padding, the first block holds two observations at least, because a conversion of one price row gives no return. AnIsEmptyErroris thrown otherwise. - A later block has the universe of the first block.
vcat_observationsthrows anArgumentErrorotherwise.
Returns
(ptr′, rd): The conversion with the block folded, and the returns of the block as aReturnsResult.
Related
partial_fit_transform(est::PriceGapFill, pr::PricesResult) -> (est′, pr′)Fills the gaps of a block of prices as the fit over the whole history does, and carries the last observed prices forward.
A column that no earlier block priced has no carried price. A gap that opens such a column stays a gap until the first observed price of the column, because the batch replay of PriceGapFill leaves it so on the training window. A price later in the block is not a seed for a gap before it. A carried price that an absence from the listing follows is not a seed either, because the next return would be a move across the absence.
Algorithm
- Check that the convention is a
CarriedPrice. - Read the asset names
namesand a copyvalsof the prices, with each gap asNaN. When the step keeps a state, check thatnamesequals the names that the state pinned, and read the carried pricesvand their flagsheldfrom the state. Otherwisevismissingandheldistruefor each asset. - Resolve the Listing Span
spanthat bounds the fill withgap_fill_span, as the batch replay does. - Find
t0, the first row of the block after the endteof the state. With no state,t0is one row past the end of the block. - For each column
j, find the last observed rowt. - Fill the gaps of column
jofvalswithgap_fill_column!. A column with a carried price and a set flag takes the price as the seed from rowt0. Any other column gets amissingseed, so its gaps fill from the earlier prices of the block only. - When
texists, the new carried pricevnew[j]isvals[t, j], and the new flag tells withgap_fill_openwhether the Listing Span ofprholds fromtto the end of the block. Otherwise the new flag is the old flag, cleared when the span breaks inside the block. - Build the filled
PricesResultfromvals, and the new state fromnames,vnew, the new flags and the last timestamp of the block.
Arguments
est: The gap fill, with its state incache, ornothingbefore the first block.pr: The block of prices.
Validation
- The convention is a
CarriedPrice. A fill by a statistic of the window changes each earlier gap when the window grows, so it has no online form.supports_partial_fitanswersfalsefor it, and aPipelinerefuses it at warm-up. AnArgumentErroris thrown otherwise. - A later block has the asset names of the first block. An
ArgumentErroris thrown otherwise. - Under
strict, a block with a gap comes from aPricesResultwith a Listing Span.gap_fill_spanthrows anArgumentErrorotherwise.
Returns
(est′, pr′): The gap fill with the block folded, and the block with its gaps filled.
Related
partial_fit_transform(mdf::MissingDataFilter, pr::PricesResult) -> (mdf′, pr)Counts the gaps of a block of prices, and returns the block unchanged.
The column filter changes only which assets survive, and a longer window can change that set. So the verb leaves the column filter to the call with no data. There, fit_preprocessing with no data gives the MissingDataFilterResult of the whole history, and a Pipeline applies it as a view. At row_thr = 1 the row filter drops no row, so every row passes.
Algorithm
- Check that
row_thris one. - Count the missing observations of each asset in the block, the cells of
asset_gap_cells, givingmiss. - With no state, make a state from the asset names,
missand the number of observations of the block. With a state, check that the asset names equal the names that it pinned, and addmissand the number of observations to its counts. - Return the step with the new state, and
prunchanged.
Arguments
mdf: The filter, with its state incache, ornothingbefore the first block.pr: The block of prices.
Validation
row_thr == 1. A lower threshold drops a row that an earlier step kept when a column leaves the universe, and the returns on each side of that row change.supports_partial_fitanswersfalsefor it, and aPipelinerefuses it at warm-up. AnArgumentErroris thrown otherwise.- A later block has the asset names of the first block. An
ArgumentErroris thrown otherwise.
Returns
(mdf′, pr): The filter with the block folded, and the block itself.
Related
PortfolioOptimisers.series_values — Function
series_values(pr::PricesResult) -> Any
Puts the series columns of the price data pr side by side: the assets, then the factors, then the benchmark, then the Exogenous Series.
series_values_returns lays the columns of a ReturnsResult out in the same order, so a column index of one matrix names the same series in the other.
Algorithm
- Read the asset prices
Xwith each gap written asNaN, giving the first block ofcols. - When
prholds factor pricesF, append them tocolsin the same way. - When
prholds benchmark pricesB, append them tocolsin the same way. - When
prholds an Exogenous SeriesE, append it tocolsin the same way. - Concatenate
colshorizontally.
Returns
P::AbstractMatrix: The prices,observations × series, with each gap asNaN.
Related
PortfolioOptimisers.advance_anchor — Function
advance_anchor(
anchor::AbstractVector,
P::AbstractMatrix
) -> Any
Moves the anchor of each series column of a PricesToReturnsState past a block.
The new anchor of a column is its last observed price in the block. A column with no observed price in the block keeps its anchor.
Algorithm
- Copy
anchorintoout. - For each column
jofP, find the last rowtwhose price is observed. When there is one, writeP[t, j]intoout[j].
Arguments
anchor: The last observed price of each series column before the block, orNaN.P: The prices of the block,observations × series.
Returns
out::AbstractVector: A new vector.anchordoes not change.
Related
PortfolioOptimisers.block_gap_return! — Function
block_gap_return!(
alg::AbstractGapReturnAlgorithm,
R::AbstractMatrix,
P::AbstractMatrix,
anchor::AbstractVector,
last::AbstractMatrix,
ret_method::Symbol
) -> AbstractMatrix
Writes the cells that a Gap Return algorithm resolves in the returns of a new block, and reads the anchor of the state for the prices before the block.
The batch conversion gives gap_return a whole column. This function gives it the column of the block with two rows in front: the anchor and the last price row kept. So the algorithm meets the observed price that a gap books its move against, and the price just before the first row of the block, as it does in the whole history. The function copies back only the cells of the new rows, and only the cells that gap_return_writable admits over the extended column. The batch conversion keeps the same rule. An anchor is an earlier observed price, and a column with no observed price admits no cell before its first observation.
Algorithm
- Make
p, a column of prices two rows longer thanP, andr, a column of returns one row longer thanR. - For each series column
j, writeanchor[j]andlast[1, j]into the first two rows ofp, and the prices of the block after them. WriteNaNinto the first row ofr, and the returns of the block after it. - Find the writable cells
wof the extended column withgap_return_writable. Go to the next column when no cell is writable. - Apply
algto the extended column withgap_return, givingout. - Copy
outintoRat each writable cell of the new rows.
Arguments
alg: The Gap Return algorithm.R: The returns of the new block,observations × series, changed in place.P: The prices of the new block,observations × series.anchor: The last observed price of each series column before the block.last: The last price row before the block,1 × series.ret_method::simpleor:log.
Validation
algreturns one value for each return cell of the extended column. ADimensionMismatchis thrown otherwise.
Returns
R: The same matrix, with the writable cells of the new rows written.
Related
PortfolioOptimisers.series_values_returns — Function
series_values_returns(rd::ReturnsResult) -> Any
Puts the series columns of the returns data rd side by side, in the order of series_values: the assets, then the factors, then the benchmark, then the Exogenous Series.
Algorithm
- Copy the asset returns
Xinto the first block ofcols. - When
rdholds factor returnsF, append a copy tocols. - When
rdholds benchmark returnsB, append a copy tocols, as one column whenBis a vector. - Concatenate
colshorizontally.
Returns
R::Matrix: The returns,observations × series. The matrix is a copy, so a change to it does not reachrd.
Related
PortfolioOptimisers.returns_with_series — Function
returns_with_series(
rd::ReturnsResult,
R::AbstractMatrix,
_::AbstractGapReturnAlgorithm
) -> Union{ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, Nothing} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K}, ReturnsResult{_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, AssetPanel{__T_pf, __T_amsk, __T_emsk}} where {_A, _B, _C, _D, _E, _F, _G, _H, _I, _J, _K, __T_pf, __T_amsk, __T_emsk}}
Rebuilds a ReturnsResult from the series matrix that series_values_returns lays out.
A Gap Return writes a finite return where the default conversion left none. So the function derives the estimation mask again, as the conversion does.
Algorithm
- Split
Rinto the asset returnsX, the factor returnsF, the benchmark returnsBand the Exogenous SeriesE, in the layout ofseries_values_returns.Bis one column when the benchmark ofrdis a vector. - When
rdholds a time-varying panel, keep its active maskamsk, and derive the estimation maskemskas the cells ofamskwhereXis finite. Whenemskis true everywhere,compress_all_truereplaces the two masks with one all-true mask. - Build the
ReturnsResultfromX,F,B,E, the panel, and the other fields ofrd.
Returns
rd′::ReturnsResult: The returns data ofR.
Related
PortfolioOptimisers.show_fields — Method
show_fields(
est::Union{MissingDataFilter, PriceGapFill, PricesToReturns}
) -> Tuple{Vararg{Symbol}}
Lists the fields that the display of a data step shows.
The display shows cache only when the step keeps a state, so a step that has folded no block shows no cache line.
Returns
fns::Tuple: The names of the fields to show.
Related