Price gap fill
The price gap fill
PriceGapFill fills the price gaps inside the listing span of an asset. It is the only fill of the price data, and it runs only if you add it. Use it to state what the price was during a suspension or a holiday. You do not need it to remove gaps, because the conversion to returns and every later step accept a NaN.
The fill writes only inside the listing span, so it never makes a price before an asset is listed or after it is delisted. You fit it on a training window and apply it to later windows. With CarriedPrice, the fill carries the last observed price forward across the gap. The fitted result stores the last price of the training window, and the fill starts from it when a later window opens inside a gap. With a Num_VecToScaM, a fixed number or a reduction such as the median, the fill writes one constant per asset, computed from the observed prices of the training window. A constant makes two price moves that the market never made, one into the gap and one out of it. CarriedPrice makes no such moves, and it is usually the fill you want.
The fill runs on prices, before PricesToReturns. After the conversion, the only repair is to set the missing returns to zero, and that loses the price move across the gap. For the prices p₀, _, _, p₃, the carried prices give the returns 0, 0, p₃/p₀ - 1. A filled cell has a finite return, and its estimation mask entry is true, because a fill states that the asset traded.
PortfolioOptimisers.CarriedPrice — Type
struct CarriedPrice <: AbstractAlgorithmStates that a price did not move across a gap, so the fill holds the last observed price forward.
CarriedPrice is the Held Price convention. Inside a Held Gap the price is the last observed price of the listed run, so every return inside the gap is zero and the return at the observation that ends the gap carries the whole move. The fill thus conserves wealth. A constant per asset cannot conserve it, because a fill with the median of an asset makes one large move into a suspension and a second one out of it.
It names a convention and not a value, so it is the one member of the fill slot of PriceGapFill that is not a number. The fill reads each price off the data. The fitted value is only the seed for a later window that opens inside a gap.
Mathematical definition
\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} p_{s_{t,\,i},\,i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ exists} \\ v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ does not exist}\,,\ t \geq t_{0}\,,\ r_{t,\,i} = 1\,,\ v_{i} \text{ exists} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ r_{t,\,i} &= \min\left\{u \leq t : a_{u',\,i} = 1 \text{ for } u \leq u' \leq t\right\}\,, \\ s_{t,\,i} &= \max\left(\mathcal{O}_{i} \cap \{r_{t,\,i}, \ldots, t - 1\}\right)\,. \end{align}\]
The third case holds a gap that no case fills, so that gap stays absent. Let $b < c$ be two consecutive members of $\mathcal{O}_{i}$ in one listed run. Then $\tilde{p}_{t,\,i} = p_{b,\,i}$ for $b < t < c$, so the returns at $b + 1, \ldots, c - 1$ are zero and the return at $c$ is $p_{c,\,i} / p_{b,\,i} - 1$. A gap after an absence from the listing takes no price from before the absence.
Where:
- $\tilde{p}_{t,\,i}$: Filled price of asset $i$ at observation $t$, the price after the fill.
- $p_{t,\,i}$: Price of asset $i$ at observation $t$ of the price clock, absent at a gap.
- $a_{t,\,i}$: Listing indicator of asset $i$ at observation $t$ of the price clock, $1$ when the asset is listed and $0$ when it is not.
- $\mathcal{O}_{i}$: Observed set of asset $i$, the observations at which its price is neither
missingnorNaN. - $r_{t,\,i}$: Run start of asset $i$ at observation $t$, the first observation of the unbroken listed run that holds $t$.
- $s_{t,\,i}$: Carry source of asset $i$ at observation $t$, the last member of $\mathcal{O}_{i}$ before $t$ in the listed run that holds $t$.
- $v_{i}$: Fitted value of asset $i$, read off its observed prices over the training window.
- $t_{0}$: Replay start, the first observation of the window after the end of the training window, or $n + 1$ when the window holds none.
- $n$: Price clock length, the number of price observations.
Constructors
CarriedPrice() -> CarriedPriceExamples
julia> CarriedPrice()CarriedPrice()Related
PortfolioOptimisers.PriceGapFill — Type
struct PriceGapFill{__T_fill, __T_strict, __T_cache} <: AbstractPricesPreprocessingEstimatorFills the price gaps inside the listing of each asset with a stated convention, and changes no other price.
PriceGapFill is the one fill of the ingestion layer, and it runs only when a caller adds it. A fill states a price convention across a suspension or a holiday. Its purpose is not to remove a gap, because the conversion keeps a gap and no later step needs it gone. The Listing Span bounds the fill, so the fill writes into Held Gaps alone and never makes a price before an asset is listed or after it is delisted.
The fill runs on prices, before PricesToReturns, because a carried price is a statement about a price and the returns cannot state it. Across a run of prices p₀, _, _, p₃ the unfilled returns are all non-finite, and a zero in their place loses the move from p₀ to p₃. The carried prices give the returns 0, 0, p₃/p₀ - 1. A filled cell thus has a finite return and a true estimation mask entry. That is the intent, because a caller who fills states that the asset traded.
Under CarriedPrice the fitted value is only a seed. A window that opens inside a gap after the training window takes the last observed training price, and every later gap of that window takes the most recent price that the window observed in the same listed run. The fill writes the seed onto no observation at or before the end of the training window. A Pipeline transforms the training window with the step that it just fitted, and on that window a gap that opens the window stays a Held Gap. It does not take the last price of the window. No fill thus reads a later price.
An asset gets no seed when an absence from its listing follows its last observed training price, or when the training window ends outside its listing. A later window cannot see that absence, so a seed from before it would book a return across the absence. The asset keeps its entry in the result, and a later window fills its gaps from the prices of that window alone.
Mathematical definition
Under CarriedPrice:
\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} p_{s_{t,\,i},\,i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ exists} \\ v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ does not exist}\,,\ t \geq t_{0}\,,\ r_{t,\,i} = 1\,,\ v_{i} \text{ exists} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ v_{i} &= p_{m_{i},\,i} \quad \text{when } a_{u,\,i} = 1 \text{ for } m_{i} \leq u \leq n^{\mathrm{tr}}\,, \\ m_{i} &= \max \mathcal{O}_{i}^{\mathrm{tr}}\,. \end{align}\]
When an absence falls between $m_{i}$ and $n^{\mathrm{tr}}$, $v_{i}$ does not exist.
Under a Num_VecToScaM:
\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ v_{i} &= \phi\left(\left\{p_{u,\,i} : u \in \mathcal{O}_{i}^{\mathrm{tr}}\right\}\right)\,. \end{align}\]
An asset with an empty $\mathcal{O}_{i}^{\mathrm{tr}}$ has no $v_{i}$, and every price of that asset stays as it is.
Where:
- $\tilde{p}_{t,\,i}$: Filled price of asset $i$ at observation $t$, the price after the fill.
- $p_{t,\,i}$: Price of asset $i$ at observation $t$ of the price clock, absent at a gap.
- $a_{t,\,i}$: Listing indicator of asset $i$ at observation $t$ of the price clock, $1$ when the asset is listed and $0$ when it is not.
- $\mathcal{O}_{i}$: Observed set of asset $i$, the observations at which its price is neither
missingnorNaN. - $\mathcal{O}_{i}^{\mathrm{tr}}$: Observed set of asset $i$ over the training window.
- $m_{i}$: Last observation of the training window at which asset $i$ has an observed price.
- $n^{\mathrm{tr}}$: Last observation of the training window.
- $r_{t,\,i}$: Run start of asset $i$ at observation $t$, the first observation of the unbroken listed run that holds $t$.
- $s_{t,\,i}$: Carry source of asset $i$ at observation $t$, the last member of $\mathcal{O}_{i}$ before $t$ in the listed run that holds $t$.
- $v_{i}$: Fitted value of asset $i$, read off its observed prices over the training window.
- $\phi$: Reduction that a
Num_VecToScaMstates, from the observed prices of an asset to one number. A plain number is the constant reduction. - $t_{0}$: Replay start, the first observation of the window after the end of the training window, or $n + 1$ when the window holds none.
- $n$: Price clock length, the number of price observations.
Algorithm
Fit
- For each asset column, find
t, the row of the last observed price of the training window. The step leaves out every entry thatis_missing_valueaccepts. - For an asset whose column holds no observed price, the step depends on the convention. Under a
Num_VecToScaMskip the asset, because there is no price to reduce. The asset gets no entry in the result, so the apply step does not change it. UnderCarriedPricethe asset keeps its entry with amissingvalue, so a later window fills its gaps from the prices of that window alone. - Collect the observed prices of the column into
obs. Read withPortfolioOptimisers.gap_fill_openwhether the Listing Span of the price data holds fromtto the end of the training window. Price data that states no span states no absence. - Reduce
obsto one value withPortfolioOptimisers.gap_fill_seed, giving the fitted value of the asset. UnderCarriedPricethe value ismissingwhen the span does not hold fromtto the end, or when the column holds no observed price. - Return a
PriceGapFillResultthat holds the fitted asset names, their values, the last timestamp of the training window, the convention andstrict.
Apply
- Copy the price values of the window into
vals, so the input does not change. - Read the Listing Span
spanthat bounds the fill withPortfolioOptimisers.gap_fill_span. Price data that states no span gives an all-falsespan, and the fill writes no cell. - Find
t0, the first observation of the window after the end of the training window. On the training window itselft0is one past the last row. On a window that follows the training window,t0is the first row. - For each fitted asset name, find its column
jin the window. Skip a name that the window does not carry. - Fill column
jofvalswithPortfolioOptimisers.gap_fill_column!, with the fitted value as the seed fromt0andspanas the bound. Amissingseed fills the gaps of the column from the prices of the window alone. - Rebuild
Xfromvalswith the same timestamps and column names, then rebuild thePricesResult. Every other field passes through unchanged.
Fields
fill: Convention for a gap inside the Listing Span.CarriedPriceholds the last observed price forward. ANum_VecToScaMreduces the observed training prices of an asset to one value, and every gap of that asset takes the value.
strict: Iftrue, a window with a gap from aPricesResultthat states no Listing Span raises anArgumentError. Iffalse, it issues a warning.
cache: Optional partial-fit state. It isnothinguntilpartial_fit!writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it.factorykeeps it unchanged, because a factory call resolves the configuration, not the sample.port_opt_viewcopies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone.obs_weights_viewremoves it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.
Constructors
PriceGapFill(; fill::Union{CarriedPrice, Num_VecToScaM} = CarriedPrice(), strict::Bool = false, cache::Option{<:AbstractPartialFitState} = nothing,) -> PriceGapFillKeywords correspond to the struct's fields.
Online form
A PriceGapFill with a CarriedPrice fill has an online step. partial_fit_transform fills a block of prices from the carried prices and moves the carried prices forward. fit_preprocessing with no data reads the PriceGapFillResult of the whole history out of cache. A statistic fill has no online form, because a longer window changes the value of every earlier gap. supports_partial_fit returns false for it, and a Pipeline refuses it at warm-up with a message that names the step.
Examples
julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4), [100.0 1.0; NaN 3.0; NaN 5.0; 130.0 7.0], ["A", "B"]);julia> res = fit_preprocessing(PriceGapFill(), PricesResult(; X = X));julia> res.v2-element Vector{Float64}: 130.0 7.0Related
PortfolioOptimisers.PriceGapFillResult — Type
struct PriceGapFillResult{__T_nx, __T_v, __T_te, __T_fill, __T_strict} <: AbstractPricesPreprocessingResultHolds the value that a price gap fill fitted for each asset, with the end of the training window and the convention.
PriceGapFill makes it, and apply_preprocessing replays it on a window. One result type serves both conventions, as AssetSelectorResult serves the whole selector family. The convention is a field and not a second type, because PortfolioOptimisers.gap_fill_column! dispatches on it and the two conventions differ in nothing else.
Fields
nx: Names of the fitted assets. UnderCarriedPricethese are all the assets of the training window. Under aNum_VecToScaMthese are the assets with an observed training price.
v: Fitted values, aligned withnx. UnderCarriedPriceeach value is the last observed training price of its asset, ormissingwhen an absence from the listing follows that price in the training window, or when the asset has no observed training price. Amissingseed is never written. Under aNum_VecToScaMeach value is the reduction of the observed training prices of its asset.
te: Last timestamp of the training window. UnderCarriedPricethe fill writes a seed ofvonly onto an observation afterte. A gap at or beforetethus never takes the seed. ANum_VecToScaMdoes not readte.
fill: Convention that readsv, copied from the estimator.
strict: Iftrue, a window with a gap from aPricesResultthat states no Listing Span raises anArgumentError. Iffalse, it issues a warning.
Related