Price gap fill

The price gap fill

PriceGapFill fills the price gaps inside the listing span of an asset. It is the only fill of the price data, and it runs only if you add it. Use it to state what the price was during a suspension or a holiday. You do not need it to remove gaps, because the conversion to returns and every later step accept a NaN.

The fill writes only inside the listing span, so it never makes a price before an asset is listed or after it is delisted. You fit it on a training window and apply it to later windows. With CarriedPrice, the fill carries the last observed price forward across the gap. The fitted result stores the last price of the training window, and the fill starts from it when a later window opens inside a gap. With a Num_VecToScaM, a fixed number or a reduction such as the median, the fill writes one constant per asset, computed from the observed prices of the training window. A constant makes two price moves that the market never made, one into the gap and one out of it. CarriedPrice makes no such moves, and it is usually the fill you want.

The fill runs on prices, before PricesToReturns. After the conversion, the only repair is to set the missing returns to zero, and that loses the price move across the gap. For the prices p₀, _, _, p₃, the carried prices give the returns 0, 0, p₃/p₀ - 1. A filled cell has a finite return, and its estimation mask entry is true, because a fill states that the asset traded.

PortfolioOptimisers.CarriedPrice — Type
struct CarriedPrice <: AbstractAlgorithm

States that a price did not move across a gap, so the fill holds the last observed price forward.

CarriedPrice is the Held Price convention. Inside a Held Gap the price is the last observed price of the listed run, so every return inside the gap is zero and the return at the observation that ends the gap carries the whole move. The fill thus conserves wealth. A constant per asset cannot conserve it, because a fill with the median of an asset makes one large move into a suspension and a second one out of it.

It names a convention and not a value, so it is the one member of the fill slot of PriceGapFill that is not a number. The fill reads each price off the data. The fitted value is only the seed for a later window that opens inside a gap.

Mathematical definition

\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} p_{s_{t,\,i},\,i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ exists} \\ v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ does not exist}\,,\ t \geq t_{0}\,,\ r_{t,\,i} = 1\,,\ v_{i} \text{ exists} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ r_{t,\,i} &= \min\left\{u \leq t : a_{u',\,i} = 1 \text{ for } u \leq u' \leq t\right\}\,, \\ s_{t,\,i} &= \max\left(\mathcal{O}_{i} \cap \{r_{t,\,i}, \ldots, t - 1\}\right)\,. \end{align}\]

The third case holds a gap that no case fills, so that gap stays absent. Let $b < c$ be two consecutive members of $\mathcal{O}_{i}$ in one listed run. Then $\tilde{p}_{t,\,i} = p_{b,\,i}$ for $b < t < c$, so the returns at $b + 1, \ldots, c - 1$ are zero and the return at $c$ is $p_{c,\,i} / p_{b,\,i} - 1$. A gap after an absence from the listing takes no price from before the absence.

Where:

  • $\tilde{p}_{t,\,i}$: Filled price of asset $i$ at observation $t$, the price after the fill.
  • $p_{t,\,i}$: Price of asset $i$ at observation $t$ of the price clock, absent at a gap.
  • $a_{t,\,i}$: Listing indicator of asset $i$ at observation $t$ of the price clock, $1$ when the asset is listed and $0$ when it is not.
  • $\mathcal{O}_{i}$: Observed set of asset $i$, the observations at which its price is neither missing nor NaN.
  • $r_{t,\,i}$: Run start of asset $i$ at observation $t$, the first observation of the unbroken listed run that holds $t$.
  • $s_{t,\,i}$: Carry source of asset $i$ at observation $t$, the last member of $\mathcal{O}_{i}$ before $t$ in the listed run that holds $t$.
  • $v_{i}$: Fitted value of asset $i$, read off its observed prices over the training window.
  • $t_{0}$: Replay start, the first observation of the window after the end of the training window, or $n + 1$ when the window holds none.
  • $n$: Price clock length, the number of price observations.

Constructors

CarriedPrice() -> CarriedPrice

Examples

julia> CarriedPrice()CarriedPrice()

Related

source
PortfolioOptimisers.PriceGapFill — Type
struct PriceGapFill{__T_fill, __T_strict, __T_cache} <: AbstractPricesPreprocessingEstimator

Fills the price gaps inside the listing of each asset with a stated convention, and changes no other price.

PriceGapFill is the one fill of the ingestion layer, and it runs only when a caller adds it. A fill states a price convention across a suspension or a holiday. Its purpose is not to remove a gap, because the conversion keeps a gap and no later step needs it gone. The Listing Span bounds the fill, so the fill writes into Held Gaps alone and never makes a price before an asset is listed or after it is delisted.

The fill runs on prices, before PricesToReturns, because a carried price is a statement about a price and the returns cannot state it. Across a run of prices p₀, _, _, p₃ the unfilled returns are all non-finite, and a zero in their place loses the move from p₀ to p₃. The carried prices give the returns 0, 0, p₃/p₀ - 1. A filled cell thus has a finite return and a true estimation mask entry. That is the intent, because a caller who fills states that the asset traded.

Under CarriedPrice the fitted value is only a seed. A window that opens inside a gap after the training window takes the last observed training price, and every later gap of that window takes the most recent price that the window observed in the same listed run. The fill writes the seed onto no observation at or before the end of the training window. A Pipeline transforms the training window with the step that it just fitted, and on that window a gap that opens the window stays a Held Gap. It does not take the last price of the window. No fill thus reads a later price.

An asset gets no seed when an absence from its listing follows its last observed training price, or when the training window ends outside its listing. A later window cannot see that absence, so a seed from before it would book a return across the absence. The asset keeps its entry in the result, and a later window fills its gaps from the prices of that window alone.

Mathematical definition

Under CarriedPrice:

\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} p_{s_{t,\,i},\,i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ exists} \\ v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i}\,,\ s_{t,\,i} \text{ does not exist}\,,\ t \geq t_{0}\,,\ r_{t,\,i} = 1\,,\ v_{i} \text{ exists} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ v_{i} &= p_{m_{i},\,i} \quad \text{when } a_{u,\,i} = 1 \text{ for } m_{i} \leq u \leq n^{\mathrm{tr}}\,, \\ m_{i} &= \max \mathcal{O}_{i}^{\mathrm{tr}}\,. \end{align}\]

When an absence falls between $m_{i}$ and $n^{\mathrm{tr}}$, $v_{i}$ does not exist.

Under a Num_VecToScaM:

\[\begin{align} \tilde{p}_{t,\,i} &= \begin{cases} v_{i} & a_{t,\,i} = 1\,,\ t \notin \mathcal{O}_{i} \\ p_{t,\,i} & \text{otherwise} \end{cases}\,, \\ v_{i} &= \phi\left(\left\{p_{u,\,i} : u \in \mathcal{O}_{i}^{\mathrm{tr}}\right\}\right)\,. \end{align}\]

An asset with an empty $\mathcal{O}_{i}^{\mathrm{tr}}$ has no $v_{i}$, and every price of that asset stays as it is.

Where:

  • $\tilde{p}_{t,\,i}$: Filled price of asset $i$ at observation $t$, the price after the fill.
  • $p_{t,\,i}$: Price of asset $i$ at observation $t$ of the price clock, absent at a gap.
  • $a_{t,\,i}$: Listing indicator of asset $i$ at observation $t$ of the price clock, $1$ when the asset is listed and $0$ when it is not.
  • $\mathcal{O}_{i}$: Observed set of asset $i$, the observations at which its price is neither missing nor NaN.
  • $\mathcal{O}_{i}^{\mathrm{tr}}$: Observed set of asset $i$ over the training window.
  • $m_{i}$: Last observation of the training window at which asset $i$ has an observed price.
  • $n^{\mathrm{tr}}$: Last observation of the training window.
  • $r_{t,\,i}$: Run start of asset $i$ at observation $t$, the first observation of the unbroken listed run that holds $t$.
  • $s_{t,\,i}$: Carry source of asset $i$ at observation $t$, the last member of $\mathcal{O}_{i}$ before $t$ in the listed run that holds $t$.
  • $v_{i}$: Fitted value of asset $i$, read off its observed prices over the training window.
  • $\phi$: Reduction that a Num_VecToScaM states, from the observed prices of an asset to one number. A plain number is the constant reduction.
  • $t_{0}$: Replay start, the first observation of the window after the end of the training window, or $n + 1$ when the window holds none.
  • $n$: Price clock length, the number of price observations.

Algorithm

Fit

  1. For each asset column, find t, the row of the last observed price of the training window. The step leaves out every entry that is_missing_value accepts.
  2. For an asset whose column holds no observed price, the step depends on the convention. Under a Num_VecToScaM skip the asset, because there is no price to reduce. The asset gets no entry in the result, so the apply step does not change it. Under CarriedPrice the asset keeps its entry with a missing value, so a later window fills its gaps from the prices of that window alone.
  3. Collect the observed prices of the column into obs. Read with PortfolioOptimisers.gap_fill_open whether the Listing Span of the price data holds from t to the end of the training window. Price data that states no span states no absence.
  4. Reduce obs to one value with PortfolioOptimisers.gap_fill_seed, giving the fitted value of the asset. Under CarriedPrice the value is missing when the span does not hold from t to the end, or when the column holds no observed price.
  5. Return a PriceGapFillResult that holds the fitted asset names, their values, the last timestamp of the training window, the convention and strict.

Apply

  1. Copy the price values of the window into vals, so the input does not change.
  2. Read the Listing Span span that bounds the fill with PortfolioOptimisers.gap_fill_span. Price data that states no span gives an all-false span, and the fill writes no cell.
  3. Find t0, the first observation of the window after the end of the training window. On the training window itself t0 is one past the last row. On a window that follows the training window, t0 is the first row.
  4. For each fitted asset name, find its column j in the window. Skip a name that the window does not carry.
  5. Fill column j of vals with PortfolioOptimisers.gap_fill_column!, with the fitted value as the seed from t0 and span as the bound. A missing seed fills the gaps of the column from the prices of the window alone.
  6. Rebuild X from vals with the same timestamps and column names, then rebuild the PricesResult. Every other field passes through unchanged.

Fields

  • fill: Convention for a gap inside the Listing Span. CarriedPrice holds the last observed price forward. A Num_VecToScaM reduces the observed training prices of an asset to one value, and every gap of that asset takes the value.
  • strict: If true, a window with a gap from a PricesResult that states no Listing Span raises an ArgumentError. If false, it issues a warning.
  • cache: Optional partial-fit state. It is nothing until partial_fit! writes one. When the caller gives no data matrix, the batch verb of the estimator reads it. Each propagation channel does one thing with it. factory keeps it unchanged, because a factory call resolves the configuration, not the sample. port_opt_view copies the entries of the selected assets into a new state, so the viewed estimator holds the state of those assets alone. obs_weights_view removes it, because a state has no slice on the observation axis. Only a type with an observation-weights field has that channel. A type with no such field falls through to the identity method, so it keeps its state. A family whose state has no exact asset slice removes it on both axes and names the reason.

Constructors

PriceGapFill(;    fill::Union{CarriedPrice, Num_VecToScaM} = CarriedPrice(),    strict::Bool = false,    cache::Option{<:AbstractPartialFitState} = nothing,) -> PriceGapFill

Keywords correspond to the struct's fields.

Online form

A PriceGapFill with a CarriedPrice fill has an online step. partial_fit_transform fills a block of prices from the carried prices and moves the carried prices forward. fit_preprocessing with no data reads the PriceGapFillResult of the whole history out of cache. A statistic fill has no online form, because a longer window changes the value of every earlier gap. supports_partial_fit returns false for it, and a Pipeline refuses it at warm-up with a message that names the step.

Examples

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 4),                     [100.0 1.0; NaN 3.0; NaN 5.0; 130.0 7.0], ["A", "B"]);julia> res = fit_preprocessing(PriceGapFill(), PricesResult(; X = X));julia> res.v2-element Vector{Float64}: 130.0   7.0

Related

source
PortfolioOptimisers.PriceGapFillResult — Type
struct PriceGapFillResult{__T_nx, __T_v, __T_te, __T_fill, __T_strict} <: AbstractPricesPreprocessingResult

Holds the value that a price gap fill fitted for each asset, with the end of the training window and the convention.

PriceGapFill makes it, and apply_preprocessing replays it on a window. One result type serves both conventions, as AssetSelectorResult serves the whole selector family. The convention is a field and not a second type, because PortfolioOptimisers.gap_fill_column! dispatches on it and the two conventions differ in nothing else.

Fields

  • nx: Names of the fitted assets. Under CarriedPrice these are all the assets of the training window. Under a Num_VecToScaM these are the assets with an observed training price.
  • v: Fitted values, aligned with nx. Under CarriedPrice each value is the last observed training price of its asset, or missing when an absence from the listing follows that price in the training window, or when the asset has no observed training price. A missing seed is never written. Under a Num_VecToScaM each value is the reduction of the observed training prices of its asset.
  • te: Last timestamp of the training window. Under CarriedPrice the fill writes a seed of v only onto an observation after te. A gap at or before te thus never takes the seed. A Num_VecToScaM does not read te.
  • fill: Convention that reads v, copied from the estimator.
  • strict: If true, a window with a gap from a PricesResult that states no Listing Span raises an ArgumentError. If false, it issues a warning.

Related

source