Prices to returns

abstract type AbstractGapReturnAlgorithm <: AbstractAlgorithm

Supertype of the policies that write a return into the cells a price gap left non-finite.

A return is the change between two consecutive observations, so a run of k gapped prices leaves k + 1 non-finite returns and the move across the gap is recorded nowhere. That is the default, and it is what nothing means on prices_to_returns. A caller who wants the move booked states one of these instead.

An algorithm may write only a non-finite cell inside the asset's Listing Span that has an earlier observed price in its column; gap_return_writable derives that set and apply_gap_return restores every other cell. So a cell computed from two observed prices is frozen whatever the algorithm returns, a gap can never spread beyond the cells that read one of its prices, and no return is invented before an asset's first price.

Interfaces

In order to implement a new concrete type that works seamlessly with the library, subtype AbstractGapReturnAlgorithm and implement the following methods:

gap_return

• gap_return(alg::AbstractGapReturnAlgorithm, p::AbstractVector, r::AbstractVector, ret_method::Symbol) -> Vector: One column's returns, with the writable cells resolved.

Arguments

• alg: The concrete subtype instance.
• p: One column's prices along the observation axis, gaps included.
• r: The returns TimeSeries.percentchange computed from p, so length(p) - length(r) is 0 under padding and 1 otherwise.
• ret_method: :simple or :log. Compute a value with gap_return_value rather than re-spelling the two branches.

Returns

• out::Vector: The same length as r. Only the cells gap_return_writable admits are read back, so a method may return the frozen cells unchanged and need not defend the invariant itself.

Related

• CatchUpGapReturn
• gap_return
• gap_return_writable
• apply_gap_return
• prices_to_returns

struct CatchUpGapReturn <: AbstractGapReturnAlgorithm

Books a Held Gap's whole move on the observation that ends it, shortening the gap to k.

A suspension of k observations leaves k + 1 non-finite returns by default. This puts $P_{t+k} / P_{t-1} - 1$ on the observation the asset resumes trading and leaves the k observations inside the gap non-finite, so wealth is conserved across the gap and the Held Gap is exactly the run of unpriced observations. An asset's inception is untouched: it has no earlier observed price to anchor on.

The estimation mask reads the values it was given and is unaware of which algorithm produced them, so the re-pricing cell is estimable and a (k + 1)-period return enters a one-period moment as one draw, at roughly $\sqrt{k + 1}$ the scale.

Constructors

CatchUpGapReturn() -> CatchUpGapReturn

Examples

julia> CatchUpGapReturn()CatchUpGapReturn()

Related

• AbstractGapReturnAlgorithm
• gap_return
• prices_to_returns

struct PricesToReturns{__T_ret_method, __T_padding, __T_gap_return_alg, __T_cache} <: AbstractPreprocessingEstimator

Preprocessing estimator converting price-level data into returns-level data.

PricesToReturns is the estimator form of prices_to_returns: it consumes a PricesResult and produces a ReturnsResult. It is stateless — applying it to any window simply runs the conversion — so its fitted object is the estimator itself.

Its three fields are the three keywords that survive the rule that a keyword belongs to the conversion if and only if it changes the arithmetic of a return. Joining and collapsing move the observation clock and are PriceIngestion's; filling is PriceGapFill's and deleting is MissingDataFilter's, both fitted steps; and every datum the conversion reads is a field of the PricesResult it consumes.

The step is stateless, and it does not need to be stateful to fix an asset universe: the carrier states one. A PricesResult that price_ingestion built carries a Listing Span, and this step projects it onto the returns clock and hands the ReturnsResult an AssetPanel whose two masks say which assets are in the universe and which of them can be estimated at each observation. The asset axis is fixed before the split, so every window of every fold carries every asset and a window can no longer silently lose a column.

Algorithm

The estimator is stateless, so both verbs are thin.

1. fit_preprocessing returns the estimator itself. There is no state to fit.
2. apply_preprocessing calls prices_to_returns with the three fields as keywords, handing it the PricesResult whole. It returns the ReturnsResult.

Every row and every column of the window reaches the conversion, because the conversion has no way to drop one. gap_return_alg is a field, because it decides what the observations a gap left non-finite carry, which is the arithmetic of a return rather than a policy about the universe.

Fields

• ret_method: Return calculation method (:simple or :log).

• padding: Whether to pad missing values in the returns calculation.

• gap_return_alg: What the observations a price gap left non-finite carry. nothing is the arithmetic, and CatchUpGapReturn books the move across the gap on the observation that ends it. See AbstractGapReturnAlgorithm.

• cache: Optional partial-fit state. It is nothing until partial_fit! writes one, and the estimator's read-out verb reads it when the caller gives no data matrix. Each propagation channel does one thing with it: factory carries it unchanged, because a factory call resolves configuration rather than the sample; port_opt_view slices it to the selected assets by index copy, so the viewed estimator answers over those assets alone; and obs_weights_view drops it, because no slice of a state exists on the observation axis. A family whose state has no exact asset slice drops it on both axes and names the reason.

Constructors

PricesToReturns(;    ret_method::Symbol = :simple,    padding::Bool = false,    gap_return_alg::Option{<:AbstractGapReturnAlgorithm} = nothing,    cache::Option{<:AbstractPartialFitState} = nothing) -> PricesToReturns

Keywords correspond to the struct's fields.

Online form

The conversion is stateless to a reader and row-local to a fold: a return reads two consecutive prices, so partial_fit_transform converts a block of prices exactly as the whole history would by keeping the last price row in cache, and under a CatchUpGapReturn the last observed price of every series column. fit_preprocessing with no data reads the estimator itself out, as the batch fit does. A caller's own Gap Return algorithm has no online form, and supports_partial_fit answers false for it.

Validation

• ret_method in (:simple, :log).

Examples

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3),                     [100.0 101.0; 102.0 103.0; 104.0 105.0], ["A", "B"]);julia> pr = PricesResult(; X = X);julia> rr = apply_preprocessing(PricesToReturns(), pr);julia> size(rr.X)(2, 2)julia> rr.nx2-element Vector{String}: "A" "B"

Related

• AbstractPreprocessingEstimator
• prices_to_returns
• AbstractGapReturnAlgorithm
• CatchUpGapReturn
• PriceIngestion
• PricesResult
• ReturnsResult

prices_to_returns(
    pr::PricesResult;
    ret_method::Symbol = :simple,
    padding::Bool = false,
    gap_return_alg::Option{<:AbstractGapReturnAlgorithm} = nothing
) -> ReturnsResult
prices_to_returns(
    X::TimeSeries.TimeArray;
    kwargs...
) -> ReturnsResult

Compute returns from the price carrier, and nothing else.

A keyword survives here if and only if it changes the arithmetic of a return, which is the rule and the reason there are three. Every datum the conversion reads — the asset prices, the factors, the benchmark, the implied volatilities, the Listing Span and the AssetPanel — is already a field of the PricesResult, so naming one as a keyword would be a second way to say what the carrier says.

The second method is the friendliest call in the library, and it is the layer's own path rather than a way around it: it runs price_ingestion with a default PriceIngestion and converts what that emits. A caller wanting a different join, a collapse, a declared span, or factor, benchmark and implied-volatility series writes the two steps.

An absent price has one spelling, NaN, and the conversion carries it into the returns rather than deleting the observation or the asset that holds one. Filling a gap is PriceGapFill's and deleting one is MissingDataFilter's, both of them fitted steps.

Mathematical definition

Returns are computed from prices $P_{t,i}$ as:

\[\begin{align} r_{t,i} &= \begin{cases} (P_{t,i} - P_{t-1,i}) / P_{t-1,i} & \text{simple} \\ \ln(P_{t,i} / P_{t-1,i}) & \text{log} \end{cases}\,. \end{align}\]

Where:

• $r_{t,i}$: Return of asset $i$ at time $t$.
• $P_{t,i}$: Price of asset $i$ at time $t$.

Both branches need a positive price, and a zero price gives $\pm\infty$.

A benchmark $B$ is converted by the same rule and carried alongside the asset returns, never subtracted from them. The subtraction that forms the excess return $\tilde{r}_{t,i} = r_{t,i} - b_{t,i}$ is a separate operation, and it is applied only when the optimisation tracks the benchmark.

Algorithm

1. Check that the asset, factor and benchmark series can still be named side by side with assert_distinct_series_names. Read the asset names and the asset timestamps from pr.X, and check pr.pnl against them with check_asset_panel and pr.span with assert_span_shape.
2. Lay the three price blocks side by side on the carrier's clock with append_carrier_block!, spelling every absent price NaN with unify_gaps. The carrier states one clock, so a factor or benchmark series is read at the asset timestamps rather than joined onto them, and one stating a different clock is refused by name: a join adds or drops observations, and price_ingestion owns every clock move. A benchmark is one shared column, or one column per asset.
3. Convert the prices to returns with TimeSeries.percentchange under ret_method and padding. This is the step that applies the formula above. It computes both branches through logarithms — the log return is $\ln P_{t,i} - \ln P_{t-1,i}$, and the simple return is expm1 of it — so the two agree with the closed forms above to floating point rather than to the last bit. When padding is true the first observation is kept and its return is NaN, so the returns keep the length of the price clock. A gap carried here does not spread. The formula reads two prices, so a run of k gapped prices makes exactly the k + 1 returns that read one of them non-finite, and every later return of that column is computed from two observed prices and is finite. A gap is confined to its own column for the same reason: no asset's return reads another's price.
4. Resolve the cells the conversion left non-finite with apply_gap_return, under gap_return_alg. nothing is the default rule, and its method returns the table untouched, so the arithmetic step 3 produced is bit-identical. An algorithm may write only a non-finite cell inside a column's Listing Span that has an earlier observed price, which is what freezes every return computed from two observed prices, and it reports an @info when it finds no such cell.
5. Name the three blocks. Step 1 refused every name two of the tables shared and the clock's own name timestamp, so the asset names nx, the factor names nf and the benchmark names nb are the lists read off the three tables, and ts is the timestamp column the DataFrames.DataFrame conversion wrote. Each is the typed vector its table held, rather than whatever is left once the other groups have taken what they recognise.
6. Spell the implied volatilities' absences with unify_gaps and index pr.iv by ts, then check them and pr.ivpa against the asset count. The returns clock is the price clock less the observation padding costs, so a carrier the layer built covers it, and an absent implied volatility is carried as NaN for the estimator that reads it to exclude.
7. Subselect the AssetPanel. Recover the surviving rows with feature_row_indices and view the panel with port_opt_view, handing it the asset names so that a square tensor Panel Field is cut on its label axis too. The conversion removes no column, so the asset axis reaches the panel whole and the subselection that bites is the observation one: a time-varying panel is cut to the surviving observations and matched back into the price timestamps.
8. State the universe. Cut pr.span to the asset axis with span_carrier_view, and hand it and the converted returns to returns_universe_masks, which projects it onto the returns clock and intersects it with finiteness. A carrier that states no span states no universe, and the conversion emits no panel. attach_universe_masks puts the pair onto the Asset Panel, keeping whatever Panel Fields it already carried, and mints one with no field when the carrier held none.
9. Build the asset, factor and benchmark matrices from the columns of each group. The asset group is always present, because the conversion removes no column; a factor or benchmark group given no column is nothing.
10. Return the ReturnsResult.

The conversion removes no observation and no asset. Deleting either is a Universe Policy, and a policy is fitted on a training window and replayed by name, which a stateless conversion cannot do; MissingDataFilter owns it, with col_thr deleting an asset and row_thr an observation. A keyword survives here if and only if it changes the arithmetic of a return.

Arguments

• pr: The price carrier, as price_ingestion emits it or a caller builds it.
• X: Asset price data (observations × assets), converted through a default PriceIngestion.
• ret_method: Return calculation method (:simple or :log).
• padding: Whether to pad missing values in returns calculation.
• gap_return_alg: What the observations a price gap left non-finite carry. nothing is the arithmetic — a return is the change between two consecutive observations, so a run of k gapped prices leaves k + 1 non-finite returns and the move across the gap is recorded nowhere — and CatchUpGapReturn books that move on the observation the asset resumes trading instead, shortening the Held Gap to k. Any algorithm may write only a non-finite cell inside an asset's Listing Span that has an earlier observed price in its column, so a return computed from two observed prices is frozen whichever one is stated. It has no cell to write over a gap-free table, and reports an @info there.

Validation

• Every price reaching step 3 is positive. TimeSeries.percentchange takes a logarithm on both branches, so a negative price raises a DomainError from inside it, on the simple branch as well.
• The asset, factor and benchmark column names are pairwise disjoint, and none of them is timestamp. Raises a ConflictingArgumentError naming the offending columns.
• If pr.F or pr.B is not nothing, its timestamps equal the asset timestamps. Raises a ConflictingArgumentError naming price_ingestion, which is what puts two series on one clock.
• If pr.iv is not nothing, the returns timestamps are a subset of TimeSeries.timestamp(pr.iv), then iv = values(unify_gaps(iv)[ts]), !isempty(iv), size(iv) == size(X), and every value is finite and non-negative where it is present (an absent one is NaN; see assert_nonneg_where_present).
• If pr.span is not nothing, size(pr.span) == size(values(pr.X)). Raises a DimensionMismatch.
• pr.ivpa is validated in that same branch, so it is checked only when pr.iv is given: all(x -> x > 0, ivpa), all(x -> isfinite(x), ivpa), and, if a vector, length(ivpa) == size(iv, 2). The bound is strict — a zero adjustment is rejected.

Returns

• rr::ReturnsResult: Struct containing asset/factor returns, names, time series, and optional implied volatility data. A converted benchmark is carried in its B field.

Examples

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3), [100 101; 102 103; 104 105],                     ["A", "B"])3×2 TimeSeries.TimeArray{Int64, 2, Dates.Date, Matrix{Int64}} 2020-01-01 to 2020-01-03┌────────────┬─────┬─────┐│            │ A   │ B   │├────────────┼─────┼─────┤│ 2020-01-01 │ 100 │ 101 ││ 2020-01-02 │ 102 │ 103 ││ 2020-01-03 │ 104 │ 105 │└────────────┴─────┴─────┘julia> prices_to_returns(X)ReturnsResult    nx ┼ Vector{String}: ["A", "B"]     X ┼ 2×2 Matrix{Float64}    nf ┼ nothing     F ┼ nothing    nb ┼ nothing     B ┼ nothing    ts ┼ Vector{Dates.Date}: [Dates.Date("2020-01-02"), Dates.Date("2020-01-03")]    iv ┼ nothing  ivpa ┼ nothing   pnl ┼ AssetPanel       │     pf ┼ Vector{PortfolioOptimisers.AbstractPanelField}: PortfolioOptimisers.AbstractPanelField[]       │   amsk ┼ 2×2 PortfolioOptimisers.AllTrueMask       │   emsk ┴ 2×2 PortfolioOptimisers.AllTrueMask

Related

• PricesResult
• ReturnsResult
• Option
• VecStr
• MatNum
• VecDate
• Num_VecNum
• TimeSeries
• apply_gap_return
• AbstractGapReturnAlgorithm
• CatchUpGapReturn
• PriceIngestion
• price_ingestion
• append_carrier_block!
• unify_gaps
• returns_universe_masks
• attach_universe_masks
• span_carrier_view
• returns_result_picker: subtracts the carried benchmark, and only when the optimisation tracks it.

gap_return(alg::CatchUpGapReturn, p::AbstractVector, r::AbstractVector, ret_method::Symbol) -> Vector

Resolve the writable cells of one column's returns.

The method Julia selects is the algorithm. Only CatchUpGapReturn ships, and it is the reason the family is an algorithm rather than a flag: a caller who wants a suspension's move spread across its observations is asking a question of the same kind, and it costs one type.

Algorithm

CatchUpGapReturn walks the observation axis carrying the row of the last observed price.

1. On a gapped price, carry nothing forward and write nothing: the observation is inside the Held Gap and stays non-finite.
2. On an observed price whose immediate predecessor was observed too, write nothing: TimeSeries.percentchange already computed that cell from two consecutive prices, and gap_return_writable freezes it in any case.
3. On an observed price whose immediate predecessor was not, write gap_return_value of it against the carried price. This is the observation that ends the gap, and the whole move across the gap lands on it.

A column's first observed price carries nothing, so nothing is written on it, which is what makes an inception and an interior gap one case.

Arguments

• alg: The Gap Return algorithm.
• p: One column's prices along the observation axis, gaps included.
• r: The returns TimeSeries.percentchange computed from p.
• ret_method: :simple or :log.

Returns

• out::Vector: The same length as r, with the writable cells resolved.

Examples

julia> PortfolioOptimisers.gap_return(CatchUpGapReturn(), [100.0, NaN, NaN, 110.0],                                      [NaN, NaN, NaN], :simple)3-element Vector{Float64}: NaN NaN   0.0999999999999999

Related

• AbstractGapReturnAlgorithm
• CatchUpGapReturn
• gap_return_writable
• gap_return_value
• apply_gap_return