Pre-processing

Prices to returns

Other than FiniteAllocationOptimisationEstimator, all optimisations work based off returns data rather than price data. These functions and types are involved in computing returns.

PortfolioOptimisers.AbstractReturnsResult Type

julia

abstract type AbstractReturnsResult <: AbstractResult

Abstract supertype for all returns result types in PortfolioOptimisers.jl.

All concrete and/or types representing the result of returns calculations should be subtypes of AbstractReturnsResult.

Related

source

PortfolioOptimisers.ReturnsResult Type

julia

struct ReturnsResult{__T_nx, __T_X, __T_nf, __T_F, __T_nb, __T_B, __T_ts, __T_iv, __T_ivpa} <: AbstractReturnsResult

A flexible container type for storing the results of asset and factor returns calculations in PortfolioOptimisers.jl.

ReturnsResult is the standard result type returned by returns-processing routines, such as prices_to_returns.

It supports both asset and factor returns, as well as optional time series and implied volatility information, and is designed for downstream compatibility with optimisation and analysis routines.

Fields

nx: Names or identifiers of asset columns (assets × 1).
X: Asset returns matrix (observations × assets).
nf: Names or identifiers of factor columns (factors × 1).
F: Factor returns matrix (observations × factors).
nb: Names or identifiers of benchmark columns (observations × 1) or (observations × assets).
B: Benchmark prices (observations × 1) or (observations × assets).
ts: Optional timestamps for each observation (observations × 1).
iv: Implied volatilities matrix (observations × assets).
ivpa: Implied volatility risk premium adjustment, if a vector (assets × 1).

Constructors

julia

ReturnsResult(;
    nx::Option{<:VecStr} = nothing,
    X::Option{<:MatNum} = nothing,
    nf::Option{<:VecStr} = nothing,
    F::Option{<:MatNum} = nothing,
    nb::Option{<:VecStr} = nothing,
    B::Option{<:VecNum_MatNum} = nothing,
    ts::Option{<:VecDate} = nothing,
    iv::Option{<:MatNum} = nothing,
    ivpa::Option{<:Num_VecNum} = nothing,
) -> ReturnsResult

Keywords correspond to the struct's fields.

Validation

If nx or X is not nothing, !isempty(nx), !isempty(X), and length(nx) == size(X, 2).
If nf or F is not nothing, !isempty(nf), !isempty(F), length(nf) == size(F, 2), and size(X, 1) == size(F, 1).
If nb or B is not nothing and B is a matrix: !isempty(nb), !isempty(B), and length(nb) == size(B, 2).
If nb or B is not nothing and B is a vector: length(nb) == 1.
If X and B are not nothing: if B is a vector, size(X, 1) == size(B, 1); if B is a matrix, size(X) == size(B).
If ts is not nothing, !isempty(ts), and length(ts) == size(X, 1).
If ts and B are not nothing: length(ts) == size(B, 1).
If iv is not nothing, !isempty(iv), all(x -> x >= 0, iv), size(iv) == size(X).
If ivpa is not nothing, all(x -> x >= 0, ivpa), all(x -> isfinite(x), ivpa); if a vector, length(ivpa) == size(iv, 2).

Examples

julia

julia> ReturnsResult(; nx = ["A", "B"], X = [0.1 0.2; 0.3 0.4])
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

Related

source

PortfolioOptimisers.check_names_and_returns_matrix Function

julia

check_names_and_returns_matrix(
    names::Union{Nothing, AbstractVector{<:AbstractString}},
    mat::Union{Nothing, AbstractMatrix{<:Union{var"#s29", var"#s28"} where {var"#s29"<:Number, var"#s28"<:AbstractJuMPScalar}}},
    names_sym::Symbol,
    mat_sym::Symbol
)

Validate that asset or factor names and their corresponding returns matrix are provided and consistent.

Arguments

names: Asset or factor names.
mat: Returns matrix.
names_sym: Symbolic name for the names argument displayed in error messages.
mat_sym: Symbolic name for the matrix argument displayed in error messages.

Returns

nothing.

Details

If either names or mat is not nothing:
- !isnothing(names) and !isnothing(mat).
- !isempty(names) and !isempty(mat).
- length(names) == size(mat, 2).

Related

ReturnsResult

source

PortfolioOptimisers.prices_to_returns Function

julia

prices_to_returns(
    X::TimeSeries.TimeArray,
    F::Option{<:TimeSeries.TimeArray} = nothing;
    B::Option{<:TimeSeries.TimeArray} = nothing,
    iv::Option{<:TimeSeries.TimeArray} = nothing,
    ivpa::Option{<:Num_VecNum} = nothing,
    ret_method::Symbol = :simple, padding::Bool = false,
    missing_col_percent::Number = 1.0,
    missing_row_percent::Option{<:Number} = 1.0,
    collapse_args::Tuple = (),
    map_func::Option{<:Function} = nothing,
    join_method::Symbol = :outer,
    impute_method::Option{<:Impute.Imputor} = nothing
) -> ReturnsResult

Convert price data (and optionally factor data) in TimeSeries.TimeArray format to returns, with flexible handling of missing data, imputation, and optional implied volatility information.

Mathematical definition

Returns are computed from prices $P_{t, i}$ as:

\begin{aligned} r_{t, i} & = {\begin{cases} (P_{t, i} - P_{t - 1, i}) / P_{t - 1, i} & simple \\ \ln (P_{t, i} / P_{t - 1, i}) & log \end{cases} . \end{aligned}

Where:

$r_{t, i}$ : Return of asset $i$ at time $t$ .
$P_{t, i}$ : Price of asset $i$ at time $t$ .

If a benchmark $B_{t, i}$ is provided, excess returns are used: ${\tilde{r}}_{t, i} = r_{t, i} - b_{t, i}$ .

Arguments

X: Asset price data (observations × assets).
F: Optional Factor price data (observations × factors).
B: Optional Benchmark price data (observations × assets) or (observations × 1).
iv: Optional Implied volatility data.
ivpa: Optional Implied volatility risk premium adjustment.
ret_method: Return calculation method (:simple or :log).
padding: Whether to pad missing values in returns calculation.
missing_col_percent: Maximum allowed fraction (0, 1] of missing values per column (asset + factor).
missing_row_percent: Maximum allowed fraction (0, 1] of missing values per row (timestamp).
collapse_args: Arguments for collapsing the time series (e.g., to lower frequency).
map_func: Optional function to apply to the data before returns calculation.
join_method: How to join asset, factor data and benchmark data (:outer, :inner, etc.).
impute_method: Optional imputation method for missing data.

Returns

rr::ReturnsResult: Struct containing asset/factor returns, names, time series, and optional implied volatility data.

Validation

!isempty(X).
0 < missing_col_percent <= 1
0 < missing_row_percent <= 1.
If F is not nothing, !isempty(F).
If B is not nothing, !isempty(B), and size(values(B), 2) in (1, size(values(X), 2)).
If iv is not nothing, the timestamp of the merged data matrix must be a subset of TimeSeries.timestamp(iv), then iv = values(iv), !isempty(iv), all(x -> x >= 0, iv), size(iv) == size(X).
If ivpa is not nothing, all(x -> x >= 0, ivpa), all(x -> isfinite(x), ivpa); if a vector, length(ivpa) == size(iv, 2).

Details

Joins asset, factor, and benchmark data as specified.
Optionally applies a mapping function and/or collapses the time series.
Handles missing values by filtering, imputation, and dropping as configured.
Computes returns using the specified method.
- If B is not nothing, it is subtracted from asset returns. Used for returns tracking error optimisations.
Returns a ReturnsResult with asset/factor names, returns, timestamps, and optional implied volatility data.

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3), [100 101; 102 103; 104 105],
                     ["A", "B"])
3×2 TimeSeries.TimeArray{Int64, 2, Dates.Date, Matrix{Int64}} 2020-01-01 to 2020-01-03
┌────────────┬─────┬─────┐
│            │ A   │ B   │
├────────────┼─────┼─────┤
│ 2020-01-01 │ 100 │ 101 │
│ 2020-01-02 │ 102 │ 103 │
│ 2020-01-03 │ 104 │ 105 │
└────────────┴─────┴─────┘

julia> prices_to_returns(X)
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ts ┼ Vector{Dates.Date}: [Dates.Date("2020-01-02"), Dates.Date("2020-01-03")]
    iv ┼ nothing
  ivpa ┴ nothing

Related

source

PortfolioOptimisers.port_opt_view Method

julia

port_opt_view(rd::ReturnsResult, i) -> ReturnsResult

Return a view of the ReturnsResult object for the assets at indices i.

This is the port_opt_view method for ReturnsResult — the View of the library's central data structure, restricting it to a subset of assets.

Warning

This two-argument method indexes assets, matching the rest of the port_opt_view family. The four-argument method port_opt_view(rd, i, j, k) indexes observations first and assets second. The two arities therefore give i different meanings; see port_opt_view(rd::ReturnsResult, i, j, k).

Arguments

rd: A ReturnsResult object containing asset and/or factor returns.
i: Indices of the assets to view.

Returns

new_rr::ReturnsResult: A new ReturnsResult containing only the data for the specified index.

Details

Extracts the asset name, returns, implied volatility, and risk premium adjustment for indices i.
Preserves factor, timestamp, and other fields from the original object.
Returns nothing for fields that are not present.

Examples

julia

julia> rd = ReturnsResult(; nx = ["A", "B"], X = [0.1 0.2; 0.3 0.4])
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

julia> PortfolioOptimisers.port_opt_view(rd, 2:2)
ReturnsResult
    nx ┼ SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}: ["B"]
     X ┼ 2×1 SubArray{Float64, 2, Matrix{Float64}, Tuple{Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}}, true}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

Related

port_opt_view( rd::ReturnsResult, i, j, k = : ) -> ReturnsResult

Return a view of the ReturnsResult object for assets at indices j, observations at indices i, and factors at indices k.

Warning

Unlike every other port_opt_view method — including port_opt_view(rd::ReturnsResult, i) — the first index of this method selects observations, not assets. Assets are the second index. Cross-validation splits observations and assets together, which is why this arity exists at all.

Arguments

rd: A ReturnsResult object containing asset and/or factor returns.
i: Index or indices of the observation(s) to view.
j: Index or indices of the assets to view.
k: Index or indices of the factors to view.

Returns

new_rr::ReturnsResult: A new ReturnsResult containing only the data for the specified indices.

Details

Extracts the asset name, returns, implied volatility, and risk premium adjustment for indices j and observation(s) i.
Extracts the factor names and returns for indices k and observations i.
Preserves factor names and returns for the selected observations.
Preserves timestamps for the selected observations.
Returns nothing for fields that are not present in the original object.

Related

Examples

julia

julia> rd = ReturnsResult(; nx = ["A", "B"], X = [0.1 0.2; 0.3 0.4; 0.5 0.6], nf = ["F1"],
                          F = [1.0; 2.0; 3.0;;])
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 3×2 Matrix{Float64}
    nf ┼ Vector{String}: ["F1"]
     F ┼ 3×1 Matrix{Float64}
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

julia> PortfolioOptimisers.port_opt_view(rd, 1:2, 2:2)
ReturnsResult
    nx ┼ SubArray{String, 1, Vector{String}, Tuple{UnitRange{Int64}}, true}: ["B"]
     X ┼ 2×1 SubArray{Float64, 2, Matrix{Float64}, Tuple{UnitRange{Int64}, UnitRange{Int64}}, false}
    nf ┼ Vector{String}: ["F1"]
     F ┼ 2×1 SubArray{Float64, 2, Matrix{Float64}, Tuple{UnitRange{Int64}, Base.Slice{Base.OneTo{Int64}}}, false}
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

port_opt_view(rd::AbstractReturnsResult, args...; kwargs...)

Erroring tripwire for AbstractReturnsResult subtypes that do not implement port_opt_view.

Without it, the universal leaf fallback port_opt_view(x, i, args...) would hand back the returns result unsubselected, and a meta-optimiser or cross-validation fold would silently train on the full universe. Returns data is never a leaf value, so an unhandled subtype is a missing method, not a pass-through.

Related

source

PortfolioOptimisers.returns_result_picker Function

julia

returns_result_picker(rd::ReturnsResult, brt::Bool) -> ReturnsResult

Return a ReturnsResult appropriate for benchmark-tracking optimisations.

This helper inspects the ReturnsResult's benchmark field B and the boolean flag brt (benchmark-tracking). If brt is true and a benchmark B is present it returns a new ReturnsResult in which asset returns X have the benchmark removed (i.e. X - B or broadcast X .- B for vector benchmarks). If brt is false or no benchmark is present, the original ReturnsResult is returned unchanged.

Arguments

rd: A ReturnsResult object containing asset, factor and/or benchmark returns.
brt: Boolean flag indicating whether benchmark-tracking behaviour should be applied. When true, asset returns are adjusted by subtracting the benchmark B (if present).

Returns

rd::ReturnsResult:
- If brt is true and a benchmark B is present: A new ReturnsResult with adjusted asset returns
- Otherwise: The rd is returned unchanged.

Details

When an adjustment is required and B is present, a new ReturnsResult is returned leaving the original rd unmodified.
If no adjustment is required or B is nothing, the original ReturnsResult is returned unchanged.
For vector benchmarks (VecNum) subtraction uses broadcasting (X .- B) to subtract the per-observation benchmark from each asset column (index tracking).
For matrix benchmarks (MatNum) subtraction uses matrix subtraction (X - B).
Other fields (nx, nf, F, ts, iv, ivpa) are preserved in the returned object.

Examples

julia

julia> rd = ReturnsResult(; nx = ["A", "B"], X = [0.10 0.20; 0.30 0.40], nb = ["BM"],
                          B = [0.01; 0.02])
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ Vector{String}: ["BM"]
     B ┼ Vector{Float64}: [0.01, 0.02]
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

julia> rd2 = returns_result_picker(rd, false)  # no change when brt is false
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ Vector{String}: ["BM"]
     B ┼ Vector{Float64}: [0.01, 0.02]
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

julia> rd === rd2
true

julia> rd3 = returns_result_picker(rd, true)
ReturnsResult
    nx ┼ Vector{String}: ["A", "B"]
     X ┼ 2×2 Matrix{Float64}
    nf ┼ nothing
     F ┼ nothing
    nb ┼ nothing
     B ┼ nothing
    ts ┼ nothing
    iv ┼ nothing
  ivpa ┴ nothing

julia> rd.X .- rd.B == rd3.X
true

Related

source

PortfolioOptimisers.Prices_RR Type

julia

Prices_RR

Union of the two data levels cross-validation folds can be computed on: returns-level (AbstractReturnsResult) and price-level (AbstractPricesResult) data.

Fold generation only needs an observation count (cv_nobs) and a timestamp vector (cv_timestamps), so Base.split and n_splits accept either level. Price-level splitting is what lets a Pipeline be cross-validated on its input rows, keeping stateful preprocessing inside the fold.

source

Price-level data

PortfolioOptimisers.AbstractPricesResult Type

julia

abstract type AbstractPricesResult <: AbstractResult

Abstract supertype for all price-level data result types in PortfolioOptimisers.jl.

All concrete types representing price-level data should be subtypes of AbstractPricesResult. Defined alongside AbstractReturnsResult so cross-validation splitting, preprocessing, and prediction can dispatch on either data level.

Related

source

PortfolioOptimisers.PricesResult Type

julia

struct PricesResult{__T_X, __T_F, __T_B, __T_iv, __T_ivpa} <: AbstractPricesResult

A container for aligned, time-indexed price-level data in PortfolioOptimisers.jl.

PricesResult is the prices-level mirror of ReturnsResult: it bundles asset prices with optional factor, benchmark, and implied volatility series, all as TimeSeries.TimeArrays. It is the input to price-level preprocessing estimators and prices-to-returns conversion, and the type that defines timestamp-window slicing for pipeline cross-validation via port_opt_view.

The asset price series X is the master clock: port_opt_view selects observation windows on X and aligns the other series to the selected timestamps.

Fields

X: Asset price data (observations × assets). The master clock for timestamp-window slicing.
F: Optional factor price data (observations × factors).
B: Optional benchmark price data (observations × 1) or (observations × assets).
iv: Optional implied volatility data (observations × assets).
ivpa: Implied volatility risk premium adjustment, if a vector (assets × 1).

Constructors

julia

PricesResult(;
    X::TimeSeries.TimeArray,
    F::Option{<:TimeSeries.TimeArray} = nothing,
    B::Option{<:TimeSeries.TimeArray} = nothing,
    iv::Option{<:TimeSeries.TimeArray} = nothing,
    ivpa::Option{<:Num_VecNum} = nothing,
) -> PricesResult

Keywords correspond to the struct's fields.

Validation

!isempty(X).
If F is not nothing: !isempty(F).
If B is not nothing: !isempty(B), and size(values(B), 2) in (1, size(values(X), 2)).
If iv is not nothing: !isempty(iv), all(x -> x >= 0, values(iv)), all(x -> isfinite(x), values(iv)), and size(values(iv), 2) == size(values(X), 2).
If ivpa is not nothing: all(x -> x > 0, ivpa), all(x -> isfinite(x), ivpa); if a vector, length(ivpa) == size(values(X), 2).

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3),
                     [100.0 101.0; 102.0 103.0; 104.0 105.0], ["A", "B"]);

julia> pr = PricesResult(; X = X);

julia> size(values(pr.X))
(3, 2)

Related

source

PortfolioOptimisers.port_opt_view Method

julia

port_opt_view(
    pr::PricesResult,
    _::Colon,
    _::Colon
) -> PricesResult

Return a view of the PricesResult for the observation window i of the asset price series X.

The asset price series is the master clock: i selects rows of X, and the factor, benchmark, and implied volatility series are aligned to the selected timestamps (rows whose timestamps are absent from a series are dropped from that series).

Arguments

pr: A PricesResult object.
i: Observation window into the rows of pr.X. Either integer indices (AbstractVector{<:Integer}, AbstractRange, or Colon) or a vector of timestamps (AbstractVector{<:Dates.AbstractTime}).

Returns

new_pr::PricesResult: A new PricesResult containing only the data for the selected window.

Details

Colon returns pr unchanged.
Integer windows index the rows of pr.X directly; the selected timestamps are then used to align F, B, and iv.
Timestamp windows are applied to all series directly.
ivpa is per-asset and passes through unchanged.

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3),
                     [100.0 101.0; 102.0 103.0; 104.0 105.0], ["A", "B"]);

julia> pr = PricesResult(; X = X);

julia> pv = PortfolioOptimisers.port_opt_view(pr, 2:3);

julia> first(timestamp(pv.X))
2020-01-02

julia> size(values(pv.X))
(2, 2)

Related

source

Preprocessing estimators

Preprocessing estimators transform price- or returns-level data under a fit/apply contract: fit_preprocessing learns whatever state the transformation needs from a training window — the surviving asset universe, imputation parameters, thresholds — and apply_preprocessing replays that state on unseen windows, so no information flows from test data back into the transformation.

They are ordinary estimators and know nothing about pipelines. A Pipeline drives them through these two verbs, exactly as it drives prior estimators through prior or optimisers through optimise.

PortfolioOptimisers.AbstractPreprocessingEstimator Type

julia

abstract type AbstractPreprocessingEstimator <: AbstractEstimator

Abstract supertype for all preprocessing estimator types in PortfolioOptimisers.jl.

Preprocessing estimators transform price or returns data (prices-to-returns conversion, missing-data filtering, imputation) under a fit/apply contract. Fitting one on training data with fit_preprocessing produces a result carrying any fitted state — imputation parameters, thresholds, and the selected asset universe — which apply_preprocessing then replays on unseen data so train and test windows are transformed consistently. Stateless preprocessing estimators carry no state, and applying them is equivalent to running them.

They are ordinary estimators: they know nothing about pipelines. A Pipeline drives them through the same fit/apply verbs any other caller would use.

All concrete preprocessing estimators should subtype one of the two data-level subtypes:

AbstractPricesPreprocessingEstimator: consumes and produces price-level data (PricesResult).
AbstractReturnsPreprocessingEstimator: consumes and produces returns-level data (ReturnsResult).

Related

source

PortfolioOptimisers.AbstractPricesPreprocessingEstimator Type

julia

abstract type AbstractPricesPreprocessingEstimator <: AbstractPreprocessingEstimator

Abstract supertype for preprocessing estimators that consume and produce price-level data.

Concrete subtypes transform a PricesResult into another PricesResult.

Related

source

PortfolioOptimisers.AbstractReturnsPreprocessingEstimator Type

julia

abstract type AbstractReturnsPreprocessingEstimator <: AbstractPreprocessingEstimator

Abstract supertype for preprocessing estimators that consume and produce returns-level data.

Concrete subtypes transform a ReturnsResult into another ReturnsResult.

Related

source

PortfolioOptimisers.AbstractPreprocessingResult Type

julia

abstract type AbstractPreprocessingResult <: AbstractResult

Abstract supertype for all preprocessing result types in PortfolioOptimisers.jl.

Preprocessing results are produced by fit_preprocessing on training data. They carry the fitted state needed to apply the same transformation to unseen data — imputation parameters, thresholds, and the selected asset universe. Stateless preprocessing estimators produce results that carry only their configuration.

All concrete preprocessing results should subtype one of the two data-level subtypes, AbstractPricesPreprocessingResult or AbstractReturnsPreprocessingResult, so a caller can replay each fitted transformation at the data level it applies to.

Related

source

PortfolioOptimisers.AbstractPricesPreprocessingResult Type

julia

abstract type AbstractPricesPreprocessingResult <: AbstractPreprocessingResult

Abstract supertype for preprocessing results that apply to price-level data (PricesResult).

Related

source

PortfolioOptimisers.AbstractReturnsPreprocessingResult Type

julia

abstract type AbstractReturnsPreprocessingResult <: AbstractPreprocessingResult

Abstract supertype for preprocessing results that apply to returns-level data (ReturnsResult).

Related

source

PortfolioOptimisers.fit_preprocessing Function

julia

fit_preprocessing(est::AbstractPreprocessingEstimator, data) -> fitted

Fit a preprocessing estimator on a data window and return the fitted object consumed by apply_preprocessing.

The fitted object carries whatever state the transformation needs to be replayed consistently on unseen data — imputation parameters, thresholds, and the selected asset universe. Stateless preprocessing estimators return themselves.

Interfaces

Concrete preprocessing estimators must implement:

fit_preprocessing(est::MyPreprocessing, data) -> fitted: Compute the fitted state from the training window.
apply_preprocessing(fitted, data) -> data′: Transform a data window with the fitted state.

Arguments

est: The preprocessing estimator.
data: The training data window (PricesResult or ReturnsResult depending on the estimator's level).

Returns

fitted: The fitted object, typically an AbstractPreprocessingResult or the estimator itself when stateless.

Related

source

julia

fit_preprocessing(
    tts::TrainTestSplit,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult

Fit a TrainTestSplit by cutting the data into its two windows.

Unlike the other preprocessing estimators, the fitted result is not replayed on unseen data: a holdout's rows are a fact about the fitting window alone, so apply_preprocessing on a TrainTestSplitResult passes the window through unchanged.

Related

source

julia

fit_preprocessing(
    sel::AbstractAssetSelector,
    rd::AbstractReturnsResult
) -> AssetSelectorResult

Fit any AbstractAssetSelector by recording the asset universe select_assets keeps.

Validation

select_assets must return a mask whose length matches the number of asset columns.
The selection must keep at least one asset; a selector that empties the universe throws rather than passing a zero-asset problem downstream (the MissingDataFilter precedent).

Related

source

PortfolioOptimisers.apply_preprocessing Function

julia

apply_preprocessing(fitted, data) -> data′

Transform a data window with a fitted preprocessing object.

Applying the fitted object produced by fit_preprocessing on the training window to an unseen (test) window replays the same transformation — the same asset universe, the same imputation parameters — so train and test data stay consistent and no information flows from test to train.

Arguments

fitted: The fitted object returned by fit_preprocessing (an AbstractPreprocessingResult, or a stateless estimator).
data: The data window to transform.

Returns

data′: The transformed data window.

Related

source

julia

apply_preprocessing(
    res::AssetSelectorResult,
    rd::AbstractReturnsResult
) -> ReturnsResult

Replay a fitted asset universe on a data window.

The surviving columns are emitted in fitted order, not in the window's own column order, because the terminal weights are indexed by the training universe and assert_universe_aligned compares the two name vectors elementwise.

Validation

Every fitted asset name must be present in the window; a missing one throws rather than silently shrinking the universe.

Related

source

PortfolioOptimisers.is_missing_value Function

julia

is_missing_value(x) -> Bool

Return true when x counts as a missing observation in price-level data.

Price-level data stores absent observations either as missing or as NaN (the two conventions prices_to_returns already unifies).

Arguments

x: The value to test.

Returns

flag::Bool: true when x is missing or a NaN number.

Related

source

PortfolioOptimisers.PricesToReturns Type

julia

struct PricesToReturns{__T_ret_method, __T_padding, __T_collapse_args, __T_map_func, __T_join_method} <: AbstractPreprocessingEstimator

Preprocessing estimator converting price-level data into returns-level data.

PricesToReturns is the estimator form of prices_to_returns: it consumes a PricesResult and produces a ReturnsResult. It is stateless — applying it to any window simply runs the conversion — so its fitted object is the estimator itself.

Missing-data filtering is deliberately not part of this estimator (the corresponding prices_to_returns keywords are held at their permissive defaults); use MissingDataFilter and Imputer as separate, independently tunable steps.

Warning

Because this step is stateless, it does not define an asset universe. prices_to_returns drops assets that are entirely missing in the window being converted, so a training window in which an asset has no history produces a different universe from a clean test window. Precede this estimator with a MissingDataFilter (which fits the universe on the training window) and an Imputer (which fills the remaining gaps with training statistics) whenever a fitted transformation must be replayed on unseen windows; a Pipeline enforces this via assert_universe_aligned.

Fields

ret_method: Return calculation method (:simple or :log).
padding: Whether to pad missing values in the returns calculation.
collapse_args: Arguments for collapsing the time series (e.g. to lower frequency).
map_func: Optional function applied to the data before the returns calculation.
join_method: How asset, factor, and benchmark data are joined (:outer, :inner, etc.).

Constructors

julia

PricesToReturns(;
    ret_method::Symbol = :simple,
    padding::Bool = false,
    collapse_args::Tuple = (),
    map_func::Option{<:Function} = nothing,
    join_method::Symbol = :outer,
) -> PricesToReturns

Keywords correspond to the struct's fields.

Validation

ret_method in (:simple, :log).

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3),
                     [100.0 101.0; 102.0 103.0; 104.0 105.0], ["A", "B"]);

julia> pr = PricesResult(; X = X);

julia> rr = apply_preprocessing(PricesToReturns(), pr);

julia> size(rr.X)
(2, 2)

julia> rr.nx
2-element Vector{String}:
 "A"
 "B"

Related

source

PortfolioOptimisers.MissingDataFilter Type

julia

struct MissingDataFilter{__T_col_thr, __T_row_thr} <: AbstractPricesPreprocessingEstimator

Preprocessing estimator dropping assets and observations with excessive missing data from price-level data.

The asset universe is fitted state: the training window decides which assets survive (per-column missing fraction at most col_thr), and applying the fitted result to an unseen window subsets it to that same universe — so train weights and test returns always refer to the same assets. Observation (row) filtering is window-local: rows whose missing fraction across the surviving assets exceeds row_thr are dropped from whichever window is being transformed.

This estimator supersedes the missing_col_percent/missing_row_percent keywords of prices_to_returns, making the thresholds fitted state and independently tunable. Only the asset series X (and the matching implied volatility columns) participate; factor and benchmark series pass through unchanged.

Fields

col_thr: Maximum allowed fraction (0, 1] of missing observations per asset column; assets above it are dropped from the universe at fit time.
row_thr: Maximum allowed fraction (0, 1] of missing assets per observation row; rows above it are dropped from the window being transformed.

Constructors

julia

MissingDataFilter(;
    col_thr::Number = 1.0,
    row_thr::Number = 1.0,
) -> MissingDataFilter

Keywords correspond to the struct's fields.

Validation

0 < col_thr <= 1.
0 < row_thr <= 1.

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3), [100.0 NaN; 102.0 NaN; 104.0 105.0],
                     ["A", "B"]);

julia> pr = PricesResult(; X = X);

julia> res = fit_preprocessing(MissingDataFilter(; col_thr = 0.5), pr);

julia> res.nx
1-element Vector{Symbol}:
 :A

Related

source

PortfolioOptimisers.MissingDataFilterResult Type

julia

struct MissingDataFilterResult{__T_nx, __T_row_thr} <: AbstractPricesPreprocessingResult

Fitted result of a MissingDataFilter.

Carries the asset universe selected on the training window plus the row threshold needed to transform further windows. Produced by fit_preprocessing, consumed by apply_preprocessing.

Fields

nx: Names of the assets that survived the training window (the fitted universe).
row_thr: Maximum allowed fraction (0, 1] of missing assets per observation row.

Related

source

PortfolioOptimisers.Imputer Type

julia

struct Imputer{__T_stat} <: AbstractPricesPreprocessingEstimator

Preprocessing estimator imputing missing price observations from per-asset statistics fitted on the training window.

The imputation parameters are fitted state: each asset's fill value is computed from the training window's observed (non-missing) prices with the configured Num_VecToScaM, and applying the fitted result to an unseen window fills that window's missing observations with the training values — never with statistics of the window being transformed, which is exactly the leakage a fit/apply contract exists to prevent.

Assets with no observed values in the training window get no fill value and are left untouched at apply time; combine with MissingDataFilter to drop them instead.

Fields

stat: Reducer computing an asset's fill value from its observed training prices (Num_VecToScaM).

Constructors

julia

Imputer(;
    stat::Num_VecToScaM = MedianValue(),
) -> Imputer

Keywords correspond to the struct's fields.

Examples

julia

julia> X = TimeArray(Date(2020, 1, 1):Day(1):Date(2020, 1, 3), [100.0 1.0; NaN 3.0; 104.0 5.0],
                     ["A", "B"]);

julia> pr = PricesResult(; X = X);

julia> res = fit_preprocessing(Imputer(), pr);

julia> pv = apply_preprocessing(res, pr);

julia> values(pv.X)[2, 1]
102.0

Related

source

PortfolioOptimisers.ImputerResult Type

julia

struct ImputerResult{__T_nx, __T_v} <: AbstractPricesPreprocessingResult

Fitted result of an Imputer.

Carries the per-asset fill values computed on the training window. Produced by fit_preprocessing, consumed by apply_preprocessing.

Fields

nx: Names of the assets with a fitted fill value.
v: Fill values, aligned with nx.

Related

source

Train/test splitting

A holdout split reserves the tail of the time-ordered observations as a test window and trains on the head. It comes in two forms: the free function train_test_split, which cuts data into a train/test pair, and the estimator TrainTestSplit (alias TTS), which carries the protocol inside a Pipeline as its first step — so every fitted step downstream sees the training window alone, and fit_predict(pipe, data) evaluates on the held-out window in one line.

Sizes are row counts (Integer) or fractions of the observations (AbstractFloat in (0, 1)). Giving one side makes the other its complement; giving both embargoes the rows between the two windows. See docs/adr/0031-holdout-split-as-a-pipeline-step.md.

The keyword form returns a bare (train, test) tuple; the estimator form, train_test_split(tts, data), returns the same TrainTestSplitResult a pipeline's split step produces, so one configured holdout can be reused inside and outside a pipeline.

PortfolioOptimisers.train_test_split Function

julia

train_test_split(rd::ReturnsResult; train_size, test_size) -> (train, test)
train_test_split(pr::PricesResult; train_size, test_size) -> (train, test)

Cut price- or returns-level data into a training window (the head) and a held-out test window (the tail).

The free-function form of TrainTestSplit; the windows are port_opt_views, so no data is copied. See safe_index for the sizing rules — complement when one side is given, embargo when both are.

Arguments

rd/pr: The data to split.
train_size: Training rows as a count (Integer) or a fraction (AbstractFloat in (0, 1)); nothing takes the complement of test_size.
test_size: Test rows, likewise; nothing takes the complement of train_size. With neither given the split is 75/25.

Returns

(train, test): The two windows, of the same type as the input.

Examples

julia

julia> rd = ReturnsResult(; nx = ["A"], X = reshape(collect(0.1:0.1:1.0), 10, 1));

julia> train, test = train_test_split(rd; test_size = 0.2);

julia> size(train.X, 1), size(test.X, 1)
(8, 2)

Related

source

julia

train_test_split(
    tts::TrainTestSplit,
    data::Union{AbstractPricesResult, AbstractReturnsResult}
) -> TrainTestSplitResult

Split data under a TrainTestSplit, returning both windows as a TrainTestSplitResult.

The estimator-form counterpart of the keyword form: train_test_split(rd; test_size = 0.2) hands back a bare (train, test) tuple, while this hands back the same fitted result a pipeline's split step produces, so a holdout configured once can be reused verbatim inside and outside a Pipeline.

Related

source

PortfolioOptimisers.TrainTestSplit Type

julia

struct TrainTestSplit{__T_train_size, __T_test_size} <: AbstractPreprocessingEstimator

Preprocessing estimator reserving the tail of the observations as a held-out test window.

The estimator form of train_test_split, and the way the holdout protocol enters a Pipeline: as the first step, it hands the training window to every step downstream and stashes the test window in its fitted TrainTestSplitResult. fit_predict(pipe, data) then evaluates the fitted workflow on that held-out window in one line.

It is the one preprocessing estimator that is not pinned to a data level: it splits whichever level the pipeline input provides, price or returns, since a holdout is a statement about rows, not about columns or units.

Replaying a fitted split on an unseen window is a pass-through — the fitted rows are training-window state, and applying them to new data would be meaningless — so predict(res, future_data) keeps working on genuinely new observations.

Warning

A pipeline containing a TrainTestSplit may not also be cross-validated: the split and the cross-validator are two evaluation protocols, and cross-validation already defines its own train/test windows. search_cross_validation rejects such a pipeline rather than silently shaving a second holdout off every fold.

Fields

train_size: Training observations as a count (Integer) or a fraction (AbstractFloat in (0, 1)); nothing takes the complement of test_size.
test_size: Test observations, likewise; nothing takes the complement of train_size.

Constructors

julia

TrainTestSplit(;
    train_size::Option{<:Number} = nothing,
    test_size::Option{<:Number} = nothing,
) -> TrainTestSplit

Keywords correspond to the struct's fields. Sizes follow safe_index: a row count (Integer) or a fraction of the observations (AbstractFloat in (0, 1)); one side given makes the other its complement; both given embargoes the rows between them; neither given splits 75/25.

Examples

julia

julia> pipe = Pipeline(;
                       steps = (TrainTestSplit(; test_size = 0.2), PricesToReturns(),
                                EmpiricalPrior(), EqualWeighted()));

julia> pipe.names
("split", "returns", "prior", "opt")

Related

source

PortfolioOptimisers.TrainTestSplitResult Type

julia

struct TrainTestSplitResult{__T_train, __T_test} <: AbstractResult

Fitted result of a TrainTestSplit, carrying both windows of the holdout.

The test window is the payoff: it is the data the fitted pipeline has never seen, and what fit_predict(pipe, data) predicts on. The train window is kept alongside it so the raw data the workflow was fitted on is retrievable from the result rather than having to be re-derived.

Both are port_opt_views of the input at whichever level the split ran (price or returns).

Fields

train: The training window: the head of the observations, and the data every downstream step is fitted on.
test: The held-out test window: the tail of the observations, which no fitted step has seen.

Related

source

PortfolioOptimisers.safe_index Function

julia

safe_index(
    lo::Union{Nothing, Number},
    hi::Union{Nothing, Number},
    N::Integer
) -> Tuple{Any, Any}
safe_index(
    lo::Union{Nothing, Number},
    hi::Union{Nothing, Number},
    N::Integer,
    D
) -> Tuple{Any, Any}

Return the (train, test) observation ranges of a holdout split over N time-ordered rows.

Training rows come from the head of the data and test rows from the tail, so the test window is always the most recent one. Each size is a row count (Integer) or a fraction of the observations (AbstractFloat in (0, 1)), resolved by split_count.

Neither given: the split falls at D (75 % train, 25 % test).
One given: the other side is its complement, so the two windows partition the data.
Both given: the head supplies lo training rows, the tail supplies hi test rows, and any rows between them are embargoed — they belong to neither window. This is how a gap between train and test is expressed.

Validation

Both windows are non-empty. A split whose sizes saturate the data on one side (train_size = N) leaves nothing to test on and throws.
The windows do not overlap: lo + hi <= N.

Related

source

PortfolioOptimisers.split_count Function

julia

split_count(s::Integer, N::Integer, name::Symbol) -> Int64

Resolve one side of a train/test split into a row count.

A size is either an Integer count of observations, or an AbstractFloat fraction of them in (0, 1). Counts saturate at N (asking for more rows than exist takes all of them); the safe_index window guards then reject a split that leaves either side empty.

Related

source

Asset selection infrastructure

Asset selectors are the returns-level preprocessing subfamily that restricts the asset universe. The universe chosen on the training window is the selector's fitted state, so a selector is safe inside cross-validation. The concrete selectors live in Asset selection; this is the seam they share.

PortfolioOptimisers.AbstractAssetSelector Type

julia

abstract type AbstractAssetSelector <: AbstractReturnsPreprocessingEstimator

Abstract supertype for returns-level preprocessing estimators that restrict the asset universe.

An asset selector answers one question on the training window — which asset columns survive? — and that answer is its fitted state. apply_preprocessing replays the fitted universe on unseen windows, so a selector is safe inside cross-validation: the selection is made on train data alone and never re-decided on test data.

Concrete subtypes implement a single method, select_assets; the family shares one fit_preprocessing and one apply_preprocessing. Selectors restrict columns only. Observation filtering is a price-level concern (MissingDataFilter), because a fitted transformation cannot decide which rows of an unseen window to drop without breaking the weights/returns alignment assert_universe_aligned enforces.

See docs/adr/0029-asset-selection-is-returns-preprocessing.md for the design rationale.

Related

source

PortfolioOptimisers.AssetSelectorResult Type

julia

struct AssetSelectorResult{__T_nx} <: AbstractReturnsPreprocessingResult

Fitted result of any AbstractAssetSelector.

Carries the asset universe selected on the training window. One result type serves the whole family: every selector differs in how it chooses the universe, never in what it stores.

Fields

nx: Names of the assets that survived the training window, in their original column order (the fitted universe).

Related

source

PortfolioOptimisers.select_assets Function

julia

select_assets(sel::AbstractAssetSelector, rd::AbstractReturnsResult) -> BitVector

Return the keep-mask over the asset columns of rd.

This is the single method a concrete AbstractAssetSelector must implement. It is called by fit_preprocessing on the training window only; the resulting universe is then replayed on every later window by apply_preprocessing.

Arguments

sel: The asset selector.
rd: The training-window returns data.

Returns

keep::BitVector: true for each asset column to retain, length(keep) == size(rd.X, 2).

Related

source

PortfolioOptimisers.find_complete_indices Function

julia

find_complete_indices(X::AbstractMatrix; dims::Int = 1) -> VecInt

Return the indices of columns (or rows) in matrix X that do not contain any missing or NaN values.

This function scans the specified dimension of the input matrix and returns the indices of columns (or rows) that are complete, i.e., contain no missing or NaN values.

Internal machinery — the caller-facing form is CompleteAssetSelector, which wraps the dims = 1 (complete-column) mode as a fit/apply estimator. The dims = 2 (complete-row) mode has no estimator form: dropping observations is a price-level concern (MissingDataFilter).

Arguments

X: Data matrix observations × features if the dims keyword does not exist or dims = 1, features × observations when dims = 2.
dims: Dimension along which to perform the computation.

Validation

dims in (1, 2).

Returns

res::VecInt: Indices of columns (or rows) in X that are complete.

Details

If dims == 2, the matrix is transposed and columns are checked.
Any column (or row) containing at least one missing or NaN value is excluded.
The result is a vector of indices of complete columns (or rows).

Examples

julia

julia> X = [1.0 2.0 NaN; 4.0 missing 6.0];

julia> PortfolioOptimisers.find_complete_indices(X)
1-element Vector{Int64}:
 1

julia> PortfolioOptimisers.find_complete_indices(X; dims = 2)
Int64[]

Related

source

Pre-processing ​

Prices to returns ​

Price-level data ​

Preprocessing estimators ​

Train/test splitting ​

Asset selection infrastructure ​

Pre-processing

Prices to returns

Price-level data

Preprocessing estimators

Train/test splitting

Asset selection infrastructure