Low-dimension ensemble prior

struct LowDimensionEnsemblePrior{T1<:Integer, T2<:Integer, T3<:Real, T4<:(Union{Nothing, var"#s1796"} where var"#s1796"<:Posdef), T5<:AbstractRNG, T6<:(Union{Nothing, var"#s1795"} where var"#s1795"<:Integer)} <: AbstractLowOrderPriorEstimator_A

Forecasts the next price relative and its covariance by averaging many low-dimensional lagged regressions, weighted by their in-sample fit.

This is the prior of Xi, Li, Song and Ning (2023). It treats the market as one high-dimensional system and draws many small subsystems of subsystem_size assets from it at random. In each subsystem, the price relative of every asset is regressed on the price relatives of the subsystem's assets one period earlier. The forecast of a subsystem counts in proportion to a Gaussian kernel of its in-sample mean squared error. Each subsystem also gives a predictive covariance, and the aggregate weighs it by the product of the kernel weights of the two assets.

The mean and the covariance come from the same regressions. So the risk term of a programme that reads this prior measures the uncertainty of the forecast that its return term reads. A window of $w$ regression pairs needs $w + 1$ rows.

Mathematical definition

The rows are the periods $t - w$ to $t$. Subsystem $l$ holds the assets $\mathcal{A}_l = (l_1, \ldots, l_s)$. For every asset $k$, every subsystem $l$ and $i = 0, \ldots, w - 1$,

\[\begin{align} x_{t-i,k} &= \sum_{j = 1}^{s} \beta^{(l)}_{k,j}\, x_{t-i-1,l_j} + \varepsilon^{(l)}_{t-i,k}\,,\\ \hat{\boldsymbol{\beta}}^{(l)}_k &= \left(\mathbf{X}^{(l)}\right)^{+} \boldsymbol{y}_k\,,\\ \hat{x}^{(l)}_{t+1,k} &= \boldsymbol{x}_{t,\mathcal{A}_l}^\intercal \hat{\boldsymbol{\beta}}^{(l)}_k\,,\\ R^{(l)}_k &= \frac{1}{w} \sum_{i = 0}^{w - 1} \left(x_{t-i,k} - \boldsymbol{x}_{t-i-1,\mathcal{A}_l}^\intercal \hat{\boldsymbol{\beta}}^{(l)}_k\right)^2\,,\\ v^{(l)}_k &= \frac{\exp\left(-R^{(l)}_k / \sigma^2\right)}{\sum_{m = 1}^{L} \exp\left(-R^{(m)}_k / \sigma^2\right)}\,,\\ \hat{x}_{t+1,k} &= \sum_{l = 1}^{L} v^{(l)}_k\, \hat{x}^{(l)}_{t+1,k}\,,\\ \hat{\boldsymbol{\mu}} &= \hat{\boldsymbol{x}}_{t+1} - \boldsymbol{1}\,,\\ S^{(l)}_{pq} &= \frac{1}{w - 1} \sum_{i = 0}^{w - 1} \left(x_{t-i,l_p} - \bar{x}_{l_p}\right) \left(x_{t-i,l_q} - \bar{x}_{l_q}\right)\,,\\ \hat{\mathbf{\Sigma}}^{(l)} &= \mathbf{B}^{(l)} \mathbf{S}^{(l)} \mathbf{B}^{(l)\intercal}\,,\\ \left(\hat{\mathbf{\Sigma}}_{t+1}\right)_{kh} &= \frac{\sum_{l = 1}^{L} v^{(l)}_k v^{(l)}_h\, \hat{\Sigma}^{(l)}_{kh}}{\sum_{l = 1}^{L} v^{(l)}_k v^{(l)}_h}\,. \end{align}\]

Where:

• $\boldsymbol{x}_t$: Price relative vector of period $t$, each asset's price over its price one period earlier.
• $x_{t,k}$: Entry $k$ of $\boldsymbol{x}_t$, and $\boldsymbol{x}_{t,\mathcal{A}_l}$ the entries of the assets of $\mathcal{A}_l$.
• $t$: Period index, counted from one.
• $w$: Regression window, the number of regression pairs. The rows number $w + 1$.
• $L$: Number of subsystems.
• $s$: Number of assets in each subsystem.
• $\mathcal{A}_l$: Assets of subsystem $l$, drawn uniformly without replacement. The same subsystems serve every asset $k$.
• $\beta^{(l)}_{k,j}$, $\varepsilon^{(l)}_{t-i,k}$: Coefficient and error term of the regression of asset $k$ on subsystem $l$, with no intercept.
• $\mathbf{X}^{(l)}$: $w \times s$ matrix whose row $i + 1$ is $\boldsymbol{x}_{t-i-1,\mathcal{A}_l}^\intercal$.
• $(\cdot)^{+}$: Moore–Penrose pseudo-inverse.
• $\boldsymbol{y}_k = (x_{t,k}, \ldots, x_{t-w+1,k})^\intercal$: Targets of the regression of asset $k$.
• $\hat{\boldsymbol{\beta}}^{(l)}_k$: Least-squares coefficient vector of asset $k$ on subsystem $l$.
• $\hat{x}^{(l)}_{t+1,k}$: Forecast of subsystem $l$ for asset $k$.
• $R^{(l)}_k$: In-sample mean squared error of that regression.
• $\sigma$: Kernel bandwidth.
• $v^{(l)}_k$: Kernel weight of subsystem $l$ for asset $k$. The weights of one asset sum to one.
• $\hat{\boldsymbol{x}}_{t+1}$: Price Relative Forecast for period $t + 1$, made after the row of period $t$.
• $\hat{x}_{t+1,k}$: Entry $k$ of $\hat{\boldsymbol{x}}_{t+1}$.
• $\hat{\boldsymbol{\mu}}$: Forecast return vector, the Price Relative Forecast less one.
• $\mathbf{S}^{(l)}$: $s \times s$ sample covariance of the price relatives of the assets of $\mathcal{A}_l$ over the last $w$ rows, with entries $S^{(l)}_{pq}$.
• $\bar{x}_{l_p} = \frac{1}{w} \sum_{i = 0}^{w - 1} x_{t-i,l_p}$: Mean of the price relatives of asset $l_p$ over the last $w$ rows.
• $\mathbf{B}^{(l)}$: $N \times s$ matrix whose row $k$ is $\hat{\boldsymbol{\beta}}^{(l)\intercal}_k$.
• $\hat{\mathbf{\Sigma}}^{(l)}$: Predictive covariance of subsystem $l$, with entries $\hat{\Sigma}^{(l)}_{kh}$.
• $\hat{\mathbf{\Sigma}}_{t+1}$: Predictive covariance of the price relatives of period $t + 1$, made after the row of period $t$.
• $N$: Number of assets.

These are the equations 4 to 10 of the paper. When $\mathbf{X}^{(l)}$ has full column rank, the pseudo-inverse gives the paper's $(\mathbf{X}^{(l)\intercal} \mathbf{X}^{(l)})^{-1} \mathbf{X}^{(l)\intercal} \boldsymbol{y}_k$. When the window has fewer pairs than the subsystem has assets, it gives the least-squares coefficients of least norm.

Every $\hat{\mathbf{\Sigma}}^{(l)}$ is positive semidefinite. But $\hat{\mathbf{\Sigma}}_{t+1}$ averages them under weights that differ from entry to entry, so it need not be positive semidefinite itself.

Fields

• n_subsystems: The number of subsystems that every fit draws, the paper's $L$.

• subsystem_size: The number of assets in every subsystem, the paper's $s$. When the rows hold fewer assets, the fit uses all of them.

• sigma: The kernel bandwidth of the subsystem weights, the paper's $\sigma$.

• pdm: The positive definite repair of the aggregated covariance, or nothing to leave the aggregate as it is. On short windows the aggregate is usually indefinite.

• rng: Random number generator.

• seed: Optional seed. If set, the draws come from a copy of rng seeded with it, so rng does not advance and every call draws the same values. If nothing, the draws come from rng itself.

Constructors

LowDimensionEnsemblePrior(; n_subsystems::Integer = 300, subsystem_size::Integer = 3, sigma::Real = 0.025, pdm::Option{<:Posdef} = Posdef(), rng::Random.AbstractRNG = Random.default_rng(), seed::Option{<:Integer} = nothing) -> LowDimensionEnsemblePrior

Keywords correspond to the struct's fields. The defaults are the paper's. The paper scans $L$ from 100 to 2000 and finds that the mean squared prediction error converges once $L$ exceeds 300. It sets three assets in each subsystem and a kernel bandwidth of 0.025.

Validation

• n_subsystems >= 1. A DomainError is thrown otherwise.
• subsystem_size >= 1. A DomainError is thrown otherwise.
• sigma > 0 and finite. A DomainError is thrown otherwise.

Examples

julia> LowDimensionEnsemblePrior()LowDimensionEnsemblePrior    n_subsystems ┼ Int64: 300  subsystem_size ┼ Int64: 3           sigma ┼ Float64: 0.025             pdm ┼ Posdef                 │      alg ┼ UnionAll: NearestCorrelationMatrix.Newton                 │   kwargs ┴ @NamedTuple{}: NamedTuple()             rng ┼ Random.TaskLocalRNG: Random.TaskLocalRNG()            seed ┴ nothing

Related

• AbstractLowOrderPriorEstimator_A
• LowOrderPrior
• prior
• Posdef
• LowDimensionEnsemblePortfolio
• PriorExpectedReturns

References

• [110] W. Xi, Z. Li, X. Song and H. Ning. Online portfolio selection with predictive instantaneous risk assessment. Pattern Recognition 144, 109872 (2023). Equations 4 to 10 and Section 5.3.

prior(pe::LowDimensionEnsemblePrior, X::MatNum, F::Option{<:MatNum} = nothing, pnl::Option{<:AssetPanel} = nothing; dims::Int = 1, kwargs...)

Computes the low-dimension ensemble prior of the rows, with the forecast return as mu and the aggregated predictive covariance as sigma.

The fit reads the Coverage Universe of the rows, as every prior does. An asset with a non-finite return, or with an inactive row of the panel, at any observation is left out. Both moments are expanded back to the full universe with NaN outside it, so the Investable Mask of a consumer reads the departure. The returned rows keep every column as given.

Algorithm

1. Orient X to observations by assets, giving Xf.
2. Reduce Xf to its Coverage Universe with coverage_reduction, giving the mask cmsk and the reduced returns X of T rows and N assets.
3. Check that T >= 3.
4. Form the price relatives P = 1 + X. Split them into the targets Y, every row but the first, the regressors Z, every row but the last, and the last row z.
5. Clip the subsystem size to N, giving s.
6. Resolve the random number generator from rng and seed, and draw L subsystems idxs of s assets each, without replacement. One draw serves every target asset. With a seed, every call draws the same subsystems. Without one, every call draws new ones, so a rule that refits the prior at every period draws new subsystems at every period, as the paper does.
7. Solve one least-squares problem for each subsystem, for all the targets at once, giving the s × N coefficient matrices Bs. Julia's \ gives the coefficients of least norm when the window has fewer pairs than s.
8. Compute the L × N in-sample mean squared errors R and the subsystem forecasts F.
9. Subtract the smallest error of each asset from its column of R, giving E. This leaves every kernel ratio unchanged. It also stops the kernel from underflowing to zero for every subsystem when the errors are large against $\sigma^2$.
10. Compute the kernel weights V from E, and scale each column to sum to one.
11. Compute mu as the weighted forecasts less one.
12. For each pair of assets, find the smallest sum m of their two shifted errors over the subsystems. The covariance weight of subsystem l for the pair is the kernel of its sum less m. The two shifts cancel in the ratio, and the largest weight of each pair is one, so no denominator underflows to zero.
13. Accumulate the weighted subsystem covariances in num and the weights in den, and divide, giving sigma.
14. Symmetrise sigma, because a matrix product need not be symmetric to the bit.
15. Repair sigma with posdef under pdm, and expand mu and sigma to the full universe with expand_moment.

Arguments

• pe: The ensemble prior estimator.
• X: Asset returns matrix (observations × assets).
• F: Factor returns matrix, ignored.
• pnl: Asset Panel, whose active mask enters the Coverage Universe.
• dims: Dimension along which to perform the computation.
• kwargs...: Ignored.

Validation

• size(X, 1) >= 3 after orientation, because two regression pairs are the fewest whose regressor covariance exists. An ArgumentError is thrown otherwise.

Returns

• pr::LowOrderPrior: The rows as given, the forecast mean and the predictive covariance, with NaN outside the Coverage Universe.

Related

• LowDimensionEnsemblePrior
• LowOrderPrior
• prior
• coverage_reduction
• expand_moment