Cross-Sectional Regression (a): private API

Types

PortfolioOptimisers.CrossSectionalLoopBuffers — Type
struct CrossSectionalLoopBuffers{__T_w, __T_A, __T_y, __T_Ac, __T_yc, __T_D, __T_Q, __T_G, __T_b, __T_gram, __T_v}

Holds the buffers that one task of the loop of cross_sectional_regression gathers each observation into.

This type is an implementation detail and is not intended for direct use. cross_sectional_loop_buffers makes it, and the loop of cross_sectional_block_regression reads it. Each buffer holds every asset, and an observation reads its first rows. The loop gives the solve the fields b, gram and v as the workspace that cross_sectional_solve reads.

Fields

  • w: w: The gathered weights of the observation.
  • A: A: The gathered exposures of the observation, assets × factors.
  • y: y: The gathered returns of the observation.
  • Ac: Ac: The centred exposures of an intercept, assets × factors. It has no row without an intercept.
  • yc: yc: The centred returns of an intercept. It has no entry without an intercept.
  • D: D: The weighted design, assets × factors.
  • Q: Q: The factors of the column-pivoted QR of the weighted design, assets × factors.
  • G: G: The pivoted columns that the leverage check divides, assets × factors.
  • b: b: The work vector of a solve through the QR, of length max(assets, factors).
  • gram: gram: The Gram matrix of a solve of the normal equations, factors × factors.
  • v: v: The right side of a solve of the normal equations, of length factors.

Related

source

Functions

PortfolioOptimisers.cross_sectional_design_mask — Function
cross_sectional_design_mask(
    Z::AbstractArray{<:Union{Number, AbstractJuMPScalar}, 3},
    X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    W::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> BitMatrix

Return the eligibility mask of a cross-sectional design, after checking its three arrays.

An (observation, asset) pair is eligible when its cross-sectional weight is positive. The weight alone decides. A pair that a zero weight excludes may carry a missing return or a missing exposure, and a pair with a positive weight may not.

Arguments

  • Z::Arr3Num: Exposure tensor assets × factors × observations.
  • X::MatNum: Asset returns matrix observations × assets.
  • W::MatNum: Cross-sectional weights matrix observations × assets.

Validation

  • !isempty(Z), !isempty(X) and !isempty(W).
  • size(Z, 3) == size(X, 1) and size(Z, 1) == size(X, 2).
  • size(W) == size(X).
  • all(isfinite, W) and all(x -> x >= 0, W).
  • Every pair with a positive weight carries a finite return and finite exposures. The IsNonFiniteError names the observation, the asset and the weight.

Returns

  • act::BitMatrix: Eligibility mask observations × assets, true where the weight is positive.

Related

source
PortfolioOptimisers.cross_sectional_residual_dof — Function
cross_sectional_residual_dof(edof::Nothing, n::AbstractVector{<:Integer}, neff::VecNum)
    -> Nothing
cross_sectional_residual_dof(edof::VecNum, n::AbstractVector{<:Integer}, neff::VecNum)
    -> VecNum

Return the degrees of freedom of each variance that reads the residuals of a cross-sectional fit, or nothing when the fit records no residual count.

The fit spends its parameters on the assets of each observation, not on the observations of each asset. So the $\sum_{t} n_{t}$ residuals keep the fraction $\phi$ of their degrees of freedom, and a variance that reads $n_{i}^{\mathrm{eff}}$ effective observations of asset $i$ keeps that fraction of them. The fraction spreads the spend evenly over the assets.

Mathematical definition

\[\begin{align} \phi &= \dfrac{\sum_{t} \nu_{t}}{\sum_{t} n_{t}}\,, \\ \nu_{i} &= \phi \, n_{i}^{\mathrm{eff}}\,. \end{align}\]

Where:

  • $\phi$: Fraction of the degrees of freedom that the fit leaves.
  • $\nu_{t}$: Residual count of the fit of observation $t$, the entry t of edof.
  • $n_{t}$: Number of eligible assets at observation $t$, the entry t of n.
  • $\nu_{i}$: Degrees of freedom of the variance of asset $i$.
  • $n_{i}^{\mathrm{eff}}$: Effective count of the observations of asset $i$, the entry i of neff.

On a balanced panel of $T$ observations and $N$ assets with no intercept and designs of full rank, $\nu_{i} = T (N - K) / N$ when $n_{i}^{\mathrm{eff}} = T$.

Algorithm

The method that Julia selects is the algorithm.

  1. edof is nothing: the fit records no count, as a fit that a caller builds by hand does, so the degrees of freedom are not known.
  2. edof is a vector: take $\phi$, and scale neff by it.

Arguments

  • edof: The residual count of each observation, the field of CrossSectionalRegression, or nothing.
  • n: The count of eligible assets of each observation.
  • neff: The effective count of the observations of each asset that the variance reads.

Returns

  • nu::Option{<:VecNum}: The degrees of freedom of each variance, or nothing.

Related

source
PortfolioOptimisers.cross_sectional_finite_exposures — Function
cross_sectional_finite_exposures(
    Z::AbstractArray{<:Union{Number, AbstractJuMPScalar}, 3}
) -> BitMatrix

Return the mask of the (observation, asset) pairs whose exposure to every factor is finite.

The pass reads Z in its memory order: the observation axis is the outer loop, and the asset axis is the inner loop.

Arguments

  • Z::Arr3Num: Exposure tensor assets × factors × observations.

Returns

  • fin::BitMatrix: Mask observations × assets, true where every exposure of the pair is finite.

Related

source
PortfolioOptimisers.cross_sectional_block_regression — Function
cross_sectional_block_regression(cre::Union{CrossSectionalLinearRegression,
                                            CrossSectionalTargetRegression},
                                 Z::Arr3Num, X::MatNum, W::MatNum,
                                 t0::Integer) -> CrossSectionalRegression
cross_sectional_block_regression(cre::AbstractCrossSectionalRegressionEstimator,
                                 Z::Arr3Num, X::MatNum, W::MatNum,
                                 t0::Integer) -> CrossSectionalRegression

Fit one regression per observation of a block that follows t0 fitted observations, as cross_sectional_regression fits it.

A refusal names the observation at the row t of X as observation t0 + t. So the carry fold of a CrossSectionalFactorPrior, which fits the new observations of a step alone, names the observation that the batch fit names. The method for an estimator that the library does not know calls cross_sectional_regression on the block, and the estimator numbers the observations of the block itself.

Algorithm

The algorithm of cross_sectional_regression, with t0 + t as the index of the observation that cross_sectional_coefficients receives.

Arguments

  • cre: Cross-sectional regression estimator.
  • Z::Arr3Num: Exposure tensor assets × factors × observations.
  • X::MatNum: Asset returns matrix observations × assets.
  • W::MatNum: Cross-sectional weights matrix observations × assets.
  • t0::Integer: Number of fitted observations before the block.

Validation

Returns

  • csr::CrossSectionalRegression: The fit of the block.

Related

source
PortfolioOptimisers.cross_sectional_coefficients — Function
cross_sectional_coefficients(
    cre::CrossSectionalLinearRegression,
    A::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    w::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    t::Integer;
    D,
    F,
    ws
) -> CrossSectionalSolveResult

Return the factor returns and the residual count of one observation, through the member's own solve.

CrossSectionalLinearRegression scales the design and the target by sqrt.(w) and hands them to cross_sectional_solve. The square root of a Rational weight is a float, so that solve runs in floating point for a Rational panel. CrossSectionalTargetRegression refuses an empty cross-section, and hands the unscaled pair to cross_sectional_target_solve, with w as the observation weights of the target and the weighted design sqrt.(w) .* A for the rank test of its solve algorithm.

Arguments

  • cre: Cross-sectional regression estimator.
  • A::MatNum: Exposures of the eligible assets, eligible assets × factors, already demeaned when an intercept is fitted.
  • y::VecNum: Returns of the eligible assets, already demeaned when an intercept is fitted.
  • w::VecNum: Cross-sectional weights of the eligible assets.
  • t::Integer: Index of the observation.
  • D::MatNum: Weighted design sqrt.(w) .* A. A caller that weighted the design already passes it.
  • F: Column-pivoted QR of D that the caller already took, or nothing. The solve reads it when its member reads a QR.
  • ws::NamedTuple: Workspace of the solve, as cross_sectional_solve reads it, or (;). The target regression does not read it.

Validation

Returns

  • res::CrossSectionalSolveResult: f, the factor returns of the observation, of length size(A, 2), and edof, the residual count of the fit of the factors, with no account of an intercept.

Related

source
PortfolioOptimisers.cross_sectional_column_drop — Function
cross_sectional_column_drop(
    fit_cols,
    D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    r::Integer;
    F
) -> CrossSectionalSolveResult

Return the factor returns and the residual count of a rank-deficient observation that fits only the columns its rank test keeps.

Algorithm

  1. Return zero for every factor when r is zero. The fit spends no parameter, so its residual count is size(D, 1).
  2. Keep the columns that the first r pivots of the column-pivoted QR F of D name, in their order on the factor axis, with cross_sectional_kept_columns.
  3. Fit the kept columns with fit_cols, and give every other factor a return of zero. The residual count is the count of that fit.

Arguments

  • fit_cols: Function of the kept column indices that returns their coefficients and the residual count of their fit, a CrossSectionalSolveResult.
  • D::MatNum: Weighted design of one observation, eligible assets × factors.
  • y::VecNum: Target of the observation. Its type and the type of D give the type of the zero answer of rank zero.
  • r::Integer: Rank of D, read by cross_sectional_rank, below size(D, 2).
  • F: Column-pivoted QR of D, as LinearAlgebra.qr(D, LinearAlgebra.ColumnNorm()) returns it. A caller that read the rank off a factorisation passes it, so the design is factorised once.

Returns

  • res::CrossSectionalSolveResult: f, the factor returns of the observation, of length size(D, 2), zero at each dropped factor, and edof, the residual count of the fit of the kept columns.

Related

source
PortfolioOptimisers.cross_sectional_kept_columns — Function
cross_sectional_kept_columns(
    D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    r::Integer;
    F
) -> Vector{Int64}

Return the columns of a rank-deficient weighted design that DependentColumnDrop keeps, in their order on the factor axis.

They are the columns that the first r pivots of the column-pivoted QR of D name. cross_sectional_column_drop fits them, and cross_sectional_mark_dropped! marks the others, so the fit and its mark read one pivot.

Arguments

  • D::MatNum: Weighted design of one observation, eligible assets × factors.
  • r::Integer: Rank of D, read by cross_sectional_rank.
  • F: Column-pivoted QR of D, as LinearAlgebra.qr(D, LinearAlgebra.ColumnNorm()) returns it. A caller that read the rank off a factorisation passes it, so the design is factorised once.

Returns

  • keep::Vector{Int}: Indices of the kept columns, sorted.

Related

source
PortfolioOptimisers.cross_sectional_drop_mask — Function
cross_sectional_drop_mask(alg::AbstractCrossSectionalSolveAlgorithm, T::Integer,
                          K::Integer) -> Nothing
cross_sectional_drop_mask(alg::DependentColumnDrop, T::Integer, K::Integer) -> Matrix{Bool}

Return the empty dropped-column mask of a fit, or nothing for a solve algorithm that drops no column.

Only DependentColumnDrop gives a factor a return of zero by the rule, so only its fit states the mask. Every other member either solves every column, or answers with the pseudo-inverse, whose standard errors the diagnostics already read.

Arguments

  • alg: Cross-sectional solve algorithm.
  • T::Integer: Number of observations.
  • K::Integer: Number of factors.

Returns

  • dc::Option{<:Matrix{Bool}}: All false, T × K, or nothing.

Related

source
PortfolioOptimisers.cross_sectional_bit_mask — Function
cross_sectional_bit_mask(dc::Nothing) -> Nothing
cross_sectional_bit_mask(dc::AbstractMatrix{Bool}) -> BitMatrix

Pack a mask that the fits of the observations wrote into a BitMatrix, or keep nothing.

The fits write a Matrix{Bool}, whose entries are bytes, because two threads that write the rows of one word of a BitMatrix would race. The result stores the packed mask.

Arguments

  • dc: The mask, or nothing.

Returns

  • dc::Option{<:BitMatrix}: The packed mask, or nothing.

Related

source
PortfolioOptimisers.cross_sectional_mark_dropped! — Function
cross_sectional_mark_dropped!(dc::Nothing, A::MatNum, w::VecNum, t::Integer;
                              D = nothing, F = nothing) -> Nothing
cross_sectional_mark_dropped!(dc::AbstractMatrix{Bool}, A::MatNum, w::VecNum,
                              t::Integer; D::MatNum = A .* sqrt.(w),
                              F = nothing) -> Nothing

Mark the columns of one observation that DependentColumnDrop drops.

The weighted design is the one that the solve reads, so the rank and the pivot are those of the fit. A design of full rank drops nothing, and a design of rank zero drops every column, because the fit gives every factor a return of zero.

Arguments

  • dc: Dropped-column mask observations × factors, written in place, or nothing.
  • A::MatNum: Exposures of the eligible assets, eligible assets × factors, already demeaned when an intercept is fitted.
  • w::VecNum: Cross-sectional weights of the eligible assets.
  • t::Integer: Row of the observation in dc.
  • D::MatNum: Weighted design sqrt.(w) .* A. A caller that weighted the design already passes it.
  • F: Column-pivoted QR of D, or nothing to take it here, through cross_sectional_qr.

Returns

  • nothing.

Related

source
PortfolioOptimisers.cross_sectional_row_space — Function
cross_sectional_row_space(
    f::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    r::Integer
) -> AbstractVector{<:Union{Number, AbstractJuMPScalar}}

Return the projection of a coefficient vector onto the row space of a rank-deficient weighted design.

Two coefficient vectors whose difference lies in the null space of the design give the same linear predictor. The projection removes the part of f in the null space, so it keeps the linear predictor and it is the vector of least norm among those that give it. For a least-squares answer, it is the answer of the pseudo-inverse.

Mathematical definition

\[\begin{align} \boldsymbol{f}^{\star} &= \left(\mathbf{I} - \mathbf{N} \mathbf{N}^{\intercal}\right) \boldsymbol{f}\,. \end{align}\]

Where:

  • $\boldsymbol{f}^{\star}$: Projected coefficient vector.
  • $\boldsymbol{f}$: Coefficient vector of the fit on the kept columns.
  • $\mathbf{N}$: Right singular vectors of the weighted design past its rank r, an orthonormal basis of its null space.

Arguments

  • f::VecNum: Coefficient vector, of length size(D, 2).
  • D::MatNum: Weighted design of one observation, eligible assets × factors.
  • r::Integer: Rank of D, read by cross_sectional_rank.

Returns

  • f::VecNum: The projected coefficient vector. f itself when r is zero, because the fit gave zero to every factor.

Related

source
PortfolioOptimisers.cross_sectional_target_coef — Function
cross_sectional_target_coef(
    tgt::AbstractRegressionTarget,
    A::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
    y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
    D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
    F
) -> CrossSectionalSolveResult

Return the coefficients and the residual count of a target fitted to a weighted design of full rank, at the precision of the design rather than of its Gram matrix.

GLM factorises the Gram matrix $\mathbf{D}^{\intercal} \mathbf{D}$ of the weighted design, which squares its condition number. So a factor set whose design has a condition number above about $10^{8}$ loses its precision in GLM, or GLM drops a column by its own pivot. cross_sectional_rank keeps every column up to a condition number of about $1 / (K \epsilon)$, where $\epsilon$ is the machine epsilon of the design. That is about $10^{15}$ in Float64 for up to eight factors. A target that is_basis_invariant states gives the same coefficients on any basis of the span of the design, mapped back. So the verb fits such a target to the design whose weighted Gram matrix is the identity, and maps the answer back. The solve then reads the triangular factor of D, as the solve of CrossSectionalLinearRegression does, and a least-squares target gives its factor returns to rounding. A target that is not invariant fits A itself, because a change of basis changes its answer.

Mathematical definition

\[\begin{align} \mathbf{D} \mathbf{\Pi} &= \mathbf{Q} \mathbf{R}\,, & \mathbf{M} &= \mathbf{\Pi} \mathbf{R}^{-1}\,, & \boldsymbol{f} &= \mathbf{M} \, \hat{\boldsymbol{c}}\,. \end{align}\]

Where:

  • $\mathbf{\Pi}$, $\mathbf{Q}$, $\mathbf{R}$: Column permutation, orthonormal factor and triangular factor of the column-pivoted QR of D.
  • $\mathbf{M}$: Change of basis. The weighted Gram matrix of $\mathbf{A} \mathbf{M}$ is $\mathbf{Q}^{\intercal} \mathbf{Q} = \mathbf{I}$.
  • $\hat{\boldsymbol{c}}$: Coefficients of the target fitted to $\mathbf{A} \mathbf{M}$.
  • $\boldsymbol{f}$: Coefficients on the columns of A.

The residual count is residual_count of the fit and the design that it fitted. A target that is basis invariant gives the same fitted values on $\mathbf{A} \mathbf{M}$ as on $\mathbf{A}$, so the change of basis keeps its count.

Arguments

  • tgt: Regression target of the observation, which carries its weights.
  • A::MatNum: Exposures of the eligible assets, eligible assets × factors, of full rank.
  • y::VecNum: Returns of the eligible assets.
  • D::MatNum: Weighted design sqrt.(w) .* A.
  • F: Column-pivoted QR of D, or nothing to take it here. A caller that read the rank off a factorisation passes it, so the design is factorised once.

Returns

  • res::CrossSectionalSolveResult: f, the coefficients of the target, of length size(A, 2), and edof, the residual count of the fit.

Related

source
PortfolioOptimisers.cross_sectional_loop_buffers — Function
cross_sectional_loop_buffers(intercept::Val{false}, Z::Arr3Num, X::MatNum, W::MatNum,
                             Tf::Type) -> CrossSectionalLoopBuffers
cross_sectional_loop_buffers(intercept::Val{true}, Z::Arr3Num, X::MatNum, W::MatNum,
                             Tf::Type) -> CrossSectionalLoopBuffers

Allocate the buffers that one task of the loop of cross_sectional_regression gathers each observation into.

Each buffer holds every asset, and an observation reads its first rows, so one allocation serves every observation of the task. The gathers of Z, X and W take the working type Tf, the promotion of the three types, so a fit of mixed element types computes in one type. The gather copies the data anyway, so the conversion costs no allocation, and data of one type keep their type. The centred exposures and returns of an intercept take Tf too. The weighted design takes the type that sqrt widens Tf to, so a Rational panel factorises in floating point.

Algorithm

The method that Julia selects is the algorithm.

  1. No intercept: the gathers take the type Tf, the centred buffers have no row, and the weighted design widens Tf.
  2. An intercept: the gathers and the centred buffers hold every asset in the type Tf, and the weighted design widens Tf.

Arguments

  • intercept: Val(cre.intercept) of the estimator.
  • Z::Arr3Num: Exposure tensor assets × factors × observations.
  • X::MatNum: Asset returns matrix observations × assets.
  • W::MatNum: Cross-sectional weights matrix observations × assets.
  • Tf::Type: The working type of the fit.

Returns

  • buf::CrossSectionalLoopBuffers: The gathered weights, exposures and returns, the centred exposures and returns, the weighted design, the factors of its column-pivoted QR, the pivoted columns that the leverage check divides, the work vector of a solve through the QR, and the Gram matrix and the right side of a solve of the normal equations, as CrossSectionalLoopBuffers states them.

Related

source
PortfolioOptimisers.cross_sectional_foreach — Function
cross_sectional_foreach(f, ex::FLoops.Transducers.Executor, idx::AbstractVector,
                        init = Returns(nothing)) -> nothing
cross_sectional_foreach(f, ex::FLoops.SequentialEx, idx::AbstractVector,
                        init = Returns(nothing)) -> nothing

Call f on every entry of idx under the executor ex, and raise the error of the first entry that fails.

The calls must be independent, and each must write a slice of its own. A threaded executor wraps an error in a TaskFailedException, and the entry that fails first in time depends on the schedule. So each call records its own error, and after the loop the function raises the recorded error of the lowest position, unchanged. Every executor therefore raises the error that a serial loop raises. Under SequentialEx the function runs the serial loop itself, so the error keeps the backtrace of the call that raised it.

Each task of the executor calls init() once, and every call of f that the task runs receives that state. The state holds the buffers of the calls, so a loop over many entries allocates them once per task. A call must not keep a buffer of the state after it returns, because the next call of the task writes it. The default init gives the state nothing, and a call that needs no buffer ignores it.

Arguments

  • f: Function of one entry of idx and of the state of the task. Its return value is discarded.
  • ex: Parallel execution strategy.
  • idx: The entries, in the order of the serial loop.
  • init: Function of no argument that returns the state of one task.

Returns

  • nothing.

Related

source