Cross-Sectional Regression (a): private API
Types
PortfolioOptimisers.CrossSectionalLoopBuffers — Type
struct CrossSectionalLoopBuffers{__T_w, __T_A, __T_y, __T_Ac, __T_yc, __T_D, __T_Q, __T_G, __T_b, __T_gram, __T_v}Holds the buffers that one task of the loop of cross_sectional_regression gathers each observation into.
This type is an implementation detail and is not intended for direct use. cross_sectional_loop_buffers makes it, and the loop of cross_sectional_block_regression reads it. Each buffer holds every asset, and an observation reads its first rows. The loop gives the solve the fields b, gram and v as the workspace that cross_sectional_solve reads.
Fields
w:w: The gathered weights of the observation.
A:A: The gathered exposures of the observation,assets × factors.
y:y: The gathered returns of the observation.
Ac:Ac: The centred exposures of an intercept,assets × factors. It has no row without an intercept.
yc:yc: The centred returns of an intercept. It has no entry without an intercept.
D:D: The weighted design,assets × factors.
Q:Q: The factors of the column-pivotedQRof the weighted design,assets × factors.
G:G: The pivoted columns that the leverage check divides,assets × factors.
b:b: The work vector of a solve through theQR, of lengthmax(assets, factors).
gram:gram: The Gram matrix of a solve of the normal equations,factors × factors.
v:v: The right side of a solve of the normal equations, of lengthfactors.
Related
Functions
PortfolioOptimisers.cross_sectional_design_mask — Function
cross_sectional_design_mask(
Z::AbstractArray{<:Union{Number, AbstractJuMPScalar}, 3},
X::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
W::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}}
) -> BitMatrix
Return the eligibility mask of a cross-sectional design, after checking its three arrays.
An (observation, asset) pair is eligible when its cross-sectional weight is positive. The weight alone decides. A pair that a zero weight excludes may carry a missing return or a missing exposure, and a pair with a positive weight may not.
Arguments
Z::Arr3Num: Exposure tensorassets × factors × observations.X::MatNum: Asset returns matrixobservations × assets.W::MatNum: Cross-sectional weights matrixobservations × assets.
Validation
!isempty(Z),!isempty(X)and!isempty(W).size(Z, 3) == size(X, 1)andsize(Z, 1) == size(X, 2).size(W) == size(X).all(isfinite, W)andall(x -> x >= 0, W).- Every pair with a positive weight carries a finite return and finite exposures. The
IsNonFiniteErrornames the observation, the asset and the weight.
Returns
act::BitMatrix: Eligibility maskobservations × assets, true where the weight is positive.
Related
PortfolioOptimisers.cross_sectional_residual_dof — Function
cross_sectional_residual_dof(edof::Nothing, n::AbstractVector{<:Integer}, neff::VecNum)
-> Nothing
cross_sectional_residual_dof(edof::VecNum, n::AbstractVector{<:Integer}, neff::VecNum)
-> VecNumReturn the degrees of freedom of each variance that reads the residuals of a cross-sectional fit, or nothing when the fit records no residual count.
The fit spends its parameters on the assets of each observation, not on the observations of each asset. So the $\sum_{t} n_{t}$ residuals keep the fraction $\phi$ of their degrees of freedom, and a variance that reads $n_{i}^{\mathrm{eff}}$ effective observations of asset $i$ keeps that fraction of them. The fraction spreads the spend evenly over the assets.
Mathematical definition
\[\begin{align} \phi &= \dfrac{\sum_{t} \nu_{t}}{\sum_{t} n_{t}}\,, \\ \nu_{i} &= \phi \, n_{i}^{\mathrm{eff}}\,. \end{align}\]
Where:
- $\phi$: Fraction of the degrees of freedom that the fit leaves.
- $\nu_{t}$: Residual count of the fit of observation $t$, the entry
tofedof. - $n_{t}$: Number of eligible assets at observation $t$, the entry
tofn. - $\nu_{i}$: Degrees of freedom of the variance of asset $i$.
- $n_{i}^{\mathrm{eff}}$: Effective count of the observations of asset $i$, the entry
iofneff.
On a balanced panel of $T$ observations and $N$ assets with no intercept and designs of full rank, $\nu_{i} = T (N - K) / N$ when $n_{i}^{\mathrm{eff}} = T$.
Algorithm
The method that Julia selects is the algorithm.
edofisnothing: the fit records no count, as a fit that a caller builds by hand does, so the degrees of freedom are not known.edofis a vector: take $\phi$, and scaleneffby it.
Arguments
edof: The residual count of each observation, the field ofCrossSectionalRegression, ornothing.n: The count of eligible assets of each observation.neff: The effective count of the observations of each asset that the variance reads.
Returns
nu::Option{<:VecNum}: The degrees of freedom of each variance, ornothing.
Related
PortfolioOptimisers.cross_sectional_finite_exposures — Function
cross_sectional_finite_exposures(
Z::AbstractArray{<:Union{Number, AbstractJuMPScalar}, 3}
) -> BitMatrix
Return the mask of the (observation, asset) pairs whose exposure to every factor is finite.
The pass reads Z in its memory order: the observation axis is the outer loop, and the asset axis is the inner loop.
Arguments
Z::Arr3Num: Exposure tensorassets × factors × observations.
Returns
fin::BitMatrix: Maskobservations × assets, true where every exposure of the pair is finite.
Related
PortfolioOptimisers.cross_sectional_block_regression — Function
cross_sectional_block_regression(cre::Union{CrossSectionalLinearRegression,
CrossSectionalTargetRegression},
Z::Arr3Num, X::MatNum, W::MatNum,
t0::Integer) -> CrossSectionalRegression
cross_sectional_block_regression(cre::AbstractCrossSectionalRegressionEstimator,
Z::Arr3Num, X::MatNum, W::MatNum,
t0::Integer) -> CrossSectionalRegressionFit one regression per observation of a block that follows t0 fitted observations, as cross_sectional_regression fits it.
A refusal names the observation at the row t of X as observation t0 + t. So the carry fold of a CrossSectionalFactorPrior, which fits the new observations of a step alone, names the observation that the batch fit names. The method for an estimator that the library does not know calls cross_sectional_regression on the block, and the estimator numbers the observations of the block itself.
Algorithm
The algorithm of cross_sectional_regression, with t0 + t as the index of the observation that cross_sectional_coefficients receives.
Arguments
cre: Cross-sectional regression estimator.Z::Arr3Num: Exposure tensorassets × factors × observations.X::MatNum: Asset returns matrixobservations × assets.W::MatNum: Cross-sectional weights matrixobservations × assets.t0::Integer: Number of fitted observations before the block.
Validation
- The rules of
cross_sectional_regression.
Returns
csr::CrossSectionalRegression: The fit of the block.
Related
PortfolioOptimisers.cross_sectional_coefficients — Function
cross_sectional_coefficients(
cre::CrossSectionalLinearRegression,
A::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
w::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
t::Integer;
D,
F,
ws
) -> CrossSectionalSolveResult
Return the factor returns and the residual count of one observation, through the member's own solve.
CrossSectionalLinearRegression scales the design and the target by sqrt.(w) and hands them to cross_sectional_solve. The square root of a Rational weight is a float, so that solve runs in floating point for a Rational panel. CrossSectionalTargetRegression refuses an empty cross-section, and hands the unscaled pair to cross_sectional_target_solve, with w as the observation weights of the target and the weighted design sqrt.(w) .* A for the rank test of its solve algorithm.
Arguments
cre: Cross-sectional regression estimator.A::MatNum: Exposures of the eligible assets,eligible assets × factors, already demeaned when an intercept is fitted.y::VecNum: Returns of the eligible assets, already demeaned when an intercept is fitted.w::VecNum: Cross-sectional weights of the eligible assets.t::Integer: Index of the observation.D::MatNum: Weighted designsqrt.(w) .* A. A caller that weighted the design already passes it.F: Column-pivotedQRofDthat the caller already took, ornothing. The solve reads it when its member reads aQR.ws::NamedTuple: Workspace of the solve, ascross_sectional_solvereads it, or(;). The target regression does not read it.
Validation
- Under
CrossSectionalTargetRegression,!isempty(y). An external target has no cross-section to fit when no asset is eligible, and theArgumentErrornames the observation. - Under
CrossSectionalTargetRegression, the target has afactory(tgt, w)method that carries the weights, elsefactory(::AbstractRegressionTarget, ::ObsWeights)throws anArgumentError.
Returns
res::CrossSectionalSolveResult:f, the factor returns of the observation, of lengthsize(A, 2), andedof, the residual count of the fit of the factors, with no account of an intercept.
Related
PortfolioOptimisers.cross_sectional_column_drop — Function
cross_sectional_column_drop(
fit_cols,
D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
r::Integer;
F
) -> CrossSectionalSolveResult
Return the factor returns and the residual count of a rank-deficient observation that fits only the columns its rank test keeps.
Algorithm
- Return zero for every factor when
ris zero. The fit spends no parameter, so its residual count issize(D, 1). - Keep the columns that the first
rpivots of the column-pivotedQRFofDname, in their order on the factor axis, withcross_sectional_kept_columns. - Fit the kept columns with
fit_cols, and give every other factor a return of zero. The residual count is the count of that fit.
Arguments
fit_cols: Function of the kept column indices that returns their coefficients and the residual count of their fit, aCrossSectionalSolveResult.D::MatNum: Weighted design of one observation,eligible assets × factors.y::VecNum: Target of the observation. Its type and the type ofDgive the type of the zero answer of rank zero.r::Integer: Rank ofD, read bycross_sectional_rank, belowsize(D, 2).F: Column-pivotedQRofD, asLinearAlgebra.qr(D, LinearAlgebra.ColumnNorm())returns it. A caller that read the rank off a factorisation passes it, so the design is factorised once.
Returns
res::CrossSectionalSolveResult:f, the factor returns of the observation, of lengthsize(D, 2), zero at each dropped factor, andedof, the residual count of the fit of the kept columns.
Related
PortfolioOptimisers.cross_sectional_kept_columns — Function
cross_sectional_kept_columns(
D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
r::Integer;
F
) -> Vector{Int64}
Return the columns of a rank-deficient weighted design that DependentColumnDrop keeps, in their order on the factor axis.
They are the columns that the first r pivots of the column-pivoted QR of D name. cross_sectional_column_drop fits them, and cross_sectional_mark_dropped! marks the others, so the fit and its mark read one pivot.
Arguments
D::MatNum: Weighted design of one observation,eligible assets × factors.r::Integer: Rank ofD, read bycross_sectional_rank.F: Column-pivotedQRofD, asLinearAlgebra.qr(D, LinearAlgebra.ColumnNorm())returns it. A caller that read the rank off a factorisation passes it, so the design is factorised once.
Returns
keep::Vector{Int}: Indices of the kept columns, sorted.
Related
PortfolioOptimisers.cross_sectional_drop_mask — Function
cross_sectional_drop_mask(alg::AbstractCrossSectionalSolveAlgorithm, T::Integer,
K::Integer) -> Nothing
cross_sectional_drop_mask(alg::DependentColumnDrop, T::Integer, K::Integer) -> Matrix{Bool}Return the empty dropped-column mask of a fit, or nothing for a solve algorithm that drops no column.
Only DependentColumnDrop gives a factor a return of zero by the rule, so only its fit states the mask. Every other member either solves every column, or answers with the pseudo-inverse, whose standard errors the diagnostics already read.
Arguments
alg: Cross-sectional solve algorithm.T::Integer: Number of observations.K::Integer: Number of factors.
Returns
dc::Option{<:Matrix{Bool}}: Allfalse,T × K, ornothing.
Related
PortfolioOptimisers.cross_sectional_bit_mask — Function
cross_sectional_bit_mask(dc::Nothing) -> Nothing
cross_sectional_bit_mask(dc::AbstractMatrix{Bool}) -> BitMatrixPack a mask that the fits of the observations wrote into a BitMatrix, or keep nothing.
The fits write a Matrix{Bool}, whose entries are bytes, because two threads that write the rows of one word of a BitMatrix would race. The result stores the packed mask.
Arguments
dc: The mask, ornothing.
Returns
dc::Option{<:BitMatrix}: The packed mask, ornothing.
Related
PortfolioOptimisers.cross_sectional_mark_dropped! — Function
cross_sectional_mark_dropped!(dc::Nothing, A::MatNum, w::VecNum, t::Integer;
D = nothing, F = nothing) -> Nothing
cross_sectional_mark_dropped!(dc::AbstractMatrix{Bool}, A::MatNum, w::VecNum,
t::Integer; D::MatNum = A .* sqrt.(w),
F = nothing) -> NothingMark the columns of one observation that DependentColumnDrop drops.
The weighted design is the one that the solve reads, so the rank and the pivot are those of the fit. A design of full rank drops nothing, and a design of rank zero drops every column, because the fit gives every factor a return of zero.
Arguments
dc: Dropped-column maskobservations × factors, written in place, ornothing.A::MatNum: Exposures of the eligible assets,eligible assets × factors, already demeaned when an intercept is fitted.w::VecNum: Cross-sectional weights of the eligible assets.t::Integer: Row of the observation indc.D::MatNum: Weighted designsqrt.(w) .* A. A caller that weighted the design already passes it.F: Column-pivotedQRofD, ornothingto take it here, throughcross_sectional_qr.
Returns
nothing.
Related
PortfolioOptimisers.cross_sectional_row_space — Function
cross_sectional_row_space(
f::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
r::Integer
) -> AbstractVector{<:Union{Number, AbstractJuMPScalar}}
Return the projection of a coefficient vector onto the row space of a rank-deficient weighted design.
Two coefficient vectors whose difference lies in the null space of the design give the same linear predictor. The projection removes the part of f in the null space, so it keeps the linear predictor and it is the vector of least norm among those that give it. For a least-squares answer, it is the answer of the pseudo-inverse.
Mathematical definition
\[\begin{align} \boldsymbol{f}^{\star} &= \left(\mathbf{I} - \mathbf{N} \mathbf{N}^{\intercal}\right) \boldsymbol{f}\,. \end{align}\]
Where:
- $\boldsymbol{f}^{\star}$: Projected coefficient vector.
- $\boldsymbol{f}$: Coefficient vector of the fit on the kept columns.
- $\mathbf{N}$: Right singular vectors of the weighted design past its rank
r, an orthonormal basis of its null space.
Arguments
f::VecNum: Coefficient vector, of lengthsize(D, 2).D::MatNum: Weighted design of one observation,eligible assets × factors.r::Integer: Rank ofD, read bycross_sectional_rank.
Returns
f::VecNum: The projected coefficient vector.fitself whenris zero, because the fit gave zero to every factor.
Related
PortfolioOptimisers.cross_sectional_target_coef — Function
cross_sectional_target_coef(
tgt::AbstractRegressionTarget,
A::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}},
y::AbstractVector{<:Union{Number, AbstractJuMPScalar}},
D::AbstractMatrix{<:Union{Number, AbstractJuMPScalar}};
F
) -> CrossSectionalSolveResult
Return the coefficients and the residual count of a target fitted to a weighted design of full rank, at the precision of the design rather than of its Gram matrix.
GLM factorises the Gram matrix $\mathbf{D}^{\intercal} \mathbf{D}$ of the weighted design, which squares its condition number. So a factor set whose design has a condition number above about $10^{8}$ loses its precision in GLM, or GLM drops a column by its own pivot. cross_sectional_rank keeps every column up to a condition number of about $1 / (K \epsilon)$, where $\epsilon$ is the machine epsilon of the design. That is about $10^{15}$ in Float64 for up to eight factors. A target that is_basis_invariant states gives the same coefficients on any basis of the span of the design, mapped back. So the verb fits such a target to the design whose weighted Gram matrix is the identity, and maps the answer back. The solve then reads the triangular factor of D, as the solve of CrossSectionalLinearRegression does, and a least-squares target gives its factor returns to rounding. A target that is not invariant fits A itself, because a change of basis changes its answer.
Mathematical definition
\[\begin{align} \mathbf{D} \mathbf{\Pi} &= \mathbf{Q} \mathbf{R}\,, & \mathbf{M} &= \mathbf{\Pi} \mathbf{R}^{-1}\,, & \boldsymbol{f} &= \mathbf{M} \, \hat{\boldsymbol{c}}\,. \end{align}\]
Where:
- $\mathbf{\Pi}$, $\mathbf{Q}$, $\mathbf{R}$: Column permutation, orthonormal factor and triangular factor of the column-pivoted
QRofD. - $\mathbf{M}$: Change of basis. The weighted Gram matrix of $\mathbf{A} \mathbf{M}$ is $\mathbf{Q}^{\intercal} \mathbf{Q} = \mathbf{I}$.
- $\hat{\boldsymbol{c}}$: Coefficients of the target fitted to $\mathbf{A} \mathbf{M}$.
- $\boldsymbol{f}$: Coefficients on the columns of
A.
The residual count is residual_count of the fit and the design that it fitted. A target that is basis invariant gives the same fitted values on $\mathbf{A} \mathbf{M}$ as on $\mathbf{A}$, so the change of basis keeps its count.
Arguments
tgt: Regression target of the observation, which carries its weights.A::MatNum: Exposures of the eligible assets,eligible assets × factors, of full rank.y::VecNum: Returns of the eligible assets.D::MatNum: Weighted designsqrt.(w) .* A.F: Column-pivotedQRofD, ornothingto take it here. A caller that read the rank off a factorisation passes it, so the design is factorised once.
Returns
res::CrossSectionalSolveResult:f, the coefficients of the target, of lengthsize(A, 2), andedof, the residual count of the fit.
Related
PortfolioOptimisers.cross_sectional_loop_buffers — Function
cross_sectional_loop_buffers(intercept::Val{false}, Z::Arr3Num, X::MatNum, W::MatNum,
Tf::Type) -> CrossSectionalLoopBuffers
cross_sectional_loop_buffers(intercept::Val{true}, Z::Arr3Num, X::MatNum, W::MatNum,
Tf::Type) -> CrossSectionalLoopBuffersAllocate the buffers that one task of the loop of cross_sectional_regression gathers each observation into.
Each buffer holds every asset, and an observation reads its first rows, so one allocation serves every observation of the task. The gathers of Z, X and W take the working type Tf, the promotion of the three types, so a fit of mixed element types computes in one type. The gather copies the data anyway, so the conversion costs no allocation, and data of one type keep their type. The centred exposures and returns of an intercept take Tf too. The weighted design takes the type that sqrt widens Tf to, so a Rational panel factorises in floating point.
Algorithm
The method that Julia selects is the algorithm.
- No intercept: the gathers take the type
Tf, the centred buffers have no row, and the weighted design widensTf. - An intercept: the gathers and the centred buffers hold every asset in the type
Tf, and the weighted design widensTf.
Arguments
intercept:Val(cre.intercept)of the estimator.Z::Arr3Num: Exposure tensorassets × factors × observations.X::MatNum: Asset returns matrixobservations × assets.W::MatNum: Cross-sectional weights matrixobservations × assets.Tf::Type: The working type of the fit.
Returns
buf::CrossSectionalLoopBuffers: The gathered weights, exposures and returns, the centred exposures and returns, the weighted design, the factors of its column-pivotedQR, the pivoted columns that the leverage check divides, the work vector of a solve through theQR, and the Gram matrix and the right side of a solve of the normal equations, asCrossSectionalLoopBuffersstates them.
Related
PortfolioOptimisers.cross_sectional_foreach — Function
cross_sectional_foreach(f, ex::FLoops.Transducers.Executor, idx::AbstractVector,
init = Returns(nothing)) -> nothing
cross_sectional_foreach(f, ex::FLoops.SequentialEx, idx::AbstractVector,
init = Returns(nothing)) -> nothingCall f on every entry of idx under the executor ex, and raise the error of the first entry that fails.
The calls must be independent, and each must write a slice of its own. A threaded executor wraps an error in a TaskFailedException, and the entry that fails first in time depends on the schedule. So each call records its own error, and after the loop the function raises the recorded error of the lowest position, unchanged. Every executor therefore raises the error that a serial loop raises. Under SequentialEx the function runs the serial loop itself, so the error keeps the backtrace of the call that raised it.
Each task of the executor calls init() once, and every call of f that the task runs receives that state. The state holds the buffers of the calls, so a loop over many entries allocates them once per task. A call must not keep a buffer of the state after it returns, because the next call of the task writes it. The default init gives the state nothing, and a call that needs no buffer ignores it.
Arguments
f: Function of one entry ofidxand of the state of the task. Its return value is discarded.ex: Parallel execution strategy.idx: The entries, in the order of the serial loop.init: Function of no argument that returns the state of one task.
Returns
nothing.
Related