Cross-Sectional Regression (a)
Types
PortfolioOptimisers.AbstractCrossSectionalSolveAlgorithm — Type
abstract type AbstractCrossSectionalSolveAlgorithm <: AbstractRegressionAlgorithmAbstract supertype for all cross-sectional solve algorithm types.
A member decides what CrossSectionalLinearRegression and CrossSectionalTargetRegression do when the weighted design of an observation is rank deficient. The members differ in whether a rank test runs at all, and in what happens when it fails. The choice changes the answer, because Julia's \ answers a deficient design in three ways. A square design goes to an LU factorisation that throws LinearAlgebra.SingularException on an exactly zero pivot. A non-square one goes to a column-pivoted QR whose solve completes the orthogonal factorisation, so it returns the minimum-norm solution and agrees with a pseudo-inverse. A design that is only nearly dependent passes the rank test of every member and returns a badly conditioned answer that a pseudo-inverse would truncate.
A member means the same thing under both estimators, so the two give the same factor returns for a least-squares target. CrossSectionalTargetRegression refuses MinimumNormSolve, whose answer on a target is the answer of PseudoInverseFallback.
All concrete and/or abstract types representing cross-sectional solve algorithms should be subtypes of AbstractCrossSectionalSolveAlgorithm.
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractCrossSectionalSolveAlgorithm and implement the following methods. A member that only CrossSectionalLinearRegression reads needs only cross_sectional_solve, and a member that only CrossSectionalTargetRegression reads needs only cross_sectional_target_solve.
A new member gets no mask of dropped columns, because the regression keeps dc at nothing for every member except DependentColumnDrop. So a new member that drops a column gives it no mark, and the regression diagnostics and the factor attribution read the pseudo-inverse of the full design of each observation for it.
Each verb returns the factor returns and the residual count of the fit that it makes, because only the fit knows the parameters that it spent. A least-squares fit is a projection onto the span of the design, so it spends the rank of the design, and not its number of columns. Read that rank with cross_sectional_rank, the rank test of every member of the library, so that the count of a new member agrees with theirs and with the t-statistics of cs_regression_t_stats. A penalised fit spends fewer, and residual_count reads its count. The regression writes the count of each observation into edof of CrossSectionalRegression.
cross_sectional_solve
cross_sectional_solve(alg::AbstractCrossSectionalSolveAlgorithm, A::MatNum, y::VecNum, t::Integer, F = nothing, ws::NamedTuple = (;)): Returns the factor returns and the residual count of one observation of aCrossSectionalLinearRegression. State the six arguments in one method. A member that reads no factorisation and no workspace ignoresFandws, ascross_sectional_solve(::MySolve, A, y, t, F = nothing, ws::NamedTuple = (;)).
Arguments
alg: The concrete subtype instance.A: Weighted design of one observation,eligible assets × factors.y: Weighted target of one observation, of lengtheligible assets.t: Index of the observation, for the message of a refusal.F: Column-pivotedQRofAthat the regression took, ornothing.ws: Workspace of the task, ascross_sectional_solvestates it.
Returns
res::CrossSectionalSolveResult: The factor returns of the observation, of lengthsize(A, 2), and the residual count of the fit,size(A, 1)less the rank of the columns that the fit solves. The method builds it with the constructor ofCrossSectionalSolveResult, which checks the count.
cross_sectional_target_solve
cross_sectional_target_solve(alg::AbstractCrossSectionalSolveAlgorithm, tgt::AbstractRegressionTarget, obs::NamedTuple): Returns the factor returns and the residual count of one observation of aCrossSectionalTargetRegression.
Arguments
alg: The concrete subtype instance.tgt: Regression target of the observation, which carries its weights. Fit it withStatsAPI.fit(tgt, A, y).obs: The observation,(; A, y, D, t, F): the exposuresAand the returnsyof the eligible assets, the weighted designDfor a rank test, the indextof the observation, and the column-pivotedQRFofDthat the regression already took, ornothing.
Returns
res::CrossSectionalSolveResult: The factor returns of the observation, of lengthsize(obs.A, 2), and the residual count of the fit,residual_countof the fitted target and the design that it fitted.
assert_cross_sectional_target_solve
assert_cross_sectional_target_solve(alg::AbstractCrossSectionalSolveAlgorithm, tgt::AbstractRegressionTarget): Refuses a target that the member cannot solve, in the constructor ofCrossSectionalTargetRegression. The root passes every target, so a member that solves every target needs no method.
Arguments
alg: The concrete subtype instance.tgt: Regression target.
Returns
nothing. A refusal throws anArgumentError.
Examples
julia> struct MySolve <: PortfolioOptimisers.AbstractCrossSectionalSolveAlgorithm endjulia> CrossSectionalTargetRegression(; alg = MySolve()).algMySolve()Related
PortfolioOptimisers.PseudoInverseFallback — Type
struct PseudoInverseFallback <: AbstractCrossSectionalSolveAlgorithmSolves the full-rank design directly and pseudo-inverts a rank-deficient one.
This is the default of CrossSectionalLinearRegression and of CrossSectionalTargetRegression. It never throws on a dependent factor set. A square design that \ would refuse reaches the pseudo-inverse instead, and a non-square one reaches the same minimum-norm answer through either route. The rank test costs a second factorisation of every design that passes it.
A target has no pseudo-inverse, so on a rank-deficient design it fits the columns that DependentColumnDrop keeps, and the answer is projected onto the row space of the weighted design. Every coefficient vector that differs from the fit by a vector of the null space gives the same linear predictor, and the projection is the one of least norm among them. For a least-squares target it is the answer of CrossSectionalLinearRegression. The projection reads the fit through the linear predictor alone, which is what is_basis_invariant states of a target, so the constructor refuses a target that answers false.
Examples
julia> PseudoInverseFallback()PseudoInverseFallback()Related
PortfolioOptimisers.RankDeficiencyRefusal — Type
struct RankDeficiencyRefusal <: AbstractCrossSectionalSolveAlgorithmSolves the full-rank design directly and refuses a rank-deficient one.
The refusal names the observation, the rank it measured, the factor count and the count of eligible assets, so a caller can tell a dependent factor set apart from a cross-section that is too small. Under CrossSectionalTargetRegression the refusal runs before the target fits.
Examples
julia> RankDeficiencyRefusal()RankDeficiencyRefusal()Related
PortfolioOptimisers.UncheckedSolve — Type
struct UncheckedSolve <: AbstractCrossSectionalSolveAlgorithmRuns no rank test and takes whatever \ returns.
It is the cheapest member, because it factorises the design once instead of twice. A rank-deficient non-square design reaches a column-pivoted QR whose solve returns the minimum-norm answer, so it agrees with MinimumNormSolve there. An exactly singular square design reaches an LU factorisation instead and throws LinearAlgebra.SingularException. That is the one design on which the member fails.
Under CrossSectionalTargetRegression the target sees the full design, and its own factorisation decides. GLM drops a collinear column by its own pivot and gives it a return of zero, and its pivot need not drop the column that the rank test of DependentColumnDrop drops. A factor set that is dependent to rounding alone can pass the pivot of GLM and fail its Cholesky factorisation, which throws LinearAlgebra.PosDefException. On the carry fold of a CrossSectionalFactorPrior, a move solves an observation again when cross_sectional_rank finds it rank-deficient, so an observation that the pivot of GLM drops and the rank test keeps can keep the answer of the old basis.
Examples
julia> UncheckedSolve()UncheckedSolve()Related
PortfolioOptimisers.MinimumNormSolve — Type
struct MinimumNormSolve <: AbstractCrossSectionalSolveAlgorithmAlways pseudo-inverts, so it runs no rank test and takes no threshold.
Every observation takes the minimum-norm least-squares solution, whatever its rank, so one rule gives the factor returns of an observation that lost a factor and of one that did not. It is the most expensive member, because it takes a singular value decomposition of every design. It parts from UncheckedSolve on a square design, which \ sends to an LU factorisation, and on a design that is only nearly dependent, where the pseudo-inverse truncates a singular value that \ keeps.
CrossSectionalTargetRegression refuses it. A target has no pseudo-inverse, and the projection of PseudoInverseFallback changes nothing on a design of full rank, so the member would be a second name for that one.
Examples
julia> MinimumNormSolve()MinimumNormSolve()Related
PortfolioOptimisers.DependentColumnDrop — Type
struct DependentColumnDrop <: AbstractCrossSectionalSolveAlgorithmSolves the full-rank design directly, and drops the dependent columns of a rank-deficient one.
The member reads the rank r of the weighted design with cross_sectional_rank. When r falls short of the factor count, it keeps the columns that the first r pivots of the column-pivoted QR of the design name, solves on them, and gives every other factor a return of zero. A design of rank zero gives zero to every factor. The answer is a basic solution, not the answer of least norm. Its fitted values and its residuals are those of PseudoInverseFallback, and its factor returns are not. The pivot prefers the column of larger weighted norm, so which factor takes zero depends on the scale of the exposures and on the basis of a Factor Family.
The fit marks each dropped factor in the dc field of CrossSectionalRegression. So the t-statistics of cs_regression_t_stats and the standard errors of a factor attribution give a dropped factor no estimation error, and read the kept factors on the columns that the fit solved, rather than the pseudo-inverse of the whole design.
Examples
julia> DependentColumnDrop()DependentColumnDrop()Related
PortfolioOptimisers.NormalEquationsSolve — Type
struct NormalEquationsSolve <: AbstractCrossSectionalSolveAlgorithmSolves the normal equations of a well-conditioned design, and gives every other design to PseudoInverseFallback.
The member forms the Gram matrix $\mathbf{D}^{\intercal} \mathbf{D}$ of the weighted design and factorises it by a Cholesky factorisation with pivots. That costs about half of the column-pivoted QR that the other members take. The Gram matrix squares the condition number of the design, so the answer loses about twice the digits that the QR loses. The triangular factor of the factorisation has the condition number of the design, and cross_sectional_gram_conditioned estimates it in the 1-norm. When the factorisation has full rank and that estimate is at most $\epsilon^{-1/4}$, about $8 \times 10^{3}$ in Float64, the member solves the normal equations, and it counts every column, as the rank test of cross_sectional_rank does on such a design. Otherwise it takes the answer and the count of PseudoInverseFallback, so it never answers a nearly dependent design with the normal equations. The ratio of the pivots is not the test, because it can be far below the condition number of the design.
The normal equations lose precision with the square of the condition number. Against the QR solve of PseudoInverseFallback, the largest relative error of the factor returns over 20 random 1600 × 28 designs was 5e-15 at a condition number of 10, 2e-13 at 1e2 and 2e-11 at 1e3. At 3e3 the member solved 4 of the 20 designs by the normal equations, and at 1e4 none. The member is the default of no estimator. The leverage check of the regression takes its own QR of the design under this member, so the full gain of the member needs leverage = false on the estimator as well.
Under CrossSectionalTargetRegression, the member runs the same test on the weighted design. A design that passes goes to the target as it stands, so GLM factorises its own Gram matrix, and a design that fails takes the answer of PseudoInverseFallback. So the constructor refuses a target that is_basis_invariant answers false for, as it does under PseudoInverseFallback.
Mathematical definition
\[\begin{align} \mathbf{\Pi}^{\intercal} \mathbf{D}^{\intercal} \mathbf{D} \mathbf{\Pi} &= \mathbf{U}^{\intercal} \mathbf{U}\,, & \hat{\kappa} &\approx \lVert \mathbf{U} \rVert_{1} \lVert \mathbf{U}^{-1} \rVert_{1} \le \epsilon^{-1/4}\,. \end{align}\]
Where:
- $\mathbf{D}$: Weighted design of one observation,
eligible assets × factors. - $\mathbf{\Pi}$, $\mathbf{U}$: Permutation and triangular factor of the pivoted Cholesky factorisation of the Gram matrix.
- $\hat{\kappa}$: Estimate of the condition number of $\mathbf{U}$ in the 1-norm, within a factor of $K$ of the condition number of $\mathbf{D}$. The member solves the normal equations when it holds and the factorisation has full rank.
- $K$: Number of factors.
- $\epsilon$: Machine epsilon of the element type of $\mathbf{D}$.
Examples
julia> NormalEquationsSolve()NormalEquationsSolve()Related
PortfolioOptimisers.CrossSectionalSolveResult — Type
struct CrossSectionalSolveResult{__T_f, __T_edof} <: AbstractResultThe answer of the solve of one observation of a Cross-Sectional Regression: the factor returns of the observation and the residual count of its fit.
cross_sectional_solve and cross_sectional_target_solve return it, and a new solve algorithm builds it in its own methods, as the # Interfaces section of AbstractCrossSectionalSolveAlgorithm states. So the type is the contract of the interface, and its constructor checks the count that a solve gives. The regression writes f into its row of factor returns, and edof into its entry of edof of CrossSectionalRegression. A caller reads the fields by name, as res.f or (; f, edof) = res.
Fields
f: Factor returns of the observation, one entry per column of its design.
edof: Residual count of the fit, the number of rows of the design less the parameters that the fit spent.
Constructors
CrossSectionalSolveResult(f::VecNum, edof::Real)CrossSectionalSolveResult(; f::VecNum, edof::Real)The positional constructor checks the values. The keyword constructor takes each field by name, with no default, and calls it.
Validation
edofis finite. Otherwise, the constructor raises aDomainError. A count below zero passes, because a fit of more columns than rows spends more parameters than it has rows.
Examples
julia> res = PortfolioOptimisers.CrossSectionalSolveResult(; f = [0.5, -0.25], edof = 3);julia> (res.f, res.edof)([0.5, -0.25], 3)julia> PortfolioOptimisers.CrossSectionalSolveResult([0.5, -0.25], NaN)ERROR: DomainError with NaN:isfinite(edof) must hold. Gotedof => NaN[...]Related
PortfolioOptimisers.AbstractUnseenMemberRule — Type
abstract type AbstractUnseenMemberRule <: AbstractAlgorithmAbstract supertype for the Unseen Member rule of a CrossSectionalFactorPrior, the rule that says what return an Unseen Member gets at an observation.
An Unseen Member is a member of a constrained Factor Family that no asset of positive regression weight loads on at one observation. The exposures lag the returns, so the zero-sum condition of an observation reads the benchmark weights of an earlier one, which can still weight the member after its last asset delists. The data state nothing about its return there. The rule states the return, and with it the zero-sum condition of the observation.
The fit stores the rule on its CrossSectionalFactorModel block. The regression diagnostics and the standard errors of a factor_attribution call the verb of the rule on the stored exposures and weights, so they read the design that the fit regressed on.
Interfaces
In order to implement a new concrete type that works seamlessly with the library, subtype AbstractUnseenMemberRule and implement the following method:
unseen_member_design
unseen_member_design(rule::MyUnseenMemberRule, fcb::FactorFamilyBasis, B::Arr3Num, Zl::Arr3Num, W::MatNum) -> UnseenMemberDesignResult: Returns the design that the regression of the prior solves, and the change of each observation that maps its coefficients back to the reduced factor returns.
Arguments
rule: The member of the family.fcb: The Factor Family Basis of the lagged exposures, one row per observation ofZl.B: Lagged exposures on the raw axis,assets × factors × observations. Earlier releases held the observations on the first axis. A method written for them readsB[t, :, :], and it must now readB[:, :, t].Zl: Lagged exposures on the reduced axis, whichfcbgives fromB, in the same layout.W: Cross-sectional weights matrixobservations × assets. The pairs of positive weight are the sample of each observation.
Returns
res::UnseenMemberDesignResult:Z, the design to regress on,assets × factors × observations, andP, onet => P_tpair per observation that the rule changes, whereP_tis areduced factors × reduced factorsmatrix, and the factor returns oftareP_ttimes the coefficients oft.Pis empty when the rule changes no observation. The method builds it with the constructor ofUnseenMemberDesignResult, which checks each change against the design.
Related
PortfolioOptimisers.ZeroUnseenMember — Type
struct ZeroUnseenMember <: AbstractUnseenMemberRuleThe Unseen Member rule that gives an Unseen Member a return of zero at its observation, and holds the zero-sum condition of the observation over the other members of its family.
The data state nothing about the return of the member, so the rule states the value that an Empty Factor gets, and that the member gets at an observation that gives it no benchmark weight. The condition over the whole family still holds, because the member adds a zero term to it. The observation is identified again, so its factor returns do not depend on the member that the family drops. unseen_member_design states the change of the design.
Constructors
ZeroUnseenMember() -> ZeroUnseenMemberExamples
julia> ZeroUnseenMember()ZeroUnseenMember()Related
PortfolioOptimisers.SolvedUnseenMember — Type
struct SolvedUnseenMember <: AbstractUnseenMemberRuleThe Unseen Member rule that keeps an Unseen Member in the zero-sum condition of its observation, and lets the solve algorithm of the regression state its return.
The member keeps the benchmark weight of the lagged exposures in the condition. Its own column of the raw design is zero, so the member fixes the condition by itself, the condition no longer identifies the other members of the family, and the reduced design of the observation is rank-deficient. The solve algorithm of the Cross-Sectional Regression Estimator then gives the answer. Under PseudoInverseFallback it is the minimum-norm answer on the reduced axis, which depends on the member that the family drops. The residuals of the pairs of positive weight are the residuals of ZeroUnseenMember when the members that the sample sees span the same fitted values, which they do for a one-hot family.
A CrossSectionalFactorModel that a caller builds without a rule takes this one, because its design is then the reduced exposures themselves.
Constructors
SolvedUnseenMember() -> SolvedUnseenMemberExamples
julia> SolvedUnseenMember()SolvedUnseenMember()Related
PortfolioOptimisers.CrossSectionalLinearRegression — Type
struct CrossSectionalLinearRegression{__T_alg, __T_intercept, __T_leverage, __T_ex} <: AbstractCrossSectionalRegressionEstimatorFits one weighted least squares per observation across the assets, in closed form.
The solve runs on the weighted design $\sqrt{w_{t,i}} \, \boldsymbol{z}_{t,i}$ rather than on the normal matrix $\mathbf{Z}_{t}^{\intercal} \mathbf{W}_{t} \mathbf{Z}_{t}$. The condition number of the normal matrix is the square of the condition number of the weighted design. alg decides what happens when the weighted design is rank deficient.
The square root sets the arithmetic of the solve. A Rational weight has a floating-point square root, so the solve of a Rational panel runs in floating point. f keeps the Rational type of the panel, and holds the floating-point answer written as a Rational, not the exact least-squares answer. A caller who needs the exact answer solves the weighted normal equations in Rational arithmetic.
The leverage check and the solve read one column-pivoted QR of the weighted design, so the check costs a triangular solve on top of the factorisation of the solve. leverage = false skips the check, and with NormalEquationsSolve the fit then takes no QR of a well-conditioned design.
leverage = false states that no eligible pair has a leverage of one. On a design with such a pair, for example the only member of a level of a one-hot family, the residual of the pair keeps the round-off of the subtraction rather than an exact zero, and no reader of the fit leaves it out. Its idiosyncratic variance is then a round-off too, and a standardised residual that divides by it is noise, under both members of AbstractLeverageOneMask.
Algorithm
- Check
Z,XandW, and take the eligibility mask, per# Validationofcross_sectional_regression. - For each observation
t, gather the eligible assets, their weightsw, their exposuresAand their returnsy. - When
interceptistrue, subtract the weighted meansybarandxbarfromyand fromA, so the fit runs through the weighted centroid of the cross-section. - Scale
Aandybysqrt.(w), giving the weighted design and the weighted target. - When
leverageistrue, take the column-pivotedQRof the weighted design, and mark the pairs of leverage one throughcross_sectional_leverage_one. - Solve the weighted design through the branch
algselects, on the same factorisation when the member reads one, giving the rowtoff. - When
interceptistrue, set the entrytofbtoybar - dot(f[t, :], xbar). - Subtract the systematic part from
X, givingeps.
Fields
alg: Solve algorithm, anAbstractCrossSectionalSolveAlgorithm. It decides what the fit does with a rank-deficient weighted design.
intercept:intercept: Whether a per-observation intercept is fitted. Whenfalse, the regression runs through the origin of the cross-section.
leverage:leverage: Whether the fit marks the Leverage-One Pairs of each observation in theh1field ofCrossSectionalRegression, throughcross_sectional_leverage_one. Whenfalse, the fit runs no leverage check andh1isnothing, so every reader of the mask reads every residual as data.
ex: Parallel execution strategy. It runs the fits of the observations, which are independent problems. Each fit writes its own row, so every executor gives the same result, and a refusal is the one of the first observation that fails.
Constructors
CrossSectionalLinearRegression(; alg::AbstractCrossSectionalSolveAlgorithm = PseudoInverseFallback(), intercept::Bool = false, leverage::Bool = true, ex::FLoops.Transducers.Executor = ThreadedEx()) -> CrossSectionalLinearRegressionKeywords correspond to the struct's fields.
Examples
julia> CrossSectionalLinearRegression()CrossSectionalLinearRegression alg ┼ PseudoInverseFallback() intercept ┼ Bool: false leverage ┼ Bool: true ex ┴ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()Related
PortfolioOptimisers.CrossSectionalTargetRegression — Type
struct CrossSectionalTargetRegression{__T_tgt, __T_alg, __T_intercept, __T_leverage, __T_ex} <: AbstractCrossSectionalRegressionEstimatorFits one external regression model per observation across the assets.
The cross-sectional weights reach the model as observation weights, through factory and the target's own kwargs, so both targets of the library, LinearModel and GeneralisedLinearModel, run here unchanged. A caller's own target runs here too, when it states the methods of the # Interfaces section of AbstractRegressionTarget. A target with no weight method is refused, because its fit would ignore the cross-sectional weights. Unlike CrossSectionalLinearRegression, the fit refuses an observation with no eligible asset, because an external model has no cross-section to read.
alg decides what the fit does with a rank-deficient weighted design, as it does for CrossSectionalLinearRegression, and cross_sectional_target_solve states each member. The default PseudoInverseFallback fits the columns that the rank test keeps and projects the answer onto the row space of the weighted design, so a least-squares target gives the factor returns of CrossSectionalLinearRegression. A rank test is needed because a factor set can be dependent to rounding alone, such as a beta that shrinks fully to the mean of its industry, and the factorisation of GLM refuses such a design with a PosDefException. A rank-deficient design also comes from ordinary use. Under SolvedUnseenMember, the design of an observation with an Unseen Member is rank-deficient by construction.
Algorithm
- Check
Z,XandW, and take the eligibility mask, per# Validationofcross_sectional_regression. - For each observation
t, gather the eligible assets, their weightsw, their exposuresAand their returnsy. Refuse when no asset is eligible. - When
interceptistrue, subtract the weighted meansybarandxbarfromyand fromA. The target fits no intercept column of its own, so the intercept is recovered from the centroid rather than fitted. - When
leverageistrue, take the column-pivotedQRof the weighted designsqrt.(w) .* A, and mark the pairs of leverage one throughcross_sectional_leverage_one. The rank test ofalgreads the same factorisation. - Build the per-observation target with
factory(tgt, StatsBase.aweights(w)), fit it through the branchalgselects, and read its coefficients into the rowtoff. - When
interceptistrue, set the entrytofbtoybar - dot(f[t, :], xbar). - Subtract the systematic part from
X, givingeps.
The warning of CrossSectionalLinearRegression about leverage = false holds here too.
Fields
tgt: Regression model target.
alg: Solve algorithm, anAbstractCrossSectionalSolveAlgorithm. It decides what the fit does with a rank-deficient weighted design.
intercept:intercept: Whether a per-observation intercept is fitted. Whenfalse, the regression runs through the origin of the cross-section.
leverage:leverage: Whether the fit marks the Leverage-One Pairs of each observation in theh1field ofCrossSectionalRegression, throughcross_sectional_leverage_one. Whenfalse, the fit runs no leverage check andh1isnothing, so every reader of the mask reads every residual as data.
ex: Parallel execution strategy. It runs the fits of the observations, which are independent problems. Each fit writes its own row, so every executor gives the same result, and a refusal is the one of the first observation that fails.
Constructors
CrossSectionalTargetRegression(; tgt::AbstractRegressionTarget = LinearModel(), alg::AbstractCrossSectionalSolveAlgorithm = PseudoInverseFallback(), intercept::Bool = false, leverage::Bool = true, ex::FLoops.Transducers.Executor = ThreadedEx()) -> CrossSectionalTargetRegressionKeywords correspond to the struct's fields.
Validation
- The rules of
assert_cross_sectional_target_solve:algis notMinimumNormSolve, and underPseudoInverseFallbackandNormalEquationsSolve,is_basis_invariantanswerstruefortgt.
Examples
julia> CrossSectionalTargetRegression()CrossSectionalTargetRegression tgt ┼ LinearModel │ kwargs ┴ @NamedTuple{}: NamedTuple() alg ┼ PseudoInverseFallback() intercept ┼ Bool: false leverage ┼ Bool: true ex ┴ Transducers.ThreadedEx{@NamedTuple{}}: Transducers.ThreadedEx()Related
PortfolioOptimisers.CrossSectionalRegression — Type
struct CrossSectionalRegression{__T_f, __T_eps, __T_n, __T_edof, __T_b, __T_h1, __T_dc, __T_lv} <: AbstractCrossSectionalRegressionResultHolds the factor returns, the residuals, the eligible asset counts and the optional intercepts of a fitted cross-sectional regression.
The result is a sibling of Regression rather than a widening of it, because the two disagree on what an asset index means. A Regression holds one row per asset, and port_opt_view slices its rows. This result holds one row per observation and one column per asset, so the same index slices its columns. It carries no loadings matrix, because the exposures are the regression's input and an Exposure Estimator produces them.
Mathematical definition
\[\begin{align} \left(b_{t}, \boldsymbol{f}_{t}\right) &= \underset{b, \boldsymbol{f}}{\arg\min} \sum_{i = 1}^{N} w_{t,i} \left(x_{t,i} - b - \boldsymbol{z}_{t,i}^{\intercal} \boldsymbol{f}\right)^{2} \\ \boldsymbol{\varepsilon}_{t} &= \boldsymbol{x}_{t} - b_{t} \boldsymbol{1} - \mathbf{Z}_{t} \boldsymbol{f}_{t}\,, \\ \nu_{t} &= n_{t} - \operatorname{tr}\left(2 \mathbf{S}_{t} - \mathbf{S}_{t}^{\intercal} \mathbf{S}_{t}\right)\,. \end{align}\]
Where:
- $\boldsymbol{f}_{t}$: Factor returns of observation $t$, the $t$-th row of
f. - $\boldsymbol{x}_t$: Asset returns for observation $t$, the $t$-th row of the returns matrix.
- $\boldsymbol{\varepsilon}_{t}$: Residuals of observation $t$, the $t$-th row of
eps. - $\mathbf{Z}_{t}$: Exposure slice of observation $t$, $N \times K$, one row per asset.
- $\boldsymbol{z}_{t,i}$: Exposures of asset $i$ at observation $t$, the $i$-th row of $\mathbf{Z}_{t}$.
- $w_{t,i} \geq 0$: Cross-sectional weight of asset $i$ at observation $t$. A weight of zero excludes the pair from the fit.
- $b_{t}$: Intercept of observation $t$, the $t$-th entry of
b. Whenbis unset, the minimisation holds $b$ at zero. - $\nu_{t}$: Residual count of observation $t$, the $t$-th entry of
edof. - $n_{t}$: Number of eligible assets at observation $t$, the $t$-th entry of
n. - $\mathbf{S}_{t}$: Smoother of the fit of observation $t$, the $n_{t} \times n_{t}$ matrix that maps the returns of the eligible assets to their fitted values.
- $N$: Number of assets.
- $K$: Number of factors.
Each observation is one independent problem, so a factor return is a cross-sectional quantity and never a time-series one. When the weighted design of an observation is rank deficient, the minimiser is not unique, and the solve algorithm of the estimator picks one of them.
The minimisation is the fit of CrossSectionalLinearRegression and of a least-squares target. A CrossSectionalTargetRegression whose target is not least squares, such as a penalised target or a GeneralisedLinearModel whose link is not the identity, minimises the loss of its target with the same weights. Under an intercept, every fit reads the returns and the exposures centred on their weighted means, and $b_{t}$ is the weighted mean of the returns less the weighted mean of the exposures times $\boldsymbol{f}_{t}$. For a least-squares fit, this is the joint minimiser above.
The fit of an observation spends parameters on its eligible assets, and $\nu_{t}$ is the count that it leaves. It follows from $\mathbb{E}[\mathrm{RSS}_{t}] = \sigma^{2} \operatorname{tr}((\mathbf{I} - \mathbf{S}_{t})^{\intercal} (\mathbf{I} - \mathbf{S}_{t}))$ under a correct mean, so it is the count of the chi-squared law of the residuals. A least-squares fit is a projection onto the span of the weighted design, so $\mathbf{S}_{t}^{\intercal} \mathbf{S}_{t} = \mathbf{S}_{t}$, and it spends the rank of the design, not its number of columns. A rank-deficient observation thus spends fewer parameters than $K$, as the t-statistics of cs_regression_t_stats count it. A penalised fit spends fewer still, and residual_count reads its count. The intercept spends one more parameter at an observation with an eligible asset, because the design is centred on the weighted mean, and the fit of the centred design leaves the weighted mean alone. A Cross-Sectional Factor Prior reads edof for the degrees of freedom of each idiosyncratic variance, through cross_sectional_variance_counts.
A pair whose leverage is one has a direction of the design of its own, for example the only member of a level of a one-hot family. The fit reproduces its return whatever the return is, so its residual is zero by construction and tells nothing about its idiosyncratic risk. h1 marks these pairs, and a least-squares fit writes an exact zero there in place of the round-off of the subtraction.
A factor whose column DependentColumnDrop dropped at an observation takes a return of zero by the rule, not from the data. dc marks these pairs of observation and factor, so a diagnostic of the fit gives such a factor a variance of zero, and it reads the other factors on the columns that the fit kept.
An Empty Factor has no nonzero exposure at a pair of positive weight, so the fit cannot estimate it at any observation and gives it a return of zero. lv records which factors the fit estimated. A refit of a CrossSectionalFactorPrior under new scenario weights reads it, so it fits the factor prior on the same factors as the fit.
Fields
f: Factor returns matrixobservations × factors. Rowtholds the coefficients of the cross-sectional fit of observationt.
eps: Residual matrixobservations × assets. It is the part of the returns the exposures do not explain, and an entry of an excluded pair is whatever the arithmetic of that pair produced, so a missing return leaves a missing residual.
n: Count of assets that entered each fit, of lengthobservations. An asset enters when its cross-sectional weight is positive.
edof: Residual count of each fit, of lengthobservations, ornothing. An entry is the count of assets of the fit less the parameters that the fit spent: the rank of the weighted design for a least-squares fit, the count thatresidual_countreads for every other fit, and one more for the intercept.nothingstates no count, as a result that a caller builds by hand does, and a Cross-Sectional Factor Prior then states no degrees of freedom of its idiosyncratic variances.
b:b: Regression intercept vector.
h1: Leverage-one maskobservations × assets, ornothing. An entry is true where the pair entered the fit with a leverage of one, so the fit reproduced its return and its residual is zero by construction.nothingstates no mark, as a result that a caller builds by hand does.
dc: Dropped-column maskobservations × factors, ornothing. An entry is true whereDependentColumnDropdropped the column of the factor at the observation, so the fit gave the factor a return of zero by the rule.nothingstates no drop, as a fit under every other solve algorithm, and a result that a caller builds by hand, does.
lv: Live-factor mask, one entry per column off, ornothing. An entry is true where the factor is not an Empty Factor, so the fit estimated its return, and false where the fit gave it a return of zero because no pair of positive weight loads on it.nothingstates no record, as a result that a caller builds by hand, or that a carry fold joins from its steps, does.
Constructors
CrossSectionalRegression(; f::MatNum, eps::MatNum, n::AbstractVector{<:Integer}, edof::Option{<:VecNum} = nothing, b::Option{<:VecNum} = nothing, h1::Option{<:AbstractMatrix{Bool}} = nothing, dc::Option{<:AbstractMatrix{Bool}} = nothing, lv::Option{<:AbstractVector{Bool}} = nothing) -> CrossSectionalRegressionKeywords correspond to the struct's fields.
Validation
!isempty(f),!isempty(eps)and!isempty(n).size(f, 1) == size(eps, 1) == length(n).all(x -> x >= 0, n).- If provided,
length(edof) == length(n). - If provided,
!isempty(b), andlength(b) == size(f, 1). - If provided,
size(h1) == size(eps). - If provided,
size(dc) == size(f). - If provided,
length(lv) == size(f, 2).
View parameters
CrossSectionalRegression defines its own port_opt_view method rather than deriving one from field tags.
epsandh1are sliced on their second axis, which is the asset axis of a cross-sectional result.f,n,edof,b,dcandlvpass through unchanged. Each is indexed by observation, by factor or by both, and neither axis follows an asset selection.
Examples
julia> CrossSectionalRegression(; f = [1.0 2.0; 3.0 4.0], eps = [0.1 0.2 0.3; 0.4 0.5 0.6], n = [3, 3], edof = [1, 1])CrossSectionalRegression f ┼ 2×2 Matrix{Float64} eps ┼ 2×3 Matrix{Float64} n ┼ Vector{Int64}: [3, 3] edof ┼ Vector{Int64}: [1, 1] b ┼ nothing h1 ┼ nothing dc ┼ nothing lv ┴ nothingRelated
Functions
PortfolioOptimisers.cross_sectional_regression — Function
cross_sectional_regression(cre::Union{CrossSectionalLinearRegression,
CrossSectionalTargetRegression}, Z::Arr3Num,
X::MatNum, W::MatNum) -> CrossSectionalRegression
cross_sectional_regression(csr::CrossSectionalRegression, args...) -> CrossSectionalRegressionFit one regression per observation across the assets, or return a fitted result unchanged.
The verb is its own rather than a fourth argument of regression, because regression(re::Regression, args...) is a greedy passthrough that returns its first argument for any trailing arguments, so a time-series result handed to a cross-sectional call would return silently instead of raising.
The weight matrix W is an argument rather than a field, because a two-pass weighting scheme calls the estimator twice on one design with two different weight matrices, and a policy stored on the estimator would force a second estimator object or a mutation.
A refusal names an observation by its row of X. The verb fits cross_sectional_block_regression with no observation before X.
Algorithm
- Take the eligibility mask through
cross_sectional_design_mask. - For each observation
t, gather the eligible assets, their weightsw, their exposuresAand their returnsyinto buffers of the task, and record their count inn. The observations are independent problems, andcre.exruns them throughcross_sectional_foreach, which gives each task its buffers once. Each writes its own row, so every executor gives the same result. - When
cre.interceptistrue, take the weighted meansybarandxbarofyand ofA, and subtract them. An observation with no eligible asset takes zero for both. - Scale
Abysqrt.(w), giving the weighted design. Whencre.leverageistrueor the solve algorithm isDependentColumnDrop, take its column-pivotedQRonce. Whencre.leverageistrue, mark the eligible assets of leverage one in the rowtofh1, throughcross_sectional_leverage_oneon that factorisation. Otherwiseh1isnothing. - Take the factor returns and the residual count of the observation through
cross_sectional_coefficients, on the same factorisation, and write the factor returns into the rowtoff. An observation with no eligible asset takes zero factor returns underCrossSectionalLinearRegression, except underRankDeficiencyRefusal, whose rank test reads an empty design as rank zero and refuses it by name. - When
cre.interceptistrue, writeybar - dot(f[t, :], xbar)into the entrytofb. UnderDependentColumnDrop, mark the dropped columns of the observation in the rowtofdc, throughcross_sectional_mark_dropped!. - Write the residual count into the entry
tofedof, less one for the intercept whencre.interceptistrueand the observation has an eligible asset.edoftakes the working type off, because a penalised fit counts a real number of parameters, and the integer count of a least-squares fit is exact in that type. - Subtract the systematic part, through
cross_sectional_systematic, fromX, givingeps. - When
h1is set andcross_sectional_least_squaresanswerstrue, write an exact zero intoepsat every pair thath1marks. The fit reproduced the return of that pair, so zero is its residual in exact arithmetic.
Arguments
cre: Cross-sectional regression estimator.csr: A cross-sectional regression result.Z::Arr3Num: Exposure tensorassets × factors × observations.X::MatNum: Asset returns matrixobservations × assets.W::MatNum: Cross-sectional weights matrixobservations × assets.args...: Additional positional arguments (ignored by the passthrough).
Validation
- The rules of
cross_sectional_design_mask.
Returns
csr::CrossSectionalRegression: The fitted result, or the input result unchanged.
Examples
julia> Z = reshape([1.0, 0.0, 0.5, 0.0, 1.0, 0.5], 3, 2, 1);julia> cross_sectional_regression(CrossSectionalLinearRegression(), Z, [1.0 2.0 1.5], ones(1, 3))CrossSectionalRegression f ┼ 1×2 Matrix{Float64} eps ┼ 1×3 Matrix{Float64} n ┼ Vector{Int64}: [3] edof ┼ Vector{Float64}: [1.0] b ┼ nothing h1 ┼ 1×3 BitMatrix dc ┼ nothing lv ┴ nothingRelated
PortfolioOptimisers.port_opt_view — Method
port_opt_view(csr::CrossSectionalRegression, i, args...)Return a view of a CrossSectionalRegression result, selecting only the assets indexed by i.
Algorithm
- Take a column view of
epsoveri, giving the residuals of the selected assets. The asset axis of a cross-sectional result is the second one, because a row ofepsis one observation. - Take the same view of
h1when it is set, throughnothing_scalar_array_view_odd_order. - Build a new
CrossSectionalRegressionfrom the two views and the six untouched fields, which re-runs every guard of the constructor.
Arguments
csr: A cross-sectional regression result.i: Indices of the assets to select.args...: Additional positional arguments (ignored).
Returns
csr::CrossSectionalRegression: A new result whose residuals are restricted to the selected assets.
Examples
julia> csr = CrossSectionalRegression(; f = [1.0 2.0], eps = [0.1 0.2 0.3], n = [3], edof = [1])CrossSectionalRegression f ┼ 1×2 Matrix{Float64} eps ┼ 1×3 Matrix{Float64} n ┼ Vector{Int64}: [3] edof ┼ Vector{Int64}: [1] b ┼ nothing h1 ┼ nothing dc ┼ nothing lv ┴ nothingjulia> PortfolioOptimisers.port_opt_view(csr, [1, 3])CrossSectionalRegression f ┼ 1×2 Matrix{Float64} eps ┼ 1×2 SubArray{Float64, 2, Matrix{Float64}, Tuple{Base.Slice{Base.OneTo{Int64}}, Vector{Int64}}, false} n ┼ Vector{Int64}: [3] edof ┼ Vector{Int64}: [1] b ┼ nothing h1 ┼ nothing dc ┼ nothing lv ┴ nothingRelated
PortfolioOptimisers.cross_sectional_solve — Function
cross_sectional_solve(alg::AbstractCrossSectionalSolveAlgorithm, A::MatNum, y::VecNum,
t::Integer, F = nothing, ws::NamedTuple = (;))
-> CrossSectionalSolveResultSolve the weighted design A against the weighted target y through the branch alg selects, and count the residual degrees of freedom of the fit.
Every answer is a least-squares fit, so its fitted values are the projection of y onto the span of the columns that it solves. The projection spends the rank r of those columns, read by cross_sectional_rank, and the count is size(A, 1) - r. The t-statistics of cs_regression_t_stats read the same rank.
The fifth argument is the column-pivoted QR of A that cross_sectional_regression took, or nothing. A member of the library that reads a QR reads F, so the fit factorises the design once. The sixth argument is the workspace of the task, a NamedTuple whose optional fields a member reads with get: the work vector b of a solve through F, which cross_sectional_full_solve solves in, and the Gram matrix gram and the right side v of the normal equations, which cross_sectional_gram_factor and cross_sectional_normal_solve fill. Each gives the bits of the allocating form. Every member takes the six arguments in one method, with F = nothing and ws = (;) as defaults, and a member that reads neither ignores them.
Algorithm
UncheckedSolvereturnsA \ y, with no rank test. A squareAthat\solves has full rank. On every otherA,\solves through the column-pivotedQRofA, so the method takes that factorisation, solves through it, and reads the rank off it, with one factorisation of the design.MinimumNormSolvereturnsLinearAlgebra.pinv(A) * y, and counts the rank thatcross_sectional_rankreads.PseudoInverseFallbacktakes the column-pivotedQRofAand readscross_sectional_rankoff it. It returnscross_sectional_full_solve, the answer ofA \ ythrough the same factorisation, when the rank equals the factor count, andLinearAlgebra.pinv(A) * yotherwise.RankDeficiencyRefusaltakes the same factorisation, refuses a rank-deficientAwithassert_cross_sectional_rank, and returnscross_sectional_full_solveotherwise.DependentColumnDroptakes the same factorisation and reads the rank off it. It returnscross_sectional_full_solvewhen the rank equals the factor count, and the answer ofcross_sectional_column_dropon the pivot of the factorisation otherwise.NormalEquationsSolvetakes the pivoted Cholesky factorisation of the Gram matrix ofAthroughcross_sectional_gram_factor. It solves the normal equations through it withcross_sectional_normal_solvewhen the factorisation passes the test of the member, with a count ofsize(A, 1) - size(A, 2). Otherwise it returns the answer ofPseudoInverseFallback, which readsFandws. Its own solve reads noQR.
Arguments
alg: Cross-sectional solve algorithm.A::MatNum: Weighted design of one observation,eligible assets × factors.y::VecNum: Weighted target of one observation, of lengtheligible assets.t::Integer: Index of the observation, named by the refusal ofRankDeficiencyRefusal.F: Column-pivotedQRofA, asLinearAlgebra.qr(A, LinearAlgebra.ColumnNorm())returns it, ornothingto take it here.ws::NamedTuple: Workspace,(; b, gram, v)or any part of it, overwritten.(;)allocates every buffer.
Validation
- Under
RankDeficiencyRefusal, the rules ofassert_cross_sectional_rank.
Returns
res::CrossSectionalSolveResult:f, the factor returns of the observation, of lengthsize(A, 2), andedof, the residual count of the fit,size(A, 1)less the rank of the columns that it solves.
Related
PortfolioOptimisers.cross_sectional_target_solve — Function
cross_sectional_target_solve(alg::UncheckedSolve, tgt::AbstractRegressionTarget,
obs::NamedTuple) -> CrossSectionalSolveResult
cross_sectional_target_solve(alg::RankDeficiencyRefusal, tgt::AbstractRegressionTarget,
obs::NamedTuple) -> CrossSectionalSolveResult
cross_sectional_target_solve(alg::Union{DependentColumnDrop, PseudoInverseFallback},
tgt::AbstractRegressionTarget, obs::NamedTuple)
-> CrossSectionalSolveResult
cross_sectional_target_solve(alg::NormalEquationsSolve, tgt::AbstractRegressionTarget,
obs::NamedTuple) -> CrossSectionalSolveResultFit the target of one observation of a CrossSectionalTargetRegression through the branch alg selects, and read the residual count of the fit.
The count is residual_count of the fitted target and the design that it fitted. An unpenalised fit on the columns that the rank test keeps spends one parameter for each of them, so a rank-deficient observation counts its rank, as cross_sectional_solve counts it. A penalised fit states its own count. Under UncheckedSolve the target fits the full design, and its own count decides: a fit of GLM counts the rank that its pivot keeps, and the fallback of residual_count counts every column.
Algorithm
UncheckedSolvefitstgttoAandy, with no rank test.RankDeficiencyRefusaltakes the column-pivotedQRofD, refuses a rank-deficientDwithassert_cross_sectional_rank, and fitstgttoAandythroughcross_sectional_target_coefon the same factorisation otherwise.DependentColumnDropandPseudoInverseFallbacktake the column-pivotedQRofD, and read its rankroff it withcross_sectional_rank. Every later step reads the same factorisation. Whenrequals the factor count, they fittgttoAandythroughcross_sectional_target_coef. Otherwise they fittgtthe same way to the columns ofAthatcross_sectional_column_dropkeeps, andPseudoInverseFallbackprojects the answer onto the row space ofDwithcross_sectional_row_space.NormalEquationsSolvetestsDthroughcross_sectional_gram_factor. It fitstgttoAandywhenDpasses, so the target factorises its own Gram matrix, and it takes the answer ofPseudoInverseFallbackotherwise.
A member that takes a QR of D reads F of obs when the regression already took it. So every member but UncheckedSolve and NormalEquationsSolve fits a basis-invariant target at the rank and at the precision of D, which the fit of CrossSectionalLinearRegression reads too.
The constructor of CrossSectionalTargetRegression refuses MinimumNormSolve, so no method takes it.
Arguments
alg: Cross-sectional solve algorithm.tgt: Regression target of the observation, which carries its weights.obs::NamedTuple: The observation,(; A, y, D, t, F):A::MatNum: Exposures of the eligible assets,eligible assets × factors, already demeaned when an intercept is fitted.y::VecNum: Returns of the eligible assets, already demeaned when an intercept is fitted.D::MatNum: Weighted designsqrt.(w) .* A, which the rank test reads.t::Integer: Index of the observation, named by the refusal ofRankDeficiencyRefusal.F: Column-pivotedQRofDthat the regression took for its leverage check, ornothing. A tuple with noFreads asnothing.
Validation
- Under
RankDeficiencyRefusal, the rules ofassert_cross_sectional_rank.
Returns
res::CrossSectionalSolveResult:f, the factor returns of the observation, of lengthsize(A, 2), andedof, the residual count of the fit.
Related
PortfolioOptimisers.assert_cross_sectional_target_solve — Function
assert_cross_sectional_target_solve(alg::MinimumNormSolve, tgt::AbstractRegressionTarget)
assert_cross_sectional_target_solve(alg::Union{PseudoInverseFallback, NormalEquationsSolve},
tgt::AbstractRegressionTarget)
assert_cross_sectional_target_solve(alg::AbstractCrossSectionalSolveAlgorithm,
tgt::AbstractRegressionTarget)Check that a CrossSectionalTargetRegression can solve a rank-deficient design of tgt under alg.
Algorithm
The method that Julia selects is the algorithm.
MinimumNormSolveis refused. On a target its answer is the answer ofPseudoInverseFallback.PseudoInverseFallbackis refused whenis_basis_invariantanswersfalsefortgt. Its projection keeps the linear predictor of the fit, so it is the answer of least norm only for a target that reads the design through the linear predictor alone.NormalEquationsSolvegives a design that fails its test toPseudoInverseFallback, so the same rule refuses it.- Every other member passes.
Arguments
alg: Cross-sectional solve algorithm.tgt: Regression target.
Validation
!isa(alg, MinimumNormSolve). Raises anArgumentError.- Under
PseudoInverseFallbackandNormalEquationsSolve,is_basis_invariant(tgt). Raises anArgumentErrorthat names the member and the target.
Returns
nothing.
Related
PortfolioOptimisers.cross_sectional_rank — Function
cross_sectional_rank(A::MatNum; R::MatNum = LinearAlgebra.qr(A, LinearAlgebra.ColumnNorm()).R) -> IntReturn the numerical rank of A, read off the diagonal of a column-pivoted QR.
The test is written out rather than delegated to LinearAlgebra.rank(::QRPivoted), which needs Julia 1.12 while this package supports 1.11. A caller that already factorised A passes the triangular factor R of its factorisation, as UncheckedSolve does, so it factorises the design once.
Algorithm
- Take the triangular factor
Rof the column-pivotedLinearAlgebra.qrofA, unless the caller passes it. - Return
0whenRhas no row or no column, because the factorisation has no pivot to read. The magnitudes of the diagonal of itsRare non-increasing, so they rank the columns by how much each adds to the span. - Count the leading diagonal entries above
min(size(A)...) * eps(real(eltype(R))) * abs(R[1, 1]), giving the numerical rank. The tolerance is the oneLinearAlgebra.rankapplies to a pivotedQR.
Arguments
A::MatNum: Weighted design of one observation,eligible assets × factors.R::MatNum: Triangular factor of the column-pivotedQRofA.
Returns
r::Int: Numerical rank ofA.
Related