Elastic net: private API
These are the fitted model of ElasticNet, its counts and scores, and the steps of its fit. elastic_net_sweeps runs the coordinate descent, elastic_net_polish solves the optimum on a sign pattern exactly, and elastic_net_counts reads the effective degrees of freedom off the equations of the active set.
PortfolioOptimisers.ElasticNetFit — Type
struct ElasticNetFit{__T_b, __T_gamma, __T_df, __T_rss, __T_nobs, __T_rdof} <: AbstractResultHolds the coefficients of a fitted ElasticNet, the strength at which the fit ran, and the counts that its scores read.
StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) returns it. StatsAPI.coef returns b, and StatsAPI.predict multiplies a design by b. StatsAPI.dof, StatsAPI.nobs, StatsAPI.deviance, StatsAPI.loglikelihood, StatsAPI.aic, StatsAPI.aicc and StatsAPI.bic read the counts, in the forms that GLM uses for a fitted LinearModel. residual_count returns rdof.
Mathematical definition
\[\begin{align} \mathbf{H} &= \mathbf{P}_{\mathcal{A}} \left(\mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\right)^{-1} \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V}\,,\\ \widehat{\mathrm{df}} &= \operatorname{tr}(\mathbf{H}) + \delta\,,\\ \nu &= T - \operatorname{tr}\left(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S}\right)\,,\quad \mathbf{S} = \mathbf{V}^{1/2} \mathbf{H} \mathbf{V}^{-1/2}\,. \end{align}\]
Where:
- $\mathbf{H}$: Smoother of the fit on its active set. The fitted response is $\mathbf{H} \boldsymbol{y}$ while the active set and the strength do not move.
- $\mathbf{P}_{\mathcal{A}}$: Columns of the active set $\mathcal{A}$, the columns whose coefficient is nonzero and the intercept column.
- $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\widehat{\mathrm{df}}$: Effective degrees of freedom of the fit,
df. - $\delta$: Term of a strength that reads the response, which the
sdivfield of the target adds throughstrength_divergence. It is zero undersdiv = nothing. - $\nu$: Residual count,
rdof. - $\mathbf{S}$: Smoother of the weighted rows. It is $\mathbf{H}$ itself without weights, and it is symmetric.
- $T$: Number of observations.
Remark (The elastic net) after Equation (28) of [109] states the trace of $\mathbf{H}$ as an unbiased estimate of the degrees of freedom at a fixed strength. Remark (The lasso with intercept) after it extends the result to an unpenalised intercept.
The residual count follows from the mean of the weighted residual sum of squares. With $\boldsymbol{y} = \boldsymbol{\mu} + \boldsymbol{\varepsilon}$ and $\mathbf{V}^{1/2} \boldsymbol{\varepsilon}$ of covariance $\sigma^{2} \mathbf{I}$,
\[\begin{align} \mathbb{E}\left[\mathrm{RSS}\right] &= \sigma^{2} \operatorname{tr}\left((\mathbf{I} - \mathbf{S})^\intercal (\mathbf{I} - \mathbf{S})\right) + \lVert \mathbf{V}^{1/2} (\mathbf{I} - \mathbf{H}) \boldsymbol{\mu} \rVert^{2}\,. \end{align}\]
So when the bias term is small, $\mathrm{RSS} / \sigma^{2}$ has the mean $\nu$, and $\nu$ is the count of the chi-squared law that ResidualInflation reads. For a projection, $\mathbf{S}^\intercal \mathbf{S} = \mathbf{S}$, and $\nu = T - p$, the count of a fitted LinearModel with $p$ columns. A ridge part shrinks $\mathbf{S}$, so $\nu$ is below $T - \operatorname{tr}(\mathbf{H})$, and the count $T - \operatorname{tr}(\mathbf{H})$ overstates the residual count. The term $\delta$ does not enter $\nu$, because no source states the residual law of a fit whose strength reads the response.
The scores take the residual variance at its maximum-likelihood value $\mathrm{RSS} / n$, with $n$ the sum of the frequency weights of glm_weights. StatsAPI.dof adds one to $\widehat{\mathrm{df}}$ for that variance, as GLM adds one to the rank of a fitted LinearModel. Equation (2.5) of [69] states AICc of a linear smoother with $\operatorname{tr}(\mathbf{H})$ in place of the number of columns, and StatsAPI.aicc with this dof is $n$ times that form plus a constant. Equation (2.16) of [70] proposes BIC with the effective degrees of freedom for the lasso, and does not derive it.
Fields
b: The coefficients on the columns of the design of the caller, one per column.
gamma: The regularisation strength $\gamma$ that the strength rule gave, on the standardised columns when the target has ave.
df: The effective degrees of freedom of the fit, the trace of the smoother on the active set plus the term ofsdiv.
rss: The weighted residual sum of squares of the fit, under the frequency weights ofglm_weights.
nobs: The number of observations, the sum of the frequency weights, or the number of rows without weights.
rdof: The residual count, the number of rows less $\operatorname{tr}(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S})$.
Related
ElasticNetStatsAPI.coef(::ElasticNetFit)StatsAPI.predict(::ElasticNetFit, ::MatNum)StatsAPI.dof(::ElasticNetFit)residual_countelastic_net_countsStepwiseRegression
References
- [109] R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012). Remark (The elastic net) after Equation (28), and Remark (The lasso with intercept).
- [69] C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998). Equation (2.5). Page 273 cites Section 3.5 of [298] for the effective degrees of freedom of a linear smoother, and the residual count above follows the derivation of this docstring. The original source of this formulation was not found. The formulation follows this work and is not checked against the original.
- [70] H. Zou, T. Hastie and R. Tibshirani. On the "degrees of freedom" of the lasso. The Annals of Statistics 35, 2173–2192 (2007). Equation (2.16).
- [299] T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd edition (Springer, 2009). Section 7.6, Equation (7.32).
StatsAPI.coef — Method
StatsAPI.coef(m::ElasticNetFit) -> VecNumReturn the coefficients of a fitted ElasticNet, one per column of the design.
Arguments
m: The fitted model.
Returns
b::VecNum:m.b.
Related
StatsAPI.dof — Method
StatsAPI.dof(m::ElasticNetFit) -> Real
StatsAPI.nobs(m::ElasticNetFit) -> Real
StatsAPI.deviance(m::ElasticNetFit) -> Real
StatsAPI.loglikelihood(m::ElasticNetFit) -> Real
StatsAPI.aic(m::ElasticNetFit) -> Real
StatsAPI.aicc(m::ElasticNetFit) -> Real
StatsAPI.bic(m::ElasticNetFit) -> RealReturn the counts and the scores of a fitted ElasticNet.
ElasticNetFit is not a StatsAPI.StatisticalModel, so each score has a method of its own, in the form of StatsAPI. The forms match GLM for a fitted LinearModel under the same weights, with the effective degrees of freedom in place of the rank.
Mathematical definition
\[\begin{align} k &= \widehat{\mathrm{df}} + 1\,,\\ \ln \hat{L} &= -\frac{n}{2} \left(\ln \frac{2 \pi\, \mathrm{RSS}}{n} + 1\right)\,,\\ \mathrm{AIC} &= 2k - 2\ln\hat{L}\,,\\ \mathrm{AICC} &= \mathrm{AIC} + \frac{2k(k+1)}{n - k - 1}\,,\\ \mathrm{BIC} &= k\ln n - 2\ln\hat{L}\,. \end{align}\]
Where:
- $k$:
StatsAPI.dof, the effective degrees of freedom plus one for the residual variance. - $\widehat{\mathrm{df}}$:
m.df. - $\hat{L}$: Maximum of the Gaussian likelihood, at the residual variance $\mathrm{RSS} / n$.
- $\mathrm{RSS}$:
StatsAPI.deviance, the weighted residual sum of squaresm.rss. - $n$:
StatsAPI.nobs,m.nobs.
Equation (2.5) of [69] states AICc with the residual variance estimated, and StatsAPI.aicc is $n$ times that form plus $n \ln 2\pi$. Equation (2.16) of [70] writes BIC with a known error variance $\sigma^{2}$, as $\mathrm{RSS} / (n \sigma^{2}) + \ln(n)\, \widehat{\mathrm{df}} / n$. The form here puts the maximum-likelihood estimate $\mathrm{RSS} / n$ in place of $\sigma^{2}$, as GLM does for a fitted LinearModel.
Arguments
m: The fitted model.
Returns
x::Real: The count or the score.
Related
References
- [69] C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998). Equation (2.5).
- [70] H. Zou, T. Hastie and R. Tibshirani. On the "degrees of freedom" of the lasso. The Annals of Statistics 35, 2173–2192 (2007). Equation (2.16).
PortfolioOptimisers.elastic_net_design — Function
elastic_net_design(tgt::ElasticNet, X::MatNum) -> TupleReturn the standardised design of an ElasticNet, the design with each row weighted, the scale of the columns, and the weights of the rows.
Algorithm
- When
tgt.veisnothing, set the scalestonothing. Otherwise take the standard deviation of each column ofXundertgt.ve, and put one in place of a zero and in place of the intercept columntgt.ic. That iss. - Divide each column of
Xby its entry ofs, which givesP. Whensisnothing,PisX. - Resolve
tgt.wagainstXwithget_observation_weights, and scale it withglm_weights, which givesv. Whenvisnothing,QisP. Otherwise multiply each row ofPby its entry ofv, which givesQ.
Arguments
tgt: The regression target.X: The design matrixobservations × regressors.
Returns
(P, Q, s, v)::Tuple: The standardised design $\mathbf{P}$, the weighted design $\mathbf{V} \mathbf{P}$, the scale of the columns ornothing, and the frequency weights of the rows ornothing.
Related
PortfolioOptimisers.elastic_net_sweeps — Function
elastic_net_sweeps(tgt::ElasticNet, P::MatNum, Q::MatNum, y::VecNum, z::VecNum,
r::VecNum, nrm2::VecNum, a::VecNum, ridge::VecNum,
pen::AbstractVector{Bool}) -> VecNumReturn the coefficients of an ElasticNet by cyclic coordinate descent from z, with an exact solve at the end of each sweep.
Mathematical definition
The minimiser of the objective of ElasticNet over one coefficient, with the others fixed, is
\[\begin{align} z_k &= \frac{S\left(\mathbf{P}_k^\intercal \mathbf{V} \boldsymbol{r}_k,\ a_k\right)}{\mathbf{P}_k^\intercal \mathbf{V} \mathbf{P}_k + d_k}\,,\quad S(v, a) = \operatorname{sign}(v) \max(\lvert v \rvert - a, 0)\,,\\ a_k &= \gamma \vartheta\, \mathbb{1}[k \in \mathcal{K}]\,,\quad d_k = \gamma (1 - \vartheta)\, \mathbb{1}[k \in \mathcal{K}]\,. \end{align}\]
Where:
- $\boldsymbol{z}$: Coefficients of the elastic net, one per column of $\mathbf{P}$. Coefficient $k$ is $z_k$.
- $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\boldsymbol{r}_k$: Residual of the response $\boldsymbol{y}$ without the term of column $k$.
- $S$: Soft threshold.
- $a_k$, $d_k$: Threshold and ridge term of column $k$, both zero at the intercept column.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.
At the intercept column the update is the least-squares coefficient of that column on the residual.
Algorithm
- Take
den, the denominator of each coordinate update. - Sweep the columns in order. For each column
k, computerho, the product of the column ofQwith the residual without its own term, and apply the coordinate update, which givesznew. Updater, and recorddelta, the largest change of a coefficient in the sweep. - Try the exact solve of
elastic_net_polishon the sign pattern ofz. When it gives the optimum, return it. - Repeat steps 2 and 3 until
deltais belowtgt.tolortgt.iterssweeps have run, and then returnz.
Arguments
tgt: The regression target, which givesitersandtol.P: The standardised design,observations × columns.Q: The weighted design, $\mathbf{V} \mathbf{P}$.y: The response.z: The coefficients of the start point. The function changes them.r: The residual of the start point. The function changes it.nrm2: The weighted squared norm of each column.a: The threshold of each column.ridge: The ridge term of each column.pen: Whether each column is penalised.
Returns
z::VecNum: The coefficients on the columns ofP.
Related
PortfolioOptimisers.elastic_net_polish — Function
elastic_net_polish(P::MatNum, Q::MatNum, y::VecNum, z::VecNum, a::VecNum,
ridge::VecNum, pen::AbstractVector{Bool}) -> Option{VecNum}Returns the exact elastic-net optimum on the sign pattern of z when that pattern is the pattern of the optimum, and nothing otherwise.
A point that meets the optimality conditions below minimises the objective of ElasticNet, because the objective is convex. For theta < 1 the objective is strictly convex in the penalised coefficients, and the point is its only minimiser. At theta = 1 the objective is the lasso, and it can have more than one minimiser. Then the equations of the active columns are singular when the pattern holds more columns than there are observations, and the function returns nothing for such a pattern. The intercept column is always active, and its sign is free.
Mathematical definition
With $\mathcal{A}$ the active columns, the optimum on the sign pattern $\boldsymbol{s}$ solves
\[\begin{align} \left(\mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\right) \boldsymbol{z}_{\mathcal{A}} &= \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \boldsymbol{y} - \gamma \vartheta\, \boldsymbol{s}\,, \end{align}\]
and it is the optimum of the objective when
\[\begin{align} \operatorname{sign}(z_k) &= s_k\,,\quad k \in \mathcal{A} \cap \mathcal{K}\,,\\ \lvert \mathbf{P}_k^\intercal \mathbf{V} (\boldsymbol{y} - \mathbf{P}_{\mathcal{A}} \boldsymbol{z}_{\mathcal{A}}) \rvert &\leq \gamma \vartheta\,,\quad k \notin \mathcal{A}\,. \end{align}\]
Where:
- $\mathcal{A}$: Active set, the columns whose coefficient in $\boldsymbol{z}$ is nonzero, and the intercept column.
- $\mathbf{P}_{\mathcal{A}}$, $\boldsymbol{z}_{\mathcal{A}}$: Columns of the regressors and coefficients of the active set.
- $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
- $\boldsymbol{s}$: Signs of the coefficients of the active set in $\boldsymbol{z}$, zero for the intercept column.
- $\boldsymbol{z}$: Coefficients of the elastic net, one per column of $\mathbf{P}$. Coefficient $k$ is $z_k$.
- $\mathbf{P}$: Regressors of the elastic net, one row per observation and one column per regressor. Column $k$ is $\mathbf{P}_k$.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\boldsymbol{y}$: Response of the elastic net, one entry per observation.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet. - $\mathcal{K}$: Penalised columns of the elastic net, every column of $\mathbf{P}$ but the intercept column that an estimator names.
Algorithm
- Find
active, the nonzero coefficients ofzand the column thatpendoes not penalise, and their indicesA. WhenAis empty, returnnothing. - Take
s, the signs of the coefficients ofA, zero at the intercept column. Factorise the matrix of the equations of the active columns, which givesF. When the matrix is singular, returnnothing. - Solve the equations with
F, which giveszA. When a penalised coefficient ofzAdoes not keep its sign ins, returnnothing. - Write
zAintozp, zero elsewhere, and computecorr, the product of each column ofQwith the residual. - When an inactive column has
abs(corr[k]) > a[k], returnnothing. Otherwise returnzp.
Arguments
P: The standardised design,observations × columns.Q: The weighted design, $\mathbf{V} \mathbf{P}$.y: The response.z: The coefficients whose sign pattern the function tries.a: The threshold of each column, $\gamma \vartheta$ or zero at the intercept column.ridge: The ridge term of each column, $\gamma (1 - \vartheta)$ or zero at the intercept column.pen: Whether each column is penalised.
Returns
z'::Option{VecNum}: The optimum, ornothing.
Related
PortfolioOptimisers.ElasticNetCounts — Type
struct ElasticNetCounts{__T_df, __T_rss, __T_nobs, __T_rdof}Holds the effective degrees of freedom, the weighted residual sum of squares, the number of observations and the residual count of a fitted ElasticNet.
This type is an implementation detail and is not intended for direct use. elastic_net_counts makes it, and StatsAPI.fit(::ElasticNet, ::MatNum, ::VecNum) copies its fields into the ElasticNetFit that it returns. ElasticNetFit states each count and its derivation.
Fields
df:df: The effective degrees of freedom of the fit, the trace of the smoother on the active set plus the term ofsdiv.
rss:rss: The weighted residual sum of squares of the fit, under the frequency weights ofglm_weights.
nobs:nobs: The number of observations, the sum of the frequency weights, or the number of rows without weights.
rdof:rdof: The residual count, the number of rows less $\operatorname{tr}(2 \mathbf{S} - \mathbf{S}^\intercal \mathbf{S})$.
Related
PortfolioOptimisers.elastic_net_counts — Function
elastic_net_counts(tgt::ElasticNet, sys::NamedTuple) -> ElasticNetCountsReturn the effective degrees of freedom, the weighted residual sum of squares, the number of observations and the residual count of a fitted ElasticNet.
ElasticNetFit states the counts and their derivation. The method reads them off the equations of the active set that elastic_net_polish solves, so it needs the pseudo-inverse of one matrix with one row per active column.
Mathematical definition
\[\begin{align} \mathbf{B} &= \mathbf{M}^{-1} \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}}\,,\quad \mathbf{M} = \mathbf{P}_{\mathcal{A}}^\intercal \mathbf{V} \mathbf{P}_{\mathcal{A}} + \gamma (1 - \vartheta) \mathbf{D}_{\mathcal{A}}\,,\\ \operatorname{tr}(\mathbf{H}) &= \operatorname{tr}(\mathbf{B})\,,\quad \operatorname{tr}(\mathbf{S}^\intercal \mathbf{S}) = \operatorname{tr}(\mathbf{B}^{2})\,. \end{align}\]
Where:
- $\mathbf{B}$: Matrix with one row and one column per active column. It has the nonzero eigenvalues of $\mathbf{H}$.
- $\mathbf{M}$: Matrix of the equations of the active set.
- $\mathbf{H}$, $\mathbf{S}$: Smoother of the fit and smoother of the weighted rows, as
ElasticNetFitstates them. - $\mathbf{P}_{\mathcal{A}}$: Columns of the active set.
- $\mathbf{D}_{\mathcal{A}}$: Diagonal matrix with one for each penalised column of the active set and zero for the intercept column.
- $\mathbf{V}$: Diagonal matrix of the weights of the observations of the elastic net, the identity when the fit has no observation weights.
- $\gamma$: Regularisation strength of the elastic net.
- $\vartheta$: Share of the $L_1$ penalty in the elastic net, the
thetafield ofElasticNet.
The trace is invariant under a cyclic permutation, so $\operatorname{tr}(\mathbf{H}) = \operatorname{tr}(\mathbf{B})$. $\mathbf{S}$ is symmetric, and $\mathbf{S}^{2}$ has the trace of $\mathbf{B}^{2}$ for the same reason.
Algorithm
- Find
A, the indices of the nonzero coefficients ofsys.zand of the column thatsys.pendoes not penalise. - Build the matrix of the equations of the active set, and take its pseudo-inverse
Mi. Fortheta < 1the matrix is positive definite, andMiis its inverse. Attheta = 1the matrix is singular when the active set holds more columns than there are observations, andBis then the projection onto the row space of the active columns, whose trace is their rank. An LU solve of such a matrix does not always fail, and it returns rounding noise in place of the rank. - Multiply
Miby the weighted Gram matrix of the active set, which givesB. TaketH, the trace ofB. - Add the term of
tgt.sdivtotHwithstrength_divergence, which givesdf. - Compute the weighted residual sum of squares
rssand the number of observationsnobsundersys.w. - Take the residual count
rdof, the number of rows less2 tH - tr(B^2).
Arguments
tgt: The regression target.sys: The fit, as the# Interfacessection ofAbstractStrengthDivergencestates it, withoutAandMi.
Returns
cnt::ElasticNetCounts: The counts thatElasticNetFitholds.
Related
References
- [69]
- C. M. Hurvich, J. S. Simonoff and C.-L. Tsai. Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, 271–293 (1998).
- [70]
- H. Zou, T. Hastie and R. Tibshirani. On the ``degrees of freedom'' of the lasso. The Annals of Statistics 35, 2173–2192 (2007), arXiv:0712.0881.
- [109]
- R. J. Tibshirani and J. Taylor. Degrees of freedom in lasso problems. The Annals of Statistics 40, 1198–1232 (2012), arXiv:1111.0653.
- [298]
- T. J. Hastie and R. J. Tibshirani. Generalized Additive Models. No. 43 of Monographs on Statistics and Applied Probability (Chapman and Hall, London, 1990).
- [299]
- T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2 Edition (Springer, New York, 2009).