Stepwise Regression: private API

_regression(re::StepwiseRegression{<:PValue, <:ForwardSelection}, x::VecNum,
           F::MatNum)

Grows a factor set from empty, admitting the model whose p-values all clear the threshold.

A candidate is admissible only when every coefficient of that candidate model is significant, so a factor that is significant on its own is rejected when it makes an incumbent insignificant.

Algorithm

1. Set included empty and val to zero, so the loop runs at least once.
2. While val does not exceed re.crit.t, do steps 3 to 6.
3. Take the factors that included does not hold, giving excluded.
4. For each i of excluded, fit re.tgt to an intercept column and the columns [included; i], and read the p-value of column i, giving test_pval. Keep i as new_factor when test_pval is the smallest so far and no p-value of that candidate model exceeds re.crit.t, and keep that model's p-values as pvals.
5. Stop the loop when step 4 kept no factor. Otherwise push new_factor onto included.
6. Set val to the largest entry of pvals.
7. Call add_best_factor_after_pval_failure!, which acts only when included is still empty.

Arguments

• re: Stepwise regression estimator with a PValue criterion and the ForwardSelection algorithm.
• x: Response vector, observations × 1.
• F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2.

Returns

• included::Vector{Int}: Indices of the selected factors, in the order the loop added them, so it is not sorted. It holds at least one index.

Related

• StepwiseRegression
• PValue
• ForwardSelection
• add_best_factor_after_pval_failure!

_regression(re::StepwiseRegression{<:MinMaxValStepwiseRegressionCriterion,
                                  <:ForwardSelection}, x::VecNum, F::MatNum)

Grows a factor set from empty, adding the factor whose model scores best.

The starting score is the worst value the criterion can take, so the first addition always happens and the selection is never empty. get_forward_reg_incl_excl! reads the direction from regression_polarity, so one loop serves a minimised and a maximised criterion alike.

Algorithm

1. Set included empty and excluded to every factor index, in ascending order.
2. Read the criterion from re.crit and re.tgt with regression_criterion_func, giving criterion_func, and the starting score from regression_threshold, giving t.
3. Fill value with that same worst score, one entry per factor.
4. Do steps 5 to 7 at most once per observation.
5. For each i of excluded, fit re.tgt to an intercept column and the columns [included; i], and write criterion_func of that fit into value[i].
6. Call get_forward_reg_incl_excl!, which moves the best factor and returns the new t.
7. Stop when step 6 moved no factor.

Arguments

• re: Stepwise regression estimator with a MinMaxValStepwiseRegressionCriterion criterion and the ForwardSelection algorithm.
• x: Response vector, observations × 1.
• F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2.

Returns

• included::Vector{Int}: Indices of the selected factors, in the order the loop added them, so it is not sorted. It holds at least one index.

Related

• StepwiseRegression
• MinValStepwiseRegressionCriterion
• MaxValStepwiseRegressionCriterion
• ForwardSelection
• get_forward_reg_incl_excl!
• regression_criterion_func
• regression_threshold

_regression(re::StepwiseRegression{<:PValue, <:BackwardElimination}, x::VecNum,
           F::MatNum)

Shrinks a factor set from full, dropping the factor of largest p-value while any exceeds the threshold.

Algorithm

1. Fit re.tgt to an intercept column and every column of F, read the coefficient p-values, and set val to the largest of them.
2. Set included to every factor index and excluded empty.
3. While val exceeds re.crit.t, do steps 4 to 6.
4. Set included to the factors that excluded does not hold. Stop the loop when included is empty.
5. Fit re.tgt to an intercept column and the columns included, giving pvals.
6. Set val to the largest entry of pvals, and push that entry's factor onto excluded. The push acts only when step 3 runs the loop again, because step 4 is what reads excluded, so the last iteration's push is discarded.
7. Call add_best_factor_after_pval_failure!, which acts only when included is empty.

Arguments

• re: Stepwise regression estimator with a PValue criterion and the BackwardElimination algorithm.
• x: Response vector, observations × 1.
• F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2.

Returns

• included::VecInt: Indices of the selected factors, in ascending order. It holds at least one index. It is the 1:size(F, 2) range itself when the full model already passes the threshold, and a Vector{Int} otherwise.

Related

• StepwiseRegression
• PValue
• BackwardElimination
• add_best_factor_after_pval_failure!

_regression(re::StepwiseRegression{<:MinMaxValStepwiseRegressionCriterion,
                                  <:BackwardElimination}, x::VecNum, F::MatNum)

Shrinks a factor set from full, removing the factor whose reduced model scores best.

value[j] is the score of the model that omits j, so the reading of "best" is the same as the forward direction's and not its inverse: under a minimised criterion the code removes the factor of lowest value, and under a maximised one the factor of highest.

Algorithm

1. Set included to every factor index, in ascending order.
2. Read the criterion from re.crit and re.tgt with regression_criterion_func, giving criterion_func.
3. Fit re.tgt to an intercept column and every column of F, and set t to criterion_func of that full model.
4. Fill value with the worst score the criterion can take, one entry per factor.
5. Do steps 6 to 8 at most once per observation.
6. For each factor of included, fit re.tgt to an intercept column and the other columns of included, and write criterion_func of that fit into that factor's entry of value. Fit the intercept column alone when included holds one factor.
7. Call get_backward_reg_incl!, which removes the best factor and returns the new t.
8. Stop when step 7 removed no factor.

Arguments

• re: Stepwise regression estimator with a MinMaxValStepwiseRegressionCriterion criterion and the BackwardElimination algorithm.
• x: Response vector, observations × 1.
• F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2.

Returns

• included::Vector{Int}: Indices of the selected factors, in ascending order. It can be empty: a criterion that rewards every removal removes every factor, which is the common outcome on a response the factors do not explain. The caller regression warns on an empty return, naming the asset, and fits the intercept column alone.

Related

• StepwiseRegression
• MinValStepwiseRegressionCriterion
• MaxValStepwiseRegressionCriterion
• BackwardElimination
• get_backward_reg_incl!
• regression_criterion_func

add_best_factor_after_pval_failure!(tgt::AbstractRegressionTarget,
                                    included::VecInt, F::MatNum,
                                    x::VecNum)

Adds the factor of smallest p-value when a p-value search selected none.

Both PValue methods of _regression call it last, so a p-value search never returns an empty factor set. It does nothing when included already holds a factor, and it warns whenever it adds one, because the factor it adds failed the threshold by construction.

Algorithm

1. Return at once when included is not empty.
2. For each factor i of F, fit tgt to an intercept column and column i, and read that column's p-value, giving test_pval.
3. Keep the smallest test_pval of step 2 and the factor that carries it, giving best_pval and new_factor.
4. Warn, naming new_factor and best_pval.
5. Push new_factor onto included.

Arguments

• tgt: Regression target that fits each candidate model.
• included: Indices of the factors the search selected. Written in place.
• F: Data matrix observations × factors if the dims keyword does not exist or dims = 1, factors × observations when dims = 2.
• x: Response vector, observations × 1.

Returns

• nothing: included gains exactly one index when it was empty, and is unchanged otherwise.

Related

• StepwiseRegression
• PValue
• regression

get_forward_reg_incl_excl!(crit::MinMaxValStepwiseRegressionCriterion,
                           value::VecNum, excluded::VecInt,
                           included::VecInt, t::Number)

Moves the best excluded factor into included when it improves the criterion.

regression_polarity states the direction: findmin and < under a minimised criterion, findmax and > under a maximised one. The search reads the whole value vector, and the answer is still the best excluded factor. An entry of an included factor holds the score that selected it; t is that same score, and it only moves in the improving direction, so no included entry improves on the current t. A score that does improve on t therefore belongs to an excluded factor, and searchsortedfirst always finds its index in excluded.

Algorithm

1. Read best and improves from regression_polarity.
2. Find the best entry of value with best, giving val and its index idx.
3. Return t unchanged when val does not improve on t.
4. Otherwise find the position of idx in excluded, move that entry to the end of included, and set t to val.

Arguments

• crit: Stepwise regression criterion, whose polarity states which direction is better.
• value: Criterion score of each factor. Entry i is the score of the model that adds factor i to included.
• excluded: Indices of the factors outside the model, in ascending order. Written in place.
• included: Indices of the factors inside the model, in the order the search added them. Written in place.
• t: Threshold value.

Returns

• t::Number: The score of the factor that moved, or the input t when no factor moved.

Related

• StepwiseRegression
• regression_polarity
• MinValStepwiseRegressionCriterion
• MaxValStepwiseRegressionCriterion
• get_backward_reg_incl!
• regression

get_backward_reg_incl!(crit::MinMaxValStepwiseRegressionCriterion, value::VecNum,
                       included::VecInt, t::Number)

Removes the best included factor from included when its removal improves the criterion.

regression_polarity states the direction: findmin and < under a minimised criterion, findmax and > under a maximised one. The search reads the whole value vector, and the answer is still the best included factor. An entry of a removed factor holds the score that removed it; t is that same score, and it only moves in the improving direction, so no removed entry improves on the current t. A score that does improve on t therefore belongs to an included factor, and searchsortedfirst always finds its index in included.

Algorithm

1. Read best and improves from regression_polarity.
2. Find the best entry of value with best, giving val and its index idx. That is the best model reachable by one removal.
3. Return t unchanged when val does not improve on t.
4. Otherwise find the position of idx in included, remove that entry, and set t to val.

Arguments

• crit: Stepwise regression criterion, whose polarity states which direction is better.
• value: Criterion score of each factor. Entry j is the score of the model that omits factor j, so the best entry names the removal that helps most.
• included: Indices of the factors inside the model, in ascending order. Written in place.
• t: Threshold value.

Returns

• t::Number: The score of the model left by the removal, or the input t when no factor was removed.

Related

• StepwiseRegression
• regression_polarity
• MinValStepwiseRegressionCriterion
• MaxValStepwiseRegressionCriterion
• get_forward_reg_incl_excl!
• regression