Python Tools for Machine Learning Regression and Classification

PyMLR is a Python module designed to simplify the use of various machine learning libraries and tools for regression and classification tasks. It integrates popular frameworks like scikit-learn, XGBoost, CatBoost, LightGBM, statsmodels, SHAP, and Optuna. This makes it a versatile tool for both beginners and experienced data scientists working on machine learning projects.

Key Features

• Regression and Classification: Supports linear and logistic regression, and many other machine learning regression and classification models.
• Integration with Popular Libraries: Works seamlessly with tools like scikit-learn, XGBoost, CatBoost, LightGBM, and other packages for preprocessing and modeling, and SHAP for explainability.
• Hyperparameter Optimization: Includes functionality for hyperparameter tuning using Optuna.
• Ease of Use: Provides a streamlined interface to reduce the complexity of implementing machine learning workflows.

Use Cases

• Building predictive models for continuous numerical or categorical data.
• Automating hyperparameter tuning for better model performance.
• Explaining model predictions using SHAP values and feature importance.

Description of Functions

The PyMLR module includes the following main Python functions to facilitate the use of sklearn, XGBoost, CatBoost, LightGBM, statsmodels, shap, and optuna solvers for regression and classification:

• logistic_auto - sklearn LogisticRegression with automatic cross-validated optimization of the hyperparameters
• logistic - sklearn LogisticRegression with user-specified hyperparameters
• xgb_auto - XGBoost XGBRegressor (default) or XGBClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• xgb - XGBoost XGBRegressor (default) or XGBClassifier (if optional kwarg classify=True) with user-specified hyperparameters
• catboost_auto - CatBoostRegressor with automatic cross-validated optimization of the hyperparameters
• catboost - CatBoostRegressor with user-specified hyperparameters
• lgbm_auto - LightGBM LGBMRegressor (default) or LGBMClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• lgbm - LightGBM LGBMRegressor (default) or LGBMClassifier with user-specified hyperparameters
• ada_auto - sklearn AdaBoostRegressor (default) or AdaBoostClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• ada - sklearn AdaBoostRegressor (default) or AdaBoostClassifier (if optional keyword argument classify=True) with user-specified hyperparameters
• mlp_auto - sklearn MLPRegressor (default) or MLPClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• mlp - sklearn MLPRegressor (default) or MLPClassifier (if optional keyword argument classify=True) with user-specified hyperparameters
• svr_auto - sklearn SVR (default) or SVC (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• svr - sklearn SVR (default) or SVC (if optional keyword argument classify=True) with user-specified hyperparameters
• knn_auto - sklearn KNeighborsRegressor (default) or KNeighborsClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• knn - sklearn KNeighborsRegressor (default) or KNeighborsClassifier (if optional keyword argument classify=True) with user-specified hyperparameters
• forest_auto - sklearn RandomForestRegressor (default) or RandomForestClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• forest - sklearn RandomForestRegressor (default) or RandomForestClassifier (if optional keyword argument classify=True) with user-specified hyperparameters
• tree_auto - sklearn DecisionTreeRegressor (default) or DecisionTreeClassifier (if optional keyword argument classify=True) with automatic cross-validated optimization of the hyperparameters
• tree - sklearn DecisionTreeRegressor (default) or DecisionTreeClassifier (if optional keyword argument classify=True) with user-specified hyperparameters
• gbr_auto - sklearn GradientBoostingRegressor with automatic cross-validated optimization of the hyperparameters
• gbr - sklearn GradientBoostingRegressor with user-specified hyperparameters
• stacking - sklearn StackingRegressor
• linear_auto - sklearn LinearRegression with optimization of selected features by optuna to minimize MSE
• linear - sklearn LinearRegression using all features to minimize MSE
• lasso - sklearn LassoCV, LassoLarsCV, and LassoLarsIC using AIC/BIC
• ridge - sklearn RidgeCV and Ridge using VIF
• elastic - sklearn ElasticNetCV
• stepwise - Stepwise statsmodels linear regression to minimize MSE, AIC, or BIC
• model_agnostic - Model-agnostic analysis (e.g. Shapley Additive Explantions (SHAP), Beeswarm and Bar plots, PDB-ICE plots, Permutation Importance plots)
• show_optuna - Display the results of the hyperparameter optimization using optuna

Hybrid functions for optimization of hyperparameters in 2-stage hybrid machine learning models, either combining two models in sequence, or using the same model with Recursive Feature Elimination, based on permutation importances of stage 1, include the following:

• xgbmlp_auto - XGBoost for feature selection (stage 1) and MLP for classication or regression (stage 2)
• xgbrfe_auto - Recursive Feature Elimination using XGBoost for feature selection (stage 1) and for classication or regression (stage 2)
• adarfe_auto - Recursive Feature Elimination using AdaBoost for feature selection (stage 1) and for classication or regression (stage 2)

The functions in PyMLR allow the user to write one line of high-level python code to provide the following:

• automatic Kfold cross-validated optimization of hyperparameters by optuna (RepeatedKFold for continuous response variables, StratifiedKFold for binomial or multinomial response variables)
• automatic KFold cross-validated LassoCV, LassoLarsCV, RidgeCV, or ElasticNetCV regression
• optional automated feature selection using SelectKBest, mutual_info regression, and f_regression in the optuna optimization pipeline
• optional automatic standardization of continuous features and output of the fitted StandardScaler
• optional automatic detection and encoding of categorical features and output of the fitted OneHotEncoder
• metrics describing the model skill (e.g. r-squared, adjusted r-squared, RMSE of residuals, p-value of the F-statistic, AIC, BIC, alpha, etc., if applicable)
• model-angostic analysis (plots of predicted vs actual, predicted vs residuals, SHAP Beeswarm, SHAP Importance, Permutation Importance, PDB/ICE)
• plots of optuna optimization history, hyperparameter importance, and contour plots of relationships between hyperparameters and MSE
• plots showing other diagnostic descriptions of the analysis if applicable (e.g. coefficients vs alpha, MSE vs alpha, AIC and BIC vs alpha)
• variance inflation factors (if applicable)
• best fit intercept and model coefficients (if applicable)
• final fitted model object
• optional automatic detection and use of NVIDIA GPU for computations if available (XGBoost)
• optimization of hyperparameters for regression of continuous response variables with either XGBoost XGBRegressor (xgb_auto), CatBoostRegressor (catboost_auto), LightGBM LGBMRegressor (lgbm_auto), sklearn SVR (svr_auto), KNeighborsRegressor (knn_auto), RandomForestRegressor (forest_auto), DecisionTreeRegressor (tree_auto), AdaBoostRegressor (ada_auto), MLPRegressor (mlp_auto), LinearRegression (linear_auto), GradientBoostingRegressor (gbr_auto), or Lasso, Ridge, ElasticNet, or stepwise linear regression
• optimization of hyperparameters for classification of binomial or multinomial response variables with either XGBoost XGBClassifier (xgb_auto), CatBoostClassifier (catboost_auto), LightGBM LGBMClassifier (lgbm_auto), LogisticRegression (logistic_auto), sklearn SVC (svr_auto), KNeighborsClassifier (knn_auto), RandomForestClassifier (forest_auto), DecisionTreeClassifier (tree_auto), AdaBoostClassifier (ada_auto), or MLPClassifier (mlp_auto)
• Hybrid models for optimization of hyperparameters in 2-stage hybrid machine learning models either combining two models in sequence or using the same model with Recursive Feature Elimination based on permutation importances of stage 1:
  • XGBoost-MLP (xgbmlp_auto) uses a 2-stage method in the optuna optimization pipeline as described by Li et al 2022 (https://doi.org/10.3390/forecast4010011). Stage 1 uses feature importances from XGBoost to select features by tuning a hyperparameter for the threshold of feature importance that is optimized by optuna. Stage 2 uses MLPClassifier or MLPRegressor for classification or regression using the selected features that have feature importances greater than the threshold from stage 1.
  • XGBoost with Recursive Feature Elimination (xgbrfe_auto) is similar to the hybrid XGBoost-MLP method, except that stage 1 and 2 both use XGBoost.
  • AdaBoost with Recursive Feature Elimination (adarfe_auto) is similar to the hybrid XGBoost-MLP method, except that stage 1 and 2 both use AdaBoost.

Installation for Python or Jupyter Notebook

The PyMLR module require that you have already installed numpy, pandas, scikit-learn, tabulate, matplotlib, seaborn, statsmodels, xgboost, lightgbm, mlxtend, shap, and optuna packages. In addition, the auto-optimization functions, because they use optuna, require that you have already installed and enabled ipywidgets in your Python environment if you are working in a Jupyter notebook.

If you have not already installed PyMLR, or you are upgrading from a previous installation of PyMLR, enter the following with pip pr !pip in your notebook or terminal:

pip install git+https://github.com/gjpelletier/PyMLR.git --upgrade

---

Examples

The examples folder contains examples using PyMLR in Jupyter Notebooks

---

Additional information on selected topics

Lasso

The lasso function in the PyMLR module provides output of regression models and summary statistics using the following methods using sklearn.linear_model:

• LassoCV: Lasso using Cross-Validation with coordinate descent to optimize alpha
• LassoLarsCV: Lasso using Cross-Validation with Least Angle Regression
• LassoLarsIC using AIC: Lasso using Least Angle Regression with Akaike Information Criterion
• LassoLarsIC using BIC: Lasso using Least Angle Regression with Bayesian Information Criterion

Lasso (Least Absolute Shrinkage and Selection Operator) adds a penalty to the loss function. This penalty encourages sparsity in the model, meaning that some coefficients will be exactly zero, effectively removing the corresponding predictors from the model.

Lasso linear regression includes an L1 penalty term to the standard least squares objective function. The penalty term is a sum of the absolute values of the regression coefficients multiplied by a hyperparameter, denoted as "alpha". The lasso function finds the optimum value of alpha for each of the methods listed above. The alpha determines the amount of shrinkage applied to the model coefficients. As alpha increases, the coefficients are pushed towards zero, and some may become exactly zero, effectively eliminating those features from the model.

Lasso regression is useful for dealing with multicollinearity, where predictors are highly correlated, and when an optimal subset of the candidate features should be included in the model.

Ridge

The ridge function in the PyMLR module provides output of regression models and summary statistics using the following methods using sklearn.linear_model:

• RidgeCV: RidgeCV regression with cross-validation using the MSE as the scoring criterion to optimize alpha
• RidgeVIF: Ridge regression using target VIF to optimize alpha by trial

Ridge regression adds an L2 penalty to the loss function, which is the product of the regularization hyperparameter and the sum of the squares of the coefficients. This penalty shrinks the coefficients towards zero but does not force them to be exactly zero.

Ridge regression is useful for dealing with multicollinearity, where predictors are highly correlated, and when all candidate features should be included in the model.

Elastic Net

The elastic function in the PyMLR module provides output of the fitted regression model and summary statistics using the following method using sklearn.linear_model:

• ElasticNetCV: Elastic regression with cross-validation using the MSE as the scoring criterion to optimize alpha and the L1-ratio that balances between L1 and L2 regularization.

Elastic regression, also know as Elastic Net, is a regularization technique that combines the strengths of Lasso (L1) and Ridge (L2) regression methods. It is particularly useful for handling datasets with high-dimensional features and multicollinearity (correlated features). By blending the penalties of L1 and L2, Elastic Net provides a balance between feature selection (Lasso) and coefficient shrinkage (Ridge).

Stacking

The stacking function in the PyMLR module provides output of a fitted regression model and summary statistics using the sklearn StackingRegressor function for ensemble modeling with any combination of sklearn base regressors which can be turned 'on' or 'off' with the following optional keyword arguments:

• lasso= 'on' (default) or 'off' uses LassoCV
• ridge= 'on' (default) or 'off' uses RidgeCV
• elastic= 'on' (default) or 'off' uses ElasticNetCV
• sgd= 'on' (default) or 'off' uses SGDRegressor
• knr= 'on' (default) or 'off' uses KNeighborsRegressor
• svr= 'on' (default) or 'off' uses SVR(kernel='rbf')
• mlp= 'on' or 'off' (default) uses MLPRegressor
• gbr= 'on' (default) or 'off' uses GradientBoostingRegressor
• tree= 'on' (default) or 'off' uses DecisionTreeRegressor
• forest= 'on' (default) or 'off' uses RandomForestRegressor

The meta-model may be specifed using the optional keyword argument meta:

• meta= 'linear', 'lasso', 'ridge' (default), or 'elastic'

where 'linear' uses LinearRegression, 'lasso' uses LassoCV, 'ridge' uses RidgeCV (default), and 'elastic' uses ElasticNetCV as the meta-model for the final estimator.

Stacking regression is an ensemble machine learning technique that improves predictive accuracy by combining multiple base regression models. Instead of selecting a single best model, stacking leverages multiple models to generate a more robust final prediction.

How Stacking Regression Works

Base Regressors (Level 0 Models):

• Several regression models (e.g., Lasso, Ridge, Elastic, DecisionTree, RandomForest, etc.) are trained independently on the dataset.
• Each model learns different aspects of the data.

Meta-Model (Level 1 Model):

• A separate model (e.g. linear regression, Lasso, or Elastic) is trained to learn from the outputs of base models.
• It assigns strength weights to each base model’s predictions, determining which models contribute the most to final accuracy.

Final Prediction:

• The meta-model makes a final prediction based on the base models' combined outputs.

Advantages of Stacking Regression

– Utilizes the strengths of multiple models
– Helps mitigate overfitting or underfitting – Works well for datasets with nonlinear relationships

Stepwise

The stepwise function in the PyMLR module is intended to be a Python alternative to the R stepAIC function. The stepwise function has the option to use either forward selection (default), backward selection, or all subsets of possible combinations for the optimum set of predictor variables as follows using statsmodels OLS:

• Forward selection (default) starts with no predictors and adds predictors as long as it improves the model (reduces AIC or BIC, or increases adjusted rsquared)
• Backward selection starts with all predictors and removes predictors as long as it improves the model (reduces AIC or BIC, or increases adjusted rsquared)
• All subsets of possible combinations of predictor features to find the best of all possible models (up to 20 candidate predictors)

Either the AIC, BIC, or adjusted rsquared may be used as the criterion with forward, backward, or all subsets. In addition, there is an option to find all features with p-values less than a signficance threshold through backward elimination based only on the p-values of the coefficients. The stepwise algorithm also has the option (default) to remove any non-signficant predictors after either a forward, backward, or all subsets search using the AIC, BIC, or adjusted rsquared criterion.

The choice between Lasso, Ridge, Elastic Net, Stacking, or Stepwise regression depends on the specific context and requirements of the analysis. Stepwise regression is widely used (e.g. Murtaugh, 2009), but often criticized (e.g. Flom and Cassell, 2007). Machine Learning methods such as Lasso, Ridge, and Elastic Net regression are generally preferred for their efficiency and ability to handle large datasets without overfitting. However, Stepwise regression can be more suitable for exploratory data analysis and when the goal is to identify the most influential predictors. Ultimately, the best choice depends on the data characteristics and the researcher's objectives. Stacking regression is an ensemble machine learning technique that improves predictive accuracy by combining multiple base regression models.

Comparison of Stepwise, Lasso, Ridge, and Elastic Net

• Feature selection: Lasso performs explicit feature selection by setting some coefficients to zero, while Ridge shrinks coefficients but retains all predictors. Elastic Net balances the regularization methods of Lasso and Ridge and is able to do feature selection. Stepwise regression also performs feature selection but can be less stable than Lasso.
• Regularization: Lasso, Ridge, and Elastic Net are regularization techniques that prevent overfitting, but they do so differently. Lasso is more likely to produce sparse models, while Ridge is more likely to shrink coefficients smoothly. Elastic Net balances the capabilities of Lasso and Ridge.
• Computational cost: Stepwise regression can be computationally expensive, especially for large datasets. Lasso, Ridge, and Elastic Net can be solved more efficiently using optimization algorithms.

AIC vs BIC

Using AIC as the criterion is the default in the stepwise fuction. The user also has the option to use the BIC as the criterion instead. AIC is considered to be a useful critierion in stepwise regression. However, BIC is generally considered to be better than AIC for several reasons:

• Penalty for Complexity: BIC penalizes models more heavily for the number of parameters, making it more conservative and less likely to overfit, especially with larger sample sizes.
• Model Selection: BIC is particularly useful when the sample size is large, as it encourages simpler models that are less likely to capture noise.
• Model Recovery: Studies suggest that BIC tends to recover the true model more effectively than AIC, particularly in scenarios where the sample size is large. While both criteria are useful for model selection, BIC is often preferred for its stricter criteria, which helps in avoiding overfitting and improving model interpretability

Acceptable VIF as the target for Ridge regression

Ridge regression reduces the Variance Inflation Factors of the features by adding a penalty term to the ordinary least squares regression. The magnitude of the penalty term is related to the regularization paramter (alpha) and the sum of the squared coefficients. At very low values of alpha there is negligible penalty and the Ridge regression results are practically the same as OLS. As the alpha value is increased, the penalty increases and the VIF values of the features decreases, which decreases the magnitude of the coefficients to mitigates the problem of multicollinearity.

Cross-validated ridge regression (e.g. using RidgeCV) does not always result in acceptable multicollinearity as indicated by VIF. While cross-validation helps in fine-tuning the regression coefficients, it does not always result in VIF values close to 1. Ideally the VIF of all features should be as close as possibe to VIF=1. This can be achieved using a trial and error method of evaluating the VIF values of the model features over a range of alpha values.

The ridge function in PyMLR includes an algorithm (RidgeVIF) to find the model with the optimum value of alpha that will result in VIF values as close as possible to a user-specified target VIF (default target VIF=1.0). This assures that there will be acceptable multicollinearity for all features. The trade-off is that this algorithm reduces the model coefficients such that the target VIF will be achieved. The user has the option to specify any target for VIF to explore the balance between VIF and coefficient values.