Classification with Hundred Hammers#
In this notebook we will explain how to use the HundredHammers library to perform a basic model selection and hyperparameter optimization for a classification problem.
To do this, we will use one of the example datasets available in the scikit-learn library.
[1]:
import logging
import hundred_hammers as hh
from hundred_hammers.model_zoo import (
DummyClassifier,
KNeighborsClassifier,
RidgeClassifier,
DecisionTreeClassifier,
)
from sklearn.datasets import load_iris
First we store the data in the X (input) and y (target) variables.
[2]:
data = load_iris()
X = data.data
y = data.target
We are going to first train some models with their default configuration. If you don’t specify the models that you want to use, some classifiers will be chosen for you.
To see which models are chosen by default, you can check the DEFAULT_CLASSIFICATION_MODELS variable
[3]:
hh.model_zoo.DEFAULT_CLASSIFICATION_MODELS
[3]:
[('Dummy', DummyClassifier(strategy='most_frequent'), {}),
('Decision Tree', DecisionTreeClassifier(random_state=0), {}),
('SVC', SVC(gamma='auto'), {}),
('Linear SVC', LinearSVC(random_state=0, tol=1e-05), {}),
('Perceptron', Perceptron(), {}),
('Logistic Regression', LogisticRegression(random_state=0), {}),
('Ridge Classifier', RidgeClassifier(random_state=0), {}),
('SGD Classifier', SGDClassifier(random_state=0), {}),
('Passive Aggressive Classifier',
PassiveAggressiveClassifier(random_state=0),
{}),
('K Neighbors Classifier', KNeighborsClassifier(), {}),
('Neural Network Classifier', MLPClassifier(random_state=0), {}),
('Gaussian Process Classifier',
GaussianProcessClassifier(random_state=0),
{}),
('Random Forest Classifier', RandomForestClassifier(random_state=0), {}),
('AdaBoost Classifier', AdaBoostClassifier(random_state=0), {}),
('Gradient Boosting Classifier',
GradientBoostingClassifier(random_state=0),
{}),
('Extra Trees Classifier', ExtraTreesClassifier(random_state=0), {}),
('Bagging Classifier', BaggingClassifier(random_state=0), {})]
Notice that it is composed of a list of tuples. Each tuple contains the name we give to the classifier, an instance of the class that implements the classifier and a grid of hyperparameters (which now is empty, but will be explained later).
Those are the models that we are going to use now.
Evaluation with default models#
First create the HundredHammersClassifier object. We want to see the progress of the evaluation of the models, so we indicate it in the constructor of the class.
[4]:
hh_models = hh.HundredHammersClassifier(show_progress_bar=True)
Then evaluate the models. Apart from the actual data (the variables X and y), you can pass other parameters. optim_hyper checks whether we want to optimize the hyperparameters of the models and n_grid_points controls how many values from each hyperparameter to check in the optimization.
We will leave hyperparameter optimization for later, so for now optim_hyper is set as false.
[5]:
# configure the logger
hh.hh_logger.setLevel(logging.WARNING)
# Evaluate the models and store the results in a variable
df_results = hh_models.evaluate(X, y, optim_hyper=False)
Evaluating models...: 100%|██████████| 17/17 [00:40<00:00, 2.37s/it]
Notice the line above the evaluation of the models. This configures the logger to only show warnings (of which there should be none). The setting you most likely would want to use in an interactive enviroment would be logging.INFO, since you get information about each model in “real time”.
If you want to see more detailed information, you can set the level to logging.DEBUG. It outputs a lot of information, but it might be useful if you encounter a bug.
For the purposes of this notebook, it will be kept to logging.WARNING but you are welcome to change it if you are running this notebook locally.
We can now show the results of our execution
[6]:
df_results
[6]:
| Model | Avg ACC (Validation Train) | Std ACC (Validation Train) | Avg ACC (Validation Test) | Std ACC (Validation Test) | Avg ACC (Train) | Std ACC (Train) | Avg ACC (Test) | Std ACC (Test) | Avg F1W (Validation Train) | Std F1W (Validation Train) | Avg F1W (Validation Test) | Std F1W (Validation Test) | Avg F1W (Train) | Std F1W (Train) | Avg F1W (Test) | Std F1W (Test) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Dummy | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 |
| 1 | Decision Tree | 1.000000 | 0.000000e+00 | 0.939167 | 3.556098e-02 | 1.000000 | 0.000000e+00 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.938725 | 3.602842e-02 | 1.000000 | 0.000000e+00 | 0.966583 | 1.110223e-16 |
| 2 | SVC | 0.963750 | 1.288302e-02 | 0.948333 | 4.213075e-02 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.963732 | 1.289618e-02 | 0.947760 | 4.309094e-02 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 |
| 3 | Linear SVC | 0.966667 | 8.068715e-03 | 0.953333 | 3.876568e-02 | 0.968333 | 3.333333e-03 | 0.966667 | 1.110223e-16 | 0.966648 | 8.077152e-03 | 0.952942 | 3.920657e-02 | 0.968333 | 3.331771e-03 | 0.966583 | 1.110223e-16 |
| 4 | Perceptron | 0.715417 | 1.401023e-01 | 0.709167 | 1.526366e-01 | 0.675000 | 1.895902e-01 | 0.693333 | 2.015496e-01 | 0.640206 | 1.857964e-01 | 0.628819 | 1.991136e-01 | 0.612396 | 2.376406e-01 | 0.628800 | 2.560143e-01 |
| 5 | Logistic Regression | 0.960625 | 1.088330e-02 | 0.944167 | 4.049177e-02 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.960613 | 1.088587e-02 | 0.943516 | 4.156557e-02 | 0.966646 | 1.110223e-16 | 1.000000 | 0.000000e+00 |
| 6 | Ridge Classifier | 0.873958 | 2.322239e-02 | 0.860000 | 6.601767e-02 | 0.875000 | 0.000000e+00 | 0.833333 | 0.000000e+00 | 0.872920 | 2.371401e-02 | 0.856881 | 6.857956e-02 | 0.874036 | 1.110223e-16 | 0.829497 | 1.110223e-16 |
| 7 | SGD Classifier | 0.803542 | 1.149994e-01 | 0.789167 | 1.158933e-01 | 0.846667 | 1.013246e-01 | 0.863333 | 1.058825e-01 | 0.760521 | 1.589352e-01 | 0.742811 | 1.569199e-01 | 0.819405 | 1.344465e-01 | 0.840678 | 1.381196e-01 |
| 8 | Passive Aggressive Classifier | 0.822083 | 9.702251e-02 | 0.814167 | 1.000729e-01 | 0.845000 | 9.137833e-02 | 0.866667 | 9.067647e-02 | 0.790508 | 1.306401e-01 | 0.778717 | 1.387220e-01 | 0.823307 | 1.176445e-01 | 0.851415 | 1.100022e-01 |
| 9 | K Neighbors Classifier | 0.962708 | 1.251562e-02 | 0.942500 | 4.073116e-02 | 0.958333 | 0.000000e+00 | 1.000000 | 0.000000e+00 | 0.962680 | 1.252658e-02 | 0.941871 | 4.140596e-02 | 0.958327 | 1.110223e-16 | 1.000000 | 0.000000e+00 |
| 10 | Neural Network Classifier | 0.969167 | 8.829008e-03 | 0.965833 | 3.404450e-02 | 0.970000 | 4.082483e-03 | 0.996667 | 1.000000e-02 | 0.969096 | 8.875182e-03 | 0.965445 | 3.464766e-02 | 0.969945 | 4.104926e-03 | 0.996658 | 1.002506e-02 |
| 11 | Gaussian Process Classifier | 0.957292 | 1.321727e-02 | 0.937500 | 4.583333e-02 | 0.950000 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.957271 | 1.323461e-02 | 0.936587 | 4.725891e-02 | 0.949969 | 1.110223e-16 | 1.000000 | 0.000000e+00 |
| 12 | Random Forest Classifier | 1.000000 | 0.000000e+00 | 0.945000 | 4.034573e-02 | 1.000000 | 0.000000e+00 | 0.953333 | 1.632993e-02 | 1.000000 | 0.000000e+00 | 0.944500 | 4.096522e-02 | 1.000000 | 0.000000e+00 | 0.953014 | 1.661890e-02 |
| 13 | AdaBoost Classifier | 0.941458 | 7.079932e-02 | 0.920000 | 8.284859e-02 | 0.958333 | 0.000000e+00 | 0.933333 | 0.000000e+00 | 0.936119 | 8.755802e-02 | 0.912969 | 1.020773e-01 | 0.958327 | 1.110223e-16 | 0.932660 | 1.110223e-16 |
| 14 | Gradient Boosting Classifier | 1.000000 | 0.000000e+00 | 0.944167 | 4.049177e-02 | 1.000000 | 0.000000e+00 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.943731 | 4.094640e-02 | 1.000000 | 0.000000e+00 | 0.966583 | 1.110223e-16 |
| 15 | Extra Trees Classifier | 1.000000 | 0.000000e+00 | 0.940000 | 4.262237e-02 | 1.000000 | 0.000000e+00 | 0.986667 | 1.632993e-02 | 1.000000 | 0.000000e+00 | 0.939543 | 4.294486e-02 | 1.000000 | 0.000000e+00 | 0.986633 | 1.637086e-02 |
| 16 | Bagging Classifier | 0.993333 | 8.003905e-03 | 0.945833 | 3.841477e-02 | 0.993333 | 6.236096e-03 | 0.956667 | 2.134375e-02 | 0.993331 | 8.006943e-03 | 0.945411 | 3.890909e-02 | 0.993332 | 6.237907e-03 | 0.956356 | 2.161894e-02 |
That’s an ok way of displaying the result, but tables can sometimes be hard to read, this is why we also implement a couple of functions to display the information of the table in a more readable format.
[7]:
hh.plot_batch_results(df_results, metric_name="ACC", title="Iris Dataset", display=False)
[8]:
hh.plot_confusion_matrix(
X,
y,
class_dict={0: "Setosa", 1: "Versicolor", 2: "Virginica"},
model=KNeighborsClassifier(),
title="Iris Dataset",
display=False,
)
In case we needed to use one of the trained models, we can take it from the trained_models attribute from the HundredHammersClassifier class. This value will consist on a list with tuples containing the name of the model and the trained model.
[9]:
hh_models.trained_models
[9]:
[('Dummy', DummyClassifier(random_state=9, strategy='most_frequent'), {}),
('Decision Tree', DecisionTreeClassifier(random_state=9), {}),
('SVC', SVC(gamma='auto', random_state=9), {}),
('Linear SVC', LinearSVC(random_state=9, tol=1e-05), {}),
('Perceptron', Perceptron(random_state=9), {}),
('Logistic Regression', LogisticRegression(random_state=9), {}),
('Ridge Classifier', RidgeClassifier(random_state=9), {}),
('SGD Classifier', SGDClassifier(random_state=9), {}),
('Passive Aggressive Classifier',
PassiveAggressiveClassifier(random_state=9),
{}),
('K Neighbors Classifier', KNeighborsClassifier(), {}),
('Neural Network Classifier', MLPClassifier(random_state=9), {}),
('Gaussian Process Classifier',
GaussianProcessClassifier(random_state=9),
{}),
('Random Forest Classifier', RandomForestClassifier(random_state=9), {}),
('AdaBoost Classifier', AdaBoostClassifier(random_state=9), {}),
('Gradient Boosting Classifier',
GradientBoostingClassifier(random_state=9),
{}),
('Extra Trees Classifier', ExtraTreesClassifier(random_state=9), {}),
('Bagging Classifier', BaggingClassifier(random_state=9), {})]
Automatic optimization of hyperparameters#
Now, let us take a look at how the hyperparameter optimization works. Let us assume that we do not want to use every model that we saw earlier, but only a select few: in this case we need to create a model list and pass it to the HundredHammersClassifier class.
For this example, we will use four simple classifier models.
[10]:
models_to_check = [
("Dummy", DummyClassifier(), None),
("Decision Tree", DecisionTreeClassifier(random_state=0), None),
("Ridge Classifer", RidgeClassifier(random_state=0), None),
("KNN", KNeighborsClassifier(), None),
]
Each model has a name and an object that implements it. The third position in the tuple represents the user-specified grid of hyperparameters, however, we will let them be automatially generated.
This will only happen for already configured models, if you want automatic generation of hyperparameters for a model that is not already added, check the “example_add_model.ipynb” notebook.
We can now proceed passing these models to the HundredHammersClassifier class.
[11]:
hh_models = hh.HundredHammersClassifier(models=models_to_check, show_progress_bar=True)
This time, since we want to optimize the hyperparameters of our models, we set the appropriate parameter to True.
We can configure how many parameters to check in the GridSearch step, n_grid_points will indicate how many values each of the hyperparameters will take. In this case, we will take 8 values for each one.
(In the case of categorical values, if there are less than 8 values, only those will be taken.)
[12]:
df_results = hh_models.evaluate(X, y, optim_hyper=True, n_grid_points=8)
Evaluating models...: 100%|██████████| 4/4 [00:00<00:00, 4.65it/s]
[13]:
df_results
[13]:
| Model | Avg ACC (Validation Train) | Std ACC (Validation Train) | Avg ACC (Validation Test) | Std ACC (Validation Test) | Avg ACC (Train) | Std ACC (Train) | Avg ACC (Test) | Std ACC (Test) | Avg F1W (Validation Train) | Std F1W (Validation Train) | Avg F1W (Validation Test) | Std F1W (Validation Test) | Avg F1W (Train) | Std F1W (Train) | Avg F1W (Test) | Std F1W (Test) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Dummy | 0.337083 | 0.046991 | 0.344167 | 0.098886 | 0.3375 | 0.056673 | 0.290000 | 9.315459e-02 | 0.334173 | 0.048363 | 0.335000 | 0.099751 | 0.336118 | 5.640639e-02 | 0.281269 | 9.491669e-02 |
| 1 | Decision Tree | 0.997917 | 0.004167 | 0.938333 | 0.039299 | 1.0000 | 0.000000 | 0.966667 | 1.110223e-16 | 0.997916 | 0.004168 | 0.937882 | 0.039750 | 1.000000 | 0.000000e+00 | 0.966583 | 1.110223e-16 |
| 2 | Ridge Classifer | 0.872083 | 0.018286 | 0.863333 | 0.061260 | 0.8750 | 0.000000 | 0.800000 | 0.000000e+00 | 0.871381 | 0.018568 | 0.861076 | 0.063023 | 0.874510 | 1.110223e-16 | 0.797980 | 1.110223e-16 |
| 3 | KNN | 1.000000 | 0.000000 | 0.967500 | 0.026732 | 1.0000 | 0.000000 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000 | 0.967359 | 0.026863 | 1.000000 | 0.000000e+00 | 0.966583 | 1.110223e-16 |
Now that we have optimized the hyperparameters of the models, we can check which hyperparameters were chosen for each. This is done by checking the best_params attribute.
[14]:
hh_models.best_params
[14]:
[('Dummy', {'strategy': 'stratified'}),
('Decision Tree', {'criterion': 'entropy', 'max_depth': 6}),
('Ridge Classifer', {'alpha': 1e-10}),
('KNN', {'metric': 'cosine', 'n_neighbors': 1})]
If we use the in-built visualizations to compare the models:
[15]:
hh.plot_batch_results(df_results, metric_name="ACC", title="Iris Dataset", display=False)
Optimization of hyperparameters with custom parameter grids#
For this example, we will use four simple classifier models with grids of hyperparameters.
These grid will contain all the parameters that the gridsearch optimization will use.
[16]:
models_to_check = [
("Dummy", DummyClassifier(), {"strategy": ["most_frequent"]}),
(
"Decision Tree",
DecisionTreeClassifier(random_state=0),
{
"criterion": ["gini", "entropy", "log_loss"],
"max_depth": [1, 2, 3, 4, 5, 6, 7],
},
),
(
"Ridge Classifer",
RidgeClassifier(random_state=0),
{"alpha": [1e-4, 1e-3, 1e-2, 0.1, 1, 10]},
),
(
"KNN",
KNeighborsClassifier(),
{"n_neighbors": [1, 3, 5, 7, 9, 11], "metric": ["manhattan", "euclidean"]},
),
]
We can now proceed passing these models to the HundredHammersClassifier class.
[17]:
hh_models = hh.HundredHammersClassifier(models=models_to_check, show_progress_bar=True)
Since we want to optimize the hyperparameters of our models, we set the appropiate parameter to True.
We don’t need to set the n_grid_points parameter since we have already chosen which parameters to take in the GridSearch step.
[18]:
df_results = hh_models.evaluate(X, y, optim_hyper=True)
Evaluating models...: 100%|██████████| 4/4 [00:00<00:00, 5.08it/s]
[19]:
df_results
[19]:
| Model | Avg ACC (Validation Train) | Std ACC (Validation Train) | Avg ACC (Validation Test) | Std ACC (Validation Test) | Avg ACC (Train) | Std ACC (Train) | Avg ACC (Test) | Std ACC (Test) | Avg F1W (Validation Train) | Std F1W (Validation Train) | Avg F1W (Validation Test) | Std F1W (Validation Test) | Avg F1W (Train) | Std F1W (Train) | Avg F1W (Test) | Std F1W (Test) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Dummy | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.333333 | 5.551115e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 | 0.166667 | 2.775558e-17 |
| 1 | Decision Tree | 0.968125 | 8.692769e-03 | 0.958333 | 3.632416e-02 | 0.966667 | 1.110223e-16 | 0.933333 | 0.000000e+00 | 0.968099 | 8.705309e-03 | 0.958156 | 3.648369e-02 | 0.966646 | 1.110223e-16 | 0.932660 | 1.110223e-16 |
| 2 | Ridge Classifer | 0.872083 | 1.828592e-02 | 0.863333 | 6.125992e-02 | 0.875000 | 0.000000e+00 | 0.800000 | 0.000000e+00 | 0.871381 | 1.856780e-02 | 0.861076 | 6.302271e-02 | 0.874510 | 1.110223e-16 | 0.797980 | 1.110223e-16 |
| 3 | KNN | 0.963542 | 1.201309e-02 | 0.946667 | 4.333333e-02 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 | 0.963528 | 1.202337e-02 | 0.946001 | 4.432124e-02 | 0.966667 | 1.110223e-16 | 1.000000 | 0.000000e+00 |
Now that we have optimized the hyperparameters of the models, we can check which hyperparameters were chosen for each. This is done by checking the best_params attribute.
[20]:
hh_models.best_params
[20]:
[('Dummy', {'strategy': 'most_frequent'}),
('Decision Tree', {'criterion': 'gini', 'max_depth': 2}),
('Ridge Classifer', {'alpha': 0.0001}),
('KNN', {'metric': 'euclidean', 'n_neighbors': 11})]
We can also show the plots like last time.
[21]:
hh.plot_batch_results(df_results, metric_name="ACC", title="Iris Dataset", display=False)
