Start

In this guide, we’ll show how you can use EvalML to automatically find the best pipeline for predicting whether a patient has breast cancer. Along the way, we’ll highlight EvalML’s built-in tools and features for understanding and interacting with the search process.

[1]:
import evalml
from evalml import AutoMLSearch

First, we load in the features and outcomes we want to use to train our model.

[2]:
X, y = evalml.demos.load_breast_cancer()

EvalML has many options to configure the pipeline search. At the minimum, we need to define an objective function. For simplicity, we will use the F1 score in this example. However, the real power of EvalML is in using domain-specific objective functions or building your own.

Below EvalML utilizes Bayesian optimization (EvalML’s default optimizer) to search and find the best pipeline defined by the given objective.

EvalML provides a number of parameters to control the search process. max_iterations is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of pipelines to train and evaluate. In this example, max_iterations is set to 5.

** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.

** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.

[3]:
automl = AutoMLSearch(problem_type="binary", objective="f1", max_iterations=5)

In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.

[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=.2)

To provide data to EvalML, it is recommended that you create a DataTable object using the Woodwork project.

EvalML also accepts and works well with pandas DataFrames. But using the DataTable makes it easy to control how EvalML will treat each feature, as a numeric feature, a categorical feature, a text feature or other type of feature. Woodwork’s DataTable includes features like inferring when a categorical feature should be treated as a text feature. For this reason, if you don’t provide Woodwork objects, EvalML will raise a warning.

[5]:
import woodwork as ww
X_train_dt = ww.DataTable(X_train)
y_train_dc = ww.DataColumn(y_train)

When we call search(), the search for the best pipeline will begin. There is no need to wrangle with missing data or categorical variables as EvalML includes various preprocessing steps (like imputation, one-hot encoding, feature selection) to ensure you’re getting the best results. As long as your data is in a single table, EvalML can handle it. If not, you can reduce your data to a single table by utilizing Featuretools and its Entity Sets.

You can find more information on pipeline components and how to integrate your own custom pipelines into EvalML here.

[6]:
automl.search(X_train_dt, y_train_dc)
Generating pipelines to search over...
*****************************
* Beginning pipeline search *
*****************************

Optimizing for F1.
Greater score is better.

Searching up to 5 pipelines.
Allowed model families: decision_tree, lightgbm, linear_model, xgboost, catboost, extra_trees, random_forest

Batch 1: (1/5) Mode Baseline Binary Classification P... Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean F1: 0.000
2/5) Decision Tree Classifier w/ Imputer      Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean F1: 0.902
3/5) LightGBM Classifier w/ Imputer           Elapsed:00:00
        Starting cross validation
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9
[LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0
        Finished cross validation - mean F1: 0.943
4/5) Extra Trees Classifier w/ Imputer        Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean F1: 0.946
5/5) Elastic Net Classifier w/ Imputer + S... Elapsed:00:01
        Starting cross validation
        Finished cross validation - mean F1: 0.620

Search finished after 00:02
Best pipeline: Extra Trees Classifier w/ Imputer
Best pipeline F1: 0.945557

After the search is finished we can view all of the pipelines searched, ranked by score. Internally, EvalML performs cross validation to score the pipelines. If it notices a high variance across cross validation folds, it will warn you. EvalML also provides additional data checks to analyze your data to assist you in producing the best performing pipeline.

[7]:
automl.rankings
[7]:
id pipeline_name score validation_score percent_better_than_baseline high_variance_cv parameters
0 3 Extra Trees Classifier w/ Imputer 0.945557 0.973451 NaN False {'Imputer': {'categorical_impute_strategy': 'm...
1 2 LightGBM Classifier w/ Imputer 0.942937 0.982456 NaN False {'Imputer': {'categorical_impute_strategy': 'm...
2 1 Decision Tree Classifier w/ Imputer 0.901955 0.921739 NaN False {'Imputer': {'categorical_impute_strategy': 'm...
3 4 Elastic Net Classifier w/ Imputer + Standard S... 0.620287 0.609756 NaN False {'Imputer': {'categorical_impute_strategy': 'm...
4 0 Mode Baseline Binary Classification Pipeline 0.000000 0.000000 NaN False {'Baseline Classifier': {'strategy': 'mode'}}

If we are interested in see more details about the pipeline, we can view a summary description using the id from the rankings table:

[8]:
automl.describe_pipeline(3)
*************************************
* Extra Trees Classifier w/ Imputer *
*************************************

Problem Type: binary
Model Family: Extra Trees

Pipeline Steps
==============
1. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : mean
         * categorical_fill_value : None
         * numeric_fill_value : None
2. Extra Trees Classifier
         * n_estimators : 100
         * max_features : auto
         * max_depth : 6
         * min_samples_split : 2
         * min_weight_fraction_leaf : 0.0
         * n_jobs : -1

Training
========
Training for binary problems.
Total training time (including CV): 1.2 seconds

Cross Validation
----------------
               F1  MCC Binary  Log Loss Binary   AUC  Precision  Balanced Accuracy Binary  Accuracy Binary # Training # Testing
0           0.973       0.958            0.128 0.999      0.982                     0.977            0.980    303.000   152.000
1           0.909       0.859            0.162 0.989      0.943                     0.923            0.934    303.000   152.000
2           0.954       0.929            0.136 0.992      0.981                     0.959            0.967    304.000   151.000
mean        0.946       0.915            0.142 0.993      0.969                     0.953            0.960          -         -
std         0.033       0.051            0.018 0.005      0.022                     0.028            0.024          -         -
coef of var 0.035       0.056            0.124 0.005      0.023                     0.029            0.025          -         -

We can also view the pipeline parameters directly:

[9]:
pipeline = automl.get_pipeline(3)
print(pipeline.parameters)
{'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Extra Trees Classifier': {'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}

We can now select the best pipeline and score it on our holdout data:

[10]:
pipeline = automl.best_pipeline
pipeline.fit(X_train, y_train)
pipeline.score(X_holdout, y_holdout, ["f1"])
[10]:
OrderedDict([('F1', 0.9523809523809523)])

We can also visualize the structure of the components contained by the pipeline:

[11]:
pipeline.graph()
[11]:
_images/start_22_0.svg