Machine learning (ML) is the process of constructing a mathematical model of a system based on a sample dataset collected from that system.
One of the main goals of training an ML model is to teach the model to separate the signal present in the data from the noise inherent in system and in the data collection process. If this is done effectively, the model can then be used to make accurate predictions about the system when presented with new, similar data. Additionally, introspecting on an ML model can reveal key information about the system being modeled, such as which inputs and transformations of the inputs are most useful to the ML model for learning the signal in the data, and are therefore the most predictive.
There are a variety of ML problem types. Supervised learning describes the case where the collected data contains an output value to be modeled and a set of inputs with which to train the model. EvalML focuses on training supervised learning models.
EvalML supports three common supervised ML problem types. The first is regression, where the target value to model is a continuous numeric value. Next are binary and multiclass classification, where the target value to model consists of two or more discrete values or categories. The choice of which supervised ML problem type is most appropriate depends on domain expertise and on how the model will be evaluated and used.
AutoML is the process of automating the construction, training and evaluation of ML models. Given a data and some configuration, AutoML searches for the most effective and accurate ML model or models to fit the dataset. During the search, AutoML will explore different combinations of model type, model parameters and model architecture.
An effective AutoML solution offers several advantages over constructing and tuning ML models by hand. AutoML can assist with many of the difficult aspects of ML, such as avoiding overfitting and underfitting, imbalanced data, detecting data leakage and other potential issues with the problem setup, and automatically applying best-practice data cleaning, feature engineering, feature selection and various modeling techniques. AutoML can also leverage search algorithms to optimally sweep the hyperparameter search space, resulting in model performance which would be difficult to achieve by manual training.
EvalML supports all of the above and more.
In its simplest usage, the AutoML search interface requires only the input data, the target data and a problem_type specifying what kind of supervised ML problem to model.
problem_type
** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.
** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.
Note: To provide data to EvalML, it is recommended that you create a DataTable object using the Woodwork project.
DataTable
EvalML also accepts pandas input, and will run type inference on top of the input pandas data. If you’d like to change the types inferred by EvalML, you can use the infer_feature_types utility method as follows. The infer_feature_types utility method takes pandas or numpy input and converts it to a Woodwork data structure. It takes in a feature_types parameter which can be used to specify what types specific columns should be. In the example below, we specify that the provider, which would have otherwise been inferred as a column with natural language, is a categorical column.
pandas
infer_feature_types
feature_types
[1]:
import evalml from evalml.utils import infer_feature_types X, y = evalml.demos.load_fraud(return_pandas=True) X = infer_feature_types(X, feature_types={'provider': 'categorical'})
Number of Features Boolean 1 Categorical 6 Numeric 5 Number of training examples: 99992 Targets False 84.82% True 15.18% Name: fraud, dtype: object
[2]:
automl = evalml.automl.AutoMLSearch(X_train=X, y_train=y, problem_type='binary') automl.search()
Using default limit of max_batches=1. Generating pipelines to search over... ***************************** * Beginning pipeline search * ***************************** Optimizing for Log Loss Binary. Lower score is better. Searching up to 1 batches for a total of 9 pipelines. Allowed model families: linear_model, decision_tree, random_forest, extra_trees, catboost, xgboost, lightgbm
Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 5.243 Batch 1: (2/9) Logistic Regression Classifier w/ Imp... Elapsed:00:01 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.326 Batch 1: (3/9) Random Forest Classifier w/ Imputer +... Elapsed:00:29 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.252 Batch 1: (4/9) XGBoost Classifier w/ Imputer + DateT... Elapsed:01:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.189 Batch 1: (5/9) CatBoost Classifier w/ Imputer + Date... Elapsed:02:02 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.507 Batch 1: (6/9) Elastic Net Classifier w/ Imputer + D... Elapsed:02:20 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.426 Batch 1: (7/9) Extra Trees Classifier w/ Imputer + D... Elapsed:02:44 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.355 Batch 1: (8/9) LightGBM Classifier w/ Imputer + Date... Elapsed:03:11 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.190 Batch 1: (9/9) Decision Tree Classifier w/ Imputer +... Elapsed:03:35 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.265 High coefficient of variation (cv >= 0.2) within cross validation scores. Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder may not perform as estimated on unseen data. Search finished after 03:55 Best pipeline: XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder Best pipeline Log Loss Binary: 0.188964
The AutoML search will log its progress, reporting each pipeline and parameter set evaluated during the search.
There are a number of mechanisms to control the AutoML search time. One way is to set the max_batches parameter which controls the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. Another way is to set the max_iterations parameter which controls the maximum number of candidate models to be evaluated during AutoML. By default, AutoML will search for a single batch. The first pipeline to be evaluated will always be a baseline model representing a trivial solution.
max_batches
max_iterations
The AutoML interface supports a variety of other parameters. For a comprehensive list, please refer to the API reference.
EvalML includes a simple method, detect_problem_type, to help determine the problem type given the target data.
detect_problem_type
This function can return the predicted problem type as a ProblemType enum, choosing from ProblemType.BINARY, ProblemType.MULTICLASS, and ProblemType.REGRESSION. If the target data is invalid (for instance when there is only 1 unique label), the function will throw an error instead.
[3]:
import pandas as pd from evalml.problem_types import detect_problem_type y = pd.Series([0, 1, 1, 0, 1, 1]) detect_problem_type(y)
<ProblemTypes.BINARY: 'binary'>
AutoMLSearch takes in an objective parameter to determine which objective to optimize for. By default, this parameter is set to auto, which allows AutoML to choose LogLossBinary for binary classification problems, LogLossMulticlass for multiclass classification problems, and R2 for regression problems.
objective
auto
LogLossBinary
LogLossMulticlass
R2
It should be noted that the objective parameter is only used in ranking and helping choose the pipelines to iterate over, but is not used to optimize each individual pipeline during fit-time.
To get the default objective for each problem type, you can use the get_default_primary_search_objective function.
get_default_primary_search_objective
[4]:
from evalml.automl import get_default_primary_search_objective binary_objective = get_default_primary_search_objective("binary") multiclass_objective = get_default_primary_search_objective("multiclass") regression_objective = get_default_primary_search_objective("regression") print(binary_objective.name) print(multiclass_objective.name) print(regression_objective.name)
Log Loss Binary Log Loss Multiclass R2
AutoMLSearch.search runs a set of data checks before beginning the search process to ensure that the input data being passed will not run into some common issues before running a potentially time-consuming search. If the data checks find any potential errors, an exception will be thrown before the search begins, allowing users to inspect their data to avoid confusing errors that may arise later during the search process.
AutoMLSearch.search
This behavior is controlled by the data_checks parameter which can take in either a DataChecks object, a list of DataCheck objects, None, or valid string inputs ("disabled", "auto"). By default, this parameter is set to auto, which runs the default collection of data sets defined in the DefaultDataChecks class. If set to "disabled" or None, no data checks will run.
data_checks
DataChecks
DataCheck
None
"disabled"
"auto"
DefaultDataChecks
EvalML’s AutoML algorithm generates a set of pipelines to search with. To provide a custom set instead, set allowed_pipelines to a list of custom pipeline classes. Note: this will prevent AutoML from generating other pipelines to search over.
[5]:
from evalml.pipelines import MulticlassClassificationPipeline class CustomMulticlassClassificationPipeline(MulticlassClassificationPipeline): component_graph = ['Simple Imputer', 'Random Forest Classifier'] automl_custom = evalml.automl.AutoMLSearch(X_train=X, y_train=y, problem_type='multiclass', allowed_pipelines=[CustomMulticlassClassificationPipeline])
Using default limit of max_batches=1.
To stop the search early, hit Ctrl-C. This will bring up a prompt asking for confirmation. Responding with y will immediately stop the search. Responding with n will continue the search.
Ctrl-C
y
n
A summary of all the pipelines built can be returned as a pandas DataFrame which is sorted by score. The score column contains the average score across all cross-validation folds while the validation_score column is computed from the first cross-validation fold.
[6]:
automl.rankings
Each pipeline is given an id. We can get more information about any particular pipeline using that id. Here, we will get more information about the pipeline with id = 1.
id
id = 1
[7]:
automl.describe_pipeline(1)
******************************************************************************************************************** * Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + Standard Scaler * ******************************************************************************************************************** Problem Type: binary Model Family: Linear Pipeline Steps ============== 1. Imputer * categorical_impute_strategy : most_frequent * numeric_impute_strategy : mean * categorical_fill_value : None * numeric_fill_value : None 2. DateTime Featurization Component * features_to_extract : ['year', 'month', 'day_of_week', 'hour'] * encode_as_categories : False 3. One Hot Encoder * top_n : 10 * features_to_encode : None * categories : None * drop : None * handle_unknown : ignore * handle_missing : error 4. Standard Scaler 5. Logistic Regression Classifier * penalty : l2 * C : 1.0 * n_jobs : -1 * multi_class : auto * solver : lbfgs Training ======== Training for binary problems. Total training time (including CV): 28.5 seconds Cross Validation ---------------- Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary # Training # Validation 0 0.329 0.552 0.831 0.997 0.509 0.671 0.900 66661.000 33331.000 1 0.324 0.533 0.836 0.990 0.486 0.661 0.897 66661.000 33331.000 2 0.325 0.533 0.835 0.993 0.486 0.661 0.897 66662.000 33330.000 mean 0.326 0.539 0.834 0.994 0.494 0.664 0.898 - - std 0.002 0.011 0.003 0.003 0.013 0.006 0.002 - - coef of var 0.007 0.020 0.003 0.003 0.027 0.009 0.002 - -
We can get the object of any pipeline via their id as well:
[8]:
pipeline = automl.get_pipeline(1) print(pipeline.name) print(pipeline.parameters)
Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + Standard Scaler {'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': None, 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Logistic Regression Classifier': {'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'}}
If you specifically want to get the best pipeline, there is a convenient accessor for that. The pipeline returned is already fitted on the input X, y data that we passed to AutoMLSearch. To turn off this default behavior, set train_best_pipeline=False when initializing AutoMLSearch.
train_best_pipeline=False
[9]:
best_pipeline = automl.best_pipeline print(best_pipeline.name) print(best_pipeline.parameters) best_pipeline.predict(X)
XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder {'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': None, 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'XGBoost Classifier': {'eta': 0.1, 'max_depth': 6, 'min_child_weight': 1, 'n_estimators': 100}}
0 False 1 False 2 False 3 False 4 False ... 99987 False 99988 False 99989 False 99990 False 99991 True Length: 99992, dtype: bool
The AutoML search algorithm first trains each component in the pipeline with their default values. After the first iteration, it then tweaks the parameters of these components using the pre-defined hyperparameter ranges that these components have. To limit the search over certain hyperparameter ranges, use make_pipeline to define a pipeline with a custom hyperparameter range. Hyperparameter ranges can be found through the API reference for each component. Note: The default value of the component must be included in any specified hyperparameter range for AutoMLSearch to succeed. Additionally, the parameter value must be specified as a list, even for just one value.
make_pipeline
[10]:
from evalml import AutoMLSearch from evalml.demos import load_fraud from evalml.pipelines.components.utils import get_estimators from evalml.model_family import ModelFamily from evalml.pipelines.utils import make_pipeline import woodwork as ww X, y = load_fraud() # example of setting parameter to just one value custom_hyperparameters = {'Imputer': { 'numeric_impute_strategy': ['mean'] }} # limit the numeric impute strategy to include only `mean` and `median` # `mean` is the default value for this argument, and it needs to be included in the specified hyperparameter range custom_hyperparameters = {'Imputer': { 'numeric_impute_strategy': ['mean', 'median'] }} estimators = get_estimators('binary', [ModelFamily.EXTRA_TREES]) pipelines_with_custom_hyperparameters = [make_pipeline(X, y, estimator, 'binary', custom_hyperparameters) for estimator in estimators] automl = AutoMLSearch(X_train=X, y_train=y, problem_type='binary', allowed_pipelines=pipelines_with_custom_hyperparameters) automl.search() automl.best_pipeline.hyperparameters
Number of Features Boolean 1 Categorical 6 Numeric 5 Number of training examples: 99992 Targets False 84.82% True 15.18% Name: fraud, dtype: object Using default limit of max_batches=1. ***************************** * Beginning pipeline search * ***************************** Optimizing for Log Loss Binary. Lower score is better. Searching up to 1 batches for a total of 2 pipelines. Allowed model families: extra_trees
Batch 1: (1/2) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 5.243 Batch 1: (2/2) Extra Trees Classifier w/ Imputer + D... Elapsed:00:01 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.360 Search finished after 00:24 Best pipeline: Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder Best pipeline Log Loss Binary: 0.360414
{'Imputer': {'categorical_impute_strategy': ['most_frequent'], 'numeric_impute_strategy': ['mean', 'median']}, 'DateTime Featurization Component': {}, 'One Hot Encoder': {}, 'Extra Trees Classifier': {'n_estimators': Integer(low=10, high=1000, prior='uniform', transform='identity'), 'max_features': ['auto', 'sqrt', 'log2'], 'max_depth': Integer(low=4, high=10, prior='uniform', transform='identity')}}
The AutoMLSearch class records detailed results information under the results field, including information about the cross-validation scoring and parameters.
AutoMLSearch
results
[11]:
automl.results
{'pipeline_results': {0: {'id': 0, 'pipeline_name': 'Mode Baseline Binary Classification Pipeline', 'pipeline_class': evalml.pipelines.classification.baseline_binary.ModeBaselineBinaryPipeline, 'pipeline_summary': 'Baseline Classifier', 'parameters': {'Baseline Classifier': {'strategy': 'mode'}}, 'score': 5.243060307948458, 'high_variance_cv': False, 'training_time': 1.3677144050598145, 'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary', 5.243353291477845), ('MCC Binary', 0.0), ('AUC', 0.5), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.848189373256128), ('# Training', 66661), ('# Validation', 33331)]), 'score': 5.243353291477845, 'binary_classification_threshold': 0.5}, {'all_objective_scores': OrderedDict([('Log Loss Binary', 5.243353291477846), ('MCC Binary', 0.0), ('AUC', 0.5), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.848189373256128), ('# Training', 66661), ('# Validation', 33331)]), 'score': 5.243353291477846, 'binary_classification_threshold': 0.5}, {'all_objective_scores': OrderedDict([('Log Loss Binary', 5.242474340889683), ('MCC Binary', 0.0), ('AUC', 0.5), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.8482148214821482), ('# Training', 66662), ('# Validation', 33330)]), 'score': 5.242474340889683, 'binary_classification_threshold': 0.5}], 'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 0, 'MCC Binary': nan, 'AUC': 0, 'Precision': nan, 'F1': nan, 'Balanced Accuracy Binary': 0, 'Accuracy Binary': 0}, 'percent_better_than_baseline': 0, 'validation_score': 5.243353291477845}, 1: {'id': 1, 'pipeline_name': 'Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder', 'pipeline_class': evalml.pipelines.utils.make_pipeline.<locals>.GeneratedPipeline, 'pipeline_summary': 'Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder', 'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': None, 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Extra Trees Classifier': {'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}, 'score': 0.3604142375161434, 'high_variance_cv': False, 'training_time': 23.351277828216553, 'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary', 0.3572252327829294), ('MCC Binary', 0.0), ('AUC', 0.8327890925252948), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.848189373256128), ('# Training', 66661), ('# Validation', 33331)]), 'score': 0.3572252327829294, 'binary_classification_threshold': 0.5}, {'all_objective_scores': OrderedDict([('Log Loss Binary', 0.35710265483449705), ('MCC Binary', 0.0), ('AUC', 0.8375894242385562), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.848189373256128), ('# Training', 66661), ('# Validation', 33331)]), 'score': 0.35710265483449705, 'binary_classification_threshold': 0.5}, {'all_objective_scores': OrderedDict([('Log Loss Binary', 0.3669148249310036), ('MCC Binary', 0.0), ('AUC', 0.8349621647188481), ('Precision', 0.0), ('F1', 0.0), ('Balanced Accuracy Binary', 0.5), ('Accuracy Binary', 0.8482148214821482), ('# Training', 66662), ('# Validation', 33330)]), 'score': 0.3669148249310036, 'binary_classification_threshold': 0.5}], 'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 93.12588037620402, 'MCC Binary': nan, 'AUC': 67.0227120988466, 'Precision': nan, 'F1': nan, 'Balanced Accuracy Binary': 0, 'Accuracy Binary': 0}, 'percent_better_than_baseline': 93.12588037620402, 'validation_score': 0.3572252327829294}}, 'search_order': [0, 1], 'errors': []}
Stacking is an ensemble machine learning algorithm that involves training a model to best combine the predictions of several base learning algorithms. First, each base learning algorithms is trained using the given data. Then, the combining algorithm or meta-learner is trained on the predictions made by those base learning algorithms to make a final prediction.
AutoML enables stacking using the ensembling flag during initalization; this is set to False by default. The stacking ensemble pipeline runs in its own batch after a whole cycle of training has occurred (each allowed pipeline trains for one batch). Note that this means a large number of iterations may need to run before the stacking ensemble runs. It is also important to note that only the first CV fold is calculated for stacking ensembles because the model internally uses CV folds.
ensembling
False
[12]:
X, y = evalml.demos.load_breast_cancer() automl_with_ensembling = AutoMLSearch(X_train=X, y_train=y, problem_type="binary", allowed_model_families=[ModelFamily.RANDOM_FOREST, ModelFamily.LINEAR_MODEL], max_batches=5, ensembling=True) automl_with_ensembling.search()
Generating pipelines to search over... Ensembling will run every 4 batches. ***************************** * Beginning pipeline search * ***************************** Optimizing for Log Loss Binary. Lower score is better. Searching up to 5 batches for a total of 20 pipelines. Allowed model families: linear_model, random_forest
Batch 1: (1/20) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 12.868 Batch 1: (2/20) Logistic Regression Classifier w/ Imp... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.076 High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data. Batch 1: (3/20) Random Forest Classifier w/ Imputer Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.174 High coefficient of variation (cv >= 0.2) within cross validation scores. Random Forest Classifier w/ Imputer may not perform as estimated on unseen data. Batch 1: (4/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:02 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.503 Batch 2: (5/20) Logistic Regression Classifier w/ Imp... Elapsed:00:02 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.118 High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data. Batch 2: (6/20) Logistic Regression Classifier w/ Imp... Elapsed:00:03 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.104 High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data. Batch 2: (7/20) Logistic Regression Classifier w/ Imp... Elapsed:00:03 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.092 Batch 2: (8/20) Logistic Regression Classifier w/ Imp... Elapsed:00:04 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.115 High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data. Batch 2: (9/20) Logistic Regression Classifier w/ Imp... Elapsed:00:05 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.097 High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data. Batch 3: (10/20) Random Forest Classifier w/ Imputer Elapsed:00:05 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.137 Batch 3: (11/20) Random Forest Classifier w/ Imputer Elapsed:00:08 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.158 Batch 3: (12/20) Random Forest Classifier w/ Imputer Elapsed:00:09 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.119 Batch 3: (13/20) Random Forest Classifier w/ Imputer Elapsed:00:11 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.117 Batch 3: (14/20) Random Forest Classifier w/ Imputer Elapsed:00:14 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.120 Batch 4: (15/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:15 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.660 Batch 4: (16/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:15 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.627 Batch 4: (17/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:16 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.588 Batch 4: (18/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:17 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.341 Batch 4: (19/20) Elastic Net Classifier w/ Imputer + S... Elapsed:00:17 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.534 Batch 5: (20/20) Stacked Ensemble Classification Pipeline Elapsed:00:18 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.073 Search finished after 00:21 Best pipeline: Stacked Ensemble Classification Pipeline Best pipeline Log Loss Binary: 0.073164
We can view more information about the stacking ensemble pipeline (which was the best performing pipeline) by calling .describe().
.describe()
[13]:
automl_with_ensembling.best_pipeline.describe()
******************************************** * Stacked Ensemble Classification Pipeline * ******************************************** Problem Type: binary Model Family: Ensemble Number of features: 30 Pipeline Steps ============== 1. Stacked Ensemble Classifier * input_pipelines : [GeneratedPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Logistic Regression Classifier':{'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'},}), GeneratedPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Random Forest Classifier':{'n_estimators': 301, 'max_depth': 9, 'n_jobs': -1},})] * final_estimator : None * cv : None * n_jobs : 1