{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright (c) Microsoft Corporation. All rights reserved.\n", "\n", "Licensed under the MIT License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Automated Machine Learning\n", "_**Classification with Deployment using a Bank Marketing Dataset**_\n", "\n", "## Contents\n", "1. [Introduction](#Introduction)\n", "1. [Setup](#Setup)\n", "1. [Train](#Train)\n", "1. [Results](#Results)\n", "1. [Deploy](#Deploy)\n", "1. [Test](#Test)\n", "1. [Acknowledgements](#Acknowledgements)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "In this example we use the UCI Bank Marketing dataset to showcase how you can use AutoML for a classification problem and deploy it to an Azure Container Instance (ACI). The classification goal is to predict if the client will subscribe to a term deposit with the bank.\n", "\n", "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n", "\n", "Please find the ONNX related documentations [here](https://github.com/onnx/onnx).\n", "\n", "In this notebook you will learn how to:\n", "1. Create an experiment using an existing workspace.\n", "2. Configure AutoML using `AutoMLConfig`.\n", "3. Train the model using local compute with ONNX compatible config on.\n", "4. Explore the results, featurization transparency options and save the ONNX model\n", "5. Inference with the ONNX model.\n", "6. Register the model.\n", "7. Create a container image.\n", "8. Create an Azure Container Instance (ACI) service.\n", "9. Test the ACI service.\n", "\n", "In addition this notebook showcases the following features\n", "- **Blocking** certain pipelines\n", "- Specifying **target metrics** to indicate stopping criteria\n", "- Handling **missing data** in the input" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "As part of the setup you have already created an Azure ML `Workspace` object. For AutoML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import logging\n", "\n", "from matplotlib import pyplot as plt\n", "import pandas as pd\n", "import os\n", "\n", "import azureml.core\n", "from azureml.core.experiment import Experiment\n", "from azureml.core.workspace import Workspace\n", "from azureml.automl.core.featurization import FeaturizationConfig\n", "from azureml.core.dataset import Dataset\n", "from azureml.train.automl import AutoMLConfig\n", "from azureml.interpret import ExplanationClient" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This sample notebook may use features that are not available in previous versions of the Azure ML SDK." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n", "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Accessing the Azure ML workspace requires authentication with Azure.\n", "\n", "The default authentication is interactive authentication using the default tenant. Executing the `ws = Workspace.from_config()` line in the cell below will prompt for authentication the first time that it is run.\n", "\n", "If you have multiple Azure tenants, you can specify the tenant by replacing the `ws = Workspace.from_config()` line in the cell below with the following:\n", "\n", "```\n", "from azureml.core.authentication import InteractiveLoginAuthentication\n", "auth = InteractiveLoginAuthentication(tenant_id = 'mytenantid')\n", "ws = Workspace.from_config(auth = auth)\n", "```\n", "\n", "If you need to run in an environment where interactive login is not possible, you can use Service Principal authentication by replacing the `ws = Workspace.from_config()` line in the cell below with the following:\n", "\n", "```\n", "from azureml.core.authentication import ServicePrincipalAuthentication\n", "auth = auth = ServicePrincipalAuthentication('mytenantid', 'myappid', 'mypassword')\n", "ws = Workspace.from_config(auth = auth)\n", "```\n", "For more details, see [aka.ms/aml-notebook-auth](http://aka.ms/aml-notebook-auth)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ws = Workspace.from_config()\n", "\n", "# choose a name for experiment\n", "experiment_name = 'automl-classification-bmarketing-all'\n", "\n", "experiment=Experiment(ws, experiment_name)\n", "\n", "output = {}\n", "output['Subscription ID'] = ws.subscription_id\n", "output['Workspace'] = ws.name\n", "output['Resource Group'] = ws.resource_group\n", "output['Location'] = ws.location\n", "output['Experiment Name'] = experiment.name\n", "pd.set_option('display.max_colwidth', -1)\n", "outputDf = pd.DataFrame(data = output, index = [''])\n", "outputDf.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create or Attach existing AmlCompute\n", "You will need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n", "\n", "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n", "\n", "#### Creation of AmlCompute takes approximately 5 minutes. \n", "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n", "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from azureml.core.compute import ComputeTarget, AmlCompute\n", "from azureml.core.compute_target import ComputeTargetException\n", "\n", "# Choose a name for your CPU cluster\n", "cpu_cluster_name = \"cpu-cluster-4\"\n", "\n", "# Verify that cluster does not exist already\n", "try:\n", " compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n", " print('Found existing cluster, use it.')\n", "except ComputeTargetException:\n", " compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n", " max_nodes=6)\n", " compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n", "\n", "compute_target.wait_for_completion(show_output=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Data\n", "\n", "Leverage azure compute to load the bank marketing dataset as a Tabular Dataset into the dataset variable. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training Data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = pd.read_csv(\"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv\")\n", "data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Add missing values in 75% of the lines.\n", "import numpy as np\n", "\n", "missing_rate = 0.75\n", "n_missing_samples = int(np.floor(data.shape[0] * missing_rate))\n", "missing_samples = np.hstack((np.zeros(data.shape[0] - n_missing_samples, dtype=np.bool), np.ones(n_missing_samples, dtype=np.bool)))\n", "rng = np.random.RandomState(0)\n", "rng.shuffle(missing_samples)\n", "missing_features = rng.randint(0, data.shape[1], n_missing_samples)\n", "data.values[np.where(missing_samples)[0], missing_features] = np.nan" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if not os.path.isdir('data'):\n", " os.mkdir('data')\n", " \n", "# Save the train data to a csv to be uploaded to the datastore\n", "pd.DataFrame(data).to_csv(\"data/train_data.csv\", index=False)\n", "\n", "ds = ws.get_default_datastore()\n", "ds.upload(src_dir='./data', target_path='bankmarketing', overwrite=True, show_progress=True)\n", "\n", " \n", "\n", "# Upload the training data as a tabular dataset for access during training on remote compute\n", "train_data = Dataset.Tabular.from_delimited_files(path=ds.path('bankmarketing/train_data.csv'))\n", "label = \"y\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Validation Data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "validation_data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_validate.csv\"\n", "validation_dataset = Dataset.Tabular.from_delimited_files(validation_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Test Data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_test.csv\"\n", "test_dataset = Dataset.Tabular.from_delimited_files(test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train\n", "\n", "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n", "\n", "|Property|Description|\n", "|-|-|\n", "|**task**|classification or regression or forecasting|\n", "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics:
accuracy
AUC_weighted
average_precision_score_weighted
norm_macro_recall
precision_score_weighted|\n", "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n", "|**blocked_models** | *List* of *strings* indicating machine learning algorithms for AutoML to avoid in this run.

Allowed values for **Classification**
LogisticRegression
SGD
MultinomialNaiveBayes
BernoulliNaiveBayes
SVM
LinearSVM
KNN
DecisionTree
RandomForest
ExtremeRandomTrees
LightGBM
GradientBoosting
TensorFlowDNN
TensorFlowLinearClassifier

Allowed values for **Regression**
ElasticNet
GradientBoosting
DecisionTree
KNN
LassoLars
SGD
RandomForest
ExtremeRandomTrees
LightGBM
TensorFlowLinearRegressor
TensorFlowDNN

Allowed values for **Forecasting**
ElasticNet
GradientBoosting
DecisionTree
KNN
LassoLars
SGD
RandomForest
ExtremeRandomTrees
LightGBM
TensorFlowLinearRegressor
TensorFlowDNN
Arima
Prophet|\n", "|**allowed_models** | *List* of *strings* indicating machine learning algorithms for AutoML to use in this run. Same values listed above for **blocked_models** allowed for **allowed_models**.|\n", "|**experiment_exit_score**| Value indicating the target for *primary_metric*.
Once the target is surpassed the run terminates.|\n", "|**experiment_timeout_hours**| Maximum amount of time in hours that all iterations combined can take before the experiment terminates.|\n", "|**enable_early_stopping**| Flag to enble early termination if the score is not improving in the short term.|\n", "|**featurization**| 'auto' / 'off' Indicator for whether featurization step should be done automatically or not. Note: If the input data is sparse, featurization cannot be turned on.|\n", "|**n_cross_validations**|Number of cross validation splits.|\n", "|**training_data**|Input dataset, containing both features and label column.|\n", "|**label_column_name**|The name of the label column.|\n", "\n", "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "automl_settings = {\n", " \"experiment_timeout_hours\" : 0.3,\n", " \"enable_early_stopping\" : True,\n", " \"iteration_timeout_minutes\": 5,\n", " \"max_concurrent_iterations\": 4,\n", " \"max_cores_per_iteration\": -1,\n", " #\"n_cross_validations\": 2,\n", " \"primary_metric\": 'AUC_weighted',\n", " \"featurization\": 'auto',\n", " \"verbosity\": logging.INFO,\n", "}\n", "\n", "automl_config = AutoMLConfig(task = 'classification',\n", " debug_log = 'automl_errors.log',\n", " compute_target=compute_target,\n", " experiment_exit_score = 0.9984,\n", " blocked_models = ['KNN','LinearSVM'],\n", " enable_onnx_compatible_models=True,\n", " training_data = train_data,\n", " label_column_name = label,\n", " validation_data = validation_dataset,\n", " **automl_settings\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Call the `submit` method on the experiment object and pass the run configuration. Execution of local runs is synchronous. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "remote_run = experiment.submit(automl_config, show_output = False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the following cell to access previous runs. Uncomment the cell below and update the run_id." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#from azureml.train.automl.run import AutoMLRun\n", "#remote_run = AutoMLRun(experiment=experiment, run_id='thresh else 'black')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Delete a Web Service\n", "\n", "Deletes the specified web service." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "aci_service.delete()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Acknowledgements" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This Bank Marketing dataset is made available under the Creative Commons (CCO: Public Domain) License: https://creativecommons.org/publicdomain/zero/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: https://creativecommons.org/publicdomain/zero/1.0/ and is available at: https://www.kaggle.com/janiobachmann/bank-marketing-dataset .\n", "\n", "_**Acknowledgements**_\n", "This data set is originally available within the UCI Machine Learning Database: https://archive.ics.uci.edu/ml/datasets/bank+marketing\n", "\n", "[Moro et al., 2014] S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014" ] } ], "metadata": { "authors": [ { "name": "ratanase" } ], "category": "tutorial", "compute": [ "AML" ], "datasets": [ "Bankmarketing" ], "deployment": [ "ACI" ], "exclude_from_index": false, "framework": [ "None" ], "friendly_name": "Automated ML run with basic edition features.", "index_order": 5, "kernelspec": { "display_name": "Python 3.6", "language": "python", "name": "python36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" }, "tags": [ "featurization", "explainability", "remote_run", "AutomatedML" ], "task": "Classification" }, "nbformat": 4, "nbformat_minor": 2 }