MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Automated Machine Learning\n",
        "_**Text Classification Using Deep Learning**_\n",
        "\n",
        "## Contents\n",
        "1. [Introduction](#Introduction)\n",
        "1. [Setup](#Setup)\n",
        "1. [Data](#Data)\n",
        "1. [Train](#Train)\n",
        "1. [Evaluate](#Evaluate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Introduction\n",
        "This notebook demonstrates classification with text data using deep learning in AutoML.\n",
        "\n",
        "AutoML highlights here include using deep neural networks (DNNs) to create embedded features from text data. Depending on the compute cluster the user provides, AutoML tried out Bidirectional Encoder Representations from Transformers (BERT) when a GPU compute is used, and Bidirectional Long-Short Term neural network (BiLSTM) when a CPU compute is used, thereby optimizing the choice of DNN for the uesr's setup.\n",
        "\n",
        "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
        "\n",
        "Notebook synopsis:\n",
        "\n",
        "1. Creating an Experiment in an existing Workspace\n",
        "2. Configuration and remote run of AutoML for a text dataset (20 Newsgroups dataset from scikit-learn) for classification\n",
        "3. Registering the best model for future use\n",
        "4. Evaluating the final model on a test set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "import logging\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "import azureml.core\n",
        "from azureml.core.experiment import Experiment\n",
        "from azureml.core.workspace import Workspace\n",
        "from azureml.core.dataset import Dataset\n",
        "from azureml.core.compute import AmlCompute\n",
        "from azureml.core.compute import ComputeTarget\n",
        "from azureml.core.run import Run\n",
        "from azureml.widgets import RunDetails\n",
        "from azureml.core.model import Model\n",
        "from helper import run_inference, get_result_df\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "from sklearn.datasets import fetch_20newsgroups"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ws = Workspace.from_config()\n",
        "\n",
        "# Choose an experiment name.\n",
        "experiment_name = \"automl-classification-text-dnn\"\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "\n",
        "output = {}\n",
        "output[\"Subscription ID\"] = ws.subscription_id\n",
        "output[\"Workspace Name\"] = ws.name\n",
        "output[\"Resource Group\"] = ws.resource_group\n",
        "output[\"Location\"] = ws.location\n",
        "output[\"Experiment Name\"] = experiment.name\n",
        "pd.set_option(\"display.max_colwidth\", None)\n",
        "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
        "outputDf.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Set up a compute cluster\n",
        "This section uses a user-provided compute cluster (named \"dnntext-cluster\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments.\n",
        "\n",
        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
        "\n",
        "Whether you provide/select a CPU or GPU cluster, AutoML will choose the appropriate DNN for that setup - BiLSTM or BERT text featurizer will be included in the candidate featurizers on CPU and GPU respectively.  If your goal is to obtain the most accurate model, we recommend you use GPU clusters since BERT featurizers usually outperform BiLSTM featurizers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "num_nodes = 2\n",
        "\n",
        "# Choose a name for your cluster.\n",
        "amlcompute_cluster_name = \"dnntext-cluster\"\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print(\"Found existing cluster, use it.\")\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(\n",
        "        vm_size=\"STANDARD_NC6\",  # CPU for BiLSTM, such as \"STANDARD_D2_V2\"\n",
        "        # To use BERT (this is recommended for best performance), select a GPU such as \"STANDARD_NC6\"\n",
        "        # or similar GPU option\n",
        "        # available in your workspace\n",
        "        idle_seconds_before_scaledown=60,\n",
        "        max_nodes=num_nodes,\n",
        "    )\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Get data\n",
        "For this notebook we will use 20 Newsgroups data from scikit-learn. We filter the data to contain four classes and take a sample as training data. Please note that for accuracy improvement, more data is needed. For this notebook we provide a small-data example so that you can use this template to use with your larger sized data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "data_dir = \"text-dnn-data\"  # Local directory to store data\n",
        "blobstore_datadir = data_dir  # Blob store directory to store data in\n",
        "target_column_name = \"y\"\n",
        "feature_column_name = \"X\"\n",
        "\n",
        "\n",
        "def get_20newsgroups_data():\n",
        "    \"\"\"Fetches 20 Newsgroups data from scikit-learn\n",
        "    Returns them in form of pandas dataframes\n",
        "    \"\"\"\n",
        "    remove = (\"headers\", \"footers\", \"quotes\")\n",
        "    categories = [\n",
        "        \"rec.sport.baseball\",\n",
        "        \"rec.sport.hockey\",\n",
        "        \"comp.graphics\",\n",
        "        \"sci.space\",\n",
        "    ]\n",
        "\n",
        "    data = fetch_20newsgroups(\n",
        "        subset=\"train\",\n",
        "        categories=categories,\n",
        "        shuffle=True,\n",
        "        random_state=42,\n",
        "        remove=remove,\n",
        "    )\n",
        "    data = pd.DataFrame(\n",
        "        {feature_column_name: data.data, target_column_name: data.target}\n",
        "    )\n",
        "\n",
        "    data_train = data[:200]\n",
        "    data_test = data[200:300]\n",
        "\n",
        "    data_train = remove_blanks_20news(\n",
        "        data_train, feature_column_name, target_column_name\n",
        "    )\n",
        "    data_test = remove_blanks_20news(data_test, feature_column_name, target_column_name)\n",
        "\n",
        "    return data_train, data_test\n",
        "\n",
        "\n",
        "def remove_blanks_20news(data, feature_column_name, target_column_name):\n",
        "\n",
        "    data[feature_column_name] = (\n",
        "        data[feature_column_name]\n",
        "        .replace(r\"\\n\", \" \", regex=True)\n",
        "        .apply(lambda x: x.strip())\n",
        "    )\n",
        "    data = data[data[feature_column_name] != \"\"]\n",
        "\n",
        "    return data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Fetch data and upload to datastore for use in training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "data_train, data_test = get_20newsgroups_data()\n",
        "\n",
        "if not os.path.isdir(data_dir):\n",
        "    os.mkdir(data_dir)\n",
        "\n",
        "train_data_fname = data_dir + \"/train_data.csv\"\n",
        "test_data_fname = data_dir + \"/test_data.csv\"\n",
        "\n",
        "data_train.to_csv(train_data_fname, index=False)\n",
        "data_test.to_csv(test_data_fname, index=False)\n",
        "\n",
        "datastore = ws.get_default_datastore()\n",
        "datastore.upload(src_dir=data_dir, target_path=blobstore_datadir, overwrite=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "train_dataset = Dataset.Tabular.from_delimited_files(\n",
        "    path=[(datastore, blobstore_datadir + \"/train_data.csv\")]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Prepare AutoML run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook uses the blocked_models parameter to exclude some models that can take a longer time to train on some text datasets. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "automl_settings = {\n",
        "    \"experiment_timeout_minutes\": 30,\n",
        "    \"primary_metric\": \"accuracy\",\n",
        "    \"max_concurrent_iterations\": num_nodes,\n",
        "    \"max_cores_per_iteration\": -1,\n",
        "    \"enable_dnn\": True,\n",
        "    \"enable_early_stopping\": True,\n",
        "    \"validation_size\": 0.3,\n",
        "    \"verbosity\": logging.INFO,\n",
        "    \"enable_voting_ensemble\": False,\n",
        "    \"enable_stack_ensemble\": False,\n",
        "}\n",
        "\n",
        "automl_config = AutoMLConfig(\n",
        "    task=\"classification\",\n",
        "    debug_log=\"automl_errors.log\",\n",
        "    compute_target=compute_target,\n",
        "    training_data=train_dataset,\n",
        "    label_column_name=target_column_name,\n",
        "    blocked_models=[\"LightGBM\", \"XGBoostClassifier\"],\n",
        "    **automl_settings,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Submit AutoML Run"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "automl_run = experiment.submit(automl_config, show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Displaying the run objects gives you links to the visual tools in the Azure Portal. Go try them!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieve the Best Model\n",
        "Below we select the best model pipeline from our iterations, use it to test on test data on the same compute cluster."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For local inferencing, you can load the model locally via. the method `remote_run.get_output()`. For more information on the arguments expected by this method, you can run `remote_run.get_output??`.\n",
        "Note that when the model contains BERT, this step will require pytorch and pytorch-transformers installed in your local environment. The exact versions of these packages can be found in the **automl_env.yml** file located in the local copy of your azureml-examples folder here: \"azureml-examples/python-sdk/tutorials/automl-with-azureml\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Retrieve the best Run object\n",
        "best_run = automl_run.get_best_child()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can now see what text transformations are used to convert text data to features for this dataset, including deep learning transformations based on BiLSTM or Transformer (BERT is one implementation of a Transformer) models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Download the featurization summary JSON file locally\n",
        "best_run.download_file(\n",
        "    \"outputs/featurization_summary.json\", \"featurization_summary.json\"\n",
        ")\n",
        "\n",
        "# Render the JSON as a pandas DataFrame\n",
        "with open(\"featurization_summary.json\", \"r\") as f:\n",
        "    records = json.load(f)\n",
        "\n",
        "featurization_summary = pd.DataFrame.from_records(records)\n",
        "featurization_summary[\"Transformations\"].tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Registering the best model\n",
        "We now register the best fitted model from the AutoML Run for use in future deployments.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Get results stats, extract the best model from AutoML run, download and register the resultant best model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "summary_df = get_result_df(automl_run)\n",
        "best_dnn_run_id = summary_df[\"run_id\"].iloc[0]\n",
        "best_dnn_run = Run(experiment, best_dnn_run_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_dir = \"Model\"  # Local folder where the model will be stored temporarily\n",
        "if not os.path.isdir(model_dir):\n",
        "    os.mkdir(model_dir)\n",
        "\n",
        "best_dnn_run.download_file(\"outputs/model.pkl\", model_dir + \"/model.pkl\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Register the model in your Azure Machine Learning Workspace. If you previously registered a model, please make sure to delete it so as to replace it with this new model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Register the model\n",
        "model_name = \"textDNN-20News\"\n",
        "model = Model.register(\n",
        "    model_path=model_dir + \"/model.pkl\", model_name=model_name, tags=None, workspace=ws\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Evaluate on Test Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now use the best fitted model from the AutoML Run to make predictions on the test set.  \n",
        "\n",
        "Test set schema should match that of the training set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_dataset = Dataset.Tabular.from_delimited_files(\n",
        "    path=[(datastore, blobstore_datadir + \"/test_data.csv\")]\n",
        ")\n",
        "\n",
        "# preview the first 3 rows of the dataset\n",
        "test_dataset.take(3).to_pandas_dataframe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_experiment = Experiment(ws, experiment_name + \"_test\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "script_folder = os.path.join(os.getcwd(), \"inference\")\n",
        "os.makedirs(script_folder, exist_ok=True)\n",
        "shutil.copy(\"infer.py\", script_folder)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_run = run_inference(\n",
        "    test_experiment,\n",
        "    compute_target,\n",
        "    script_folder,\n",
        "    best_dnn_run,\n",
        "    test_dataset,\n",
        "    target_column_name,\n",
        "    model_name,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Display computed metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_run"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "RunDetails(test_run).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "test_run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "pd.Series(test_run.get_metrics())"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "anshirga"
      }
    ],
    "compute": [
      "AML Compute"
    ],
    "datasets": [
      "None"
    ],
    "deployment": [
      "None"
    ],
    "exclude_from_index": false,
    "framework": [
      "None"
    ],
    "friendly_name": "DNN Text Featurization",
    "index_order": 2,
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "tags": [
      "None"
    ],
    "task": "Text featurization using DNNs for classification"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}