MachineLearningNotebooks/automl/04.auto-ml-remote-execution-text-data-blob-store.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Auto ML 04: Remote Execution with Text Data from Azure Blob Storage\n",
    "\n",
    "In this example we use the [Burning Man 2016 dataset](https://innovate.burningman.org/datasets-page/) to showcase how you can use AutoML to handle text data from Azure Blob Storage.\n",
    "\n",
    "Make sure you have executed the [00.configuration](00.configuration.ipynb) before running this notebook.\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. Create an `Experiment` in an existing `Workspace`.\n",
    "2. Attach an existing DSVM to a workspace.\n",
    "3. Configure AutoML using `AutoMLConfig`.\n",
    "4. Train the model using the DSVM.\n",
    "5. Explore the results.\n",
    "6. Test the best fitted model.\n",
    "\n",
    "In addition this notebook showcases the following features\n",
    "- **Parallel** executions for iterations\n",
    "- **Asynchronous** tracking of progress\n",
    "- **Cancellation** of individual iterations or the entire run\n",
    "- Retrieving models for any iteration or logged metric\n",
    "- Specifying AutoML settings as `**kwargs`\n",
    "- Handling **text** data using the `preprocess` flag\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an Experiment\n",
    "\n",
    "As part of the setup you have already created an Azure ML `Workspace` object. For AutoML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import random\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.train.automl.run import AutoMLRun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# Choose a name for the run history container in the workspace.\n",
    "experiment_name = 'automl-remote-dsvm-blobstore'\n",
    "project_folder = './sample_projects/automl-remote-dsvm-blobstore'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['SDK version'] = azureml.core.VERSION\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Project Directory'] = project_folder\n",
    "output['Experiment Name'] = experiment.name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(data=output, index=['']).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diagnostics\n",
    "\n",
    "Opt-in diagnostics for better experience, quality, and security of future releases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.telemetry import set_diagnostics_collection\n",
    "set_diagnostics_collection(send_diagnostics = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attach a Remote Linux DSVM\n",
    "To use a remote Docker compute target:\n",
    "1. Create a Linux DSVM in Azure, following these [quick instructions](https://docs.microsoft.com/en-us/azure/machine-learning/desktop-workbench/how-to-create-dsvm-hdi). Make sure you use the Ubuntu flavor (not CentOS). Make sure that disk space is available under `/tmp` because AutoML creates files under `/tmp/azureml_run`s. The DSVM should have more cores than the number of parallel runs that you plan to enable. It should also have at least 4GB per core.\n",
    "2. Enter the IP address, user name and password below.\n",
    "\n",
    "**Note:** By default, SSH runs on port 22 and you don't need to change the port number below. If you've configured SSH to use a different port, change `dsvm_ssh_port` accordinglyaddress. [Read more](https://render.githubusercontent.com/documentation/sdk/ssh-issue.md) on changing SSH ports for security reasons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import RemoteCompute\n",
    "import time\n",
    "\n",
    "# Add your VM information below\n",
    "# If a compute with the specified compute_name already exists, it will be used and the dsvm_ip_addr, dsvm_ssh_port, \n",
    "# dsvm_username and dsvm_password will be ignored.\n",
    "compute_name  = 'mydsvmb'\n",
    "dsvm_ip_addr  = '<<ip_addr>>'\n",
    "dsvm_ssh_port = 22\n",
    "dsvm_username = '<<username>>'\n",
    "dsvm_password = '<<password>>'\n",
    "\n",
    "if compute_name in ws.compute_targets:\n",
    "    print('Using existing compute.')\n",
    "    dsvm_compute = ws.compute_targets[compute_name]\n",
    "else:\n",
    "    RemoteCompute.attach(workspace=ws, name=compute_name, address=dsvm_ip_addr, username=dsvm_username, password=dsvm_password, ssh_port=dsvm_ssh_port)\n",
    "\n",
    "    while ws.compute_targets[compute_name].provisioning_state == 'Creating':\n",
    "        time.sleep(1)\n",
    "\n",
    "    dsvm_compute = ws.compute_targets[compute_name]\n",
    "    \n",
    "    if dsvm_compute.provisioning_state == 'Failed':\n",
    "        print('Attached failed.')\n",
    "        print(dsvm_compute.provisioning_errors)\n",
    "        dsvm_compute.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Get Data File\n",
    "For remote executions you should author a `get_data.py` file containing a `get_data()` function. This file should be in the root directory of the project. You can encapsulate code to read data either from a blob storage or local disk in this file.\n",
    "In this example, the `get_data()` function returns a [dictionary](README.md#getdata)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(project_folder):\n",
    "    os.makedirs(project_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $project_folder/get_data.py\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "def get_data():\n",
    "    # Load Burning Man 2016 data.\n",
    "    df = pd.read_csv(\"https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv\",\n",
    "                     delimiter=\"\\t\", quotechar='\"')\n",
    "    # Get integer labels.\n",
    "    le = LabelEncoder()\n",
    "    le.fit(df[\"Label\"].values)\n",
    "    y = le.transform(df[\"Label\"].values)\n",
    "    X = df.drop([\"Label\"], axis=1)\n",
    "\n",
    "    X_train, _, y_train, _ = train_test_split(X, y, test_size = 0.1, random_state = 42)\n",
    "\n",
    "    return { \"X\" : X_train, \"y\" : y_train }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View data\n",
    "\n",
    "You can execute the `get_data()` function locally to view the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run  $project_folder/get_data.py\n",
    "data_dict = get_data()\n",
    "df = data_dict[\"X\"]\n",
    "y = data_dict[\"y\"]\n",
    "pd.set_option('display.max_colwidth', 15)\n",
    "df['Label'] = pd.Series(y, index=df.index)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure AutoML <a class=\"anchor\" id=\"Instatiate-AutoML-Remote-DSVM\"></a>\n",
    "\n",
    "You can specify `automl_settings` as `**kwargs` as well. Also note that you can use a `get_data()` function for local excutions too.\n",
    "\n",
    "**Note:** When using Remote DSVM, you can't pass Numpy arrays directly to the fit method.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>balanced_accuracy</i><br><i>average_precision_score_weighted</i><br><i>precision_score_weighted</i>|\n",
    "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
    "|**iterations**|Number of iterations. In each iteration AutoML trains a specific pipeline with the data.|\n",
    "|**n_cross_validations**|Number of cross validation splits.|\n",
    "|**max_concurrent_iterations**|Maximum number of iterations that would be executed in parallel. This should be less than the number of cores on the DSVM.|\n",
    "|**preprocess**|Setting this to *True* enables AutoML to perform preprocessing on the input to handle *missing data*, and to perform some common *feature extraction*.|\n",
    "|**max_cores_per_iteration**|Indicates how many cores on the compute target would be used to train a single pipeline.<br>Default is *1*; you can set it to *-1* to use all cores.|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"iteration_timeout_minutes\": 60,\n",
    "    \"iterations\": 4,\n",
    "    \"n_cross_validations\": 5,\n",
    "    \"primary_metric\": 'AUC_weighted',\n",
    "    \"preprocess\": True,\n",
    "    \"max_cores_per_iteration\": 2\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(task = 'classification',\n",
    "                             path = project_folder,\n",
    "                             compute_target = dsvm_compute,\n",
    "                             data_script = project_folder + \"/get_data.py\",\n",
    "                             **automl_settings\n",
    "                            )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Models <a class=\"anchor\" id=\"Training-the-model-Remote-DSVM\"></a>\n",
    "\n",
    "Call the `submit` method on the experiment object and pass the run configuration. For remote runs the execution is asynchronous, so you will see the iterations get populated as they complete. You can interact with the widgets and models even when the experiment is running to retrieve the best model up to that point. Once you are satisfied with the model, you can cancel a particular iteration or the whole run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_run = experiment.submit(automl_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Results <a class=\"anchor\" id=\"Exploring-the-Results-Remote-DSVM\"></a>\n",
    "#### Widget for Monitoring Runs\n",
    "\n",
    "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
    "\n",
    "You can click on a pipeline to see run properties and output logs.  Logs are also available on the DSVM under `/tmp/azureml_run/{iterationid}/azureml-logs`\n",
    "\n",
    "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(remote_run).show() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until the run finishes.\n",
    "remote_run.wait_for_completion(show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Retrieve All Child Runs\n",
    "You can also use SDK methods to fetch all the child runs and see individual metrics that we log. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "children = list(remote_run.get_children())\n",
    "metricslist = {}\n",
    "for run in children:\n",
    "    properties = run.get_properties()\n",
    "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}\n",
    "    metricslist[int(properties['iteration'])] = metrics\n",
    "\n",
    "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
    "rundata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cancelling Runs\n",
    "You can cancel ongoing remote runs using the `cancel` and `cancel_iteration` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cancel the ongoing experiment and stop scheduling new iterations.\n",
    "remote_run.cancel()\n",
    "\n",
    "# Cancel iteration 1 and move onto iteration 2.\n",
    "# remote_run.cancel_iteration(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model\n",
    "\n",
    "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = remote_run.get_output()\n",
    "print(best_run)\n",
    "print(fitted_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Best Model Based on Any Other Metric\n",
    "Show the run and the model which has the smallest `accuracy` value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lookup_metric = \"accuracy\"\n",
    "# best_run, fitted_model = remote_run.get_output(metric = lookup_metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model from a Specific Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iteration = 0\n",
    "zero_run, zero_model = remote_run.get_output(iteration = iteration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the Fitted Model <a class=\"anchor\" id=\"Testing-the-Fitted-Model-Remote-DSVM\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from pandas_ml import ConfusionMatrix\n",
    "\n",
    "df = pd.read_csv(\"https://automldemods.blob.core.windows.net/datasets/PlayaEvents2016,_1.6MB,_3.4k-rows.cleaned.2.tsv\",\n",
    "                     delimiter=\"\\t\", quotechar='\"')\n",
    "\n",
    "# get integer labels\n",
    "le = LabelEncoder()\n",
    "le.fit(df[\"Label\"].values)\n",
    "y = le.transform(df[\"Label\"].values)\n",
    "X = df.drop([\"Label\"], axis=1)\n",
    "\n",
    "_, X_test, _, y_test = train_test_split(X, y, test_size=0.1, random_state=42)\n",
    "\n",
    "\n",
    "ypred = fitted_model.predict(X_test.values)\n",
    "\n",
    "\n",
    "ypred_strings = le.inverse_transform(ypred)\n",
    "ytest_strings = le.inverse_transform(y_test)\n",
    "\n",
    "cm = ConfusionMatrix(ytest_strings, ypred_strings)\n",
    "\n",
    "print(cm)\n",
    "\n",
    "cm.plot()"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "savitam"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}