MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-attach/auto-ml-remote-attach.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**Remote Execution using attach**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Results](#Results)\n",
    "1. [Test](#Test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this example we use the scikit-learn's [20newsgroup](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_20newsgroups.html) to showcase how you can use AutoML to handle text data with remote attach.\n",
    "\n",
    "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. Create an `Experiment` in an existing `Workspace`.\n",
    "2. Attach an existing DSVM to a workspace.\n",
    "3. Configure AutoML using `AutoMLConfig`.\n",
    "4. Train the model using the DSVM.\n",
    "5. Explore the results.\n",
    "6. Test the best fitted model.\n",
    "\n",
    "In addition this notebook showcases the following features\n",
    "- **Parallel** executions for iterations\n",
    "- **Asynchronous** tracking of progress\n",
    "- **Cancellation** of individual iterations or the entire run\n",
    "- Retrieving models for any iteration or logged metric\n",
    "- Specifying AutoML settings as `**kwargs`\n",
    "- Handling **text** data using the `preprocess` flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "As part of the setup you have already created an Azure ML `Workspace` object. For AutoML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import random\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.train.automl.run import AutoMLRun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# Choose a name for the run history container in the workspace.\n",
    "experiment_name = 'automl-remote-attach'\n",
    "project_folder = './sample_projects/automl-remote-attach'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['SDK version'] = azureml.core.VERSION\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Project Directory'] = project_folder\n",
    "output['Experiment Name'] = experiment.name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(data=output, index=['']).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Opt-in diagnostics for better experience, quality, and security of future releases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.telemetry import set_diagnostics_collection\n",
    "set_diagnostics_collection(send_diagnostics = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attach a Remote Linux DSVM\n",
    "To use a remote Docker compute target:\n",
    "1. Create a Linux DSVM in Azure, following these [quick instructions](https://docs.microsoft.com/en-us/azure/machine-learning/desktop-workbench/how-to-create-dsvm-hdi). Make sure you use the Ubuntu flavor (not CentOS). Make sure that disk space is available under `/tmp` because AutoML creates files under `/tmp/azureml_run`s. The DSVM should have more cores than the number of parallel runs that you plan to enable. It should also have at least 4GB per core.\n",
    "2. Enter the IP address, user name and password below.\n",
    "\n",
    "**Note:** By default, SSH runs on port 22 and you don't need to change the port number below. If you've configured SSH to use a different port, change `dsvm_ssh_port` accordinglyaddress. [Read more](https://render.githubusercontent.com/documentation/sdk/ssh-issue.md) on changing SSH ports for security reasons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, RemoteCompute\n",
    "import time\n",
    "\n",
    "# Add your VM information below\n",
    "# If a compute with the specified compute_name already exists, it will be used and the dsvm_ip_addr, dsvm_ssh_port, \n",
    "# dsvm_username and dsvm_password will be ignored.\n",
    "compute_name  = 'mydsvmb'\n",
    "dsvm_ip_addr  = '<<ip_addr>>'\n",
    "dsvm_ssh_port = 22\n",
    "dsvm_username = '<<username>>'\n",
    "dsvm_password = '<<password>>'\n",
    "\n",
    "if compute_name in ws.compute_targets:\n",
    "    print('Using existing compute.')\n",
    "    dsvm_compute = ws.compute_targets[compute_name]\n",
    "else:\n",
    "    attach_config = RemoteCompute.attach_configuration(address=dsvm_ip_addr, username=dsvm_username, password=dsvm_password, ssh_port=dsvm_ssh_port)\n",
    "    ComputeTarget.attach(workspace=ws, name=compute_name, attach_configuration=attach_config)\n",
    "\n",
    "    while ws.compute_targets[compute_name].provisioning_state == 'Creating':\n",
    "        time.sleep(1)\n",
    "\n",
    "    dsvm_compute = ws.compute_targets[compute_name]\n",
    "    \n",
    "    if dsvm_compute.provisioning_state == 'Failed':\n",
    "        print('Attached failed.')\n",
    "        print(dsvm_compute.provisioning_errors)\n",
    "        dsvm_compute.detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.runconfig import RunConfiguration\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "\n",
    "# create a new RunConfig object\n",
    "conda_run_config = RunConfiguration(framework=\"python\")\n",
    "\n",
    "# Set compute target to the Linux DSVM\n",
    "conda_run_config.target = dsvm_compute\n",
    "\n",
    "cd = CondaDependencies.create(pip_packages=['azureml-sdk[automl]'], conda_packages=['numpy'])\n",
    "conda_run_config.environment.python.conda_dependencies = cd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "For remote executions you should author a `get_data.py` file containing a `get_data()` function. This file should be in the root directory of the project. You can encapsulate code to read data either from a blob storage or local disk in this file.\n",
    "In this example, the `get_data()` function returns a [dictionary](README.md#getdata)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(project_folder):\n",
    "    os.makedirs(project_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $project_folder/get_data.py\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "def get_data():\n",
    "    remove = ('headers', 'footers', 'quotes')\n",
    "    categories = [\n",
    "        'alt.atheism',\n",
    "        'talk.religion.misc',\n",
    "        'comp.graphics',\n",
    "        'sci.space',\n",
    "    ]\n",
    "    data_train = fetch_20newsgroups(subset = 'train', categories = categories,\n",
    "                                    shuffle = True, random_state = 42,\n",
    "                                    remove = remove)\n",
    "    \n",
    "    X_train = np.array(data_train.data).reshape((len(data_train.data),1))\n",
    "    y_train = np.array(data_train.target)\n",
    "    \n",
    "    return { \"X\" : X_train, \"y\" : y_train }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "You can specify `automl_settings` as `**kwargs` as well. Also note that you can use a `get_data()` function for local excutions too.\n",
    "\n",
    "**Note:** When using Remote DSVM, you can't pass Numpy arrays directly to the fit method.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
    "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
    "|**iterations**|Number of iterations. In each iteration AutoML trains a specific pipeline with the data.|\n",
    "|**n_cross_validations**|Number of cross validation splits.|\n",
    "|**max_concurrent_iterations**|Maximum number of iterations that would be executed in parallel. This should be less than the number of cores on the DSVM.|\n",
    "|**preprocess**|Setting this to *True* enables AutoML to perform preprocessing on the input to handle *missing data*, and to perform some common *feature extraction*.|\n",
    "|**enable_cache**|Setting this to *True* enables preprocess done once and reuse the same preprocessed data for all the iterations. Default value is True.\n",
    "|**max_cores_per_iteration**|Indicates how many cores on the compute target would be used to train a single pipeline.<br>Default is *1*; you can set it to *-1* to use all cores.|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"iteration_timeout_minutes\": 60,\n",
    "    \"iterations\": 4,\n",
    "    \"n_cross_validations\": 5,\n",
    "    \"primary_metric\": 'AUC_weighted',\n",
    "    \"preprocess\": True,\n",
    "    \"max_cores_per_iteration\": 2\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(task = 'classification',\n",
    "                             path = project_folder,\n",
    "                             run_configuration=conda_run_config,\n",
    "                             data_script = project_folder + \"/get_data.py\",\n",
    "                             **automl_settings\n",
    "                            )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the `submit` method on the experiment object and pass the run configuration. For remote runs the execution is asynchronous, so you will see the iterations get populated as they complete. You can interact with the widgets and models even when the experiment is running to retrieve the best model up to that point. Once you are satisfied with the model, you can cancel a particular iteration or the whole run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_run = experiment.submit(automl_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "#### Widget for Monitoring Runs\n",
    "\n",
    "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
    "\n",
    "You can click on a pipeline to see run properties and output logs.  Logs are also available on the DSVM under `/tmp/azureml_run/{iterationid}/azureml-logs`\n",
    "\n",
    "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(remote_run).show() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until the run finishes.\n",
    "remote_run.wait_for_completion(show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-process cache cleanup\n",
    "The preprocess data gets cache at user default file store. When the run is completed the cache can be cleaned by running below cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remote_run.clean_preprocessor_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Retrieve All Child Runs\n",
    "You can also use SDK methods to fetch all the child runs and see individual metrics that we log. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "children = list(remote_run.get_children())\n",
    "metricslist = {}\n",
    "for run in children:\n",
    "    properties = run.get_properties()\n",
    "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}\n",
    "    metricslist[int(properties['iteration'])] = metrics\n",
    "\n",
    "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
    "rundata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cancelling Runs\n",
    "You can cancel ongoing remote runs using the `cancel` and `cancel_iteration` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cancel the ongoing experiment and stop scheduling new iterations.\n",
    "# remote_run.cancel()\n",
    "\n",
    "# Cancel iteration 1 and move onto iteration 2.\n",
    "# remote_run.cancel_iteration(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model\n",
    "\n",
    "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = remote_run.get_output()\n",
    "print(best_run)\n",
    "print(fitted_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Best Model Based on Any Other Metric\n",
    "Show the run and the model which has the smallest `accuracy` value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lookup_metric = \"accuracy\"\n",
    "# best_run, fitted_model = remote_run.get_output(metric = lookup_metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model from a Specific Iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iteration = 0\n",
    "zero_run, zero_model = remote_run.get_output(iteration = iteration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load test data.\n",
    "from pandas_ml import ConfusionMatrix\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "remove = ('headers', 'footers', 'quotes')\n",
    "categories = [\n",
    "        'alt.atheism',\n",
    "        'talk.religion.misc',\n",
    "        'comp.graphics',\n",
    "        'sci.space',\n",
    "    ]\n",
    "\n",
    "data_test = fetch_20newsgroups(subset = 'test', categories = categories,\n",
    "                               shuffle = True, random_state = 42,\n",
    "                               remove = remove)\n",
    "\n",
    "X_test = np.array(data_test.data).reshape((len(data_test.data),1))\n",
    "y_test = data_test.target\n",
    "\n",
    "# Test our best pipeline.\n",
    "\n",
    "y_pred = fitted_model.predict(X_test)\n",
    "y_pred_strings = [data_test.target_names[i] for i in y_pred]\n",
    "y_test_strings = [data_test.target_names[i] for i in y_test]\n",
    "\n",
    "cm = ConfusionMatrix(y_test_strings, y_pred_strings)\n",
    "print(cm)\n",
    "cm.plot()"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "savitam"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}