MachineLearningNotebooks/automl/09.auto-ml-classification-with-deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML 09: Classification with deployment\n",
    "\n",
    "In this example we use the scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) to showcase how you can use AutoML for a simple classification problem.\n",
    "\n",
    "Make sure you have executed the [00.configuration](00.configuration.ipynb) before running this notebook.\n",
    "\n",
    "In this notebook you would see\n",
    "1. Creating an Experiment using an existing Workspace\n",
    "2. Instantiating AutoMLConfig\n",
    "3. Training the Model using local compute\n",
    "4. Exploring the results\n",
    "5. Registering the model\n",
    "6. Creating Image and creating aci service\n",
    "7. Testing the aci service\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Experiment\n",
    "\n",
    "As part of the setup you have already created a <b>Workspace</b>. For AutoML you would need to create an <b>Experiment</b>. An <b>Experiment</b> is a named object in a <b>Workspace</b>, which is used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "import os\n",
    "import random\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.train.automl.run import AutoMLRun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# choose a name for experiment\n",
    "experiment_name =  'automl-local-classification'\n",
    "# project folder\n",
    "project_folder = './sample_projects/automl-local-classification'\n",
    "\n",
    "experiment=Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['SDK version'] = azureml.core.VERSION\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Project Directory'] = project_folder\n",
    "output['Experiment Name'] = experiment.name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(data=output, index=['']).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diagnostics\n",
    "\n",
    "Opt-in diagnostics for better experience, quality, and security of future releases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.telemetry import set_diagnostics_collection\n",
    "set_diagnostics_collection(send_diagnostics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiate Auto ML Config\n",
    "\n",
    "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**task**|classification or regression|\n",
    "|**primary_metric**|This is the metric that you want to optimize.<br> Classification supports the following primary metrics <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>balanced_accuracy</i><br><i>average_precision_score_weighted</i><br><i>precision_score_weighted</i>|\n",
    "|**max_time_sec**|Time limit in seconds for each iteration|\n",
    "|**iterations**|Number of iterations. In each iteration Auto ML trains a specific pipeline with the data|\n",
    "|**n_cross_validations**|Number of cross validation splits|\n",
    "|**X**|(sparse) array-like, shape = [n_samples, n_features]|\n",
    "|**y**|(sparse) array-like, shape = [n_samples, ], [n_samples, n_classes]<br>Multi-class targets. An indicator matrix turns on multilabel classification.  This should be an array of integers. |\n",
    "|**path**|Relative path to the project folder.  AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = datasets.load_digits()\n",
    "X_digits = digits.data[10:,:]\n",
    "y_digits = digits.target[10:]\n",
    "\n",
    "automl_config = AutoMLConfig(task = 'classification',\n",
    "                             name=experiment_name,\n",
    "                             debug_log='automl_errors.log',\n",
    "                             primary_metric='AUC_weighted',\n",
    "                             max_time_sec=1200,\n",
    "                             iterations=10,\n",
    "                             n_cross_validations=2,\n",
    "                             verbosity=logging.INFO,\n",
    "                             X = X_digits, \n",
    "                             y = y_digits,\n",
    "                             path=project_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model\n",
    "\n",
    "You can call the submit method on the experiment object and pass the run configuration. For Local runs the execution is synchronous. Depending on the data and number of iterations this can run for while.\n",
    "You will see the currently running iterations printing to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_run = experiment.submit(automl_config, show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model\n",
    "\n",
    "Below we select the best pipeline from our iterations. The *get_output* method on automl_classifier returns the best run and the fitted model for the last *fit* invocation. There are overloads on *get_output* that allow you to retrieve the best run and fitted model for *any* logged metric or a particular *iteration*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = local_run.get_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register fitted model for deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "description = 'AutoML Model'\n",
    "tags = None\n",
    "model = local_run.register_model(description=description, tags=tags, iteration=8)\n",
    "local_run.model_id # This will be written to the script file later in the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Scoring script ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile score.py\n",
    "import pickle\n",
    "import json\n",
    "import numpy\n",
    "from sklearn.externals import joblib\n",
    "from azureml.core.model import Model\n",
    "\n",
    "\n",
    "def init():\n",
    "    global model\n",
    "    model_path = Model.get_model_path(model_name = '<<modelid>>') # this name is model.id of model that we want to deploy\n",
    "    # deserialize the model file back into a sklearn model\n",
    "    model = joblib.load(model_path)\n",
    "\n",
    "def run(rawdata):\n",
    "    try:\n",
    "        data = json.loads(rawdata)['data']\n",
    "        data = numpy.array(data)\n",
    "        result = model.predict(data)\n",
    "    except Exception as e:\n",
    "        result = str(e)\n",
    "        return json.dumps({\"error\": result})\n",
    "    return json.dumps({\"result\":result.tolist()})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create yml file for env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure the consistence the fit results with the training results, the sdk dependence versions need to be the same as the environment that trains the model. Details about retrieving the versions can be found in notebook 12.auto-ml-retrieve-the-training-sdk-versions.ipynb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_name = 'automl-local-classification'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "ml_run = AutoMLRun(experiment=experiment, run_id=local_run.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dependencies = ml_run.get_run_sdk_dependencies(iteration=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in ['azureml-train-automl', 'azureml-sdk', 'azureml-core']:\n",
    "    print('{}\\t{}'.format(p, dependencies[p]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile myenv.yml\n",
    "name: myenv\n",
    "channels:\n",
    "  - defaults\n",
    "dependencies:\n",
    "  - pip:\n",
    "    - numpy==1.14.2\n",
    "    - scikit-learn==0.19.2\n",
    "    - azureml-sdk[notebooks,automl]==<<azureml-version>> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Substitute the actual version number in the environment file.\n",
    "\n",
    "conda_env_file_name = 'myenv.yml'\n",
    "\n",
    "with open(conda_env_file_name, 'r') as cefr:\n",
    "    content = cefr.read()\n",
    "\n",
    "with open(conda_env_file_name, 'w') as cefw:\n",
    "    cefw.write(content.replace('<<azureml-version>>', dependencies['azureml-sdk']))\n",
    "\n",
    "# Substitute the actual model id in the script file.\n",
    "\n",
    "script_file_name = 'score.py'\n",
    "\n",
    "with open(script_file_name, 'r') as cefr:\n",
    "    content = cefr.read()\n",
    "\n",
    "with open(script_file_name, 'w') as cefw:\n",
    "    cefw.write(content.replace('<<modelid>>', local_run.model_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Image ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.image import Image, ContainerImage\n",
    "\n",
    "image_config = ContainerImage.image_configuration(runtime= \"python\",\n",
    "                                 execution_script = script_file_name,\n",
    "                                 conda_file = conda_env_file_name,\n",
    "                                 tags = {'area': \"digits\", 'type': \"automl_classification\"},\n",
    "                                 description = \"Image for automl classification sample\")\n",
    "\n",
    "image = Image.create(name = \"automlsampleimage\",\n",
    "                     # this is the model object \n",
    "                     models = [model],\n",
    "                     image_config = image_config, \n",
    "                     workspace = ws)\n",
    "\n",
    "image.wait_for_creation(show_output = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Image as web service on Azure Container Instance ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
    "                                               memory_gb = 1, \n",
    "                                               tags = {'area': \"digits\", 'type': \"automl_classification\"}, \n",
    "                                               description = 'sample service for Automl Classification')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import Webservice\n",
    "\n",
    "aci_service_name = 'automl-sample-01'\n",
    "print(aci_service_name)\n",
    "aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
    "                                           image = image,\n",
    "                                           name = aci_service_name,\n",
    "                                           workspace = ws)\n",
    "aci_service.wait_for_deployment(True)\n",
    "print(aci_service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To delete a service ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#aci_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To get logs from deployed service ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#aci_service.get_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Web Service ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Randomly select digits and test\n",
    "digits = datasets.load_digits()\n",
    "X_digits = digits.data[:10, :]\n",
    "y_digits = digits.target[:10]\n",
    "images = digits.images[:10]\n",
    "\n",
    "for index in np.random.choice(len(y_digits), 3):\n",
    "    print(index)\n",
    "    test_sample = json.dumps({'data':X_digits[index:index + 1].tolist()})\n",
    "    predicted = aci_service.run(input_data = test_sample)\n",
    "    label = y_digits[index]\n",
    "    predictedDict = json.loads(predicted)\n",
    "    title = \"Label value = %d  Predicted value = %s \" % ( label,predictedDict['result'][0])\n",
    "    fig = plt.figure(1, figsize=(3,3))\n",
    "    ax1 = fig.add_axes((0,0,.8,.8))\n",
    "    ax1.set_title(title)\n",
    "    plt.imshow(images[index], cmap=plt.cm.gray_r, interpolation='nearest')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}