MachineLearningNotebooks/01.getting-started/01.train-within-notebook/01.train-within-notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 01. Train in the Notebook & Deploy Model to ACI\n",
        "\n",
        "* Load workspace\n",
        "* Train a simple regression model directly in the Notebook python kernel\n",
        "* Record run history\n",
        "* Find the best model in run history and download it.\n",
        "* Deploy the model as an Azure Container Instance (ACI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites\n",
        "1. Make sure you go through the [00. Installation and Configuration](00.configuration.ipynb) Notebook first if you haven't. \n",
        "\n",
        "2. Install following pre-requisite libraries to your conda environment and restart notebook.\n",
        "```shell\n",
        "(myenv) $ conda install -y matplotlib tqdm scikit-learn\n",
        "```\n",
        "\n",
        "3. Check that ACI is registered for your Azure Subscription.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!az provider show -n Microsoft.ContainerInstance -o table"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If ACI is not registered, run following command to register it. Note that you have to be a subscription owner, or this command will fail."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!az provider register -n Microsoft.ContainerInstance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Validate Azure ML SDK installation and get version number for debugging purposes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "install"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment, Run, Workspace\n",
        "import azureml.core\n",
        "\n",
        "# Check core SDK version number\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Initialize Workspace\n",
        "\n",
        "Initialize a workspace object from persisted configuration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "create workspace"
        ]
      },
      "outputs": [],
      "source": [
        "ws = Workspace.from_config()\n",
        "print('Workspace name: ' + ws.name, \n",
        "      'Azure region: ' + ws.location, \n",
        "      'Subscription id: ' + ws.subscription_id, \n",
        "      'Resource group: ' + ws.resource_group, sep='\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Set experiment name\n",
        "Choose a name for experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "experiment_name = 'train-in-notebook'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Start a training run in local Notebook"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# load diabetes dataset, a well-known small dataset that comes with scikit-learn\n",
        "from sklearn.datasets import load_diabetes\n",
        "from sklearn.linear_model import Ridge\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.externals import joblib\n",
        "\n",
        "X, y = load_diabetes(return_X_y = True)\n",
        "columns = ['age', 'gender', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
        "data = {\n",
        "    \"train\":{\"X\": X_train, \"y\": y_train},        \n",
        "    \"test\":{\"X\": X_test, \"y\": y_test}\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Train a simple Ridge model\n",
        "Train a very simple Ridge regression model in scikit-learn, and save it as a pickle file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "reg = Ridge(alpha = 0.03)\n",
        "reg.fit(X=data['train']['X'], y=data['train']['y'])\n",
        "preds = reg.predict(data['test']['X'])\n",
        "print('Mean Squared Error is', mean_squared_error(data['test']['y'], preds))\n",
        "joblib.dump(value=reg, filename='model.pkl');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Add experiment tracking\n",
        "Now, let's add Azure ML experiment logging, and upload persisted model into run record as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "local run",
          "outputs upload"
        ]
      },
      "outputs": [],
      "source": [
        "experiment = Experiment(workspace=ws, name=experiment_name)\n",
        "run = experiment.start_logging()\n",
        "\n",
        "run.tag(\"Description\",\"My first run!\")\n",
        "run.log('alpha', 0.03)\n",
        "reg = Ridge(alpha=0.03)\n",
        "reg.fit(data['train']['X'], data['train']['y'])\n",
        "preds = reg.predict(data['test']['X'])\n",
        "run.log('mse', mean_squared_error(data['test']['y'], preds))\n",
        "joblib.dump(value=reg, filename='model.pkl')\n",
        "run.upload_file(name='outputs/model.pkl', path_or_stream='./model.pkl')\n",
        "\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can browse to the recorded run. Please make sure you use Chrome to navigate the run history page."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Simple parameter sweep\n",
        "Sweep over alpha values of a sklearn ridge model, and capture metrics and trained model in the Azure ML experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import os\n",
        "from tqdm import tqdm\n",
        "\n",
        "model_name = \"model.pkl\"\n",
        "\n",
        "# list of numbers from 0 to 1.0 with a 0.05 interval\n",
        "alphas = np.arange(0.0, 1.0, 0.05)\n",
        "\n",
        "# try a bunch of alpha values in a Linear Regression (Ridge) model\n",
        "for alpha in tqdm(alphas):\n",
        "    # create a bunch of runs, each train a model with a different alpha value\n",
        "    with experiment.start_logging() as run:\n",
        "        # Use Ridge algorithm to build a regression model\n",
        "        reg = Ridge(alpha=alpha)\n",
        "        reg.fit(X=data[\"train\"][\"X\"], y=data[\"train\"][\"y\"])\n",
        "        preds = reg.predict(X=data[\"test\"][\"X\"])\n",
        "        mse = mean_squared_error(y_true=data[\"test\"][\"y\"], y_pred=preds)\n",
        "\n",
        "        # log alpha, mean_squared_error and feature names in run history\n",
        "        run.log(name=\"alpha\", value=alpha)\n",
        "        run.log(name=\"mse\", value=mse)\n",
        "        run.log_list(name=\"columns\", value=columns)\n",
        "\n",
        "        with open(model_name, \"wb\") as file:\n",
        "            joblib.dump(value=reg, filename=file)\n",
        "        \n",
        "        # upload the serialized model into run history record\n",
        "        run.upload_file(name=\"outputs/\" + model_name, path_or_stream=model_name)\n",
        "\n",
        "        # now delete the serialized model from local folder since it is already uploaded to run history \n",
        "        os.remove(path=model_name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# now let's take a look at the experiment in Azure portal.\n",
        "experiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Select best model from the experiment\n",
        "Load all experiment run metrics recursively from the experiment into a dictionary object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "runs = {}\n",
        "run_metrics = {}\n",
        "\n",
        "for r in tqdm(experiment.get_runs()):\n",
        "    metrics = r.get_metrics()\n",
        "    if 'mse' in metrics.keys():\n",
        "        runs[r.id] = r\n",
        "        run_metrics[r.id] = metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now find the run with the lowest Mean Squared Error value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run_id = min(run_metrics, key = lambda k: run_metrics[k]['mse'])\n",
        "best_run = runs[best_run_id]\n",
        "print('Best run is:', best_run_id)\n",
        "print('Metrics:', run_metrics[best_run_id])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can add tags to your runs to make them easier to catalog"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "query history"
        ]
      },
      "outputs": [],
      "source": [
        "best_run.tag(key=\"Description\", value=\"The best one\")\n",
        "best_run.get_tags()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Plot MSE over alpha\n",
        "\n",
        "Let's observe the best model visually by plotting the MSE values over alpha values:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "best_alpha = run_metrics[best_run_id]['alpha']\n",
        "min_mse = run_metrics[best_run_id]['mse']\n",
        "\n",
        "alpha_mse = np.array([(run_metrics[k]['alpha'], run_metrics[k]['mse']) for k in run_metrics.keys()])\n",
        "sorted_alpha_mse = alpha_mse[alpha_mse[:,0].argsort()]\n",
        "\n",
        "plt.plot(sorted_alpha_mse[:,0], sorted_alpha_mse[:,1], 'r--')\n",
        "plt.plot(sorted_alpha_mse[:,0], sorted_alpha_mse[:,1], 'bo')\n",
        "\n",
        "plt.xlabel('alpha', fontsize = 14)\n",
        "plt.ylabel('mean squared error', fontsize = 14)\n",
        "plt.title('MSE over alpha', fontsize = 16)\n",
        "\n",
        "# plot arrow\n",
        "plt.arrow(x = best_alpha, y = min_mse + 39, dx = 0, dy = -26, ls = '-', lw = 0.4,\n",
        "          width = 0, head_width = .03, head_length = 8)\n",
        "\n",
        "# plot \"best run\" text\n",
        "plt.text(x = best_alpha - 0.08, y = min_mse + 50, s = 'Best Run', fontsize = 14)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Register the best model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Find the model file saved in the run record of best run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "query history"
        ]
      },
      "outputs": [],
      "source": [
        "for f in best_run.get_file_names():\n",
        "    print(f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can register this model in the model registry of the workspace"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "register model from history"
        ]
      },
      "outputs": [],
      "source": [
        "model = best_run.register_model(model_name='best_model', model_path='outputs/model.pkl')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Verify that the model has been registered properly. If you have done this several times you'd see the version number auto-increases each time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "register model from history"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core.model import Model\n",
        "models = Model.list(workspace=ws, name='best_model')\n",
        "for m in models:\n",
        "    print(m.name, m.version)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can also download the registered model. Afterwards, you should see a `model.pkl` file in the current directory. You can then use it for local testing if you'd like."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "download file"
        ]
      },
      "outputs": [],
      "source": [
        "# remove the model file if it is already on disk\n",
        "if os.path.isfile('model.pkl'): \n",
        "    os.remove('model.pkl')\n",
        "# download the model\n",
        "model.download(target_dir=\"./\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Scoring script\n",
        "\n",
        "Now we are ready to build a Docker image and deploy the model in it as a web service. The first step is creating the scoring script. For convenience, we have created the scoring script for you. It is printed below as text, but you can also run `%pfile ./score.py` in a cell to show the file.\n",
        "\n",
        "Tbe scoring script consists of two functions: `init` that is used to load the model to memory when starting the container, and `run` that makes the prediction when web service is called. Please pay special attention to how the model is loaded in the `init()` function. When Docker image is built for this model, the actual model file is downloaded and placed on disk, and `get_model_path` function returns the local path where the model is placed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "with open('./score.py', 'r') as scoring_script:\n",
        "    print(scoring_script.read())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create environment dependency file\n",
        "\n",
        "We need a environment dependency file `myenv.yml` to specify which libraries are needed by the scoring script when building the Docker image for web service deployment. We can manually create this file, or we can use the `CondaDependencies` API to automatically create this file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.conda_dependencies import CondaDependencies \n",
        "\n",
        "myenv = CondaDependencies()\n",
        "myenv.add_conda_package(\"scikit-learn\")\n",
        "print(myenv.serialize_to_string())\n",
        "\n",
        "with open(\"myenv.yml\",\"w\") as f:\n",
        "    f.write(myenv.serialize_to_string())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy web service into an Azure Container Instance\n",
        "The deployment process takes the registered model and your scoring scrip, and builds a Docker image. It then deploys the Docker image into Azure Container Instance as a running container with an HTTP endpoint readying for scoring calls. Read more about [Azure Container Instance](https://azure.microsoft.com/en-us/services/container-instances/).\n",
        "\n",
        "Note ACI is great for quick and cost-effective dev/test deployment scenarios. For production workloads, please use [Azure Kubernentes Service (AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/) instead. Please follow in struction in [this notebook](11.production-deploy-to-aks.ipynb) to see how that can be done from Azure ML.\n",
        " \n",
        "** Note: ** The web service creation can take 6-7 minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core.webservice import AciWebservice, Webservice\n",
        "\n",
        "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n",
        "                                               memory_gb=1, \n",
        "                                               tags={'sample name': 'AML 101'}, \n",
        "                                               description='This is a great example.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Note the below `WebService.deploy_from_model()` function takes a model object registered under the workspace. It then bakes the model file in the Docker image so it can be looked-up using the `Model.get_model_path()` function in `score.py`. \n",
        "\n",
        "If you have a local model file instead of a registered model object, you can also use the `WebService.deploy()` function which would register the model and then deploy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core.image import ContainerImage\n",
        "image_config = ContainerImage.image_configuration(execution_script=\"score.py\", \n",
        "                                    runtime=\"python\", \n",
        "                                    conda_file=\"myenv.yml\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "# this will take 5-10 minutes to finish\n",
        "# you can also use \"az container list\" command to find the ACI being deployed\n",
        "service = Webservice.deploy_from_model(name='my-aci-svc',\n",
        "                                       deployment_config=aciconfig,\n",
        "                                       models=[model],\n",
        "                                       image_config=image_config,\n",
        "                                       workspace=ws)\n",
        "\n",
        "service.wait_for_deployment(show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Test web service"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "print('web service is hosted in ACI:', service.scoring_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use the `run` API to call the web service with one row of data to get a prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "import json\n",
        "# score the first row from the test set.\n",
        "test_samples = json.dumps({\"data\": X_test[0:1, :].tolist()})\n",
        "service.run(input_data = test_samples)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Feed the entire test set and calculate the errors (residual values)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "# score the entire test set.\n",
        "test_samples = json.dumps({'data': X_test.tolist()})\n",
        "\n",
        "result = json.loads(service.run(input_data = test_samples))['result']\n",
        "residual = result - y_test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can also send raw HTTP request to test the web service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import json\n",
        "\n",
        "# 2 rows of input data, each with 10 made-up numerical features\n",
        "input_data = \"{\\\"data\\\": [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]}\"\n",
        "\n",
        "headers = {'Content-Type':'application/json'}\n",
        "\n",
        "# for AKS deployment you'd need to the service key in the header as well\n",
        "# api_key = service.get_key()\n",
        "# headers = {'Content-Type':'application/json',  'Authorization':('Bearer '+ api_key)} \n",
        "\n",
        "resp = requests.post(service.scoring_uri, input_data, headers = headers)\n",
        "print(resp.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Residual graph\n",
        "Plot a residual value graph to chart the errors on the entire test set. Observe the nice bell curve."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "f, (a0, a1) = plt.subplots(1, 2, gridspec_kw={'width_ratios':[3, 1], 'wspace':0, 'hspace': 0})\n",
        "f.suptitle('Residual Values', fontsize = 18)\n",
        "\n",
        "f.set_figheight(6)\n",
        "f.set_figwidth(14)\n",
        "\n",
        "a0.plot(residual, 'bo', alpha=0.4);\n",
        "a0.plot([0,90], [0,0], 'r', lw=2)\n",
        "a0.set_ylabel('residue values', fontsize=14)\n",
        "a0.set_xlabel('test data set', fontsize=14)\n",
        "\n",
        "a1.hist(residual, orientation='horizontal', color='blue', bins=10, histtype='step');\n",
        "a1.hist(residual, orientation='horizontal', color='blue', alpha=0.2, bins=10);\n",
        "a1.set_yticklabels([])\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Delete ACI to clean up"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Deleting ACI is super fast!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "deploy service",
          "aci"
        ]
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "service.delete()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "roastala"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}