MachineLearningNotebooks/project-brainwave/project-brainwave-transfer-learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Model Development"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This example shows how to build, train, evaluate and deploy a model running on FPGA. Only Windows is supported. We use TensorFlow and Keras to build our model. We are going to use transfer learning, with ResNet152 as a featurizer. We don't use the last layer of ResNet152 in this case and instead add and train our own classification layer.\n",
        "\n",
        "We will use the Kaggle Cats and Dogs dataset to train the classifier. The dataset can be downloaded [here](https://www.microsoft.com/en-us/download/details.aspx?id=54765). Download the zip and extract to a directory named 'catsanddogs' under your user directory (\"~/catsanddogs\").\n",
        "\n",
        "Please set up your environment as described in the [quick start](project-brainwave-quickstart.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "import tensorflow as tf\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Model Construction\n",
        "Load the files we are going to use for training and testing. By default this notebook uses only a very small subset of the Cats and Dogs dataset. That makes it run quickly, but doesn't create a very accurate classifier. You can improve the classifier by using more of the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import glob\n",
        "import imghdr\n",
        "datadir = os.path.expanduser(\"~/catsanddogs\")\n",
        "\n",
        "cat_files = glob.glob(os.path.join(datadir, 'PetImages', 'Cat', '*.jpg'))\n",
        "dog_files = glob.glob(os.path.join(datadir, 'PetImages', 'Dog', '*.jpg'))\n",
        "\n",
        "# Limit the data set to make the notebook execute quickly.\n",
        "cat_files = cat_files[:64]\n",
        "dog_files = dog_files[:64]\n",
        "\n",
        "# The data set has a few images that are not jpeg. Remove them.\n",
        "cat_files = [f for f in cat_files if imghdr.what(f) == 'jpeg']\n",
        "dog_files = [f for f in dog_files if imghdr.what(f) == 'jpeg']\n",
        "\n",
        "if(not len(cat_files) or not len(dog_files)):\n",
        "    print(\"Please download the Kaggle Cats and Dogs dataset form https://www.microsoft.com/en-us/download/details.aspx?id=54765 and extract the zip to \" + datadir)    \n",
        "    raise ValueError(\"Data not found\")\n",
        "else:\n",
        "    print(cat_files[0])\n",
        "    print(dog_files[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# constructing a numpy array as labels\n",
        "image_paths = cat_files + dog_files\n",
        "total_files = len(cat_files) + len(dog_files)\n",
        "labels = np.zeros(total_files)\n",
        "labels[len(cat_files):] = 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We need to preprocess the input file to get it into the form expected by ResNet152. We've provided a default implementation of the preprocessing that you can use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Input images as a two-dimensional tensor containing an arbitrary number of images represented a strings\n",
        "import azureml.contrib.brainwave.models.utils as utils\n",
        "in_images = tf.placeholder(tf.string)\n",
        "image_tensors = utils.preprocess_array(in_images)\n",
        "print(image_tensors.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Alternatively, if you would like to customize the preprocessing, you can write your own preprocessor using TensorFlow operations.\n",
        "\n",
        "The input to the classifier we are training is the set of features produced by ResNet50. To train the classifier we need to \n",
        "featurize the images using ResNet50. You can also run the featurizer locally on CPU or GPU. We import the featurizer as frozen, so that we are only training the classifier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.contrib.brainwave.models import QuantizedResnet152\n",
        "model_path = os.path.expanduser('~/models')\n",
        "bwmodel = QuantizedResnet152(model_path, is_frozen = True)\n",
        "print(bwmodel.version)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Calling import_graph_def on the featurizer will create a service that runs the featurizer on FPGA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "features = bwmodel.import_graph_def(input_tensor=image_tensors)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Pre-compute features\n",
        "Load the data set and compute the features. These can be precomputed because they don't change during training. This can take a while to run on CPU."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "def chunks(l, n):\n",
        "    \"\"\"Yield successive n-sized chunks from l.\"\"\"\n",
        "    for i in range(0, len(l), n):\n",
        "        yield l[i:i + n]\n",
        "\n",
        "def read_files(files):\n",
        "    contents = []\n",
        "    for path in files:\n",
        "        with open(path, 'rb') as f:\n",
        "            contents.append(f.read())\n",
        "    return contents\n",
        "        \n",
        "feature_list = []\n",
        "with tf.Session() as sess:\n",
        "    for chunk in tqdm(chunks(image_paths, 5)):\n",
        "        contents = read_files(chunk)\n",
        "        result = sess.run([features], feed_dict={in_images: contents})\n",
        "        feature_list.extend(result[0])\n",
        "\n",
        "feature_results = np.array(feature_list)\n",
        "print(feature_results.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Add and Train the classifier\n",
        "We use Keras to define and train a simple classifier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dropout, Dense, Flatten\n",
        "from keras import optimizers\n",
        "\n",
        "FC_SIZE = 1024\n",
        "NUM_CLASSES = 2\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Dropout(0.2, input_shape=(1, 1, 2048,)))\n",
        "model.add(Dense(FC_SIZE, activation='relu', input_dim=(1, 1, 2048,)))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(NUM_CLASSES, activation='sigmoid', input_dim=FC_SIZE))\n",
        "\n",
        "model.compile(optimizer=optimizers.SGD(lr=1e-4,momentum=0.9), loss='binary_crossentropy', metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Prepare the train and test data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "onehot_labels = np.array([[0,1] if i else [1,0] for i in labels])\n",
        "X_train, X_test, y_train, y_test = train_test_split(feature_results, onehot_labels, random_state=42, shuffle=True)\n",
        "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Train the classifier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "model.fit(X_train, y_train, epochs=16, batch_size=32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Test the Classifier\n",
        "Let's test the classifier and see how well it does. Since we only trained on a few images, we are not expecting to win a Kaggle competition, but it will likely get most of the images correct. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from numpy import argmax\n",
        "\n",
        "y_probs = model.predict(X_test)\n",
        "y_prob_max = np.argmax(y_probs, 1)\n",
        "y_test_max = np.argmax(y_test, 1)\n",
        "print(y_prob_max)\n",
        "print(y_test_max)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.metrics import confusion_matrix, roc_auc_score, accuracy_score, precision_score, recall_score, f1_score\n",
        "import itertools\n",
        "import matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "# compute a bunch of classification metrics \n",
        "def classification_metrics(y_true, y_pred, y_prob):\n",
        "    cm_dict = {}\n",
        "    cm_dict['Accuracy'] = accuracy_score(y_true, y_pred)\n",
        "    cm_dict['Precision'] =  precision_score(y_true, y_pred)\n",
        "    cm_dict['Recall'] =  recall_score(y_true, y_pred)\n",
        "    cm_dict['F1'] =  f1_score(y_true, y_pred) \n",
        "    cm_dict['AUC'] = roc_auc_score(y_true, y_prob[:,0])\n",
        "    cm_dict['Confusion Matrix'] = confusion_matrix(y_true, y_pred).tolist()\n",
        "    return cm_dict\n",
        "\n",
        "def plot_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues):\n",
        "    \"\"\"Plots a confusion matrix.\n",
        "    Source: http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
        "    New BSD License - see appendix\n",
        "    \"\"\"\n",
        "    cm_max = cm.max()\n",
        "    cm_min = cm.min()\n",
        "    if cm_min > 0: cm_min = 0\n",
        "    if normalize:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "        cm_max = 1\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=45)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "    thresh = cm_max / 2.\n",
        "    plt.clim(cm_min, cm_max)\n",
        "\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i,\n",
        "                 round(cm[i, j], 3),  # round to 3 decimals if they are float\n",
        "                 horizontalalignment=\"center\",\n",
        "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "    plt.ylabel('True label')\n",
        "    plt.xlabel('Predicted label')\n",
        "    plt.show()\n",
        "    \n",
        "cm_dict = classification_metrics(y_test_max, y_prob_max, y_probs)\n",
        "for m in cm_dict:\n",
        "    print(m, cm_dict[m])\n",
        "cm = np.asarray(cm_dict['Confusion Matrix'])\n",
        "plot_confusion_matrix(cm, ['fail','pass'], normalize=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Service Definition\n",
        "Like in the QuickStart notebook our service definition pipeline consists of three stages. Because the preprocessing and featurizing stage don't contain any variables, we can use a default session.\n",
        "Here we use the Keras classifier as the final stage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.contrib.brainwave.pipeline import ModelDefinition, TensorflowStage, BrainWaveStage, KerasStage\n",
        "\n",
        "model_def = ModelDefinition()\n",
        "model_def.pipeline.append(TensorflowStage(tf.Session(), in_images, image_tensors))\n",
        "model_def.pipeline.append(BrainWaveStage(tf.Session(), bwmodel))\n",
        "model_def.pipeline.append(KerasStage(model))\n",
        "\n",
        "model_def_path = os.path.join(datadir, 'save', 'model_def')\n",
        "model_def.save(model_def_path)\n",
        "print(model_def_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.model import Model\n",
        "from azureml.core import Workspace\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')\n",
        "model_name = \"catsanddogs-model\"\n",
        "service_name = \"modelbuild-service\"\n",
        "\n",
        "registered_model = Model.register(ws, model_def_path, model_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The first time the code below runs it will create a new service running your model. If you want to change the model you can make changes above in this notebook and save a new service definition. Then this code will update the running service in place to run the new model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.webservice import Webservice\n",
        "from azureml.exceptions import WebserviceException\n",
        "from azureml.contrib.brainwave import BrainwaveWebservice, BrainwaveImage\n",
        "try:\n",
        "    service = Webservice(ws, service_name)\n",
        "except WebserviceException:\n",
        "    image_config = BrainwaveImage.image_configuration()\n",
        "    deployment_config = BrainwaveWebservice.deploy_configuration()\n",
        "    service = Webservice.deploy_from_model(ws, service_name, [registered_model], image_config, deployment_config)\n",
        "    service.wait_for_deployment(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The service is now running in Azure and ready to serve requests. We can check the address and port."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(service.ipAddress + ':' + str(service.port))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Client\n",
        "There is a simple test client at amlrealtimeai.PredictionClient which can be used for testing. We'll use this client to score an image with our new service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.contrib.brainwave.client import PredictionClient\n",
        "client = PredictionClient(service.ipAddress, service.port)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can adapt the client [code](../../pythonlib/amlrealtimeai/client.py) to meet your needs. There is also an example C# [client](../../sample-clients/csharp).\n",
        "\n",
        "The service provides an API that is compatible with TensorFlow Serving. There are instructions to download a sample client [here](https://www.tensorflow.org/serving/setup)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Request\n",
        "Let's see how our service does on a few images. It may get a few wrong."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Specify an image to classify\n",
        "print('CATS')\n",
        "for image_file in cat_files[:8]:\n",
        "    results = client.score_image(image_file)\n",
        "    result = 'CORRECT ' if results[0] > results[1] else 'WRONG '\n",
        "    print(result + str(results))\n",
        "print('DOGS')\n",
        "for image_file in dog_files[:8]:\n",
        "    results = client.score_image(image_file)\n",
        "    result = 'CORRECT ' if results[1] > results[0] else 'WRONG '\n",
        "    print(result + str(results))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Cleanup\n",
        "Run the cell below to delete your service. In the [next notebook](project-brainwave-custom-weights.ipynb) you will learn how to retrain all the weights of one of the models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "service.delete()\n",
        "    \n",
        "registered_model.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Appendix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "License for plot_confusion_matrix:\n",
        "\n",
        "New BSD License\n",
        "\n",
        "Copyright (c) 2007\u00e2\u20ac\u201c2018 The scikit-learn developers.\n",
        "All rights reserved.\n",
        "\n",
        "\n",
        "Redistribution and use in source and binary forms, with or without\n",
        "modification, are permitted provided that the following conditions are met:\n",
        "\n",
        "  a. Redistributions of source code must retain the above copyright notice,\n",
        "     this list of conditions and the following disclaimer.\n",
        "  b. Redistributions in binary form must reproduce the above copyright\n",
        "     notice, this list of conditions and the following disclaimer in the\n",
        "     documentation and/or other materials provided with the distribution.\n",
        "  c. Neither the name of the Scikit-learn Developers  nor the names of\n",
        "     its contributors may be used to endorse or promote products\n",
        "     derived from this software without specific prior written\n",
        "     permission. \n",
        "\n",
        "\n",
        "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n",
        "AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n",
        "IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n",
        "ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR\n",
        "ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n",
        "DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n",
        "SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n",
        "CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n",
        "LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n",
        "OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH\n",
        "DAMAGE.\n"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "coverste"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}