MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# YOLO Real-time Object Detection using ONNX on AzureML\n",
        "\n",
        "This example shows how to convert the TinyYOLO model from CoreML to ONNX and operationalize it as a web service using Azure Machine Learning services and the ONNX Runtime.\n",
        "\n",
        "## What is ONNX\n",
        "ONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai).\n",
        "\n",
        "## YOLO Details\n",
        "You Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. For more information about YOLO, please visit the [YOLO website](https://pjreddie.com/darknet/yolo/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites\n",
        "\n",
        "To make the best use of your time, make sure you have done the following:\n",
        "\n",
        "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook to:\n",
        "    * install the AML SDK\n",
        "    * create a workspace and its configuration file (config.json)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Check core SDK version number\n",
        "import azureml.core\n",
        "\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Install necessary packages\n",
        "\n",
        "You'll need to run the following commands to use this tutorial:\n",
        "\n",
        "```sh\n",
        "pip install onnxmltools\n",
        "pip install coremltools  # use this on Linux and Mac\n",
        "pip install git+https://github.com/apple/coremltools  # use this on Windows\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Convert model to ONNX\n",
        "\n",
        "First we download the CoreML model. We use the CoreML model from [Matthijs Hollemans's tutorial](https://github.com/hollance/YOLO-CoreML-MPSNNGraph). This may take a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import urllib.request\n",
        "\n",
        "coreml_model_url = \"https://github.com/hollance/YOLO-CoreML-MPSNNGraph/raw/master/TinyYOLO-CoreML/TinyYOLO-CoreML/TinyYOLO.mlmodel\"\n",
        "urllib.request.urlretrieve(coreml_model_url, filename=\"TinyYOLO.mlmodel\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then we use ONNXMLTools to convert the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import onnxmltools\n",
        "import coremltools\n",
        "\n",
        "# Load a CoreML model\n",
        "coreml_model = coremltools.utils.load_spec('TinyYOLO.mlmodel')\n",
        "\n",
        "# Convert from CoreML into ONNX\n",
        "onnx_model = onnxmltools.convert_coreml(coreml_model, 'TinyYOLOv2')\n",
        "\n",
        "# Fix the preprocessor bias in the ImageScaler\n",
        "for init in onnx_model.graph.initializer:\n",
        "    if init.name == 'scalerPreprocessor_bias':\n",
        "        init.dims[1] = 1\n",
        "\n",
        "# Save ONNX model\n",
        "onnxmltools.utils.save_model(onnx_model, 'tinyyolov2.onnx')\n",
        "\n",
        "import os\n",
        "print(os.path.getsize('tinyyolov2.onnx'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploying as a web service with Azure ML\n",
        "\n",
        "### Load Azure ML workspace\n",
        "\n",
        "We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Workspace\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Registering your model with Azure ML\n",
        "\n",
        "Now we upload the model and register it in the workspace."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.model import Model\n",
        "\n",
        "model = Model.register(model_path = \"tinyyolov2.onnx\",\n",
        "                       model_name = \"tinyyolov2\",\n",
        "                       tags = {\"onnx\": \"demo\"},\n",
        "                       description = \"TinyYOLO\",\n",
        "                       workspace = ws)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Displaying your registered models\n",
        "\n",
        "You can optionally list out all the models that you have registered in this workspace."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "models = ws.models\n",
        "for name, m in models.items():\n",
        "    print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Write scoring file\n",
        "\n",
        "We are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile score.py\n",
        "import json\n",
        "import time\n",
        "import sys\n",
        "import os\n",
        "from azureml.core.model import Model\n",
        "import numpy as np    # we're going to use numpy to process input and output data\n",
        "import onnxruntime    # to inference ONNX models, we use the ONNX Runtime\n",
        "\n",
        "def init():\n",
        "    global session\n",
        "    model = Model.get_model_path(model_name = 'tinyyolov2')\n",
        "    session = onnxruntime.InferenceSession(model)\n",
        "\n",
        "def preprocess(input_data_json):\n",
        "    # convert the JSON data into the tensor input\n",
        "    return np.array(json.loads(input_data_json)['data']).astype('float32')\n",
        "\n",
        "def postprocess(result):\n",
        "    return np.array(result).tolist()\n",
        "\n",
        "def run(input_data_json):\n",
        "    try:\n",
        "        start = time.time()   # start timer\n",
        "        input_data = preprocess(input_data_json)\n",
        "        input_name = session.get_inputs()[0].name  # get the id of the first input of the model   \n",
        "        result = session.run([], {input_name: input_data})\n",
        "        end = time.time()     # stop timer\n",
        "        return {\"result\": postprocess(result),\n",
        "                \"time\": end - start}\n",
        "    except Exception as e:\n",
        "        result = str(e)\n",
        "        return {\"error\": result}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setting up inference configuration\n",
        "First we create a YAML file that specifies which dependencies we would like to see in our container. Please note that you must include azureml-defaults with verion >= 1.0.45 as a pip dependency, because it contains the functionality needed to host the model as a web service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.conda_dependencies import CondaDependencies \n",
        "\n",
        "myenv = CondaDependencies.create(pip_packages=[\"numpy\", \"onnxruntime==1.15.1\", \"azureml-core\", \"azureml-defaults\"])\n",
        "\n",
        "with open(\"myenv.yml\",\"w\") as f:\n",
        "    f.write(myenv.serialize_to_string())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then we create the inference configuration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.model import InferenceConfig\n",
        "from azureml.core.environment import Environment\n",
        "\n",
        "\n",
        "myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"myenv.yml\")\n",
        "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Deploy the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.webservice import AciWebservice\n",
        "\n",
        "aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
        "                                               memory_gb = 1, \n",
        "                                               tags = {'demo': 'onnx'}, \n",
        "                                               description = 'web service for TinyYOLO ONNX model')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following cell will take a few minutes to run as the model gets packaged up and deployed to ACI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "aci_service_name = 'my-aci-service-tiny-yolo'\n",
        "print(\"Service\", aci_service_name)\n",
        "aci_service = Model.deploy(ws, aci_service_name, [model], inference_config, aciconfig)\n",
        "aci_service.wait_for_deployment(True)\n",
        "print(aci_service.state)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In case the deployment fails, you can check the logs. Make sure to delete your aci_service before trying again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if aci_service.state != 'Healthy':\n",
        "    # run this command for debugging.\n",
        "    print(aci_service.get_logs())\n",
        "    aci_service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Success!\n",
        "\n",
        "If you've made it this far, you've deployed a working web service that does object detection using an ONNX model. You can get the URL for the webservice with the code below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(aci_service.scoring_uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When you are eventually done using the web service, remember to delete it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "aci_service.delete()"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "viswamy"
      }
    ],
    "category": "deployment",
    "compute": [
      "local"
    ],
    "datasets": [
      "PASCAL VOC"
    ],
    "deployment": [
      "Azure Container Instance"
    ],
    "exclude_from_index": false,
    "framework": [
      "ONNX"
    ],
    "friendly_name": "Convert and deploy TinyYolo with ONNX Runtime",
    "index_order": 5,
    "kernelspec": {
      "display_name": "Python 3.8 - AzureML",
      "language": "python",
      "name": "python38-azureml"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "star_tag": [
      "featured"
    ],
    "tags": [
      "ONNX Converter"
    ],
    "task": "Object Detection"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}