Compare commits
1 Commits
release_up
...
release_up
| Author | SHA1 | Date | |
|---|---|---|---|
|
a6088928ab |
@@ -1,383 +1,383 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Configuration\n",
|
"# Configuration\n",
|
||||||
"\n",
|
"\n",
|
||||||
"_**Setting up your Azure Machine Learning services workspace and configuring your notebook library**_\n",
|
"_**Setting up your Azure Machine Learning services workspace and configuring your notebook library**_\n",
|
||||||
"\n",
|
"\n",
|
||||||
"---\n",
|
"---\n",
|
||||||
"---\n",
|
"---\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Table of Contents\n",
|
"## Table of Contents\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. [Introduction](#Introduction)\n",
|
"1. [Introduction](#Introduction)\n",
|
||||||
" 1. What is an Azure Machine Learning workspace\n",
|
" 1. What is an Azure Machine Learning workspace\n",
|
||||||
"1. [Setup](#Setup)\n",
|
"1. [Setup](#Setup)\n",
|
||||||
" 1. Azure subscription\n",
|
" 1. Azure subscription\n",
|
||||||
" 1. Azure ML SDK and other library installation\n",
|
" 1. Azure ML SDK and other library installation\n",
|
||||||
" 1. Azure Container Instance registration\n",
|
" 1. Azure Container Instance registration\n",
|
||||||
"1. [Configure your Azure ML Workspace](#Configure%20your%20Azure%20ML%20workspace)\n",
|
"1. [Configure your Azure ML Workspace](#Configure%20your%20Azure%20ML%20workspace)\n",
|
||||||
" 1. Workspace parameters\n",
|
" 1. Workspace parameters\n",
|
||||||
" 1. Access your workspace\n",
|
" 1. Access your workspace\n",
|
||||||
" 1. Create a new workspace\n",
|
" 1. Create a new workspace\n",
|
||||||
" 1. Create compute resources\n",
|
" 1. Create compute resources\n",
|
||||||
"1. [Next steps](#Next%20steps)\n",
|
"1. [Next steps](#Next%20steps)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"---\n",
|
"---\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Introduction\n",
|
"## Introduction\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This notebook configures your library of notebooks to connect to an Azure Machine Learning (ML) workspace. In this case, a library contains all of the notebooks in the current folder and any nested folders. You can configure this notebook library to use an existing workspace or create a new workspace.\n",
|
"This notebook configures your library of notebooks to connect to an Azure Machine Learning (ML) workspace. In this case, a library contains all of the notebooks in the current folder and any nested folders. You can configure this notebook library to use an existing workspace or create a new workspace.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Typically you will need to run this notebook only once per notebook library as all other notebooks will use connection information that is written here. If you want to redirect your notebook library to work with a different workspace, then you should re-run this notebook.\n",
|
"Typically you will need to run this notebook only once per notebook library as all other notebooks will use connection information that is written here. If you want to redirect your notebook library to work with a different workspace, then you should re-run this notebook.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In this notebook you will\n",
|
"In this notebook you will\n",
|
||||||
"* Learn about getting an Azure subscription\n",
|
"* Learn about getting an Azure subscription\n",
|
||||||
"* Specify your workspace parameters\n",
|
"* Specify your workspace parameters\n",
|
||||||
"* Access or create your workspace\n",
|
"* Access or create your workspace\n",
|
||||||
"* Add a default compute cluster for your workspace\n",
|
"* Add a default compute cluster for your workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### What is an Azure Machine Learning workspace\n",
|
"### What is an Azure Machine Learning workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"An Azure ML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models."
|
"An Azure ML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Setup\n",
|
"## Setup\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This section describes activities required before you can access any Azure ML services functionality."
|
"This section describes activities required before you can access any Azure ML services functionality."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### 1. Azure Subscription\n",
|
"### 1. Azure Subscription\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In order to create an Azure ML Workspace, first you need access to an Azure subscription. An Azure subscription allows you to manage storage, compute, and other assets in the Azure cloud. You can [create a new subscription](https://azure.microsoft.com/en-us/free/) or access existing subscription information from the [Azure portal](https://portal.azure.com). Later in this notebook you will need information such as your subscription ID in order to create and access AML workspaces.\n",
|
"In order to create an Azure ML Workspace, first you need access to an Azure subscription. An Azure subscription allows you to manage storage, compute, and other assets in the Azure cloud. You can [create a new subscription](https://azure.microsoft.com/en-us/free/) or access existing subscription information from the [Azure portal](https://portal.azure.com). Later in this notebook you will need information such as your subscription ID in order to create and access AML workspaces.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### 2. Azure ML SDK and other library installation\n",
|
"### 2. Azure ML SDK and other library installation\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you are running in your own environment, follow [SDK installation instructions](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment). If you are running in Azure Notebooks or another Microsoft managed environment, the SDK is already installed.\n",
|
"If you are running in your own environment, follow [SDK installation instructions](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment). If you are running in Azure Notebooks or another Microsoft managed environment, the SDK is already installed.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Also install following libraries to your environment. Many of the example notebooks depend on them\n",
|
"Also install following libraries to your environment. Many of the example notebooks depend on them\n",
|
||||||
"\n",
|
"\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"(myenv) $ conda install -y matplotlib tqdm scikit-learn\n",
|
"(myenv) $ conda install -y matplotlib tqdm scikit-learn\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Once installation is complete, the following cell checks the Azure ML SDK version:"
|
"Once installation is complete, the following cell checks the Azure ML SDK version:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"install"
|
"install"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"This notebook was created using version AZUREML-SDK-VERSION of the Azure ML SDK\")\n",
|
"print(\"This notebook was created using version 1.0.43 of the Azure ML SDK\")\n",
|
||||||
"print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
|
"print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"If you are using an older version of the SDK then this notebook was created using, you should upgrade your SDK.\n",
|
"If you are using an older version of the SDK then this notebook was created using, you should upgrade your SDK.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### 3. Azure Container Instance registration\n",
|
"### 3. Azure Container Instance registration\n",
|
||||||
"Azure Machine Learning uses of [Azure Container Instance (ACI)](https://azure.microsoft.com/services/container-instances) to deploy dev/test web services. An Azure subscription needs to be registered to use ACI. If you or the subscription owner have not yet registered ACI on your subscription, you will need to use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and execute the following commands. Note that if you ran through the AML [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) you have already registered ACI. \n",
|
"Azure Machine Learning uses of [Azure Container Instance (ACI)](https://azure.microsoft.com/services/container-instances) to deploy dev/test web services. An Azure subscription needs to be registered to use ACI. If you or the subscription owner have not yet registered ACI on your subscription, you will need to use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and execute the following commands. Note that if you ran through the AML [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) you have already registered ACI. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"```shell\n",
|
"```shell\n",
|
||||||
"# check to see if ACI is already registered\n",
|
"# check to see if ACI is already registered\n",
|
||||||
"(myenv) $ az provider show -n Microsoft.ContainerInstance -o table\n",
|
"(myenv) $ az provider show -n Microsoft.ContainerInstance -o table\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# if ACI is not registered, run this command.\n",
|
"# if ACI is not registered, run this command.\n",
|
||||||
"# note you need to be the subscription owner in order to execute this command successfully.\n",
|
"# note you need to be the subscription owner in order to execute this command successfully.\n",
|
||||||
"(myenv) $ az provider register -n Microsoft.ContainerInstance\n",
|
"(myenv) $ az provider register -n Microsoft.ContainerInstance\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"---"
|
"---"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Configure your Azure ML workspace\n",
|
"## Configure your Azure ML workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Workspace parameters\n",
|
"### Workspace parameters\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To use an AML Workspace, you will need to import the Azure ML SDK and supply the following information:\n",
|
"To use an AML Workspace, you will need to import the Azure ML SDK and supply the following information:\n",
|
||||||
"* Your subscription id\n",
|
"* Your subscription id\n",
|
||||||
"* A resource group name\n",
|
"* A resource group name\n",
|
||||||
"* (optional) The region that will host your workspace\n",
|
"* (optional) The region that will host your workspace\n",
|
||||||
"* A name for your workspace\n",
|
"* A name for your workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can get your subscription ID from the [Azure portal](https://portal.azure.com).\n",
|
"You can get your subscription ID from the [Azure portal](https://portal.azure.com).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You will also need access to a [_resource group_](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), which organizes Azure resources and provides a default region for the resources in a group. You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com). If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
|
"You will also need access to a [_resource group_](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), which organizes Azure resources and provides a default region for the resources in a group. You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com). If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The region to host your workspace will be used if you are creating a new workspace. You do not need to specify this if you are using an existing workspace. You can find the list of supported regions [here](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=machine-learning-service). You should pick a region that is close to your location or that contains your data.\n",
|
"The region to host your workspace will be used if you are creating a new workspace. You do not need to specify this if you are using an existing workspace. You can find the list of supported regions [here](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=machine-learning-service). You should pick a region that is close to your location or that contains your data.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The name for your workspace is unique within the subscription and should be descriptive enough to discern among other AML Workspaces. The subscription may be used only by you, or it may be used by your department or your entire enterprise, so choose a name that makes sense for your situation.\n",
|
"The name for your workspace is unique within the subscription and should be descriptive enough to discern among other AML Workspaces. The subscription may be used only by you, or it may be used by your department or your entire enterprise, so choose a name that makes sense for your situation.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The following cell allows you to specify your workspace parameters. This cell uses the python method `os.getenv` to read values from environment variables which is useful for automation. If no environment variable exists, the parameters will be set to the specified default values. \n",
|
"The following cell allows you to specify your workspace parameters. This cell uses the python method `os.getenv` to read values from environment variables which is useful for automation. If no environment variable exists, the parameters will be set to the specified default values. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace! You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n",
|
"If you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace! You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Replace the default values in the cell below with your workspace parameters"
|
"Replace the default values in the cell below with your workspace parameters"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"subscription_id = os.getenv(\"SUBSCRIPTION_ID\", default=\"<my-subscription-id>\")\n",
|
"subscription_id = os.getenv(\"SUBSCRIPTION_ID\", default=\"<my-subscription-id>\")\n",
|
||||||
"resource_group = os.getenv(\"RESOURCE_GROUP\", default=\"<my-resource-group>\")\n",
|
"resource_group = os.getenv(\"RESOURCE_GROUP\", default=\"<my-resource-group>\")\n",
|
||||||
"workspace_name = os.getenv(\"WORKSPACE_NAME\", default=\"<my-workspace-name>\")\n",
|
"workspace_name = os.getenv(\"WORKSPACE_NAME\", default=\"<my-workspace-name>\")\n",
|
||||||
"workspace_region = os.getenv(\"WORKSPACE_REGION\", default=\"eastus2\")"
|
"workspace_region = os.getenv(\"WORKSPACE_REGION\", default=\"eastus2\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Access your workspace\n",
|
"### Access your workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The following cell uses the Azure ML SDK to attempt to load the workspace specified by your parameters. If this cell succeeds, your notebook library will be configured to access the workspace from all notebooks using the `Workspace.from_config()` method. The cell can fail if the specified workspace doesn't exist or you don't have permissions to access it. "
|
"The following cell uses the Azure ML SDK to attempt to load the workspace specified by your parameters. If this cell succeeds, your notebook library will be configured to access the workspace from all notebooks using the `Workspace.from_config()` method. The cell can fail if the specified workspace doesn't exist or you don't have permissions to access it. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n",
|
" ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n",
|
||||||
" # write the details of the workspace to a configuration file to the notebook library\n",
|
" # write the details of the workspace to a configuration file to the notebook library\n",
|
||||||
" ws.write_config()\n",
|
" ws.write_config()\n",
|
||||||
" print(\"Workspace configuration succeeded. Skip the workspace creation steps below\")\n",
|
" print(\"Workspace configuration succeeded. Skip the workspace creation steps below\")\n",
|
||||||
"except:\n",
|
"except:\n",
|
||||||
" print(\"Workspace not accessible. Change your parameters or create a new workspace below\")"
|
" print(\"Workspace not accessible. Change your parameters or create a new workspace below\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a new workspace\n",
|
"### Create a new workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you don't have an existing workspace and are the owner of the subscription or resource group, you can create a new workspace. If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
|
"If you don't have an existing workspace and are the owner of the subscription or resource group, you can create a new workspace. If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Note**: As with other Azure services, there are limits on certain resources (for example AmlCompute quota) associated with the Azure ML service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
|
"**Note**: As with other Azure services, there are limits on certain resources (for example AmlCompute quota) associated with the Azure ML service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This cell will create an Azure ML workspace for you in a subscription provided you have the correct permissions.\n",
|
"This cell will create an Azure ML workspace for you in a subscription provided you have the correct permissions.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This will fail if:\n",
|
"This will fail if:\n",
|
||||||
"* You do not have permission to create a workspace in the resource group\n",
|
"* You do not have permission to create a workspace in the resource group\n",
|
||||||
"* You do not have permission to create a resource group if it's non-existing.\n",
|
"* You do not have permission to create a resource group if it's non-existing.\n",
|
||||||
"* You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
|
"* You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources."
|
"If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create workspace"
|
"create workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Create the workspace using the specified parameters\n",
|
"# Create the workspace using the specified parameters\n",
|
||||||
"ws = Workspace.create(name = workspace_name,\n",
|
"ws = Workspace.create(name = workspace_name,\n",
|
||||||
" subscription_id = subscription_id,\n",
|
" subscription_id = subscription_id,\n",
|
||||||
" resource_group = resource_group, \n",
|
" resource_group = resource_group, \n",
|
||||||
" location = workspace_region,\n",
|
" location = workspace_region,\n",
|
||||||
" create_resource_group = True,\n",
|
" create_resource_group = True,\n",
|
||||||
" exist_ok = True)\n",
|
" exist_ok = True)\n",
|
||||||
"ws.get_details()\n",
|
"ws.get_details()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# write the details of the workspace to a configuration file to the notebook library\n",
|
"# write the details of the workspace to a configuration file to the notebook library\n",
|
||||||
"ws.write_config()"
|
"ws.write_config()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create compute resources for your training experiments\n",
|
"### Create compute resources for your training experiments\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Many of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n",
|
"Many of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors. Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n",
|
"To create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors. Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The cluster parameters are:\n",
|
"The cluster parameters are:\n",
|
||||||
"* vm_size - this describes the virtual machine type and size used in the cluster. All machines in the cluster are the same type. You can get the list of vm sizes available in your region by using the CLI command\n",
|
"* vm_size - this describes the virtual machine type and size used in the cluster. All machines in the cluster are the same type. You can get the list of vm sizes available in your region by using the CLI command\n",
|
||||||
"\n",
|
"\n",
|
||||||
"```shell\n",
|
"```shell\n",
|
||||||
"az vm list-skus -o tsv\n",
|
"az vm list-skus -o tsv\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"* min_nodes - this sets the minimum size of the cluster. If you set the minimum to 0 the cluster will shut down all nodes while not in use. Setting this number to a value higher than 0 will allow for faster start-up times, but you will also be billed when the cluster is not in use.\n",
|
"* min_nodes - this sets the minimum size of the cluster. If you set the minimum to 0 the cluster will shut down all nodes while note in use. Setting this number to a value higher than 0 will allow for faster start-up times, but you will also be billed when the cluster is not in use.\n",
|
||||||
"* max_nodes - this sets the maximum size of the cluster. Setting this to a larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n",
|
"* max_nodes - this sets the maximum size of the cluster. Setting this to a larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To create a **CPU** cluster now, run the cell below. The autoscale settings mean that the cluster will scale down to 0 nodes when inactive and up to 4 nodes when busy."
|
"To create a **CPU** cluster now, run the cell below. The autoscale settings mean that the cluster will scale down to 0 nodes when inactive and up to 4 nodes when busy."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Choose a name for your CPU cluster\n",
|
"# Choose a name for your CPU cluster\n",
|
||||||
"cpu_cluster_name = \"cpu-cluster\"\n",
|
"cpu_cluster_name = \"cpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Verify that cluster does not exist already\n",
|
"# Verify that cluster does not exist already\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
||||||
" print(\"Found existing cpu-cluster\")\n",
|
" print(\"Found existing cpu-cluster\")\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print(\"Creating new cpu-cluster\")\n",
|
" print(\"Creating new cpu-cluster\")\n",
|
||||||
" \n",
|
" \n",
|
||||||
" # Specify the configuration for the new cluster\n",
|
" # Specify the configuration for the new cluster\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n",
|
||||||
" min_nodes=0,\n",
|
" min_nodes=0,\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Create the cluster with the specified name and configuration\n",
|
" # Create the cluster with the specified name and configuration\n",
|
||||||
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
||||||
" \n",
|
" \n",
|
||||||
" # Wait for the cluster to complete, show the output log\n",
|
" # Wait for the cluster to complete, show the output log\n",
|
||||||
" cpu_cluster.wait_for_completion(show_output=True)"
|
" cpu_cluster.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"To create a **GPU** cluster, run the cell below. Note that your subscription must have sufficient quota for GPU VMs or the command will fail. To increase quota, see [these instructions](https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request). "
|
"To create a **GPU** cluster, run the cell below. Note that your subscription must have sufficient quota for GPU VMs or the command will fail. To increase quota, see [these instructions](https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request). "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Choose a name for your GPU cluster\n",
|
"# Choose a name for your GPU cluster\n",
|
||||||
"gpu_cluster_name = \"gpu-cluster\"\n",
|
"gpu_cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Verify that cluster does not exist already\n",
|
"# Verify that cluster does not exist already\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
|
" gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
|
||||||
" print(\"Found existing gpu cluster\")\n",
|
" print(\"Found existing gpu cluster\")\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print(\"Creating new gpu-cluster\")\n",
|
" print(\"Creating new gpu-cluster\")\n",
|
||||||
" \n",
|
" \n",
|
||||||
" # Specify the configuration for the new cluster\n",
|
" # Specify the configuration for the new cluster\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
|
||||||
" min_nodes=0,\n",
|
" min_nodes=0,\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
" # Create the cluster with the specified name and configuration\n",
|
" # Create the cluster with the specified name and configuration\n",
|
||||||
" gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n",
|
" gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # Wait for the cluster to complete, show the output log\n",
|
" # Wait for the cluster to complete, show the output log\n",
|
||||||
" gpu_cluster.wait_for_completion(show_output=True)"
|
" gpu_cluster.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"---\n",
|
"---\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Next steps\n",
|
"## Next steps\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In this notebook you configured this notebook library to connect easily to an Azure ML workspace. You can copy this notebook to your own libraries to connect them to you workspace, or use it to bootstrap new workspaces completely.\n",
|
"In this notebook you configured this notebook library to connect easily to an Azure ML workspace. You can copy this notebook to your own libraries to connect them to you workspace, or use it to bootstrap new workspaces completely.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you came here from another notebook, you can return there and complete that exercise, or you can try out the [Tutorials](./tutorials) or jump into \"how-to\" notebooks and start creating and deploying models. A good place to start is the [train within notebook](./how-to-use-azureml/training/train-within-notebook) example that walks through a simplified but complete end to end machine learning process."
|
"If you came here from another notebook, you can return there and complete that exercise, or you can try out the [Tutorials](./tutorials) or jump into \"how-to\" notebooks and start creating and deploying models. A good place to start is the [train within notebook](./how-to-use-azureml/training/train-within-notebook) example that walks through a simplified but complete end to end machine learning process."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": []
|
"source": []
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "roastala"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "roastala"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.5"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,498 +1,498 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Enabling App Insights for Services in Production\n",
|
"# Enabling App Insights for Services in Production\n",
|
||||||
"With this notebook, you can learn how to enable App Insights for standard service monitoring, plus, we provide examples for doing custom logging within a scoring files in a model. \n",
|
"With this notebook, you can learn how to enable App Insights for standard service monitoring, plus, we provide examples for doing custom logging within a scoring files in a model. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What does Application Insights monitor?\n",
|
"## What does Application Insights monitor?\n",
|
||||||
"It monitors request rates, response times, failure rates, etc. For more information visit [App Insights docs.](https://docs.microsoft.com/en-us/azure/application-insights/app-insights-overview)\n",
|
"It monitors request rates, response times, failure rates, etc. For more information visit [App Insights docs.](https://docs.microsoft.com/en-us/azure/application-insights/app-insights-overview)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What is different compared to standard production deployment process?\n",
|
"## What is different compared to standard production deployment process?\n",
|
||||||
"If you want to enable generic App Insights for a service run:\n",
|
"If you want to enable generic App Insights for a service run:\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"aks_service= Webservice(ws, \"aks-w-dc2\")\n",
|
"aks_service= Webservice(ws, \"aks-w-dc2\")\n",
|
||||||
"aks_service.update(enable_app_insights=True)```\n",
|
"aks_service.update(enable_app_insights=True)```\n",
|
||||||
"Where \"aks-w-dc2\" is your service name. You can also do this from the Azure Portal under your Workspace--> deployments--> Select deployment--> Edit--> Advanced Settings--> Select \"Enable AppInsights diagnostics\"\n",
|
"Where \"aks-w-dc2\" is your service name. You can also do this from the Azure Portal under your Workspace--> deployments--> Select deployment--> Edit--> Advanced Settings--> Select \"Enable AppInsights diagnostics\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you want to log custom traces, you will follow the standard deplyment process for AKS and you will:\n",
|
"If you want to log custom traces, you will follow the standard deplyment process for AKS and you will:\n",
|
||||||
"1. Update scoring file.\n",
|
"1. Update scoring file.\n",
|
||||||
"2. Update aks configuration.\n",
|
"2. Update aks configuration.\n",
|
||||||
"3. Build new image and deploy it. "
|
"3. Build new image and deploy it. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 1. Import your dependencies"
|
"## 1. Import your dependencies"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
||||||
"from azureml.core.webservice import AksWebservice\n",
|
"from azureml.core.webservice import AksWebservice\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"print(azureml.core.VERSION)"
|
"print(azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 2. Set up your configuration and create a workspace\n"
|
"## 2. Set up your configuration and create a workspace\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 3. Register Model\n",
|
"## 3. Register Model\n",
|
||||||
"Register an existing trained model, add descirption and tags."
|
"Register an existing trained model, add descirption and tags."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Register the model\n",
|
"#Register the model\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
||||||
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
||||||
" description = \"Ridge regression model to predict diabetes\",\n",
|
" description = \"Ridge regression model to predict diabetes\",\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(model.name, model.description, model.version)"
|
"print(model.name, model.description, model.version)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 4. *Update your scoring file with custom print statements*\n",
|
"## 4. *Update your scoring file with custom print statements*\n",
|
||||||
"Here is an example:\n",
|
"Here is an example:\n",
|
||||||
"### a. In your init function add:\n",
|
"### a. In your init function add:\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))```\n",
|
"print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### b. In your run function add:\n",
|
"### b. In your run function add:\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"print (\"Prediction created\" + time.strftime(\"%H:%M:%S\"))```"
|
"print (\"Prediction created\" + time.strftime(\"%H:%M:%S\"))```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import pickle\n",
|
"import pickle\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import numpy \n",
|
"import numpy \n",
|
||||||
"from sklearn.externals import joblib\n",
|
"from sklearn.externals import joblib\n",
|
||||||
"from sklearn.linear_model import Ridge\n",
|
"from sklearn.linear_model import Ridge\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"import time\n",
|
"import time\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global model\n",
|
" global model\n",
|
||||||
" #Print statement for appinsights custom traces:\n",
|
" #Print statement for appinsights custom traces:\n",
|
||||||
" print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))\n",
|
" print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" \n",
|
" \n",
|
||||||
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under the workspace\n",
|
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under the workspace\n",
|
||||||
" # this call should return the path to the model.pkl file on the local disk.\n",
|
" # this call should return the path to the model.pkl file on the local disk.\n",
|
||||||
" model_path = Model.get_model_path(model_name = 'sklearn_regression_model.pkl')\n",
|
" model_path = Model.get_model_path(model_name = 'sklearn_regression_model.pkl')\n",
|
||||||
" \n",
|
" \n",
|
||||||
" # deserialize the model file back into a sklearn model\n",
|
" # deserialize the model file back into a sklearn model\n",
|
||||||
" model = joblib.load(model_path)\n",
|
" model = joblib.load(model_path)\n",
|
||||||
" \n",
|
" \n",
|
||||||
"\n",
|
"\n",
|
||||||
"# note you can pass in multiple rows for scoring\n",
|
"# note you can pass in multiple rows for scoring\n",
|
||||||
"def run(raw_data):\n",
|
"def run(raw_data):\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" data = json.loads(raw_data)['data']\n",
|
" data = json.loads(raw_data)['data']\n",
|
||||||
" data = numpy.array(data)\n",
|
" data = numpy.array(data)\n",
|
||||||
" result = model.predict(data)\n",
|
" result = model.predict(data)\n",
|
||||||
" print (\"Prediction created\" + time.strftime(\"%H:%M:%S\"))\n",
|
" print (\"Prediction created\" + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" # you can return any datatype as long as it is JSON-serializable\n",
|
" # you can return any datatype as long as it is JSON-serializable\n",
|
||||||
" return result.tolist()\n",
|
" return result.tolist()\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" error = str(e)\n",
|
" error = str(e)\n",
|
||||||
" print (error + time.strftime(\"%H:%M:%S\"))\n",
|
" print (error + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" return error"
|
" return error"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 5. *Create myenv.yml file*"
|
"## 5. *Create myenv.yml file*"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 6. Create your new Image"
|
"## 6. Create your new Image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import ContainerImage\n",
|
"from azureml.core.image import ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
||||||
" runtime = \"python\",\n",
|
" runtime = \"python\",\n",
|
||||||
" conda_file = \"myenv.yml\",\n",
|
" conda_file = \"myenv.yml\",\n",
|
||||||
" description = \"Image with ridge regression model\",\n",
|
" description = \"Image with ridge regression model\",\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = ContainerImage.create(name = \"myimage1\",\n",
|
"image = ContainerImage.create(name = \"myimage1\",\n",
|
||||||
" # this is the model object\n",
|
" # this is the model object\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config,\n",
|
" image_config = image_config,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Deploy to ACI (Optional)"
|
"## Deploy to ACI (Optional)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import AciWebservice\n",
|
"from azureml.core.webservice import AciWebservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
||||||
" memory_gb = 1, \n",
|
" memory_gb = 1, \n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"}, \n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"}, \n",
|
||||||
" description = 'Predict diabetes using regression model',\n",
|
" description = 'Predict diabetes using regression model',\n",
|
||||||
" enable_app_insights = True)"
|
" enable_app_insights = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import Webservice\n",
|
"from azureml.core.webservice import Webservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service_name = 'my-aci-service-4'\n",
|
"aci_service_name = 'my-aci-service-4'\n",
|
||||||
"print(aci_service_name)\n",
|
"print(aci_service_name)\n",
|
||||||
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" name = aci_service_name,\n",
|
" name = aci_service_name,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"aci_service.wait_for_deployment(True)\n",
|
"aci_service.wait_for_deployment(True)\n",
|
||||||
"print(aci_service.state)"
|
"print(aci_service.state)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,28,13,45,54,6,57,8,8,10], \n",
|
" [1,28,13,45,54,6,57,8,8,10], \n",
|
||||||
" [101,9,8,37,6,45,4,3,2,41]\n",
|
" [101,9,8,37,6,45,4,3,2,41]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding='utf8')"
|
"test_sample = bytes(test_sample,encoding='utf8')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"if aci_service.state == \"Healthy\":\n",
|
"if aci_service.state == \"Healthy\":\n",
|
||||||
" prediction = aci_service.run(input_data=test_sample)\n",
|
" prediction = aci_service.run(input_data=test_sample)\n",
|
||||||
" print(prediction)\n",
|
" print(prediction)\n",
|
||||||
"else:\n",
|
"else:\n",
|
||||||
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aci_service.error)"
|
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aci_service.error)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 7. Deploy to AKS service"
|
"## 7. Deploy to AKS service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create AKS compute if you haven't done so."
|
"### Create AKS compute if you haven't done so."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Use the default configuration (can also provide parameters to customize)\n",
|
"# Use the default configuration (can also provide parameters to customize)\n",
|
||||||
"prov_config = AksCompute.provisioning_configuration()\n",
|
"prov_config = AksCompute.provisioning_configuration()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aks_name = 'my-aks-test3' \n",
|
"aks_name = 'my-aks-test3' \n",
|
||||||
"# Create the cluster\n",
|
"# Create the cluster\n",
|
||||||
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
||||||
" name = aks_name, \n",
|
" name = aks_name, \n",
|
||||||
" provisioning_configuration = prov_config)"
|
" provisioning_configuration = prov_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_target.wait_for_completion(show_output = True)"
|
"aks_target.wait_for_completion(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(aks_target.provisioning_state)\n",
|
"print(aks_target.provisioning_state)\n",
|
||||||
"print(aks_target.provisioning_errors)"
|
"print(aks_target.provisioning_errors)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"If you already have a cluster you can attach the service to it:"
|
"If you already have a cluster you can attach the service to it:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"```python \n",
|
"```python \n",
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"resource_id = '/subscriptions/<subscriptionid>/resourcegroups/<resourcegroupname>/providers/Microsoft.ContainerService/managedClusters/<aksservername>'\n",
|
"resource_id = '/subscriptions/<subscriptionid>/resourcegroups/<resourcegroupname>/providers/Microsoft.ContainerService/managedClusters/<aksservername>'\n",
|
||||||
"create_name= 'myaks4'\n",
|
"create_name= 'myaks4'\n",
|
||||||
"attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
"attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
||||||
"aks_target = ComputeTarget.attach(workspace = ws, \n",
|
"aks_target = ComputeTarget.attach(workspace = ws, \n",
|
||||||
" name = create_name, \n",
|
" name = create_name, \n",
|
||||||
" attach_configuration=attach_config)\n",
|
" attach_configuration=attach_config)\n",
|
||||||
"## Wait for the operation to complete\n",
|
"## Wait for the operation to complete\n",
|
||||||
"aks_target.wait_for_provisioning(True)```"
|
"aks_target.wait_for_provisioning(True)```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### a. *Activate App Insights through updating AKS Webservice configuration*\n",
|
"### a. *Activate App Insights through updating AKS Webservice configuration*\n",
|
||||||
"In order to enable App Insights in your service you will need to update your AKS configuration file:"
|
"In order to enable App Insights in your service you will need to update your AKS configuration file:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Set the web service configuration\n",
|
"#Set the web service configuration\n",
|
||||||
"aks_config = AksWebservice.deploy_configuration(enable_app_insights=True)"
|
"aks_config = AksWebservice.deploy_configuration(enable_app_insights=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### b. Deploy your service"
|
"### b. Deploy your service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"if aks_target.provisioning_state== \"Succeeded\": \n",
|
"if aks_target.provisioning_state== \"Succeeded\": \n",
|
||||||
" aks_service_name ='aks-w-dc5'\n",
|
" aks_service_name ='aks-w-dc5'\n",
|
||||||
" aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
" aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
||||||
" name = aks_service_name,\n",
|
" name = aks_service_name,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" deployment_config = aks_config,\n",
|
" deployment_config = aks_config,\n",
|
||||||
" deployment_target = aks_target\n",
|
" deployment_target = aks_target\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
" aks_service.wait_for_deployment(show_output = True)\n",
|
" aks_service.wait_for_deployment(show_output = True)\n",
|
||||||
" print(aks_service.state)\n",
|
" print(aks_service.state)\n",
|
||||||
"else:\n",
|
"else:\n",
|
||||||
" raise ValueError(\"AKS provisioning failed. Error: \", aks_service.error)"
|
" raise ValueError(\"AKS provisioning failed. Error: \", aks_service.error)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 8. Test your service "
|
"## 8. Test your service "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,28,13,45,54,6,57,8,8,10], \n",
|
" [1,28,13,45,54,6,57,8,8,10], \n",
|
||||||
" [101,9,8,37,6,45,4,3,2,41]\n",
|
" [101,9,8,37,6,45,4,3,2,41]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding='utf8')\n",
|
"test_sample = bytes(test_sample,encoding='utf8')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"if aks_service.state == \"Healthy\":\n",
|
"if aks_service.state == \"Healthy\":\n",
|
||||||
" prediction = aks_service.run(input_data=test_sample)\n",
|
" prediction = aks_service.run(input_data=test_sample)\n",
|
||||||
" print(prediction)\n",
|
" print(prediction)\n",
|
||||||
"else:\n",
|
"else:\n",
|
||||||
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aks_service.error)"
|
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aks_service.error)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 9. See your service telemetry in App Insights\n",
|
"## 9. See your service telemetry in App Insights\n",
|
||||||
"1. Go to the [Azure Portal](https://portal.azure.com/)\n",
|
"1. Go to the [Azure Portal](https://portal.azure.com/)\n",
|
||||||
"2. All resources--> Select the subscription/resource group where you created your Workspace--> Select the App Insights type\n",
|
"2. All resources--> Select the subscription/resource group where you created your Workspace--> Select the App Insights type\n",
|
||||||
"3. Click on the AppInsights resource. You'll see a highlevel dashboard with information on Requests, Server response time and availability.\n",
|
"3. Click on the AppInsights resource. You'll see a highlevel dashboard with information on Requests, Server response time and availability.\n",
|
||||||
"4. Click on the top banner \"Analytics\"\n",
|
"4. Click on the top banner \"Analytics\"\n",
|
||||||
"5. In the \"Schema\" section select \"traces\" and run your query.\n",
|
"5. In the \"Schema\" section select \"traces\" and run your query.\n",
|
||||||
"6. Voila! All your custom traces should be there."
|
"6. Voila! All your custom traces should be there."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Disable App Insights"
|
"# Disable App Insights"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"aks_service.update(enable_app_insights=False)"
|
"aks_service.update(enable_app_insights=False)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Clean up"
|
"## Clean up"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_service.delete()\n",
|
"aks_service.delete()\n",
|
||||||
"aci_service.delete()\n",
|
"aci_service.delete()\n",
|
||||||
"image.delete()\n",
|
"image.delete()\n",
|
||||||
"model.delete()"
|
"model.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "shipatel"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "shipatel"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.3"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.3"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,478 +1,478 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Enabling Data Collection for Models in Production\n",
|
"# Enabling Data Collection for Models in Production\n",
|
||||||
"With this notebook, you can learn how to collect input model data from your Azure Machine Learning service in an Azure Blob storage. Once enabled, this data collected gives you the opportunity:\n",
|
"With this notebook, you can learn how to collect input model data from your Azure Machine Learning service in an Azure Blob storage. Once enabled, this data collected gives you the opportunity:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Monitor data drifts as production data enters your model\n",
|
"* Monitor data drifts as production data enters your model\n",
|
||||||
"* Make better decisions on when to retrain or optimize your model\n",
|
"* Make better decisions on when to retrain or optimize your model\n",
|
||||||
"* Retrain your model with the data collected\n",
|
"* Retrain your model with the data collected\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What data is collected?\n",
|
"## What data is collected?\n",
|
||||||
"* Model input data (voice, images, and video are not supported) from services deployed in Azure Kubernetes Cluster (AKS)\n",
|
"* Model input data (voice, images, and video are not supported) from services deployed in Azure Kubernetes Cluster (AKS)\n",
|
||||||
"* Model predictions using production input data.\n",
|
"* Model predictions using production input data.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Note:** pre-aggregation or pre-calculations on this data are done by user and not included in this version of the product.\n",
|
"**Note:** pre-aggregation or pre-calculations on this data are done by user and not included in this version of the product.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What is different compared to standard production deployment process?\n",
|
"## What is different compared to standard production deployment process?\n",
|
||||||
"1. Update scoring file.\n",
|
"1. Update scoring file.\n",
|
||||||
"2. Update yml file with new dependency.\n",
|
"2. Update yml file with new dependency.\n",
|
||||||
"3. Update aks configuration.\n",
|
"3. Update aks configuration.\n",
|
||||||
"4. Build new image and deploy it. "
|
"4. Build new image and deploy it. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 1. Import your dependencies"
|
"## 1. Import your dependencies"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
||||||
"from azureml.core.webservice import Webservice, AksWebservice\n",
|
"from azureml.core.webservice import Webservice, AksWebservice\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"print(azureml.core.VERSION)"
|
"print(azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 2. Set up your configuration and create a workspace"
|
"## 2. Set up your configuration and create a workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 3. Register Model\n",
|
"## 3. Register Model\n",
|
||||||
"Register an existing trained model, add descirption and tags."
|
"Register an existing trained model, add descirption and tags."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Register the model\n",
|
"#Register the model\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
||||||
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
||||||
" description = \"Ridge regression model to predict diabetes\",\n",
|
" description = \"Ridge regression model to predict diabetes\",\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(model.name, model.description, model.version)"
|
"print(model.name, model.description, model.version)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 4. *Update your scoring file with Data Collection*\n",
|
"## 4. *Update your scoring file with Data Collection*\n",
|
||||||
"The file below, compared to the file used in notebook 11, has the following changes:\n",
|
"The file below, compared to the file used in notebook 11, has the following changes:\n",
|
||||||
"### a. Import the module\n",
|
"### a. Import the module\n",
|
||||||
"```python \n",
|
"```python \n",
|
||||||
"from azureml.monitoring import ModelDataCollector```\n",
|
"from azureml.monitoring import ModelDataCollector```\n",
|
||||||
"### b. In your init function add:\n",
|
"### b. In your init function add:\n",
|
||||||
"```python \n",
|
"```python \n",
|
||||||
"global inputs_dc, prediction_d\n",
|
"global inputs_dc, prediction_d\n",
|
||||||
"inputs_dc = ModelDataCollector(\"best_model\", identifier=\"inputs\", feature_names=[\"feat1\", \"feat2\", \"feat3\", \"feat4\", \"feat5\", \"Feat6\"])\n",
|
"inputs_dc = ModelDataCollector(\"best_model\", identifier=\"inputs\", feature_names=[\"feat1\", \"feat2\", \"feat3\", \"feat4\", \"feat5\", \"Feat6\"])\n",
|
||||||
"prediction_dc = ModelDataCollector(\"best_model\", identifier=\"predictions\", feature_names=[\"prediction1\", \"prediction2\"])```\n",
|
"prediction_dc = ModelDataCollector(\"best_model\", identifier=\"predictions\", feature_names=[\"prediction1\", \"prediction2\"])```\n",
|
||||||
" \n",
|
" \n",
|
||||||
"* Identifier: Identifier is later used for building the folder structure in your Blob, it can be used to divide \"raw\" data versus \"processed\".\n",
|
"* Identifier: Identifier is later used for building the folder structure in your Blob, it can be used to divide \"raw\" data versus \"processed\".\n",
|
||||||
"* CorrelationId: is an optional parameter, you do not need to set it up if your model doesn't require it. Having a correlationId in place does help you for easier mapping with other data. (Examples include: LoanNumber, CustomerId, etc.)\n",
|
"* CorrelationId: is an optional parameter, you do not need to set it up if your model doesn't require it. Having a correlationId in place does help you for easier mapping with other data. (Examples include: LoanNumber, CustomerId, etc.)\n",
|
||||||
"* Feature Names: These need to be set up in the order of your features in order for them to have column names when the .csv is created.\n",
|
"* Feature Names: These need to be set up in the order of your features in order for them to have column names when the .csv is created.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### c. In your run function add:\n",
|
"### c. In your run function add:\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"inputs_dc.collect(data)\n",
|
"inputs_dc.collect(data)\n",
|
||||||
"prediction_dc.collect(result)```"
|
"prediction_dc.collect(result)```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import pickle\n",
|
"import pickle\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import numpy \n",
|
"import numpy \n",
|
||||||
"from sklearn.externals import joblib\n",
|
"from sklearn.externals import joblib\n",
|
||||||
"from sklearn.linear_model import Ridge\n",
|
"from sklearn.linear_model import Ridge\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"from azureml.monitoring import ModelDataCollector\n",
|
"from azureml.monitoring import ModelDataCollector\n",
|
||||||
"import time\n",
|
"import time\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global model\n",
|
" global model\n",
|
||||||
" print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))\n",
|
" print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under the workspace\n",
|
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under the workspace\n",
|
||||||
" # this call should return the path to the model.pkl file on the local disk.\n",
|
" # this call should return the path to the model.pkl file on the local disk.\n",
|
||||||
" model_path = Model.get_model_path(model_name = 'sklearn_regression_model.pkl')\n",
|
" model_path = Model.get_model_path(model_name = 'sklearn_regression_model.pkl')\n",
|
||||||
" # deserialize the model file back into a sklearn model\n",
|
" # deserialize the model file back into a sklearn model\n",
|
||||||
" model = joblib.load(model_path)\n",
|
" model = joblib.load(model_path)\n",
|
||||||
" global inputs_dc, prediction_dc\n",
|
" global inputs_dc, prediction_dc\n",
|
||||||
" # this setup will help us save our inputs under the \"inputs\" path in our Azure Blob\n",
|
" # this setup will help us save our inputs under the \"inputs\" path in our Azure Blob\n",
|
||||||
" inputs_dc = ModelDataCollector(model_name=\"sklearn_regression_model\", identifier=\"inputs\", feature_names=[\"feat1\", \"feat2\"]) \n",
|
" inputs_dc = ModelDataCollector(model_name=\"sklearn_regression_model\", identifier=\"inputs\", feature_names=[\"feat1\", \"feat2\"]) \n",
|
||||||
" # this setup will help us save our ipredictions under the \"predictions\" path in our Azure Blob\n",
|
" # this setup will help us save our ipredictions under the \"predictions\" path in our Azure Blob\n",
|
||||||
" prediction_dc = ModelDataCollector(\"sklearn_regression_model\", identifier=\"predictions\", feature_names=[\"prediction1\", \"prediction2\"]) \n",
|
" prediction_dc = ModelDataCollector(\"sklearn_regression_model\", identifier=\"predictions\", feature_names=[\"prediction1\", \"prediction2\"]) \n",
|
||||||
" \n",
|
" \n",
|
||||||
"# note you can pass in multiple rows for scoring\n",
|
"# note you can pass in multiple rows for scoring\n",
|
||||||
"def run(raw_data):\n",
|
"def run(raw_data):\n",
|
||||||
" global inputs_dc, prediction_dc\n",
|
" global inputs_dc, prediction_dc\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" data = json.loads(raw_data)['data']\n",
|
" data = json.loads(raw_data)['data']\n",
|
||||||
" data = numpy.array(data)\n",
|
" data = numpy.array(data)\n",
|
||||||
" result = model.predict(data)\n",
|
" result = model.predict(data)\n",
|
||||||
" print (\"saving input data\" + time.strftime(\"%H:%M:%S\"))\n",
|
" print (\"saving input data\" + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" inputs_dc.collect(data) #this call is saving our input data into our blob\n",
|
" inputs_dc.collect(data) #this call is saving our input data into our blob\n",
|
||||||
" prediction_dc.collect(result)#this call is saving our prediction data into our blob\n",
|
" prediction_dc.collect(result)#this call is saving our prediction data into our blob\n",
|
||||||
" print (\"saving prediction data\" + time.strftime(\"%H:%M:%S\"))\n",
|
" print (\"saving prediction data\" + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" # you can return any data type as long as it is JSON-serializable\n",
|
" # you can return any data type as long as it is JSON-serializable\n",
|
||||||
" return result.tolist()\n",
|
" return result.tolist()\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" error = str(e)\n",
|
" error = str(e)\n",
|
||||||
" print (error + time.strftime(\"%H:%M:%S\"))\n",
|
" print (error + time.strftime(\"%H:%M:%S\"))\n",
|
||||||
" return error"
|
" return error"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 5. *Update your myenv.yml file with the required module*"
|
"## 5. *Update your myenv.yml file with the required module*"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
||||||
"myenv.add_pip_package(\"azureml-monitoring\")\n",
|
"myenv.add_pip_package(\"azureml-monitoring\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 6. Create your new Image"
|
"## 6. Create your new Image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import ContainerImage\n",
|
"from azureml.core.image import ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
||||||
" runtime = \"python\",\n",
|
" runtime = \"python\",\n",
|
||||||
" conda_file = \"myenv.yml\",\n",
|
" conda_file = \"myenv.yml\",\n",
|
||||||
" description = \"Image with ridge regression model\",\n",
|
" description = \"Image with ridge regression model\",\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = ContainerImage.create(name = \"myimage1\",\n",
|
"image = ContainerImage.create(name = \"myimage1\",\n",
|
||||||
" # this is the model object\n",
|
" # this is the model object\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config,\n",
|
" image_config = image_config,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(model.name, model.description, model.version)"
|
"print(model.name, model.description, model.version)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 7. Deploy to AKS service"
|
"## 7. Deploy to AKS service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create AKS compute if you haven't done so."
|
"### Create AKS compute if you haven't done so."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Use the default configuration (can also provide parameters to customize)\n",
|
"# Use the default configuration (can also provide parameters to customize)\n",
|
||||||
"prov_config = AksCompute.provisioning_configuration()\n",
|
"prov_config = AksCompute.provisioning_configuration()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aks_name = 'my-aks-test1' \n",
|
"aks_name = 'my-aks-test1' \n",
|
||||||
"# Create the cluster\n",
|
"# Create the cluster\n",
|
||||||
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
||||||
" name = aks_name, \n",
|
" name = aks_name, \n",
|
||||||
" provisioning_configuration = prov_config)"
|
" provisioning_configuration = prov_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_target.wait_for_completion(show_output = True)\n",
|
"aks_target.wait_for_completion(show_output = True)\n",
|
||||||
"print(aks_target.provisioning_state)\n",
|
"print(aks_target.provisioning_state)\n",
|
||||||
"print(aks_target.provisioning_errors)"
|
"print(aks_target.provisioning_errors)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"If you already have a cluster you can attach the service to it:"
|
"If you already have a cluster you can attach the service to it:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"```python \n",
|
"```python \n",
|
||||||
" %%time\n",
|
" %%time\n",
|
||||||
" resource_id = '/subscriptions/<subscriptionid>/resourcegroups/<resourcegroupname>/providers/Microsoft.ContainerService/managedClusters/<aksservername>'\n",
|
" resource_id = '/subscriptions/<subscriptionid>/resourcegroups/<resourcegroupname>/providers/Microsoft.ContainerService/managedClusters/<aksservername>'\n",
|
||||||
" create_name= 'myaks4'\n",
|
" create_name= 'myaks4'\n",
|
||||||
" attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
" attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
||||||
" aks_target = ComputeTarget.attach(workspace = ws, \n",
|
" aks_target = ComputeTarget.attach(workspace = ws, \n",
|
||||||
" name = create_name, \n",
|
" name = create_name, \n",
|
||||||
" attach_configuration=attach_config)\n",
|
" attach_configuration=attach_config)\n",
|
||||||
" ## Wait for the operation to complete\n",
|
" ## Wait for the operation to complete\n",
|
||||||
" aks_target.wait_for_provisioning(True)```"
|
" aks_target.wait_for_provisioning(True)```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### a. *Activate Data Collection and App Insights through updating AKS Webservice configuration*\n",
|
"### a. *Activate Data Collection and App Insights through updating AKS Webservice configuration*\n",
|
||||||
"In order to enable Data Collection and App Insights in your service you will need to update your AKS configuration file:"
|
"In order to enable Data Collection and App Insights in your service you will need to update your AKS configuration file:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Set the web service configuration\n",
|
"#Set the web service configuration\n",
|
||||||
"aks_config = AksWebservice.deploy_configuration(collect_model_data=True, enable_app_insights=True)"
|
"aks_config = AksWebservice.deploy_configuration(collect_model_data=True, enable_app_insights=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### b. Deploy your service"
|
"### b. Deploy your service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"if aks_target.provisioning_state== \"Succeeded\": \n",
|
"if aks_target.provisioning_state== \"Succeeded\": \n",
|
||||||
" aks_service_name ='aks-w-dc0'\n",
|
" aks_service_name ='aks-w-dc0'\n",
|
||||||
" aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
" aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
||||||
" name = aks_service_name,\n",
|
" name = aks_service_name,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" deployment_config = aks_config,\n",
|
" deployment_config = aks_config,\n",
|
||||||
" deployment_target = aks_target\n",
|
" deployment_target = aks_target\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
" aks_service.wait_for_deployment(show_output = True)\n",
|
" aks_service.wait_for_deployment(show_output = True)\n",
|
||||||
" print(aks_service.state)\n",
|
" print(aks_service.state)\n",
|
||||||
"else: \n",
|
"else: \n",
|
||||||
" raise ValueError(\"aks provisioning failed, can't deploy service. Error: \", aks_service.error)"
|
" raise ValueError(\"aks provisioning failed, can't deploy service. Error: \", aks_service.error)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 8. Test your service and send some data\n",
|
"## 8. Test your service and send some data\n",
|
||||||
"**Note**: It will take around 15 mins for your data to appear in your blob.\n",
|
"**Note**: It will take around 15 mins for your data to appear in your blob.\n",
|
||||||
"The data will appear in your Azure Blob following this format:\n",
|
"The data will appear in your Azure Blob following this format:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"/modeldata/subscriptionid/resourcegroupname/workspacename/webservicename/modelname/modelversion/identifier/year/month/day/data.csv "
|
"/modeldata/subscriptionid/resourcegroupname/workspacename/webservicename/modelname/modelversion/identifier/year/month/day/data.csv "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,2,3,4,54,6,7,8,88,10], \n",
|
" [1,2,3,4,54,6,7,8,88,10], \n",
|
||||||
" [10,9,8,37,36,45,4,33,2,1]\n",
|
" [10,9,8,37,36,45,4,33,2,1]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"if aks_service.state == \"Healthy\":\n",
|
"if aks_service.state == \"Healthy\":\n",
|
||||||
" prediction = aks_service.run(input_data=test_sample)\n",
|
" prediction = aks_service.run(input_data=test_sample)\n",
|
||||||
" print(prediction)\n",
|
" print(prediction)\n",
|
||||||
"else:\n",
|
"else:\n",
|
||||||
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aks_service.error)"
|
" raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aks_service.error)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## 9. Validate you data and analyze it\n",
|
"## 9. Validate you data and analyze it\n",
|
||||||
"You can look into your data following this path format in your Azure Blob (it takes up to 15 minutes for the data to appear):\n",
|
"You can look into your data following this path format in your Azure Blob (it takes up to 15 minutes for the data to appear):\n",
|
||||||
"\n",
|
"\n",
|
||||||
"/modeldata/**subscriptionid>**/**resourcegroupname>**/**workspacename>**/**webservicename>**/**modelname>**/**modelversion>>**/**identifier>**/*year/month/day*/data.csv \n",
|
"/modeldata/**subscriptionid>**/**resourcegroupname>**/**workspacename>**/**webservicename>**/**modelname>**/**modelversion>>**/**identifier>**/*year/month/day*/data.csv \n",
|
||||||
"\n",
|
"\n",
|
||||||
"For doing further analysis you have multiple options:"
|
"For doing further analysis you have multiple options:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### a. Create DataBricks cluter and connect it to your blob\n",
|
"### a. Create DataBricks cluter and connect it to your blob\n",
|
||||||
"https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal or in your databricks workspace you can look for the template \"Azure Blob Storage Import Example Notebook\".\n",
|
"https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal or in your databricks workspace you can look for the template \"Azure Blob Storage Import Example Notebook\".\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Here is an example for setting up the file location to extract the relevant data:\n",
|
"Here is an example for setting up the file location to extract the relevant data:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"<code> file_location = \"wasbs://mycontainer@storageaccountname.blob.core.windows.net/unknown/unknown/unknown-bigdataset-unknown/my_iterate_parking_inputs/2018/°/°/data.csv\" \n",
|
"<code> file_location = \"wasbs://mycontainer@storageaccountname.blob.core.windows.net/unknown/unknown/unknown-bigdataset-unknown/my_iterate_parking_inputs/2018/°/°/data.csv\" \n",
|
||||||
"file_type = \"csv\"</code>\n"
|
"file_type = \"csv\"</code>\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### b. Connect Blob to Power Bi (Small Data only)\n",
|
"### b. Connect Blob to Power Bi (Small Data only)\n",
|
||||||
"1. Download and Open PowerBi Desktop\n",
|
"1. Download and Open PowerBi Desktop\n",
|
||||||
"2. Select \"Get Data\" and click on \"Azure Blob Storage\" >> Connect\n",
|
"2. Select \"Get Data\" and click on \"Azure Blob Storage\" >> Connect\n",
|
||||||
"3. Add your storage account and enter your storage key.\n",
|
"3. Add your storage account and enter your storage key.\n",
|
||||||
"4. Select the container where your Data Collection is stored and click on Edit. \n",
|
"4. Select the container where your Data Collection is stored and click on Edit. \n",
|
||||||
"5. In the query editor, click under \"Name\" column and add your Storage account Model path into the filter. Note: if you want to only look into files from a specific year or month, just expand the filter path. For example, just look into March data: /modeldata/subscriptionid>/resourcegroupname>/workspacename>/webservicename>/modelname>/modelversion>/identifier>/year>/3\n",
|
"5. In the query editor, click under \"Name\" column and add your Storage account Model path into the filter. Note: if you want to only look into files from a specific year or month, just expand the filter path. For example, just look into March data: /modeldata/subscriptionid>/resourcegroupname>/workspacename>/webservicename>/modelname>/modelversion>/identifier>/year>/3\n",
|
||||||
"6. Click on the double arrow aside the \"Content\" column to combine the files. \n",
|
"6. Click on the double arrow aside the \"Content\" column to combine the files. \n",
|
||||||
"7. Click OK and the data will preload.\n",
|
"7. Click OK and the data will preload.\n",
|
||||||
"8. You can now click Close and Apply and start building your custom reports on your Model Input data."
|
"8. You can now click Close and Apply and start building your custom reports on your Model Input data."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Disable Data Collection"
|
"# Disable Data Collection"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"aks_service.update(collect_model_data=False)"
|
"aks_service.update(collect_model_data=False)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Clean up"
|
"## Clean up"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_service.delete()\n",
|
"aks_service.delete()\n",
|
||||||
"image.delete()\n",
|
"image.delete()\n",
|
||||||
"model.delete()"
|
"model.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "shipatel"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "shipatel"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.3"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.3"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,443 +1,443 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# YOLO Real-time Object Detection using ONNX on AzureML\n",
|
"# YOLO Real-time Object Detection using ONNX on AzureML\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This example shows how to convert the TinyYOLO model from CoreML to ONNX and operationalize it as a web service using Azure Machine Learning services and the ONNX Runtime.\n",
|
"This example shows how to convert the TinyYOLO model from CoreML to ONNX and operationalize it as a web service using Azure Machine Learning services and the ONNX Runtime.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What is ONNX\n",
|
"## What is ONNX\n",
|
||||||
"ONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai).\n",
|
"ONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## YOLO Details\n",
|
"## YOLO Details\n",
|
||||||
"You Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. For more information about YOLO, please visit the [YOLO website](https://pjreddie.com/darknet/yolo/)."
|
"You Only Look Once (YOLO) is a state-of-the-art, real-time object detection system. For more information about YOLO, please visit the [YOLO website](https://pjreddie.com/darknet/yolo/)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To make the best use of your time, make sure you have done the following:\n",
|
"To make the best use of your time, make sure you have done the following:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (config.json)"
|
" * create a workspace and its configuration file (config.json)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Install necessary packages\n",
|
"#### Install necessary packages\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You'll need to run the following commands to use this tutorial:\n",
|
"You'll need to run the following commands to use this tutorial:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"```sh\n",
|
"```sh\n",
|
||||||
"pip install onnxmltools\n",
|
"pip install onnxmltools\n",
|
||||||
"pip install coremltools # use this on Linux and Mac\n",
|
"pip install coremltools # use this on Linux and Mac\n",
|
||||||
"pip install git+https://github.com/apple/coremltools # use this on Windows\n",
|
"pip install git+https://github.com/apple/coremltools # use this on Windows\n",
|
||||||
"```"
|
"```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Convert model to ONNX\n",
|
"## Convert model to ONNX\n",
|
||||||
"\n",
|
"\n",
|
||||||
"First we download the CoreML model. We use the CoreML model from [Matthijs Hollemans's tutorial](https://github.com/hollance/YOLO-CoreML-MPSNNGraph). This may take a few minutes."
|
"First we download the CoreML model. We use the CoreML model from [Matthijs Hollemans's tutorial](https://github.com/hollance/YOLO-CoreML-MPSNNGraph). This may take a few minutes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import urllib.request\n",
|
"import urllib.request\n",
|
||||||
"\n",
|
"\n",
|
||||||
"coreml_model_url = \"https://github.com/hollance/YOLO-CoreML-MPSNNGraph/raw/master/TinyYOLO-CoreML/TinyYOLO-CoreML/TinyYOLO.mlmodel\"\n",
|
"coreml_model_url = \"https://github.com/hollance/YOLO-CoreML-MPSNNGraph/raw/master/TinyYOLO-CoreML/TinyYOLO-CoreML/TinyYOLO.mlmodel\"\n",
|
||||||
"urllib.request.urlretrieve(coreml_model_url, filename=\"TinyYOLO.mlmodel\")\n"
|
"urllib.request.urlretrieve(coreml_model_url, filename=\"TinyYOLO.mlmodel\")\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Then we use ONNXMLTools to convert the model."
|
"Then we use ONNXMLTools to convert the model."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import onnxmltools\n",
|
"import onnxmltools\n",
|
||||||
"import coremltools\n",
|
"import coremltools\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Load a CoreML model\n",
|
"# Load a CoreML model\n",
|
||||||
"coreml_model = coremltools.utils.load_spec('TinyYOLO.mlmodel')\n",
|
"coreml_model = coremltools.utils.load_spec('TinyYOLO.mlmodel')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Convert from CoreML into ONNX\n",
|
"# Convert from CoreML into ONNX\n",
|
||||||
"onnx_model = onnxmltools.convert_coreml(coreml_model, 'TinyYOLOv2')\n",
|
"onnx_model = onnxmltools.convert_coreml(coreml_model, 'TinyYOLOv2')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Save ONNX model\n",
|
"# Save ONNX model\n",
|
||||||
"onnxmltools.utils.save_model(onnx_model, 'tinyyolov2.onnx')\n",
|
"onnxmltools.utils.save_model(onnx_model, 'tinyyolov2.onnx')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"print(os.path.getsize('tinyyolov2.onnx'))"
|
"print(os.path.getsize('tinyyolov2.onnx'))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Deploying as a web service with Azure ML\n",
|
"## Deploying as a web service with Azure ML\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Load Azure ML workspace\n",
|
"### Load Azure ML workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
|
"We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
|
"print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Registering your model with Azure ML\n",
|
"### Registering your model with Azure ML\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Now we upload the model and register it in the workspace."
|
"Now we upload the model and register it in the workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"\n",
|
"\n",
|
||||||
"model = Model.register(model_path = \"tinyyolov2.onnx\",\n",
|
"model = Model.register(model_path = \"tinyyolov2.onnx\",\n",
|
||||||
" model_name = \"tinyyolov2\",\n",
|
" model_name = \"tinyyolov2\",\n",
|
||||||
" tags = {\"onnx\": \"demo\"},\n",
|
" tags = {\"onnx\": \"demo\"},\n",
|
||||||
" description = \"TinyYOLO\",\n",
|
" description = \"TinyYOLO\",\n",
|
||||||
" workspace = ws)"
|
" workspace = ws)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Displaying your registered models\n",
|
"#### Displaying your registered models\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can optionally list out all the models that you have registered in this workspace."
|
"You can optionally list out all the models that you have registered in this workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"models = ws.models\n",
|
"models = ws.models\n",
|
||||||
"for name, m in models.items():\n",
|
"for name, m in models.items():\n",
|
||||||
" print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
" print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Write scoring file\n",
|
"### Write scoring file\n",
|
||||||
"\n",
|
"\n",
|
||||||
"We are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object."
|
"We are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import time\n",
|
"import time\n",
|
||||||
"import sys\n",
|
"import sys\n",
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"import numpy as np # we're going to use numpy to process input and output data\n",
|
"import numpy as np # we're going to use numpy to process input and output data\n",
|
||||||
"import onnxruntime # to inference ONNX models, we use the ONNX Runtime\n",
|
"import onnxruntime # to inference ONNX models, we use the ONNX Runtime\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global session\n",
|
" global session\n",
|
||||||
" model = Model.get_model_path(model_name = 'tinyyolov2')\n",
|
" model = Model.get_model_path(model_name = 'tinyyolov2')\n",
|
||||||
" session = onnxruntime.InferenceSession(model)\n",
|
" session = onnxruntime.InferenceSession(model)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def preprocess(input_data_json):\n",
|
"def preprocess(input_data_json):\n",
|
||||||
" # convert the JSON data into the tensor input\n",
|
" # convert the JSON data into the tensor input\n",
|
||||||
" return np.array(json.loads(input_data_json)['data']).astype('float32')\n",
|
" return np.array(json.loads(input_data_json)['data']).astype('float32')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def postprocess(result):\n",
|
"def postprocess(result):\n",
|
||||||
" return np.array(result).tolist()\n",
|
" return np.array(result).tolist()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def run(input_data_json):\n",
|
"def run(input_data_json):\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" start = time.time() # start timer\n",
|
" start = time.time() # start timer\n",
|
||||||
" input_data = preprocess(input_data_json)\n",
|
" input_data = preprocess(input_data_json)\n",
|
||||||
" input_name = session.get_inputs()[0].name # get the id of the first input of the model \n",
|
" input_name = session.get_inputs()[0].name # get the id of the first input of the model \n",
|
||||||
" result = session.run([], {input_name: input_data})\n",
|
" result = session.run([], {input_name: input_data})\n",
|
||||||
" end = time.time() # stop timer\n",
|
" end = time.time() # stop timer\n",
|
||||||
" return {\"result\": postprocess(result),\n",
|
" return {\"result\": postprocess(result),\n",
|
||||||
" \"time\": end - start}\n",
|
" \"time\": end - start}\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" result = str(e)\n",
|
" result = str(e)\n",
|
||||||
" return {\"error\": result}"
|
" return {\"error\": result}"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create container image\n",
|
"### Create container image\n",
|
||||||
"First we create a YAML file that specifies which dependencies we would like to see in our container."
|
"First we create a YAML file that specifies which dependencies we would like to see in our container."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime==0.4.0\",\"azureml-core\"])\n",
|
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime==0.4.0\",\"azureml-core\"])\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Then we have Azure ML create the container. This step will likely take a few minutes."
|
"Then we have Azure ML create the container. This step will likely take a few minutes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import ContainerImage\n",
|
"from azureml.core.image import ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
||||||
" runtime = \"python\",\n",
|
" runtime = \"python\",\n",
|
||||||
" conda_file = \"myenv.yml\",\n",
|
" conda_file = \"myenv.yml\",\n",
|
||||||
" docker_file = \"Dockerfile\",\n",
|
" docker_file = \"Dockerfile\",\n",
|
||||||
" description = \"TinyYOLO ONNX Demo\",\n",
|
" description = \"TinyYOLO ONNX Demo\",\n",
|
||||||
" tags = {\"demo\": \"onnx\"}\n",
|
" tags = {\"demo\": \"onnx\"}\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = ContainerImage.create(name = \"onnxyolo\",\n",
|
"image = ContainerImage.create(name = \"onnxyolo\",\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config,\n",
|
" image_config = image_config,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"In case you need to debug your code, the next line of code accesses the log file."
|
"In case you need to debug your code, the next line of code accesses the log file."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(image.image_build_log_uri)"
|
"print(image.image_build_log_uri)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"We're all set! Let's get our model chugging.\n",
|
"We're all set! Let's get our model chugging.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Deploy the container image"
|
"### Deploy the container image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import AciWebservice\n",
|
"from azureml.core.webservice import AciWebservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
||||||
" memory_gb = 1, \n",
|
" memory_gb = 1, \n",
|
||||||
" tags = {'demo': 'onnx'}, \n",
|
" tags = {'demo': 'onnx'}, \n",
|
||||||
" description = 'web service for TinyYOLO ONNX model')"
|
" description = 'web service for TinyYOLO ONNX model')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The following cell will likely take a few minutes to run as well."
|
"The following cell will likely take a few minutes to run as well."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import Webservice\n",
|
"from azureml.core.webservice import Webservice\n",
|
||||||
"from random import randint\n",
|
"from random import randint\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service_name = 'onnx-tinyyolo'+str(randint(0,100))\n",
|
"aci_service_name = 'onnx-tinyyolo'+str(randint(0,100))\n",
|
||||||
"print(\"Service\", aci_service_name)\n",
|
"print(\"Service\", aci_service_name)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" name = aci_service_name,\n",
|
" name = aci_service_name,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service.wait_for_deployment(True)\n",
|
"aci_service.wait_for_deployment(True)\n",
|
||||||
"print(aci_service.state)"
|
"print(aci_service.state)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"In case the deployment fails, you can check the logs. Make sure to delete your aci_service before trying again."
|
"In case the deployment fails, you can check the logs. Make sure to delete your aci_service before trying again."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"if aci_service.state != 'Healthy':\n",
|
"if aci_service.state != 'Healthy':\n",
|
||||||
" # run this command for debugging.\n",
|
" # run this command for debugging.\n",
|
||||||
" print(aci_service.get_logs())\n",
|
" print(aci_service.get_logs())\n",
|
||||||
" aci_service.delete()"
|
" aci_service.delete()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Success!\n",
|
"## Success!\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you've made it this far, you've deployed a working web service that does object detection using an ONNX model. You can get the URL for the webservice with the code below."
|
"If you've made it this far, you've deployed a working web service that does object detection using an ONNX model. You can get the URL for the webservice with the code below."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(aci_service.scoring_uri)"
|
"print(aci_service.scoring_uri)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"When you are eventually done using the web service, remember to delete it."
|
"When you are eventually done using the web service, remember to delete it."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#aci_service.delete()"
|
"#aci_service.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "viswamy"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "viswamy"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.5"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,427 +1,427 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# ResNet50 Image Classification using ONNX and AzureML\n",
|
"# ResNet50 Image Classification using ONNX and AzureML\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This example shows how to deploy the ResNet50 ONNX model as a web service using Azure Machine Learning services and the ONNX Runtime.\n",
|
"This example shows how to deploy the ResNet50 ONNX model as a web service using Azure Machine Learning services and the ONNX Runtime.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## What is ONNX\n",
|
"## What is ONNX\n",
|
||||||
"ONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai).\n",
|
"ONNX is an open format for representing machine learning and deep learning models. ONNX enables open and interoperable AI by enabling data scientists and developers to use the tools of their choice without worrying about lock-in and flexibility to deploy to a variety of platforms. ONNX is developed and supported by a community of partners including Microsoft, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## ResNet50 Details\n",
|
"## ResNet50 Details\n",
|
||||||
"ResNet classifies the major object in an input image into a set of 1000 pre-defined classes. For more information about the ResNet50 model and how it was created can be found on the [ONNX Model Zoo github](https://github.com/onnx/models/tree/master/models/image_classification/resnet). "
|
"ResNet classifies the major object in an input image into a set of 1000 pre-defined classes. For more information about the ResNet50 model and how it was created can be found on the [ONNX Model Zoo github](https://github.com/onnx/models/tree/master/models/image_classification/resnet). "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To make the best use of your time, make sure you have done the following:\n",
|
"To make the best use of your time, make sure you have done the following:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (config.json)"
|
" * create a workspace and its configuration file (config.json)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Download pre-trained ONNX model from ONNX Model Zoo.\n",
|
"#### Download pre-trained ONNX model from ONNX Model Zoo.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Download the [ResNet50v2 model and test data](https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet50v2/resnet50v2.tar.gz) and extract it in the same folder as this tutorial notebook.\n"
|
"Download the [ResNet50v2 model and test data](https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet50v2/resnet50v2.tar.gz) and extract it in the same folder as this tutorial notebook.\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import urllib.request\n",
|
"import urllib.request\n",
|
||||||
"\n",
|
"\n",
|
||||||
"onnx_model_url = \"https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet50v2/resnet50v2.tar.gz\"\n",
|
"onnx_model_url = \"https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet50v2/resnet50v2.tar.gz\"\n",
|
||||||
"urllib.request.urlretrieve(onnx_model_url, filename=\"resnet50v2.tar.gz\")\n",
|
"urllib.request.urlretrieve(onnx_model_url, filename=\"resnet50v2.tar.gz\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"!tar xvzf resnet50v2.tar.gz"
|
"!tar xvzf resnet50v2.tar.gz"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Deploying as a web service with Azure ML"
|
"## Deploying as a web service with Azure ML"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Load your Azure ML workspace\n",
|
"### Load your Azure ML workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
|
"We begin by instantiating a workspace object from the existing workspace created earlier in the configuration notebook."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
|
"print(ws.name, ws.location, ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Register your model with Azure ML\n",
|
"### Register your model with Azure ML\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Now we upload the model and register it in the workspace."
|
"Now we upload the model and register it in the workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"\n",
|
"\n",
|
||||||
"model = Model.register(model_path = \"resnet50v2/resnet50v2.onnx\",\n",
|
"model = Model.register(model_path = \"resnet50v2/resnet50v2.onnx\",\n",
|
||||||
" model_name = \"resnet50v2\",\n",
|
" model_name = \"resnet50v2\",\n",
|
||||||
" tags = {\"onnx\": \"demo\"},\n",
|
" tags = {\"onnx\": \"demo\"},\n",
|
||||||
" description = \"ResNet50v2 from ONNX Model Zoo\",\n",
|
" description = \"ResNet50v2 from ONNX Model Zoo\",\n",
|
||||||
" workspace = ws)"
|
" workspace = ws)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Displaying your registered models\n",
|
"#### Displaying your registered models\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can optionally list out all the models that you have registered in this workspace."
|
"You can optionally list out all the models that you have registered in this workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"models = ws.models\n",
|
"models = ws.models\n",
|
||||||
"for name, m in models.items():\n",
|
"for name, m in models.items():\n",
|
||||||
" print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
" print(\"Name:\", name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Write scoring file\n",
|
"### Write scoring file\n",
|
||||||
"\n",
|
"\n",
|
||||||
"We are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object."
|
"We are now going to deploy our ONNX model on Azure ML using the ONNX Runtime. We begin by writing a score.py file that will be invoked by the web service call. The `init()` function is called once when the container is started so we load the model using the ONNX Runtime into a global session object."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import time\n",
|
"import time\n",
|
||||||
"import sys\n",
|
"import sys\n",
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"import numpy as np # we're going to use numpy to process input and output data\n",
|
"import numpy as np # we're going to use numpy to process input and output data\n",
|
||||||
"import onnxruntime # to inference ONNX models, we use the ONNX Runtime\n",
|
"import onnxruntime # to inference ONNX models, we use the ONNX Runtime\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def softmax(x):\n",
|
"def softmax(x):\n",
|
||||||
" x = x.reshape(-1)\n",
|
" x = x.reshape(-1)\n",
|
||||||
" e_x = np.exp(x - np.max(x))\n",
|
" e_x = np.exp(x - np.max(x))\n",
|
||||||
" return e_x / e_x.sum(axis=0)\n",
|
" return e_x / e_x.sum(axis=0)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global session\n",
|
" global session\n",
|
||||||
" model = Model.get_model_path(model_name = 'resnet50v2')\n",
|
" model = Model.get_model_path(model_name = 'resnet50v2')\n",
|
||||||
" session = onnxruntime.InferenceSession(model, None)\n",
|
" session = onnxruntime.InferenceSession(model, None)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def preprocess(input_data_json):\n",
|
"def preprocess(input_data_json):\n",
|
||||||
" # convert the JSON data into the tensor input\n",
|
" # convert the JSON data into the tensor input\n",
|
||||||
" img_data = np.array(json.loads(input_data_json)['data']).astype('float32')\n",
|
" img_data = np.array(json.loads(input_data_json)['data']).astype('float32')\n",
|
||||||
" \n",
|
" \n",
|
||||||
" #normalize\n",
|
" #normalize\n",
|
||||||
" mean_vec = np.array([0.485, 0.456, 0.406])\n",
|
" mean_vec = np.array([0.485, 0.456, 0.406])\n",
|
||||||
" stddev_vec = np.array([0.229, 0.224, 0.225])\n",
|
" stddev_vec = np.array([0.229, 0.224, 0.225])\n",
|
||||||
" norm_img_data = np.zeros(img_data.shape).astype('float32')\n",
|
" norm_img_data = np.zeros(img_data.shape).astype('float32')\n",
|
||||||
" for i in range(img_data.shape[0]):\n",
|
" for i in range(img_data.shape[0]):\n",
|
||||||
" norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]\n",
|
" norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]\n",
|
||||||
"\n",
|
"\n",
|
||||||
" return norm_img_data\n",
|
" return norm_img_data\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def postprocess(result):\n",
|
"def postprocess(result):\n",
|
||||||
" return softmax(np.array(result)).tolist()\n",
|
" return softmax(np.array(result)).tolist()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def run(input_data_json):\n",
|
"def run(input_data_json):\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" start = time.time()\n",
|
" start = time.time()\n",
|
||||||
" # load in our data which is expected as NCHW 224x224 image\n",
|
" # load in our data which is expected as NCHW 224x224 image\n",
|
||||||
" input_data = preprocess(input_data_json)\n",
|
" input_data = preprocess(input_data_json)\n",
|
||||||
" input_name = session.get_inputs()[0].name # get the id of the first input of the model \n",
|
" input_name = session.get_inputs()[0].name # get the id of the first input of the model \n",
|
||||||
" result = session.run([], {input_name: input_data})\n",
|
" result = session.run([], {input_name: input_data})\n",
|
||||||
" end = time.time() # stop timer\n",
|
" end = time.time() # stop timer\n",
|
||||||
" return {\"result\": postprocess(result),\n",
|
" return {\"result\": postprocess(result),\n",
|
||||||
" \"time\": end - start}\n",
|
" \"time\": end - start}\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" result = str(e)\n",
|
" result = str(e)\n",
|
||||||
" return {\"error\": result}"
|
" return {\"error\": result}"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create container image"
|
"### Create container image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"First we create a YAML file that specifies which dependencies we would like to see in our container."
|
"First we create a YAML file that specifies which dependencies we would like to see in our container."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime\",\"azureml-core\"])\n",
|
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime\",\"azureml-core\"])\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Then we have Azure ML create the container. This step will likely take a few minutes."
|
"Then we have Azure ML create the container. This step will likely take a few minutes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import ContainerImage\n",
|
"from azureml.core.image import ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
||||||
" runtime = \"python\",\n",
|
" runtime = \"python\",\n",
|
||||||
" conda_file = \"myenv.yml\",\n",
|
" conda_file = \"myenv.yml\",\n",
|
||||||
" docker_file = \"Dockerfile\",\n",
|
" docker_file = \"Dockerfile\",\n",
|
||||||
" description = \"ONNX ResNet50 Demo\",\n",
|
" description = \"ONNX ResNet50 Demo\",\n",
|
||||||
" tags = {\"demo\": \"onnx\"}\n",
|
" tags = {\"demo\": \"onnx\"}\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = ContainerImage.create(name = \"onnxresnet50v2\",\n",
|
"image = ContainerImage.create(name = \"onnxresnet50v2\",\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config,\n",
|
" image_config = image_config,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"In case you need to debug your code, the next line of code accesses the log file."
|
"In case you need to debug your code, the next line of code accesses the log file."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(image.image_build_log_uri)"
|
"print(image.image_build_log_uri)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"We're all set! Let's get our model chugging.\n",
|
"We're all set! Let's get our model chugging.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Deploy the container image"
|
"### Deploy the container image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import AciWebservice\n",
|
"from azureml.core.webservice import AciWebservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
||||||
" memory_gb = 1, \n",
|
" memory_gb = 1, \n",
|
||||||
" tags = {'demo': 'onnx'}, \n",
|
" tags = {'demo': 'onnx'}, \n",
|
||||||
" description = 'web service for ResNet50 ONNX model')"
|
" description = 'web service for ResNet50 ONNX model')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The following cell will likely take a few minutes to run as well."
|
"The following cell will likely take a few minutes to run as well."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import Webservice\n",
|
"from azureml.core.webservice import Webservice\n",
|
||||||
"from random import randint\n",
|
"from random import randint\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service_name = 'onnx-demo-resnet50'+str(randint(0,100))\n",
|
"aci_service_name = 'onnx-demo-resnet50'+str(randint(0,100))\n",
|
||||||
"print(\"Service\", aci_service_name)\n",
|
"print(\"Service\", aci_service_name)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" name = aci_service_name,\n",
|
" name = aci_service_name,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service.wait_for_deployment(True)\n",
|
"aci_service.wait_for_deployment(True)\n",
|
||||||
"print(aci_service.state)"
|
"print(aci_service.state)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"In case the deployment fails, you can check the logs. Make sure to delete your aci_service before trying again."
|
"In case the deployment fails, you can check the logs. Make sure to delete your aci_service before trying again."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"if aci_service.state != 'Healthy':\n",
|
"if aci_service.state != 'Healthy':\n",
|
||||||
" # run this command for debugging.\n",
|
" # run this command for debugging.\n",
|
||||||
" print(aci_service.get_logs())\n",
|
" print(aci_service.get_logs())\n",
|
||||||
" aci_service.delete()"
|
" aci_service.delete()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Success!\n",
|
"## Success!\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you've made it this far, you've deployed a working web service that does image classification using an ONNX model. You can get the URL for the webservice with the code below."
|
"If you've made it this far, you've deployed a working web service that does image classification using an ONNX model. You can get the URL for the webservice with the code below."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(aci_service.scoring_uri)"
|
"print(aci_service.scoring_uri)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"When you are eventually done using the web service, remember to delete it."
|
"When you are eventually done using the web service, remember to delete it."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#aci_service.delete()"
|
"#aci_service.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "viswamy"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "viswamy"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.5"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because one or more lines are too long
@@ -1,477 +1,477 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Deploying a web service to Azure Kubernetes Service (AKS)\n",
|
"# Deploying a web service to Azure Kubernetes Service (AKS)\n",
|
||||||
"This notebook shows the steps for deploying a service: registering a model, creating an image, provisioning a cluster (one time action), and deploying a service to it. \n",
|
"This notebook shows the steps for deploying a service: registering a model, creating an image, provisioning a cluster (one time action), and deploying a service to it. \n",
|
||||||
"We then test and delete the service, image and model."
|
"We then test and delete the service, image and model."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
"from azureml.core.compute import AksCompute, ComputeTarget\n",
|
||||||
"from azureml.core.webservice import Webservice, AksWebservice\n",
|
"from azureml.core.webservice import Webservice, AksWebservice\n",
|
||||||
"from azureml.core.image import Image\n",
|
"from azureml.core.image import Image\n",
|
||||||
"from azureml.core.model import Model"
|
"from azureml.core.model import Model"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"print(azureml.core.VERSION)"
|
"print(azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Get workspace\n",
|
"# Get workspace\n",
|
||||||
"Load existing workspace from the config file info."
|
"Load existing workspace from the config file info."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Register the model\n",
|
"# Register the model\n",
|
||||||
"Register an existing trained model, add descirption and tags."
|
"Register an existing trained model, add descirption and tags."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Register the model\n",
|
"#Register the model\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
"model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
|
||||||
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
" model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
||||||
" description = \"Ridge regression model to predict diabetes\",\n",
|
" description = \"Ridge regression model to predict diabetes\",\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(model.name, model.description, model.version)"
|
"print(model.name, model.description, model.version)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Create an image\n",
|
"# Create an image\n",
|
||||||
"Create an image using the registered model the script that will load and run the model."
|
"Create an image using the registered model the script that will load and run the model."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import pickle\n",
|
"import pickle\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import numpy\n",
|
"import numpy\n",
|
||||||
"from sklearn.externals import joblib\n",
|
"from sklearn.externals import joblib\n",
|
||||||
"from sklearn.linear_model import Ridge\n",
|
"from sklearn.linear_model import Ridge\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global model\n",
|
" global model\n",
|
||||||
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under\n",
|
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under\n",
|
||||||
" # this is a different behavior than before when the code is run locally, even though the code is the same.\n",
|
" # this is a different behavior than before when the code is run locally, even though the code is the same.\n",
|
||||||
" model_path = Model.get_model_path('sklearn_regression_model.pkl')\n",
|
" model_path = Model.get_model_path('sklearn_regression_model.pkl')\n",
|
||||||
" # deserialize the model file back into a sklearn model\n",
|
" # deserialize the model file back into a sklearn model\n",
|
||||||
" model = joblib.load(model_path)\n",
|
" model = joblib.load(model_path)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# note you can pass in multiple rows for scoring\n",
|
"# note you can pass in multiple rows for scoring\n",
|
||||||
"def run(raw_data):\n",
|
"def run(raw_data):\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" data = json.loads(raw_data)['data']\n",
|
" data = json.loads(raw_data)['data']\n",
|
||||||
" data = numpy.array(data)\n",
|
" data = numpy.array(data)\n",
|
||||||
" result = model.predict(data)\n",
|
" result = model.predict(data)\n",
|
||||||
" # you can return any data type as long as it is JSON-serializable\n",
|
" # you can return any data type as long as it is JSON-serializable\n",
|
||||||
" return result.tolist()\n",
|
" return result.tolist()\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" error = str(e)\n",
|
" error = str(e)\n",
|
||||||
" return error"
|
" return error"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import ContainerImage\n",
|
"from azureml.core.image import ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
"image_config = ContainerImage.image_configuration(execution_script = \"score.py\",\n",
|
||||||
" runtime = \"python\",\n",
|
" runtime = \"python\",\n",
|
||||||
" conda_file = \"myenv.yml\",\n",
|
" conda_file = \"myenv.yml\",\n",
|
||||||
" description = \"Image with ridge regression model\",\n",
|
" description = \"Image with ridge regression model\",\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"}\n",
|
||||||
" )\n",
|
" )\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = ContainerImage.create(name = \"myimage1\",\n",
|
"image = ContainerImage.create(name = \"myimage1\",\n",
|
||||||
" # this is the model object\n",
|
" # this is the model object\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config,\n",
|
" image_config = image_config,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Use a custom Docker image\n",
|
"#### Use a custom Docker image\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can also specify a custom Docker image to be used as base image if you don't want to use the default base image provided by Azure ML. Please make sure the custom Docker image has Ubuntu >= 16.04, Conda >= 4.5.\\* and Python(3.5.\\* or 3.6.\\*).\n",
|
"You can also specify a custom Docker image to be used as base image if you don't want to use the default base image provided by Azure ML. Please make sure the custom Docker image has Ubuntu >= 16.04, Conda >= 4.5.\\* and Python(3.5.\\* or 3.6.\\*).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Only Supported for `ContainerImage`(from azureml.core.image) with `python` runtime.\n",
|
"Only Supported for `ContainerImage`(from azureml.core.image) with `python` runtime.\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"# use an image available in public Container Registry without authentication\n",
|
"# use an image available in public Container Registry without authentication\n",
|
||||||
"image_config.base_image = \"mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda\"\n",
|
"image_config.base_image = \"mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# or, use an image available in a private Container Registry\n",
|
"# or, use an image available in a private Container Registry\n",
|
||||||
"image_config.base_image = \"myregistry.azurecr.io/mycustomimage:1.0\"\n",
|
"image_config.base_image = \"myregistry.azurecr.io/mycustomimage:1.0\"\n",
|
||||||
"image_config.base_image_registry.address = \"myregistry.azurecr.io\"\n",
|
"image_config.base_image_registry.address = \"myregistry.azurecr.io\"\n",
|
||||||
"image_config.base_image_registry.username = \"username\"\n",
|
"image_config.base_image_registry.username = \"username\"\n",
|
||||||
"image_config.base_image_registry.password = \"password\"\n",
|
"image_config.base_image_registry.password = \"password\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# or, use an image built during training.\n",
|
"# or, use an image built during training.\n",
|
||||||
"image_config.base_image = run.properties[\"AzureML.DerivedImageName\"]\n",
|
"image_config.base_image = run.properties[\"AzureML.DerivedImageName\"]\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"You can get the address of training image from the properties of a Run object. Only new runs submitted with azureml-sdk>=1.0.22 to AMLCompute targets will have the 'AzureML.DerivedImageName' property. Instructions on how to get a Run can be found in [manage-runs](../../training/manage-runs/manage-runs.ipynb). \n"
|
"You can get the address of training image from the properties of a Run object. Only new runs submitted with azureml-sdk>=1.0.22 to AMLCompute targets will have the 'AzureML.DerivedImageName' property. Instructions on how to get a Run can be found in [manage-runs](../../training/manage-runs/manage-runs.ipynb). \n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Provision the AKS Cluster\n",
|
"# Provision the AKS Cluster\n",
|
||||||
"This is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it."
|
"This is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Use the default configuration (can also provide parameters to customize)\n",
|
"# Use the default configuration (can also provide parameters to customize)\n",
|
||||||
"prov_config = AksCompute.provisioning_configuration()\n",
|
"prov_config = AksCompute.provisioning_configuration()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aks_name = 'my-aks-9' \n",
|
"aks_name = 'my-aks-9' \n",
|
||||||
"# Create the cluster\n",
|
"# Create the cluster\n",
|
||||||
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
"aks_target = ComputeTarget.create(workspace = ws, \n",
|
||||||
" name = aks_name, \n",
|
" name = aks_name, \n",
|
||||||
" provisioning_configuration = prov_config)"
|
" provisioning_configuration = prov_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Create AKS Cluster in an existing virtual network (optional)\n",
|
"# Create AKS Cluster in an existing virtual network (optional)\n",
|
||||||
"See code snippet below. Check the documentation [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-enable-virtual-network#use-azure-kubernetes-service) for more details."
|
"See code snippet below. Check the documentation [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-enable-virtual-network#use-azure-kubernetes-service) for more details."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"'''\n",
|
"'''\n",
|
||||||
"from azureml.core.compute import ComputeTarget, AksCompute\n",
|
"from azureml.core.compute import ComputeTarget, AksCompute\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Create the compute configuration and set virtual network information\n",
|
"# Create the compute configuration and set virtual network information\n",
|
||||||
"config = AksCompute.provisioning_configuration(location=\"eastus2\")\n",
|
"config = AksCompute.provisioning_configuration(location=\"eastus2\")\n",
|
||||||
"config.vnet_resourcegroup_name = \"mygroup\"\n",
|
"config.vnet_resourcegroup_name = \"mygroup\"\n",
|
||||||
"config.vnet_name = \"mynetwork\"\n",
|
"config.vnet_name = \"mynetwork\"\n",
|
||||||
"config.subnet_name = \"default\"\n",
|
"config.subnet_name = \"default\"\n",
|
||||||
"config.service_cidr = \"10.0.0.0/16\"\n",
|
"config.service_cidr = \"10.0.0.0/16\"\n",
|
||||||
"config.dns_service_ip = \"10.0.0.10\"\n",
|
"config.dns_service_ip = \"10.0.0.10\"\n",
|
||||||
"config.docker_bridge_cidr = \"172.17.0.1/16\"\n",
|
"config.docker_bridge_cidr = \"172.17.0.1/16\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Create the compute target\n",
|
"# Create the compute target\n",
|
||||||
"aks_target = ComputeTarget.create(workspace = ws,\n",
|
"aks_target = ComputeTarget.create(workspace = ws,\n",
|
||||||
" name = \"myaks\",\n",
|
" name = \"myaks\",\n",
|
||||||
" provisioning_configuration = config)\n",
|
" provisioning_configuration = config)\n",
|
||||||
"'''"
|
"'''"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Enable SSL on the AKS Cluster (optional)\n",
|
"# Enable SSL on the AKS Cluster (optional)\n",
|
||||||
"See code snippet below. Check the documentation [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-secure-web-service) for more details"
|
"See code snippet below. Check the documentation [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-secure-web-service) for more details"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# provisioning_config = AksCompute.provisioning_configuration(ssl_cert_pem_file=\"cert.pem\", ssl_key_pem_file=\"key.pem\", ssl_cname=\"www.contoso.com\")"
|
"# provisioning_config = AksCompute.provisioning_configuration(ssl_cert_pem_file=\"cert.pem\", ssl_key_pem_file=\"key.pem\", ssl_cname=\"www.contoso.com\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_target.wait_for_completion(show_output = True)\n",
|
"aks_target.wait_for_completion(show_output = True)\n",
|
||||||
"print(aks_target.provisioning_state)\n",
|
"print(aks_target.provisioning_state)\n",
|
||||||
"print(aks_target.provisioning_errors)"
|
"print(aks_target.provisioning_errors)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Optional step: Attach existing AKS cluster\n",
|
"## Optional step: Attach existing AKS cluster\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you have existing AKS cluster in your Azure subscription, you can attach it to the Workspace."
|
"If you have existing AKS cluster in your Azure subscription, you can attach it to the Workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"'''\n",
|
"'''\n",
|
||||||
"# Use the default configuration (can also provide parameters to customize)\n",
|
"# Use the default configuration (can also provide parameters to customize)\n",
|
||||||
"resource_id = '/subscriptions/92c76a2f-0e1c-4216-b65e-abf7a3f34c1e/resourcegroups/raymondsdk0604/providers/Microsoft.ContainerService/managedClusters/my-aks-0605d37425356b7d01'\n",
|
"resource_id = '/subscriptions/92c76a2f-0e1c-4216-b65e-abf7a3f34c1e/resourcegroups/raymondsdk0604/providers/Microsoft.ContainerService/managedClusters/my-aks-0605d37425356b7d01'\n",
|
||||||
"\n",
|
"\n",
|
||||||
"create_name='my-existing-aks' \n",
|
"create_name='my-existing-aks' \n",
|
||||||
"# Create the cluster\n",
|
"# Create the cluster\n",
|
||||||
"attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
"attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
|
||||||
"aks_target = ComputeTarget.attach(workspace=ws, name=create_name, attach_configuration=attach_config)\n",
|
"aks_target = ComputeTarget.attach(workspace=ws, name=create_name, attach_configuration=attach_config)\n",
|
||||||
"# Wait for the operation to complete\n",
|
"# Wait for the operation to complete\n",
|
||||||
"aks_target.wait_for_completion(True)\n",
|
"aks_target.wait_for_completion(True)\n",
|
||||||
"'''"
|
"'''"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Deploy web service to AKS"
|
"# Deploy web service to AKS"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Set the web service configuration (using default here)\n",
|
"#Set the web service configuration (using default here)\n",
|
||||||
"aks_config = AksWebservice.deploy_configuration()"
|
"aks_config = AksWebservice.deploy_configuration()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_service_name ='aks-service-1'\n",
|
"aks_service_name ='aks-service-1'\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
"aks_service = Webservice.deploy_from_image(workspace = ws, \n",
|
||||||
" name = aks_service_name,\n",
|
" name = aks_service_name,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" deployment_config = aks_config,\n",
|
" deployment_config = aks_config,\n",
|
||||||
" deployment_target = aks_target)\n",
|
" deployment_target = aks_target)\n",
|
||||||
"aks_service.wait_for_deployment(show_output = True)\n",
|
"aks_service.wait_for_deployment(show_output = True)\n",
|
||||||
"print(aks_service.state)"
|
"print(aks_service.state)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Test the web service using run method\n",
|
"# Test the web service using run method\n",
|
||||||
"We test the web sevice by passing data.\n",
|
"We test the web sevice by passing data.\n",
|
||||||
"Run() method retrieves API keys behind the scenes to make sure that call is authenticated."
|
"Run() method retrieves API keys behind the scenes to make sure that call is authenticated."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,2,3,4,5,6,7,8,9,10], \n",
|
" [1,2,3,4,5,6,7,8,9,10], \n",
|
||||||
" [10,9,8,7,6,5,4,3,2,1]\n",
|
" [10,9,8,7,6,5,4,3,2,1]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"prediction = aks_service.run(input_data = test_sample)\n",
|
"prediction = aks_service.run(input_data = test_sample)\n",
|
||||||
"print(prediction)"
|
"print(prediction)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Test the web service using raw HTTP request (optional)\n",
|
"# Test the web service using raw HTTP request (optional)\n",
|
||||||
"Alternatively you can construct a raw HTTP request and send it to the service. In this case you need to explicitly pass the HTTP header. This process is shown in the next 2 cells."
|
"Alternatively you can construct a raw HTTP request and send it to the service. In this case you need to explicitly pass the HTTP header. This process is shown in the next 2 cells."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# retreive the API keys. AML generates two keys.\n",
|
"# retreive the API keys. AML generates two keys.\n",
|
||||||
"'''\n",
|
"'''\n",
|
||||||
"key1, Key2 = aks_service.get_keys()\n",
|
"key1, Key2 = aks_service.get_keys()\n",
|
||||||
"print(key1)\n",
|
"print(key1)\n",
|
||||||
"'''"
|
"'''"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# construct raw HTTP request and send to the service\n",
|
"# construct raw HTTP request and send to the service\n",
|
||||||
"'''\n",
|
"'''\n",
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"\n",
|
"\n",
|
||||||
"import requests\n",
|
"import requests\n",
|
||||||
"\n",
|
"\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,2,3,4,5,6,7,8,9,10], \n",
|
" [1,2,3,4,5,6,7,8,9,10], \n",
|
||||||
" [10,9,8,7,6,5,4,3,2,1]\n",
|
" [10,9,8,7,6,5,4,3,2,1]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Don't forget to add key to the HTTP header.\n",
|
"# Don't forget to add key to the HTTP header.\n",
|
||||||
"headers = {'Content-Type':'application/json', 'Authorization': 'Bearer ' + key1}\n",
|
"headers = {'Content-Type':'application/json', 'Authorization': 'Bearer ' + key1}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"resp = requests.post(aks_service.scoring_uri, test_sample, headers=headers)\n",
|
"resp = requests.post(aks_service.scoring_uri, test_sample, headers=headers)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"prediction:\", resp.text)\n",
|
"print(\"prediction:\", resp.text)\n",
|
||||||
"'''"
|
"'''"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Clean up\n",
|
"# Clean up\n",
|
||||||
"Delete the service, image and model."
|
"Delete the service, image and model."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"aks_service.delete()\n",
|
"aks_service.delete()\n",
|
||||||
"image.delete()\n",
|
"image.delete()\n",
|
||||||
"model.delete()"
|
"model.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "aashishb"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "aashishb"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,453 +1,453 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Register Model, Create Image and Deploy Service\n",
|
"## Register Model, Create Image and Deploy Service\n",
|
||||||
"\n",
|
"\n",
|
||||||
"This example shows how to deploy a web service in step-by-step fashion:\n",
|
"This example shows how to deploy a web service in step-by-step fashion:\n",
|
||||||
"\n",
|
"\n",
|
||||||
" 1. Register model\n",
|
" 1. Register model\n",
|
||||||
" 2. Query versions of models and select one to deploy\n",
|
" 2. Query versions of models and select one to deploy\n",
|
||||||
" 3. Create Docker image\n",
|
" 3. Create Docker image\n",
|
||||||
" 4. Query versions of images\n",
|
" 4. Query versions of images\n",
|
||||||
" 5. Deploy the image as web service\n",
|
" 5. Deploy the image as web service\n",
|
||||||
" \n",
|
" \n",
|
||||||
"**IMPORTANT**:\n",
|
"**IMPORTANT**:\n",
|
||||||
" * This notebook requires you to first complete [train-within-notebook](../../training/train-within-notebook/train-within-notebook.ipynb) example\n",
|
" * This notebook requires you to first complete [train-within-notebook](../../training/train-within-notebook/train-within-notebook.ipynb) example\n",
|
||||||
" \n",
|
" \n",
|
||||||
"The train-within-notebook example taught you how to deploy a web service directly from model in one step. This Notebook shows a more advanced approach that gives you more control over model versions and Docker image versions. "
|
"The train-within-notebook example taught you how to deploy a web service directly from model in one step. This Notebook shows a more advanced approach that gives you more control over model versions and Docker image versions. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
|
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize Workspace\n",
|
"## Initialize Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a workspace object from persisted configuration."
|
"Initialize a workspace object from persisted configuration."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create workspace"
|
"create workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Register Model"
|
"### Register Model"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can add tags and descriptions to your models. Note you need to have a `sklearn_linreg_model.pkl` file in the current directory. This file is generated by the 01 notebook. The below call registers that file as a model with the same name `sklearn_linreg_model.pkl` in the workspace.\n",
|
"You can add tags and descriptions to your models. Note you need to have a `sklearn_linreg_model.pkl` file in the current directory. This file is generated by the 01 notebook. The below call registers that file as a model with the same name `sklearn_linreg_model.pkl` in the workspace.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Using tags, you can track useful information such as the name and version of the machine learning library used to train the model. Note that tags must be alphanumeric."
|
"Using tags, you can track useful information such as the name and version of the machine learning library used to train the model. Note that tags must be alphanumeric."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"register model from file"
|
"register model from file"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"import sklearn\n",
|
"import sklearn\n",
|
||||||
"\n",
|
"\n",
|
||||||
"library_version = \"sklearn\"+sklearn.__version__.replace(\".\",\"x\")\n",
|
"library_version = \"sklearn\"+sklearn.__version__.replace(\".\",\"x\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"model = Model.register(model_path = \"sklearn_regression_model.pkl\",\n",
|
"model = Model.register(model_path = \"sklearn_regression_model.pkl\",\n",
|
||||||
" model_name = \"sklearn_regression_model.pkl\",\n",
|
" model_name = \"sklearn_regression_model.pkl\",\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\", 'version': library_version},\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\", 'version': library_version},\n",
|
||||||
" description = \"Ridge regression model to predict diabetes\",\n",
|
" description = \"Ridge regression model to predict diabetes\",\n",
|
||||||
" workspace = ws)"
|
" workspace = ws)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can explore the registered models within your workspace and query by tag. Models are versioned. If you call the register_model command many times with same model name, you will get multiple versions of the model with increasing version numbers."
|
"You can explore the registered models within your workspace and query by tag. Models are versioned. If you call the register_model command many times with same model name, you will get multiple versions of the model with increasing version numbers."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"register model from file"
|
"register model from file"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"regression_models = Model.list(workspace=ws, tags=['area'])\n",
|
"regression_models = Model.list(workspace=ws, tags=['area'])\n",
|
||||||
"for m in regression_models:\n",
|
"for m in regression_models:\n",
|
||||||
" print(\"Name:\", m.name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
" print(\"Name:\", m.name,\"\\tVersion:\", m.version, \"\\tDescription:\", m.description, m.tags)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can pick a specific model to deploy"
|
"You can pick a specific model to deploy"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print(model.name, model.description, model.version, sep = '\\t')"
|
"print(model.name, model.description, model.version, sep = '\\t')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create Docker Image"
|
"### Create Docker Image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Show `score.py`. Note that the `sklearn_regression_model.pkl` in the `get_model_path` call is referring to a model named `sklearn_linreg_model.pkl` registered under the workspace. It is NOT referenceing the local file."
|
"Show `score.py`. Note that the `sklearn_regression_model.pkl` in the `get_model_path` call is referring to a model named `sklearn_linreg_model.pkl` registered under the workspace. It is NOT referenceing the local file."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%writefile score.py\n",
|
"%%writefile score.py\n",
|
||||||
"import pickle\n",
|
"import pickle\n",
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"import numpy\n",
|
"import numpy\n",
|
||||||
"from sklearn.externals import joblib\n",
|
"from sklearn.externals import joblib\n",
|
||||||
"from sklearn.linear_model import Ridge\n",
|
"from sklearn.linear_model import Ridge\n",
|
||||||
"from azureml.core.model import Model\n",
|
"from azureml.core.model import Model\n",
|
||||||
"\n",
|
"\n",
|
||||||
"def init():\n",
|
"def init():\n",
|
||||||
" global model\n",
|
" global model\n",
|
||||||
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under\n",
|
" # note here \"sklearn_regression_model.pkl\" is the name of the model registered under\n",
|
||||||
" # this is a different behavior than before when the code is run locally, even though the code is the same.\n",
|
" # this is a different behavior than before when the code is run locally, even though the code is the same.\n",
|
||||||
" model_path = Model.get_model_path('sklearn_regression_model.pkl')\n",
|
" model_path = Model.get_model_path('sklearn_regression_model.pkl')\n",
|
||||||
" # deserialize the model file back into a sklearn model\n",
|
" # deserialize the model file back into a sklearn model\n",
|
||||||
" model = joblib.load(model_path)\n",
|
" model = joblib.load(model_path)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# note you can pass in multiple rows for scoring\n",
|
"# note you can pass in multiple rows for scoring\n",
|
||||||
"def run(raw_data):\n",
|
"def run(raw_data):\n",
|
||||||
" try:\n",
|
" try:\n",
|
||||||
" data = json.loads(raw_data)['data']\n",
|
" data = json.loads(raw_data)['data']\n",
|
||||||
" data = numpy.array(data)\n",
|
" data = numpy.array(data)\n",
|
||||||
" result = model.predict(data)\n",
|
" result = model.predict(data)\n",
|
||||||
" # you can return any datatype as long as it is JSON-serializable\n",
|
" # you can return any datatype as long as it is JSON-serializable\n",
|
||||||
" return result.tolist()\n",
|
" return result.tolist()\n",
|
||||||
" except Exception as e:\n",
|
" except Exception as e:\n",
|
||||||
" error = str(e)\n",
|
" error = str(e)\n",
|
||||||
" return error"
|
" return error"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
"from azureml.core.conda_dependencies import CondaDependencies \n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
"myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'])\n",
|
||||||
"\n",
|
"\n",
|
||||||
"with open(\"myenv.yml\",\"w\") as f:\n",
|
"with open(\"myenv.yml\",\"w\") as f:\n",
|
||||||
" f.write(myenv.serialize_to_string())"
|
" f.write(myenv.serialize_to_string())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Note that following command can take few minutes. \n",
|
"Note that following command can take few minutes. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can add tags and descriptions to images. Also, an image can contain multiple models."
|
"You can add tags and descriptions to images. Also, an image can contain multiple models."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create image"
|
"create image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.image import Image, ContainerImage\n",
|
"from azureml.core.image import Image, ContainerImage\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image_config = ContainerImage.image_configuration(runtime= \"python\",\n",
|
"image_config = ContainerImage.image_configuration(runtime= \"python\",\n",
|
||||||
" execution_script=\"score.py\",\n",
|
" execution_script=\"score.py\",\n",
|
||||||
" conda_file=\"myenv.yml\",\n",
|
" conda_file=\"myenv.yml\",\n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
|
||||||
" description = \"Image with ridge regression model\")\n",
|
" description = \"Image with ridge regression model\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"image = Image.create(name = \"myimage1\",\n",
|
"image = Image.create(name = \"myimage1\",\n",
|
||||||
" # this is the model object. note you can pass in 0-n models via this list-type parameter\n",
|
" # this is the model object. note you can pass in 0-n models via this list-type parameter\n",
|
||||||
" # in case you need to reference multiple models, or none at all, in your scoring script.\n",
|
" # in case you need to reference multiple models, or none at all, in your scoring script.\n",
|
||||||
" models = [model],\n",
|
" models = [model],\n",
|
||||||
" image_config = image_config, \n",
|
" image_config = image_config, \n",
|
||||||
" workspace = ws)"
|
" workspace = ws)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create image"
|
"create image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"image.wait_for_creation(show_output = True)"
|
"image.wait_for_creation(show_output = True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"#### Use a custom Docker image\n",
|
"#### Use a custom Docker image\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can also specify a custom Docker image to be used as base image if you don't want to use the default base image provided by Azure ML. Please make sure the custom Docker image has Ubuntu >= 16.04, Conda >= 4.5.\\* and Python(3.5.\\* or 3.6.\\*).\n",
|
"You can also specify a custom Docker image to be used as base image if you don't want to use the default base image provided by Azure ML. Please make sure the custom Docker image has Ubuntu >= 16.04, Conda >= 4.5.\\* and Python(3.5.\\* or 3.6.\\*).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Only Supported for `ContainerImage`(from azureml.core.image) with `python` runtime.\n",
|
"Only Supported for `ContainerImage`(from azureml.core.image) with `python` runtime.\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"# use an image available in public Container Registry without authentication\n",
|
"# use an image available in public Container Registry without authentication\n",
|
||||||
"image_config.base_image = \"mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda\"\n",
|
"image_config.base_image = \"mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# or, use an image available in a private Container Registry\n",
|
"# or, use an image available in a private Container Registry\n",
|
||||||
"image_config.base_image = \"myregistry.azurecr.io/mycustomimage:1.0\"\n",
|
"image_config.base_image = \"myregistry.azurecr.io/mycustomimage:1.0\"\n",
|
||||||
"image_config.base_image_registry.address = \"myregistry.azurecr.io\"\n",
|
"image_config.base_image_registry.address = \"myregistry.azurecr.io\"\n",
|
||||||
"image_config.base_image_registry.username = \"username\"\n",
|
"image_config.base_image_registry.username = \"username\"\n",
|
||||||
"image_config.base_image_registry.password = \"password\"\n",
|
"image_config.base_image_registry.password = \"password\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# or, use an image built during training.\n",
|
"# or, use an image built during training.\n",
|
||||||
"image_config.base_image = run.properties[\"AzureML.DerivedImageName\"]\n",
|
"image_config.base_image = run.properties[\"AzureML.DerivedImageName\"]\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"You can get the address of training image from the properties of a Run object. Only new runs submitted with azureml-sdk>=1.0.22 to AMLCompute targets will have the 'AzureML.DerivedImageName' property. Instructions on how to get a Run can be found in [manage-runs](../../training/manage-runs/manage-runs.ipynb). \n"
|
"You can get the address of training image from the properties of a Run object. Only new runs submitted with azureml-sdk>=1.0.22 to AMLCompute targets will have the 'AzureML.DerivedImageName' property. Instructions on how to get a Run can be found in [manage-runs](../../training/manage-runs/manage-runs.ipynb). \n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"List images by tag and find out the detailed build log for debugging."
|
"List images by tag and find out the detailed build log for debugging."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create image"
|
"create image"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"for i in Image.list(workspace = ws,tags = [\"area\"]):\n",
|
"for i in Image.list(workspace = ws,tags = [\"area\"]):\n",
|
||||||
" print('{}(v.{} [{}]) stored at {} with build log {}'.format(i.name, i.version, i.creation_state, i.image_location, i.image_build_log_uri))"
|
" print('{}(v.{} [{}]) stored at {} with build log {}'.format(i.name, i.version, i.creation_state, i.image_location, i.image_build_log_uri))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Deploy image as web service on Azure Container Instance\n",
|
"### Deploy image as web service on Azure Container Instance\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Note that the service creation can take few minutes."
|
"Note that the service creation can take few minutes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"deploy service",
|
"deploy service",
|
||||||
"aci"
|
"aci"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import AciWebservice\n",
|
"from azureml.core.webservice import AciWebservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
"aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
|
||||||
" memory_gb = 1, \n",
|
" memory_gb = 1, \n",
|
||||||
" tags = {'area': \"diabetes\", 'type': \"regression\"}, \n",
|
" tags = {'area': \"diabetes\", 'type': \"regression\"}, \n",
|
||||||
" description = 'Predict diabetes using regression model')"
|
" description = 'Predict diabetes using regression model')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"deploy service",
|
"deploy service",
|
||||||
"aci"
|
"aci"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.webservice import Webservice\n",
|
"from azureml.core.webservice import Webservice\n",
|
||||||
"\n",
|
"\n",
|
||||||
"aci_service_name = 'my-aci-service-2'\n",
|
"aci_service_name = 'my-aci-service-2'\n",
|
||||||
"print(aci_service_name)\n",
|
"print(aci_service_name)\n",
|
||||||
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
"aci_service = Webservice.deploy_from_image(deployment_config = aciconfig,\n",
|
||||||
" image = image,\n",
|
" image = image,\n",
|
||||||
" name = aci_service_name,\n",
|
" name = aci_service_name,\n",
|
||||||
" workspace = ws)\n",
|
" workspace = ws)\n",
|
||||||
"aci_service.wait_for_deployment(True)\n",
|
"aci_service.wait_for_deployment(True)\n",
|
||||||
"print(aci_service.state)"
|
"print(aci_service.state)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Test web service"
|
"### Test web service"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Call the web service with some dummy input data to get a prediction."
|
"Call the web service with some dummy input data to get a prediction."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"deploy service",
|
"deploy service",
|
||||||
"aci"
|
"aci"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import json\n",
|
"import json\n",
|
||||||
"\n",
|
"\n",
|
||||||
"test_sample = json.dumps({'data': [\n",
|
"test_sample = json.dumps({'data': [\n",
|
||||||
" [1,2,3,4,5,6,7,8,9,10], \n",
|
" [1,2,3,4,5,6,7,8,9,10], \n",
|
||||||
" [10,9,8,7,6,5,4,3,2,1]\n",
|
" [10,9,8,7,6,5,4,3,2,1]\n",
|
||||||
"]})\n",
|
"]})\n",
|
||||||
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
"test_sample = bytes(test_sample,encoding = 'utf8')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"prediction = aci_service.run(input_data=test_sample)\n",
|
"prediction = aci_service.run(input_data=test_sample)\n",
|
||||||
"print(prediction)"
|
"print(prediction)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Delete ACI to clean up"
|
"### Delete ACI to clean up"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"deploy service",
|
"deploy service",
|
||||||
"aci"
|
"aci"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"aci_service.delete()"
|
"aci_service.delete()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "aashishb"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "aashishb"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,279 +1,279 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Breast cancer diagnosis classification with scikit-learn (run model explainer locally)"
|
"# Breast cancer diagnosis classification with scikit-learn (run model explainer locally)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package\n",
|
"Explain a model with the AML explain-model package\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a SVM classification model using Scikit-learn\n",
|
"1. Train a SVM classification model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
|
"2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
|
||||||
"3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
|
"3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
|
||||||
"4. Visualize the global and local explanations with the visualization dashboard."
|
"4. Visualize the global and local explanations with the visualization dashboard."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from sklearn.datasets import load_breast_cancer\n",
|
"from sklearn.datasets import load_breast_cancer\n",
|
||||||
"from sklearn import svm\n",
|
"from sklearn import svm\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 1. Run model explainer locally with full data"
|
"# 1. Run model explainer locally with full data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load the breast cancer diagnosis data"
|
"## Load the breast cancer diagnosis data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"breast_cancer_data = load_breast_cancer()\n",
|
"breast_cancer_data = load_breast_cancer()\n",
|
||||||
"classes = breast_cancer_data.target_names.tolist()"
|
"classes = breast_cancer_data.target_names.tolist()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Split data into train and test\n",
|
"# Split data into train and test\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
|
"x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train a SVM classification model, which you want to explain"
|
"## Train a SVM classification model, which you want to explain"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
||||||
"model = clf.fit(x_train, y_train)"
|
"model = clf.fit(x_train, y_train)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain predictions on your local machine"
|
"## Explain predictions on your local machine"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
|
"tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions (global explanation)"
|
"## Explain overall model predictions (global explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
||||||
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Sorted SHAP values\n",
|
"# Sorted SHAP values\n",
|
||||||
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
||||||
"# Corresponding feature names\n",
|
"# Corresponding feature names\n",
|
||||||
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
||||||
"# feature ranks (based on original order of features)\n",
|
"# feature ranks (based on original order of features)\n",
|
||||||
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
|
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
|
||||||
"# per class feature names\n",
|
"# per class feature names\n",
|
||||||
"print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
|
"print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
|
||||||
"# per class feature importance values\n",
|
"# per class feature importance values\n",
|
||||||
"print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
|
"print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# feature shap values for all features and all data points in the training data\n",
|
"# feature shap values for all features and all data points in the training data\n",
|
||||||
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain local data points (individual instances)"
|
"## Explain local data points (individual instances)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# explain the first member of the test set\n",
|
"# explain the first member of the test set\n",
|
||||||
"instance_num = 0\n",
|
"instance_num = 0\n",
|
||||||
"local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
|
"local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
||||||
"prediction_value = clf.predict(x_test)[instance_num]\n",
|
"prediction_value = clf.predict(x_test)[instance_num]\n",
|
||||||
"\n",
|
"\n",
|
||||||
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
||||||
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
|
"dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 2. Load visualization dashboard"
|
"# 2. Load visualization dashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
||||||
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"# Or, in Jupyter Labs, uncomment below\n",
|
"# Or, in Jupyter Labs, uncomment below\n",
|
||||||
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
||||||
"# jupyter labextension install microsoft-mli-widget"
|
"# jupyter labextension install microsoft-mli-widget"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ExplanationDashboard(global_explanation, model, x_test)"
|
"ExplanationDashboard(global_explanation, model, x_test)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,280 +1,280 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Iris flower classification with scikit-learn (run model explainer locally)"
|
"# Iris flower classification with scikit-learn (run model explainer locally)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package\n",
|
"Explain a model with the AML explain-model package\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a SVM classification model using Scikit-learn\n",
|
"1. Train a SVM classification model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
|
"2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
|
||||||
"3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
|
"3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
|
||||||
"4. Visualize the global and local explanations with the visualization dashboard."
|
"4. Visualize the global and local explanations with the visualization dashboard."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from sklearn.datasets import load_iris\n",
|
"from sklearn.datasets import load_iris\n",
|
||||||
"from sklearn import svm\n",
|
"from sklearn import svm\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 1. Run model explainer locally with full data"
|
"# 1. Run model explainer locally with full data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load the breast cancer diagnosis data"
|
"## Load the breast cancer diagnosis data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"iris = load_iris()\n",
|
"iris = load_iris()\n",
|
||||||
"X = iris['data']\n",
|
"X = iris['data']\n",
|
||||||
"y = iris['target']\n",
|
"y = iris['target']\n",
|
||||||
"classes = iris['target_names']\n",
|
"classes = iris['target_names']\n",
|
||||||
"feature_names = iris['feature_names']"
|
"feature_names = iris['feature_names']"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Split data into train and test\n",
|
"# Split data into train and test\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
|
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train a SVM classification model, which you want to explain"
|
"## Train a SVM classification model, which you want to explain"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
||||||
"model = clf.fit(x_train, y_train)"
|
"model = clf.fit(x_train, y_train)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain predictions on your local machine"
|
"## Explain predictions on your local machine"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tabular_explainer = TabularExplainer(model, x_train, features = feature_names, classes=classes)"
|
"tabular_explainer = TabularExplainer(model, x_train, features = feature_names, classes=classes)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions (global explanation)"
|
"## Explain overall model predictions (global explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Sorted SHAP values\n",
|
"# Sorted SHAP values\n",
|
||||||
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
||||||
"# Corresponding feature names\n",
|
"# Corresponding feature names\n",
|
||||||
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
||||||
"# feature ranks (based on original order of features)\n",
|
"# feature ranks (based on original order of features)\n",
|
||||||
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
|
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
|
||||||
"# per class feature names\n",
|
"# per class feature names\n",
|
||||||
"print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
|
"print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
|
||||||
"# per class feature importance values\n",
|
"# per class feature importance values\n",
|
||||||
"print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
|
"print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# feature shap values for all features and all data points in the training data\n",
|
"# feature shap values for all features and all data points in the training data\n",
|
||||||
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain local data points (individual instances)"
|
"## Explain local data points (individual instances)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# explain the first member of the test set\n",
|
"# explain the first member of the test set\n",
|
||||||
"instance_num = 0\n",
|
"instance_num = 0\n",
|
||||||
"local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
|
"local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
||||||
"prediction_value = clf.predict(x_test)[instance_num]\n",
|
"prediction_value = clf.predict(x_test)[instance_num]\n",
|
||||||
"\n",
|
"\n",
|
||||||
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
||||||
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
|
"dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load visualization dashboard"
|
"## Load visualization dashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
||||||
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"# Or, in Jupyter Labs, uncomment below\n",
|
"# Or, in Jupyter Labs, uncomment below\n",
|
||||||
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
||||||
"# jupyter labextension install microsoft-mli-widget"
|
"# jupyter labextension install microsoft-mli-widget"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ExplanationDashboard(global_explanation, model, x_test)"
|
"ExplanationDashboard(global_explanation, model, x_test)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,272 +1,272 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Boston Housing Price Prediction with scikit-learn (run model explainer locally)"
|
"# Boston Housing Price Prediction with scikit-learn (run model explainer locally)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package\n",
|
"Explain a model with the AML explain-model package\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a GradientBoosting regression model using Scikit-learn\n",
|
"1. Train a GradientBoosting regression model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
|
"2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
|
||||||
"3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
|
"3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
|
||||||
"4. Visualize the global and local explanations with the visualization dashboard."
|
"4. Visualize the global and local explanations with the visualization dashboard."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from sklearn import datasets\n",
|
"from sklearn import datasets\n",
|
||||||
"from sklearn.ensemble import GradientBoostingRegressor\n",
|
"from sklearn.ensemble import GradientBoostingRegressor\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 1. Run model explainer locally with full data"
|
"# 1. Run model explainer locally with full data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load the Boston house price data"
|
"## Load the Boston house price data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"boston_data = datasets.load_boston()"
|
"boston_data = datasets.load_boston()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Split data into train and test\n",
|
"# Split data into train and test\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
|
"x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train a GradientBoosting Regression model, which you want to explain"
|
"## Train a GradientBoosting Regression model, which you want to explain"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"reg = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
|
"reg = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
|
||||||
" learning_rate=0.1, loss='huber',\n",
|
" learning_rate=0.1, loss='huber',\n",
|
||||||
" random_state=1)\n",
|
" random_state=1)\n",
|
||||||
"model = reg.fit(x_train, y_train)"
|
"model = reg.fit(x_train, y_train)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain predictions on your local machine"
|
"## Explain predictions on your local machine"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tabular_explainer = TabularExplainer(model, x_train, features = boston_data.feature_names)"
|
"tabular_explainer = TabularExplainer(model, x_train, features = boston_data.feature_names)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions (global explanation)"
|
"## Explain overall model predictions (global explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
||||||
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Sorted SHAP values \n",
|
"# Sorted SHAP values \n",
|
||||||
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
"print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
|
||||||
"# Corresponding feature names\n",
|
"# Corresponding feature names\n",
|
||||||
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
"print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
|
||||||
"# feature ranks (based on original order of features)\n",
|
"# feature ranks (based on original order of features)\n",
|
||||||
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))"
|
"print('global importance rank: {}'.format(global_explanation.global_importance_rank))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
"dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# feature shap values for all features and all data points in the training data\n",
|
"# feature shap values for all features and all data points in the training data\n",
|
||||||
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
"print('local importance values: {}'.format(global_explanation.local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain local data points (individual instances)"
|
"## Explain local data points (individual instances)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"local_explanation = tabular_explainer.explain_local(x_test[0,:])"
|
"local_explanation = tabular_explainer.explain_local(x_test[0,:])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# sorted local feature importance information; reflects the original feature order\n",
|
"# sorted local feature importance information; reflects the original feature order\n",
|
||||||
"sorted_local_importance_names = local_explanation.get_ranked_local_names()\n",
|
"sorted_local_importance_names = local_explanation.get_ranked_local_names()\n",
|
||||||
"sorted_local_importance_values = local_explanation.get_ranked_local_values()\n",
|
"sorted_local_importance_values = local_explanation.get_ranked_local_values()\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print('sorted local importance names: {}'.format(sorted_local_importance_names))\n",
|
"print('sorted local importance names: {}'.format(sorted_local_importance_names))\n",
|
||||||
"print('sorted local importance values: {}'.format(sorted_local_importance_values))"
|
"print('sorted local importance values: {}'.format(sorted_local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load visualization dashboard"
|
"## Load visualization dashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
||||||
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
"# Or, in Jupyter Labs, uncomment below\n",
|
"# Or, in Jupyter Labs, uncomment below\n",
|
||||||
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
||||||
"# jupyter labextension install microsoft-mli-widget"
|
"# jupyter labextension install microsoft-mli-widget"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ExplanationDashboard(global_explanation, model, x_test)"
|
"ExplanationDashboard(global_explanation, model, x_test)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,337 +1,302 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Summary\n",
|
"# Summary\n",
|
||||||
"From raw data that is a mixture of categoricals and numeric, featurize the categoricals using one hot encoding. Use tabular explainer to get explain object and then get raw feature importances"
|
"From raw data that is a mixture of categoricals and numeric, featurize the categoricals using one hot encoding. Use tabular explainer to get explain object and then get raw feature importances"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package on raw features\n",
|
"Explain a model with the AML explain-model package on raw features\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a Logistic Regression model using Scikit-learn\n",
|
"1. Train a Logistic Regression model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
|
"2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
|
||||||
"3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
|
"3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
|
||||||
"4. Visualize the global and local explanations with the visualization dashboard."
|
"4. Visualize the global and local explanations with the visualization dashboard."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "markdown",
|
||||||
"execution_count": null,
|
"metadata": {},
|
||||||
"metadata": {},
|
"source": [
|
||||||
"outputs": [],
|
"This example needs sklearn-pandas. If it is not installed, uncomment and run the following line."
|
||||||
"source": [
|
]
|
||||||
"from sklearn.pipeline import Pipeline\n",
|
},
|
||||||
"from sklearn.impute import SimpleImputer\n",
|
{
|
||||||
"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
|
"cell_type": "code",
|
||||||
"from sklearn.linear_model import LogisticRegression\n",
|
"execution_count": null,
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
"metadata": {},
|
||||||
"import pandas as pd\n",
|
"outputs": [],
|
||||||
"import numpy as np"
|
"source": [
|
||||||
]
|
"#!pip install sklearn-pandas"
|
||||||
},
|
]
|
||||||
{
|
},
|
||||||
"cell_type": "code",
|
{
|
||||||
"execution_count": null,
|
"cell_type": "code",
|
||||||
"metadata": {},
|
"execution_count": null,
|
||||||
"outputs": [],
|
"metadata": {},
|
||||||
"source": [
|
"outputs": [],
|
||||||
"titanic_url = ('https://raw.githubusercontent.com/amueller/'\n",
|
"source": [
|
||||||
" 'scipy-2017-sklearn/091d371/notebooks/datasets/titanic3.csv')\n",
|
"from sklearn.pipeline import Pipeline\n",
|
||||||
"data = pd.read_csv(titanic_url)\n",
|
"from sklearn.impute import SimpleImputer\n",
|
||||||
"# fill missing values\n",
|
"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
|
||||||
"data = data.fillna(method=\"ffill\")\n",
|
"from sklearn.linear_model import LogisticRegression\n",
|
||||||
"data = data.fillna(method=\"bfill\")"
|
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
||||||
]
|
"from sklearn_pandas import DataFrameMapper\n",
|
||||||
},
|
"import pandas as pd\n",
|
||||||
{
|
"import numpy as np"
|
||||||
"cell_type": "markdown",
|
]
|
||||||
"metadata": {},
|
},
|
||||||
"source": [
|
{
|
||||||
"# 1. Run model explainer locally with full data"
|
"cell_type": "code",
|
||||||
]
|
"execution_count": null,
|
||||||
},
|
"metadata": {},
|
||||||
{
|
"outputs": [],
|
||||||
"cell_type": "markdown",
|
"source": [
|
||||||
"metadata": {},
|
"titanic_url = ('https://raw.githubusercontent.com/amueller/'\n",
|
||||||
"source": [
|
" 'scipy-2017-sklearn/091d371/notebooks/datasets/titanic3.csv')\n",
|
||||||
"Similar to example [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py), use a subset of columns"
|
"data = pd.read_csv(titanic_url)\n",
|
||||||
]
|
"# fill missing values\n",
|
||||||
},
|
"data = data.fillna(method=\"ffill\")\n",
|
||||||
{
|
"data = data.fillna(method=\"bfill\")"
|
||||||
"cell_type": "code",
|
]
|
||||||
"execution_count": null,
|
},
|
||||||
"metadata": {},
|
{
|
||||||
"outputs": [],
|
"cell_type": "markdown",
|
||||||
"source": [
|
"metadata": {},
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"source": [
|
||||||
"\n",
|
"# 1. Run model explainer locally with full data"
|
||||||
"numeric_features = ['age', 'fare']\n",
|
]
|
||||||
"categorical_features = ['embarked', 'sex', 'pclass']\n",
|
},
|
||||||
"\n",
|
{
|
||||||
"y = data['survived'].values\n",
|
"cell_type": "markdown",
|
||||||
"X = data[categorical_features + numeric_features]\n",
|
"metadata": {},
|
||||||
"\n",
|
"source": [
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
|
"Similar to example [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py), use a subset of columns"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "code",
|
||||||
"metadata": {},
|
"execution_count": null,
|
||||||
"source": [
|
"metadata": {},
|
||||||
"sklearn imports"
|
"outputs": [],
|
||||||
]
|
"source": [
|
||||||
},
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
{
|
"\n",
|
||||||
"cell_type": "code",
|
"numeric_features = ['age', 'fare']\n",
|
||||||
"execution_count": null,
|
"categorical_features = ['embarked', 'sex', 'pclass']\n",
|
||||||
"metadata": {},
|
"\n",
|
||||||
"outputs": [],
|
"y = data['survived'].values\n",
|
||||||
"source": [
|
"X = data[categorical_features + numeric_features]\n",
|
||||||
"from sklearn.pipeline import Pipeline\n",
|
"\n",
|
||||||
"from sklearn.impute import SimpleImputer\n",
|
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
|
||||||
"from sklearn.preprocessing import StandardScaler, OneHotEncoder"
|
]
|
||||||
]
|
},
|
||||||
},
|
{
|
||||||
{
|
"cell_type": "code",
|
||||||
"cell_type": "markdown",
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"outputs": [],
|
||||||
"We can explain raw features by either using a `sklearn.compose.ColumnTransformer` or a list of fitted transformer tuples. The cell below uses `sklearn.compose.ColumnTransformer`. In case you want to run the example with the list of fitted transformer tuples, comment the cell below and uncomment the cell that follows after. "
|
"source": [
|
||||||
]
|
"from sklearn.pipeline import Pipeline\n",
|
||||||
},
|
"from sklearn.impute import SimpleImputer\n",
|
||||||
{
|
"from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
|
||||||
"cell_type": "code",
|
"from sklearn_pandas import DataFrameMapper\n",
|
||||||
"execution_count": null,
|
"\n",
|
||||||
"metadata": {},
|
"# Impute, standardize the numeric features and one-hot encode the categorical features. \n",
|
||||||
"outputs": [],
|
"\n",
|
||||||
"source": [
|
"transformations = [\n",
|
||||||
"from sklearn.compose import ColumnTransformer\n",
|
" ([\"age\", \"fare\"], Pipeline(steps=[\n",
|
||||||
"\n",
|
" ('imputer', SimpleImputer(strategy='median')),\n",
|
||||||
"transformations = ColumnTransformer([\n",
|
" ('scaler', StandardScaler())\n",
|
||||||
" (\"age_fare\", Pipeline(steps=[\n",
|
" ])),\n",
|
||||||
" ('imputer', SimpleImputer(strategy='median')),\n",
|
" ([\"embarked\"], Pipeline(steps=[\n",
|
||||||
" ('scaler', StandardScaler())\n",
|
" (\"imputer\", SimpleImputer(strategy='constant', fill_value='missing')), \n",
|
||||||
" ]), [\"age\", \"fare\"]),\n",
|
" (\"encoder\", OneHotEncoder(sparse=False))])),\n",
|
||||||
" (\"embarked\", Pipeline(steps=[\n",
|
" ([\"sex\", \"pclass\"], OneHotEncoder(sparse=False)) \n",
|
||||||
" (\"imputer\", SimpleImputer(strategy='constant', fill_value='missing')), \n",
|
"]\n",
|
||||||
" (\"encoder\", OneHotEncoder(sparse=False))]), [\"embarked\"]),\n",
|
"\n",
|
||||||
" (\"sex_pclass\", OneHotEncoder(sparse=False), [\"sex\", \"pclass\"]) \n",
|
"\n",
|
||||||
"])\n",
|
"# Append classifier to preprocessing pipeline.\n",
|
||||||
"\n",
|
"# Now we have a full prediction pipeline.\n",
|
||||||
"\n",
|
"clf = Pipeline(steps=[('preprocessor', DataFrameMapper(transformations)),\n",
|
||||||
"# Append classifier to preprocessing pipeline.\n",
|
" ('classifier', LogisticRegression(solver='lbfgs'))])"
|
||||||
"# Now we have a full prediction pipeline.\n",
|
]
|
||||||
"clf = Pipeline(steps=[('preprocessor', transformations),\n",
|
},
|
||||||
" ('classifier', LogisticRegression(solver='lbfgs'))])\n"
|
{
|
||||||
]
|
"cell_type": "markdown",
|
||||||
},
|
"metadata": {},
|
||||||
{
|
"source": [
|
||||||
"cell_type": "code",
|
"## Train a Logistic Regression model, which you want to explain"
|
||||||
"execution_count": null,
|
]
|
||||||
"metadata": {},
|
},
|
||||||
"outputs": [],
|
{
|
||||||
"source": [
|
"cell_type": "code",
|
||||||
"'''\n",
|
"execution_count": null,
|
||||||
"# Uncomment below if sklearn-pandas is not installed\n",
|
"metadata": {},
|
||||||
"#!pip install sklearn-pandas\n",
|
"outputs": [],
|
||||||
"from sklearn_pandas import DataFrameMapper\n",
|
"source": [
|
||||||
"\n",
|
"model = clf.fit(x_train, y_train)"
|
||||||
"# Impute, standardize the numeric features and one-hot encode the categorical features. \n",
|
]
|
||||||
"\n",
|
},
|
||||||
"transformations = [\n",
|
{
|
||||||
" ([\"age\", \"fare\"], Pipeline(steps=[\n",
|
"cell_type": "markdown",
|
||||||
" ('imputer', SimpleImputer(strategy='median')),\n",
|
"metadata": {},
|
||||||
" ('scaler', StandardScaler())\n",
|
"source": [
|
||||||
" ])),\n",
|
"## Explain predictions on your local machine"
|
||||||
" ([\"embarked\"], Pipeline(steps=[\n",
|
]
|
||||||
" (\"imputer\", SimpleImputer(strategy='constant', fill_value='missing')), \n",
|
},
|
||||||
" (\"encoder\", OneHotEncoder(sparse=False))])),\n",
|
{
|
||||||
" ([\"sex\", \"pclass\"], OneHotEncoder(sparse=False)) \n",
|
"cell_type": "code",
|
||||||
"]\n",
|
"execution_count": null,
|
||||||
"\n",
|
"metadata": {},
|
||||||
"\n",
|
"outputs": [],
|
||||||
"# Append classifier to preprocessing pipeline.\n",
|
"source": [
|
||||||
"# Now we have a full prediction pipeline.\n",
|
"tabular_explainer = TabularExplainer(clf.steps[-1][1], initialization_examples=x_train, features=x_train.columns, transformations=transformations)"
|
||||||
"clf = Pipeline(steps=[('preprocessor', DataFrameMapper(transformations)),\n",
|
]
|
||||||
" ('classifier', LogisticRegression(solver='lbfgs'))])\n",
|
},
|
||||||
"'''"
|
{
|
||||||
]
|
"cell_type": "code",
|
||||||
},
|
"execution_count": null,
|
||||||
{
|
"metadata": {},
|
||||||
"cell_type": "markdown",
|
"outputs": [],
|
||||||
"metadata": {},
|
"source": [
|
||||||
"source": [
|
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
||||||
"## Train a Logistic Regression model, which you want to explain"
|
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
||||||
]
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
},
|
]
|
||||||
{
|
},
|
||||||
"cell_type": "code",
|
{
|
||||||
"execution_count": null,
|
"cell_type": "code",
|
||||||
"metadata": {},
|
"execution_count": null,
|
||||||
"outputs": [],
|
"metadata": {},
|
||||||
"source": [
|
"outputs": [],
|
||||||
"model = clf.fit(x_train, y_train)"
|
"source": [
|
||||||
]
|
"sorted_global_importance_values = global_explanation.get_ranked_global_values()\n",
|
||||||
},
|
"sorted_global_importance_names = global_explanation.get_ranked_global_names()\n",
|
||||||
{
|
"dict(zip(sorted_global_importance_names, sorted_global_importance_values))"
|
||||||
"cell_type": "markdown",
|
]
|
||||||
"metadata": {},
|
},
|
||||||
"source": [
|
{
|
||||||
"## Explain predictions on your local machine"
|
"cell_type": "markdown",
|
||||||
]
|
"metadata": {},
|
||||||
},
|
"source": [
|
||||||
{
|
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
||||||
"cell_type": "code",
|
]
|
||||||
"execution_count": null,
|
},
|
||||||
"metadata": {},
|
{
|
||||||
"outputs": [],
|
"cell_type": "code",
|
||||||
"source": [
|
"execution_count": null,
|
||||||
"tabular_explainer = TabularExplainer(clf.steps[-1][1], initialization_examples=x_train, features=x_train.columns, transformations=transformations)"
|
"metadata": {},
|
||||||
]
|
"outputs": [],
|
||||||
},
|
"source": [
|
||||||
{
|
"# explain the first member of the test set\n",
|
||||||
"cell_type": "code",
|
"local_explanation = tabular_explainer.explain_local(x_test[:1])"
|
||||||
"execution_count": null,
|
]
|
||||||
"metadata": {},
|
},
|
||||||
"outputs": [],
|
{
|
||||||
"source": [
|
"cell_type": "code",
|
||||||
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
"execution_count": null,
|
||||||
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
"metadata": {},
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"outputs": [],
|
||||||
]
|
"source": [
|
||||||
},
|
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
||||||
{
|
"prediction_value = clf.predict(x_test)[0]\n",
|
||||||
"cell_type": "code",
|
"\n",
|
||||||
"execution_count": null,
|
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
||||||
"metadata": {},
|
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
||||||
"outputs": [],
|
"\n",
|
||||||
"source": [
|
"# Sorted local SHAP values\n",
|
||||||
"sorted_global_importance_values = global_explanation.get_ranked_global_values()\n",
|
"print('ranked local importance values: {}'.format(sorted_local_importance_values))\n",
|
||||||
"sorted_global_importance_names = global_explanation.get_ranked_global_names()\n",
|
"# Corresponding feature names\n",
|
||||||
"dict(zip(sorted_global_importance_names, sorted_global_importance_values))"
|
"print('ranked local importance names: {}'.format(sorted_local_importance_names))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions as a collection of local (instance-level) explanations"
|
"# 2. Load visualization dashboard"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# explain the first member of the test set\n",
|
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
||||||
"local_explanation = tabular_explainer.explain_local(x_test[:1])"
|
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
]
|
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
||||||
},
|
"# Or, in Jupyter Labs, uncomment below\n",
|
||||||
{
|
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
||||||
"cell_type": "code",
|
"# jupyter labextension install microsoft-mli-widget"
|
||||||
"execution_count": null,
|
]
|
||||||
"metadata": {},
|
},
|
||||||
"outputs": [],
|
{
|
||||||
"source": [
|
"cell_type": "code",
|
||||||
"# get the prediction for the first member of the test set and explain why model made that prediction\n",
|
"execution_count": null,
|
||||||
"prediction_value = clf.predict(x_test)[0]\n",
|
"metadata": {},
|
||||||
"\n",
|
"outputs": [],
|
||||||
"sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
|
"source": [
|
||||||
"sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
|
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
||||||
"\n",
|
]
|
||||||
"# Sorted local SHAP values\n",
|
},
|
||||||
"print('ranked local importance values: {}'.format(sorted_local_importance_values))\n",
|
{
|
||||||
"# Corresponding feature names\n",
|
"cell_type": "code",
|
||||||
"print('ranked local importance names: {}'.format(sorted_local_importance_names))"
|
"execution_count": null,
|
||||||
]
|
"metadata": {},
|
||||||
},
|
"outputs": [],
|
||||||
{
|
"source": [
|
||||||
"cell_type": "markdown",
|
"ExplanationDashboard(global_explanation, model, x_test)"
|
||||||
"metadata": {},
|
]
|
||||||
"source": [
|
}
|
||||||
"# 2. Load visualization dashboard"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": null,
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
|
|
||||||
"!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
|
||||||
"!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
|
|
||||||
"# Or, in Jupyter Labs, uncomment below\n",
|
|
||||||
"# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
|
|
||||||
"# jupyter labextension install microsoft-mli-widget"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": null,
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"from azureml.contrib.explain.model.visualize import ExplanationDashboard"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": null,
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"ExplanationDashboard(global_explanation, model, x_test)"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,262 +1,262 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Breast cancer diagnosis classification with scikit-learn (save model explanations via AML Run History)"
|
"# Breast cancer diagnosis classification with scikit-learn (save model explanations via AML Run History)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package\n",
|
"Explain a model with the AML explain-model package\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a SVM classification model using Scikit-learn\n",
|
"1. Train a SVM classification model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
|
"2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from sklearn.datasets import load_breast_cancer\n",
|
"from sklearn.datasets import load_breast_cancer\n",
|
||||||
"from sklearn import svm\n",
|
"from sklearn import svm\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
"from azureml.explain.model.tabular_explainer import TabularExplainer"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 1. Run model explainer locally with full data"
|
"# 1. Run model explainer locally with full data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load the breast cancer diagnosis data"
|
"## Load the breast cancer diagnosis data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"breast_cancer_data = load_breast_cancer()\n",
|
"breast_cancer_data = load_breast_cancer()\n",
|
||||||
"classes = breast_cancer_data.target_names.tolist()"
|
"classes = breast_cancer_data.target_names.tolist()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Split data into train and test\n",
|
"# Split data into train and test\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
|
"x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train a SVM classification model, which you want to explain"
|
"## Train a SVM classification model, which you want to explain"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
"clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
|
||||||
"model = clf.fit(x_train, y_train)"
|
"model = clf.fit(x_train, y_train)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain predictions on your local machine"
|
"## Explain predictions on your local machine"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
|
"tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions (global explanation)"
|
"## Explain overall model predictions (global explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
||||||
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 2. Save Model Explanation With AML Run History"
|
"# 2. Save Model Explanation With AML Run History"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"from azureml.core import Workspace, Experiment, Run\n",
|
"from azureml.core import Workspace, Experiment, Run\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
||||||
"from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
|
"from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"experiment_name = 'explain_model'\n",
|
"experiment_name = 'explain_model'\n",
|
||||||
"experiment = Experiment(ws, experiment_name)\n",
|
"experiment = Experiment(ws, experiment_name)\n",
|
||||||
"run = experiment.start_logging()\n",
|
"run = experiment.start_logging()\n",
|
||||||
"client = ExplanationClient.from_run(run)"
|
"client = ExplanationClient.from_run(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Uploading model explanation data for storage or visualization in webUX\n",
|
"# Uploading model explanation data for storage or visualization in webUX\n",
|
||||||
"# The explanation can then be downloaded on any compute\n",
|
"# The explanation can then be downloaded on any compute\n",
|
||||||
"client.upload_model_explanation(global_explanation)"
|
"client.upload_model_explanation(global_explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Get model explanation data\n",
|
"# Get model explanation data\n",
|
||||||
"explanation = client.download_model_explanation()\n",
|
"explanation = client.download_model_explanation()\n",
|
||||||
"local_importance_values = explanation.local_importance_values\n",
|
"local_importance_values = explanation.local_importance_values\n",
|
||||||
"expected_values = explanation.expected_values"
|
"expected_values = explanation.expected_values"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Get the top k (e.g., 4) most important features with their importance values\n",
|
"# Get the top k (e.g., 4) most important features with their importance values\n",
|
||||||
"explanation = client.download_model_explanation(top_k=4)\n",
|
"explanation = client.download_model_explanation(top_k=4)\n",
|
||||||
"global_importance_values = explanation.get_ranked_global_values()\n",
|
"global_importance_values = explanation.get_ranked_global_values()\n",
|
||||||
"global_importance_names = explanation.get_ranked_global_names()\n",
|
"global_importance_names = explanation.get_ranked_global_names()\n",
|
||||||
"per_class_names = explanation.get_ranked_per_class_names()[0]\n",
|
"per_class_names = explanation.get_ranked_per_class_names()[0]\n",
|
||||||
"per_class_values = explanation.get_ranked_per_class_values()[0]"
|
"per_class_values = explanation.get_ranked_per_class_values()[0]"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print('per class feature importance values: {}'.format(per_class_values))\n",
|
"print('per class feature importance values: {}'.format(per_class_values))\n",
|
||||||
"print('per class feature importance names: {}'.format(per_class_names))"
|
"print('per class feature importance names: {}'.format(per_class_names))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"dict(zip(per_class_names, per_class_values))"
|
"dict(zip(per_class_names, per_class_values))"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,276 +1,276 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Boston Housing Price Prediction with scikit-learn (save model explanations via AML Run History)"
|
"# Boston Housing Price Prediction with scikit-learn (save model explanations via AML Run History)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Explain a model with the AML explain-model package\n",
|
"Explain a model with the AML explain-model package\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Train a GradientBoosting regression model using Scikit-learn\n",
|
"1. Train a GradientBoosting regression model using Scikit-learn\n",
|
||||||
"2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
|
"2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Save Model Explanation With AML Run History"
|
"# Save Model Explanation With AML Run History"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Import Iris dataset\n",
|
"#Import Iris dataset\n",
|
||||||
"from sklearn import datasets\n",
|
"from sklearn import datasets\n",
|
||||||
"from sklearn.ensemble import GradientBoostingRegressor\n",
|
"from sklearn.ensemble import GradientBoostingRegressor\n",
|
||||||
"\n",
|
"\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"from azureml.core import Workspace, Experiment, Run\n",
|
"from azureml.core import Workspace, Experiment, Run\n",
|
||||||
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
"from azureml.explain.model.tabular_explainer import TabularExplainer\n",
|
||||||
"from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
|
"from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"experiment_name = 'explain_model'\n",
|
"experiment_name = 'explain_model'\n",
|
||||||
"experiment = Experiment(ws, experiment_name)\n",
|
"experiment = Experiment(ws, experiment_name)\n",
|
||||||
"run = experiment.start_logging()\n",
|
"run = experiment.start_logging()\n",
|
||||||
"client = ExplanationClient.from_run(run)"
|
"client = ExplanationClient.from_run(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Load the Boston house price data"
|
"## Load the Boston house price data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"boston_data = datasets.load_boston()"
|
"boston_data = datasets.load_boston()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Split data into train and test\n",
|
"# Split data into train and test\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
|
"x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train a GradientBoosting Regression model, which you want to explain"
|
"## Train a GradientBoosting Regression model, which you want to explain"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"clf = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
|
"clf = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
|
||||||
" learning_rate=0.1, loss='huber',\n",
|
" learning_rate=0.1, loss='huber',\n",
|
||||||
" random_state=1)\n",
|
" random_state=1)\n",
|
||||||
"model = clf.fit(x_train, y_train)"
|
"model = clf.fit(x_train, y_train)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain predictions on your local machine"
|
"## Explain predictions on your local machine"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tabular_explainer = TabularExplainer(model, x_train, features=boston_data.feature_names)"
|
"tabular_explainer = TabularExplainer(model, x_train, features=boston_data.feature_names)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain overall model predictions (global explanation)"
|
"## Explain overall model predictions (global explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
"# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
|
||||||
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
"# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
|
||||||
"global_explanation = tabular_explainer.explain_global(x_test)"
|
"global_explanation = tabular_explainer.explain_global(x_test)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Uploading model explanation data for storage or visualization in webUX\n",
|
"# Uploading model explanation data for storage or visualization in webUX\n",
|
||||||
"# The explanation can then be downloaded on any compute\n",
|
"# The explanation can then be downloaded on any compute\n",
|
||||||
"client.upload_model_explanation(global_explanation)"
|
"client.upload_model_explanation(global_explanation)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Get model explanation data\n",
|
"# Get model explanation data\n",
|
||||||
"explanation = client.download_model_explanation()\n",
|
"explanation = client.download_model_explanation()\n",
|
||||||
"local_importance_values = explanation.local_importance_values\n",
|
"local_importance_values = explanation.local_importance_values\n",
|
||||||
"expected_values = explanation.expected_values"
|
"expected_values = explanation.expected_values"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Print the values\n",
|
"# Print the values\n",
|
||||||
"print('expected values: {}'.format(expected_values))\n",
|
"print('expected values: {}'.format(expected_values))\n",
|
||||||
"print('local importance values: {}'.format(local_importance_values))"
|
"print('local importance values: {}'.format(local_importance_values))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Get the top k (e.g., 4) most important features with their importance values\n",
|
"# Get the top k (e.g., 4) most important features with their importance values\n",
|
||||||
"explanation = client.download_model_explanation(top_k=4)\n",
|
"explanation = client.download_model_explanation(top_k=4)\n",
|
||||||
"global_importance_values = explanation.get_ranked_global_values()\n",
|
"global_importance_values = explanation.get_ranked_global_values()\n",
|
||||||
"global_importance_names = explanation.get_ranked_global_names()"
|
"global_importance_names = explanation.get_ranked_global_names()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"print('global importance values: {}'.format(global_importance_values))\n",
|
"print('global importance values: {}'.format(global_importance_values))\n",
|
||||||
"print('global importance names: {}'.format(global_importance_names))"
|
"print('global importance names: {}'.format(global_importance_names))"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Explain individual instance predictions (local explanation) ##### needs to get updated with the new build"
|
"## Explain individual instance predictions (local explanation) ##### needs to get updated with the new build"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"local_explanation = tabular_explainer.explain_local(x_test[0,:])"
|
"local_explanation = tabular_explainer.explain_local(x_test[0,:])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# local feature importance information\n",
|
"# local feature importance information\n",
|
||||||
"local_importance_values = local_explanation.local_importance_values\n",
|
"local_importance_values = local_explanation.local_importance_values\n",
|
||||||
"print('local importance values: {}'.format(local_importance_values))"
|
"print('local importance values: {}'.format(local_importance_values))"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "mesameki"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "mesameki"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,322 +1,322 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Distributed Chainer\n",
|
"# Distributed Chainer\n",
|
||||||
"In this tutorial, you will run a Chainer training example on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using ChainerMN distributed training across a GPU cluster."
|
"In this tutorial, you will run a Chainer training example on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using ChainerMN distributed training across a GPU cluster."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or attach existing AmlCompute\n",
|
"## Create or attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current AmlCompute. \n",
|
"# use get_status() to get a detailed status for the current AmlCompute. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute\n",
|
"## Train model on the remote compute\n",
|
||||||
"Now that we have the AmlCompute ready to go, let's run our distributed training job."
|
"Now that we have the AmlCompute ready to go, let's run our distributed training job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './chainer-distr'\n",
|
"project_folder = './chainer-distr'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Prepare training script\n",
|
"### Prepare training script\n",
|
||||||
"Now you will need to create your training script. In this tutorial, the script for distributed training of MNIST is already provided for you at `train_mnist.py`. In practice, you should be able to take any custom Chainer training script as is and run it with Azure ML without having to modify your code."
|
"Now you will need to create your training script. In this tutorial, the script for distributed training of MNIST is already provided for you at `train_mnist.py`. In practice, you should be able to take any custom Chainer training script as is and run it with Azure ML without having to modify your code."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Once your script is ready, copy the training script `train_mnist.py` into the project directory."
|
"Once your script is ready, copy the training script `train_mnist.py` into the project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('train_mnist.py', project_folder)"
|
"shutil.copy('train_mnist.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed Chainer tutorial. "
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed Chainer tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'chainer-distr'\n",
|
"experiment_name = 'chainer-distr'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a Chainer estimator\n",
|
"### Create a Chainer estimator\n",
|
||||||
"The Azure ML SDK's Chainer estimator enables you to easily submit Chainer training jobs for both single-node and distributed runs."
|
"The Azure ML SDK's Chainer estimator enables you to easily submit Chainer training jobs for both single-node and distributed runs."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.runconfig import MpiConfiguration\n",
|
"from azureml.core.runconfig import MpiConfiguration\n",
|
||||||
"from azureml.train.dnn import Chainer\n",
|
"from azureml.train.dnn import Chainer\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = Chainer(source_directory=project_folder,\n",
|
"estimator = Chainer(source_directory=project_folder,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" entry_script='train_mnist.py',\n",
|
" entry_script='train_mnist.py',\n",
|
||||||
" node_count=2,\n",
|
" node_count=2,\n",
|
||||||
" distributed_training=MpiConfiguration(),\n",
|
" distributed_training=MpiConfiguration(),\n",
|
||||||
" use_gpu=True)"
|
" use_gpu=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, Chainer and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `Chainer` constructor's `pip_packages` or `conda_packages` parameters."
|
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, Chainer and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `Chainer` constructor's `pip_packages` or `conda_packages` parameters."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)\n",
|
"run = experiment.submit(estimator)\n",
|
||||||
"print(run)"
|
"print(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. You can see that the widget automatically plots and visualizes the loss metric that we logged to the Azure ML run."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. You can see that the widget automatically plots and visualizes the loss metric that we logged to the Azure ML run."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "minxia"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "minxia"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
},
|
||||||
|
"msauthor": "minxia"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
},
|
|
||||||
"msauthor": "minxia"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,401 +1,401 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Distributed CNTK using custom docker images\n",
|
"# Distributed CNTK using custom docker images\n",
|
||||||
"In this tutorial, you will train a CNTK model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using a custom docker image and distributed training."
|
"In this tutorial, you will train a CNTK model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using a custom docker image and distributed training."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (`config.json`)"
|
" * create a workspace and its configuration file (`config.json`)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name,\n",
|
"print('Workspace name: ' + ws.name,\n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or Attach existing AmlCompute\n",
|
"## Create or Attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current AmlCompute\n",
|
"# use get_status() to get a detailed status for the current AmlCompute\n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Upload training data\n",
|
"## Upload training data\n",
|
||||||
"For this tutorial, we will be using the MNIST dataset.\n",
|
"For this tutorial, we will be using the MNIST dataset.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"First, let's download the dataset. We've included the `install_mnist.py` script to download the data and convert it to a CNTK-supported format. Our data files will get written to a directory named `'mnist'`."
|
"First, let's download the dataset. We've included the `install_mnist.py` script to download the data and convert it to a CNTK-supported format. Our data files will get written to a directory named `'mnist'`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import install_mnist\n",
|
"import install_mnist\n",
|
||||||
"\n",
|
"\n",
|
||||||
"install_mnist.main('mnist')"
|
"install_mnist.main('mnist')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"To make the data accessible for remote training, you will need to upload the data from your local machine to the cloud. AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data, and interact with it from your remote compute targets. \n",
|
"To make the data accessible for remote training, you will need to upload the data from your local machine to the cloud. AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data, and interact with it from your remote compute targets. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Each workspace is associated with a default datastore. In this tutorial, we will upload the training data to this default datastore, which we will then mount on the remote compute for training in the next section."
|
"Each workspace is associated with a default datastore. In this tutorial, we will upload the training data to this default datastore, which we will then mount on the remote compute for training in the next section."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ds = ws.get_default_datastore()\n",
|
"ds = ws.get_default_datastore()\n",
|
||||||
"print(ds.datastore_type, ds.account_name, ds.container_name)"
|
"print(ds.datastore_type, ds.account_name, ds.container_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The following code will upload the training data to the path `./mnist` on the default datastore."
|
"The following code will upload the training data to the path `./mnist` on the default datastore."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ds.upload(src_dir='./mnist', target_path='./mnist')"
|
"ds.upload(src_dir='./mnist', target_path='./mnist')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Now let's get a reference to the path on the datastore with the training data. We can do so using the `path` method. In the next section, we can then pass this reference to our training script's `--data_dir` argument. "
|
"Now let's get a reference to the path on the datastore with the training data. We can do so using the `path` method. In the next section, we can then pass this reference to our training script's `--data_dir` argument. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"path_on_datastore = 'mnist'\n",
|
"path_on_datastore = 'mnist'\n",
|
||||||
"ds_data = ds.path(path_on_datastore)\n",
|
"ds_data = ds.path(path_on_datastore)\n",
|
||||||
"print(ds_data)"
|
"print(ds_data)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute\n",
|
"## Train model on the remote compute\n",
|
||||||
"Now that we have the cluster ready to go, let's run our distributed training job."
|
"Now that we have the cluster ready to go, let's run our distributed training job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './cntk-distr'\n",
|
"project_folder = './cntk-distr'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copy the training script `cntk_distr_mnist.py` into this project directory."
|
"Copy the training script `cntk_distr_mnist.py` into this project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('cntk_distr_mnist.py', project_folder)"
|
"shutil.copy('cntk_distr_mnist.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed CNTK tutorial. "
|
"Create an [experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed CNTK tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'cntk-distr'\n",
|
"experiment_name = 'cntk-distr'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an Estimator\n",
|
"### Create an Estimator\n",
|
||||||
"The AML SDK's base Estimator enables you to easily submit custom scripts for both single-node and distributed runs. You should this generic estimator for training code using frameworks such as sklearn or CNTK that don't have corresponding custom estimators. For more information on using the generic estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-ml-models)."
|
"The AML SDK's base Estimator enables you to easily submit custom scripts for both single-node and distributed runs. You should this generic estimator for training code using frameworks such as sklearn or CNTK that don't have corresponding custom estimators. For more information on using the generic estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-ml-models)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.train.estimator import Estimator\n",
|
"from azureml.train.estimator import Estimator\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_params = {\n",
|
"script_params = {\n",
|
||||||
" '--num_epochs': 20,\n",
|
" '--num_epochs': 20,\n",
|
||||||
" '--data_dir': ds_data.as_mount(),\n",
|
" '--data_dir': ds_data.as_mount(),\n",
|
||||||
" '--output_dir': './outputs'\n",
|
" '--output_dir': './outputs'\n",
|
||||||
"}\n",
|
"}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = Estimator(source_directory=project_folder,\n",
|
"estimator = Estimator(source_directory=project_folder,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" entry_script='cntk_distr_mnist.py',\n",
|
" entry_script='cntk_distr_mnist.py',\n",
|
||||||
" script_params=script_params,\n",
|
" script_params=script_params,\n",
|
||||||
" node_count=2,\n",
|
" node_count=2,\n",
|
||||||
" process_count_per_node=1,\n",
|
" process_count_per_node=1,\n",
|
||||||
" distributed_backend='mpi',\n",
|
" distributed_backend='mpi',\n",
|
||||||
" pip_packages=['cntk-gpu==2.6'],\n",
|
" pip_packages=['cntk-gpu==2.6'],\n",
|
||||||
" custom_docker_image='microsoft/mmlspark:gpu-0.12',\n",
|
" custom_docker_image='microsoft/mmlspark:gpu-0.12',\n",
|
||||||
" use_gpu=True)"
|
" use_gpu=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"We would like to train our model using a [pre-built Docker container](https://hub.docker.com/r/microsoft/mmlspark/). To do so, specify the name of the docker image to the argument `custom_docker_image`. Finally, we provide the `cntk` package to `pip_packages` to install CNTK 2.6 on our custom image.\n",
|
"We would like to train our model using a [pre-built Docker container](https://hub.docker.com/r/microsoft/mmlspark/). To do so, specify the name of the docker image to the argument `custom_docker_image`. Finally, we provide the `cntk` package to `pip_packages` to install CNTK 2.6 on our custom image.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to run distributed CNTK, which uses MPI, you must provide the argument `distributed_backend='mpi'`."
|
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to run distributed CNTK, which uses MPI, you must provide the argument `distributed_backend='mpi'`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)\n",
|
"run = experiment.submit(estimator)\n",
|
||||||
"print(run)"
|
"print(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Alternatively, you can block until the script has completed training before running more code."
|
"Alternatively, you can block until the script has completed training before running more code."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "minxia"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "minxia"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,342 +1,342 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Distributed PyTorch with Horovod\n",
|
"# Distributed PyTorch with Horovod\n",
|
||||||
"In this tutorial, you will train a PyTorch model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using distributed training via [Horovod](https://github.com/uber/horovod) across a GPU cluster."
|
"In this tutorial, you will train a PyTorch model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using distributed training via [Horovod](https://github.com/uber/horovod) across a GPU cluster."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`\n",
|
||||||
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) on single-node PyTorch training using Azure Machine Learning"
|
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) on single-node PyTorch training using Azure Machine Learning"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or attach existing AmlCompute\n",
|
"## Create or attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current AmlCompute. \n",
|
"# use get_status() to get a detailed status for the current AmlCompute. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute\n",
|
"## Train model on the remote compute\n",
|
||||||
"Now that we have the AmlCompute ready to go, let's run our distributed training job."
|
"Now that we have the AmlCompute ready to go, let's run our distributed training job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './pytorch-distr-hvd'\n",
|
"project_folder = './pytorch-distr-hvd'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Prepare training script\n",
|
"### Prepare training script\n",
|
||||||
"Now you will need to create your training script. In this tutorial, the script for distributed training of MNIST is already provided for you at `pytorch_horovod_mnist.py`. In practice, you should be able to take any custom PyTorch training script as is and run it with Azure ML without having to modify your code.\n",
|
"Now you will need to create your training script. In this tutorial, the script for distributed training of MNIST is already provided for you at `pytorch_horovod_mnist.py`. In practice, you should be able to take any custom PyTorch training script as is and run it with Azure ML without having to modify your code.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"However, if you would like to use Azure ML's [metric logging](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#logging) capabilities, you will have to add a small amount of Azure ML logic inside your training script. In this example, at each logging interval, we will log the loss for that minibatch to our Azure ML run.\n",
|
"However, if you would like to use Azure ML's [metric logging](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#logging) capabilities, you will have to add a small amount of Azure ML logic inside your training script. In this example, at each logging interval, we will log the loss for that minibatch to our Azure ML run.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"To do so, in `pytorch_horovod_mnist.py`, we will first access the Azure ML `Run` object within the script:\n",
|
"To do so, in `pytorch_horovod_mnist.py`, we will first access the Azure ML `Run` object within the script:\n",
|
||||||
"```Python\n",
|
"```Python\n",
|
||||||
"from azureml.core.run import Run\n",
|
"from azureml.core.run import Run\n",
|
||||||
"run = Run.get_context()\n",
|
"run = Run.get_context()\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"Later within the script, we log the loss metric to our run:\n",
|
"Later within the script, we log the loss metric to our run:\n",
|
||||||
"```Python\n",
|
"```Python\n",
|
||||||
"run.log('loss', loss.item())\n",
|
"run.log('loss', loss.item())\n",
|
||||||
"```"
|
"```"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Once your script is ready, copy the training script `pytorch_horovod_mnist.py` into the project directory."
|
"Once your script is ready, copy the training script `pytorch_horovod_mnist.py` into the project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('pytorch_horovod_mnist.py', project_folder)"
|
"shutil.copy('pytorch_horovod_mnist.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed PyTorch tutorial. "
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed PyTorch tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'pytorch-distr-hvd'\n",
|
"experiment_name = 'pytorch-distr-hvd'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a PyTorch estimator\n",
|
"### Create a PyTorch estimator\n",
|
||||||
"The Azure ML SDK's PyTorch estimator enables you to easily submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch)."
|
"The Azure ML SDK's PyTorch estimator enables you to easily submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.runconfig import MpiConfiguration\n",
|
"from azureml.core.runconfig import MpiConfiguration\n",
|
||||||
"from azureml.train.dnn import PyTorch\n",
|
"from azureml.train.dnn import PyTorch\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = PyTorch(source_directory=project_folder,\n",
|
"estimator = PyTorch(source_directory=project_folder,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" entry_script='pytorch_horovod_mnist.py',\n",
|
" entry_script='pytorch_horovod_mnist.py',\n",
|
||||||
" node_count=2,\n",
|
" node_count=2,\n",
|
||||||
" distributed_training=MpiConfiguration(),\n",
|
" distributed_training=MpiConfiguration(),\n",
|
||||||
" use_gpu=True)"
|
" use_gpu=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, PyTorch, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `PyTorch` constructor's `pip_packages` or `conda_packages` parameters."
|
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, PyTorch, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `PyTorch` constructor's `pip_packages` or `conda_packages` parameters."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)\n",
|
"run = experiment.submit(estimator)\n",
|
||||||
"print(run)"
|
"print(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. You can see that the widget automatically plots and visualizes the loss metric that we logged to the Azure ML run."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. You can see that the widget automatically plots and visualizes the loss metric that we logged to the Azure ML run."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Alternatively, you can block until the script has completed training before running more code."
|
"Alternatively, you can block until the script has completed training before running more code."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True) # this provides a verbose log"
|
"run.wait_for_completion(show_output=True) # this provides a verbose log"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "minxia"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "minxia"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
},
|
||||||
|
"msauthor": "minxia"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
},
|
|
||||||
"msauthor": "minxia"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,411 +1,411 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Distributed Tensorflow with Horovod\n",
|
"# Distributed Tensorflow with Horovod\n",
|
||||||
"In this tutorial, you will train a word2vec model in TensorFlow using distributed training via [Horovod](https://github.com/uber/horovod)."
|
"In this tutorial, you will train a word2vec model in TensorFlow using distributed training via [Horovod](https://github.com/uber/horovod)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (`config.json`)\n",
|
" * create a workspace and its configuration file (`config.json`)\n",
|
||||||
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
|
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or Attach existing AmlCompute\n",
|
"## Create or Attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target')\n",
|
" print('Found existing compute target')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current cluster. \n",
|
"# use get_status() to get a detailed status for the current cluster. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
"The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Upload data to datastore\n",
|
"## Upload data to datastore\n",
|
||||||
"To make data accessible for remote training, AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data to Azure Storage, and interact with it from your remote compute targets. \n",
|
"To make data accessible for remote training, AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data to Azure Storage, and interact with it from your remote compute targets. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"If your data is already stored in Azure, or you download the data as part of your training script, you will not need to do this step. For this tutorial, although you can download the data in your training script, we will demonstrate how to upload the training data to a datastore and access it during training to illustrate the datastore functionality."
|
"If your data is already stored in Azure, or you download the data as part of your training script, you will not need to do this step. For this tutorial, although you can download the data in your training script, we will demonstrate how to upload the training data to a datastore and access it during training to illustrate the datastore functionality."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"First, download the training data from [here](http://mattmahoney.net/dc/text8.zip) to your local machine:"
|
"First, download the training data from [here](http://mattmahoney.net/dc/text8.zip) to your local machine:"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"import urllib\n",
|
"import urllib\n",
|
||||||
"\n",
|
"\n",
|
||||||
"os.makedirs('./data', exist_ok=True)\n",
|
"os.makedirs('./data', exist_ok=True)\n",
|
||||||
"download_url = 'http://mattmahoney.net/dc/text8.zip'\n",
|
"download_url = 'http://mattmahoney.net/dc/text8.zip'\n",
|
||||||
"urllib.request.urlretrieve(download_url, filename='./data/text8.zip')"
|
"urllib.request.urlretrieve(download_url, filename='./data/text8.zip')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Each workspace is associated with a default datastore. In this tutorial, we will upload the training data to this default datastore."
|
"Each workspace is associated with a default datastore. In this tutorial, we will upload the training data to this default datastore."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ds = ws.get_default_datastore()\n",
|
"ds = ws.get_default_datastore()\n",
|
||||||
"print(ds.datastore_type, ds.account_name, ds.container_name)"
|
"print(ds.datastore_type, ds.account_name, ds.container_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Upload the contents of the data directory to the path `./data` on the default datastore."
|
"Upload the contents of the data directory to the path `./data` on the default datastore."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"ds.upload(src_dir='data', target_path='data', overwrite=True, show_progress=True)"
|
"ds.upload(src_dir='data', target_path='data', overwrite=True, show_progress=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"For convenience, let's get a reference to the path on the datastore with the zip file of training data. We can do so using the `path` method. In the next section, we can then pass this reference to our training script's `--input_data` argument. "
|
"For convenience, let's get a reference to the path on the datastore with the zip file of training data. We can do so using the `path` method. In the next section, we can then pass this reference to our training script's `--input_data` argument. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"path_on_datastore = 'data/text8.zip'\n",
|
"path_on_datastore = 'data/text8.zip'\n",
|
||||||
"ds_data = ds.path(path_on_datastore)\n",
|
"ds_data = ds.path(path_on_datastore)\n",
|
||||||
"print(ds_data)"
|
"print(ds_data)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute"
|
"## Train model on the remote compute"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"project_folder = './tf-distr-hvd'\n",
|
"project_folder = './tf-distr-hvd'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copy the training script `tf_horovod_word2vec.py` into this project directory."
|
"Copy the training script `tf_horovod_word2vec.py` into this project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('tf_horovod_word2vec.py', project_folder)"
|
"shutil.copy('tf_horovod_word2vec.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed TensorFlow tutorial. "
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed TensorFlow tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'tf-distr-hvd'\n",
|
"experiment_name = 'tf-distr-hvd'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a TensorFlow estimator\n",
|
"### Create a TensorFlow estimator\n",
|
||||||
"The AML SDK's TensorFlow estimator enables you to easily submit TensorFlow training jobs for both single-node and distributed runs. For more information on the TensorFlow estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-tensorflow).\n",
|
"The AML SDK's TensorFlow estimator enables you to easily submit TensorFlow training jobs for both single-node and distributed runs. For more information on the TensorFlow estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-tensorflow).\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The TensorFlow estimator also takes a `framework_version` parameter -- if no version is provided, the estimator will default to the latest version supported by AzureML. Use `TensorFlow.get_supported_versions()` to get a list of all versions supported by your current SDK version or see the [SDK documentation](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn?view=azure-ml-py) for the versions supported in the most current release."
|
"The TensorFlow estimator also takes a `framework_version` parameter -- if no version is provided, the estimator will default to the latest version supported by AzureML. Use `TensorFlow.get_supported_versions()` to get a list of all versions supported by your current SDK version or see the [SDK documentation](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn?view=azure-ml-py) for the versions supported in the most current release."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.runconfig import MpiConfiguration\n",
|
"from azureml.core.runconfig import MpiConfiguration\n",
|
||||||
"from azureml.train.dnn import TensorFlow\n",
|
"from azureml.train.dnn import TensorFlow\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_params={\n",
|
"script_params={\n",
|
||||||
" '--input_data': ds_data\n",
|
" '--input_data': ds_data\n",
|
||||||
"}\n",
|
"}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator= TensorFlow(source_directory=project_folder,\n",
|
"estimator= TensorFlow(source_directory=project_folder,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" script_params=script_params,\n",
|
" script_params=script_params,\n",
|
||||||
" entry_script='tf_horovod_word2vec.py',\n",
|
" entry_script='tf_horovod_word2vec.py',\n",
|
||||||
" node_count=2,\n",
|
" node_count=2,\n",
|
||||||
" distributed_training=MpiConfiguration(),\n",
|
" distributed_training=MpiConfiguration(),\n",
|
||||||
" framework_version='1.13')"
|
" framework_version='1.13')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, TensorFlow, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters.\n",
|
"The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, TensorFlow, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Note that we passed our training data reference `ds_data` to our script's `--input_data` argument. This will 1) mount our datastore on the remote compute and 2) provide the path to the data zip file on our datastore."
|
"Note that we passed our training data reference `ds_data` to our script's `--input_data` argument. This will 1) mount our datastore on the remote compute and 2) provide the path to the data zip file on our datastore."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)\n",
|
"run = experiment.submit(estimator)\n",
|
||||||
"print(run)"
|
"print(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Alternatively, you can block until the script has completed training before running more code."
|
"Alternatively, you can block until the script has completed training before running more code."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "roastala"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "roastala"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
},
|
||||||
|
"msauthor": "minxia"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
},
|
|
||||||
"msauthor": "minxia"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,326 +1,326 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Distributed TensorFlow with parameter server\n",
|
"# Distributed TensorFlow with parameter server\n",
|
||||||
"In this tutorial, you will train a TensorFlow model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using native [distributed TensorFlow](https://www.tensorflow.org/deploy/distributed)."
|
"In this tutorial, you will train a TensorFlow model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using native [distributed TensorFlow](https://www.tensorflow.org/deploy/distributed)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (`config.json`)\n",
|
" * create a workspace and its configuration file (`config.json`)\n",
|
||||||
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
|
"* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or Attach existing AmlCompute\n",
|
"## Create or Attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current cluster. \n",
|
"# use get_status() to get a detailed status for the current cluster. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute\n",
|
"## Train model on the remote compute\n",
|
||||||
"Now that we have the cluster ready to go, let's run our distributed training job."
|
"Now that we have the cluster ready to go, let's run our distributed training job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './tf-distr-ps'\n",
|
"project_folder = './tf-distr-ps'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copy the training script `tf_mnist_replica.py` into this project directory."
|
"Copy the training script `tf_mnist_replica.py` into this project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('tf_mnist_replica.py', project_folder)"
|
"shutil.copy('tf_mnist_replica.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed TensorFlow tutorial. "
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed TensorFlow tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'tf-distr-ps'\n",
|
"experiment_name = 'tf-distr-ps'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a TensorFlow estimator\n",
|
"### Create a TensorFlow estimator\n",
|
||||||
"The AML SDK's TensorFlow estimator enables you to easily submit TensorFlow training jobs for both single-node and distributed runs. For more information on the TensorFlow estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-tensorflow)."
|
"The AML SDK's TensorFlow estimator enables you to easily submit TensorFlow training jobs for both single-node and distributed runs. For more information on the TensorFlow estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-tensorflow)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.runconfig import TensorflowConfiguration\n",
|
"from azureml.core.runconfig import TensorflowConfiguration\n",
|
||||||
"from azureml.train.dnn import TensorFlow\n",
|
"from azureml.train.dnn import TensorFlow\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_params={\n",
|
"script_params={\n",
|
||||||
" '--num_gpus': 1,\n",
|
" '--num_gpus': 1,\n",
|
||||||
" '--train_steps': 500\n",
|
" '--train_steps': 500\n",
|
||||||
"}\n",
|
"}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"distributed_training = TensorflowConfiguration()\n",
|
"distributed_training = TensorflowConfiguration()\n",
|
||||||
"distributed_training.worker_count = 2\n",
|
"distributed_training.worker_count = 2\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = TensorFlow(source_directory=project_folder,\n",
|
"estimator = TensorFlow(source_directory=project_folder,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" script_params=script_params,\n",
|
" script_params=script_params,\n",
|
||||||
" entry_script='tf_mnist_replica.py',\n",
|
" entry_script='tf_mnist_replica.py',\n",
|
||||||
" node_count=2,\n",
|
" node_count=2,\n",
|
||||||
" distributed_training=distributed_training,\n",
|
" distributed_training=distributed_training,\n",
|
||||||
" use_gpu=True)"
|
" use_gpu=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code specifies that we will run our training script on `2` nodes, with two workers and one parameter server. In order to execute a native distributed TensorFlow run, you must provide the argument `distributed_backend='ps'`. Using this estimator with these settings, TensorFlow and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters."
|
"The above code specifies that we will run our training script on `2` nodes, with two workers and one parameter server. In order to execute a native distributed TensorFlow run, you must provide the argument `distributed_backend='ps'`. Using this estimator with these settings, TensorFlow and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)\n",
|
"run = experiment.submit(estimator)\n",
|
||||||
"print(run)"
|
"print(run)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Alternatively, you can block until the script has completed training before running more code."
|
"Alternatively, you can block until the script has completed training before running more code."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True) # this provides a verbose log"
|
"run.wait_for_completion(show_output=True) # this provides a verbose log"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "minxia"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "minxia"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
},
|
||||||
|
"msauthor": "minxia"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
},
|
|
||||||
"msauthor": "minxia"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,256 +1,256 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Export Run History as Tensorboard logs\n",
|
"# Export Run History as Tensorboard logs\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. Run some training and log some metrics into Run History\n",
|
"1. Run some training and log some metrics into Run History\n",
|
||||||
"2. Export the run history to some directory as Tensorboard logs\n",
|
"2. Export the run history to some directory as Tensorboard logs\n",
|
||||||
"3. Launch a local Tensorboard to view the run history"
|
"3. Launch a local Tensorboard to view the run history"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
"* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
|
||||||
" * install the AML SDK\n",
|
" * install the AML SDK\n",
|
||||||
" * create a workspace and its configuration file (`config.json`)"
|
" * create a workspace and its configuration file (`config.json`)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize Workspace\n",
|
"## Initialize Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a workspace object from persisted configuration."
|
"Initialize a workspace object from persisted configuration."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace, Experiment\n",
|
"from azureml.core import Workspace, Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
" 'Resource group: ' + ws.resource_group, sep='\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Set experiment name and start the run"
|
"## Set experiment name and start the run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"experiment_name = 'export-to-tensorboard'\n",
|
"experiment_name = 'export-to-tensorboard'\n",
|
||||||
"exp = Experiment(ws, experiment_name)\n",
|
"exp = Experiment(ws, experiment_name)\n",
|
||||||
"root_run = exp.start_logging()"
|
"root_run = exp.start_logging()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# load diabetes dataset, a well-known built-in small dataset that comes with scikit-learn\n",
|
"# load diabetes dataset, a well-known built-in small dataset that comes with scikit-learn\n",
|
||||||
"from sklearn.datasets import load_diabetes\n",
|
"from sklearn.datasets import load_diabetes\n",
|
||||||
"from sklearn.linear_model import Ridge\n",
|
"from sklearn.linear_model import Ridge\n",
|
||||||
"from sklearn.metrics import mean_squared_error\n",
|
"from sklearn.metrics import mean_squared_error\n",
|
||||||
"from sklearn.model_selection import train_test_split\n",
|
"from sklearn.model_selection import train_test_split\n",
|
||||||
"\n",
|
"\n",
|
||||||
"X, y = load_diabetes(return_X_y=True)\n",
|
"X, y = load_diabetes(return_X_y=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"columns = ['age', 'gender', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']\n",
|
"columns = ['age', 'gender', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']\n",
|
||||||
"\n",
|
"\n",
|
||||||
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
|
"x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
|
||||||
"data = {\n",
|
"data = {\n",
|
||||||
" \"train\":{\"x\":x_train, \"y\":y_train}, \n",
|
" \"train\":{\"x\":x_train, \"y\":y_train}, \n",
|
||||||
" \"test\":{\"x\":x_test, \"y\":y_test}\n",
|
" \"test\":{\"x\":x_test, \"y\":y_test}\n",
|
||||||
"}"
|
"}"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Example experiment\n",
|
"# Example experiment\n",
|
||||||
"from tqdm import tqdm\n",
|
"from tqdm import tqdm\n",
|
||||||
"\n",
|
"\n",
|
||||||
"alphas = [.1, .2, .3, .4, .5, .6 , .7]\n",
|
"alphas = [.1, .2, .3, .4, .5, .6 , .7]\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# try a bunch of alpha values in a Linear Regression (Ridge) model\n",
|
"# try a bunch of alpha values in a Linear Regression (Ridge) model\n",
|
||||||
"for alpha in tqdm(alphas):\n",
|
"for alpha in tqdm(alphas):\n",
|
||||||
" # create a bunch of child runs\n",
|
" # create a bunch of child runs\n",
|
||||||
" with root_run.child_run(\"alpha\" + str(alpha)) as run:\n",
|
" with root_run.child_run(\"alpha\" + str(alpha)) as run:\n",
|
||||||
" # More data science stuff\n",
|
" # More data science stuff\n",
|
||||||
" reg = Ridge(alpha=alpha)\n",
|
" reg = Ridge(alpha=alpha)\n",
|
||||||
" reg.fit(data[\"train\"][\"x\"], data[\"train\"][\"y\"])\n",
|
" reg.fit(data[\"train\"][\"x\"], data[\"train\"][\"y\"])\n",
|
||||||
" \n",
|
" \n",
|
||||||
" preds = reg.predict(data[\"test\"][\"x\"])\n",
|
" preds = reg.predict(data[\"test\"][\"x\"])\n",
|
||||||
" mse = mean_squared_error(preds, data[\"test\"][\"y\"])\n",
|
" mse = mean_squared_error(preds, data[\"test\"][\"y\"])\n",
|
||||||
" # End train and eval\n",
|
" # End train and eval\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # log alpha, mean_squared_error and feature names in run history\n",
|
" # log alpha, mean_squared_error and feature names in run history\n",
|
||||||
" root_run.log(\"alpha\", alpha)\n",
|
" root_run.log(\"alpha\", alpha)\n",
|
||||||
" root_run.log(\"mse\", mse)"
|
" root_run.log(\"mse\", mse)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Export Run History to Tensorboard logs"
|
"## Export Run History to Tensorboard logs"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Export Run History to Tensorboard logs\n",
|
"# Export Run History to Tensorboard logs\n",
|
||||||
"from azureml.tensorboard.export import export_to_tensorboard\n",
|
"from azureml.tensorboard.export import export_to_tensorboard\n",
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"logdir = 'exportedTBlogs'\n",
|
"logdir = 'exportedTBlogs'\n",
|
||||||
"log_path = os.path.join(os.getcwd(), logdir)\n",
|
"log_path = os.path.join(os.getcwd(), logdir)\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" os.stat(log_path)\n",
|
" os.stat(log_path)\n",
|
||||||
"except os.error:\n",
|
"except os.error:\n",
|
||||||
" os.mkdir(log_path)\n",
|
" os.mkdir(log_path)\n",
|
||||||
"print(logdir)\n",
|
"print(logdir)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# export run history for the project\n",
|
"# export run history for the project\n",
|
||||||
"export_to_tensorboard(root_run, logdir)\n",
|
"export_to_tensorboard(root_run, logdir)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# or export a particular run\n",
|
"# or export a particular run\n",
|
||||||
"# export_to_tensorboard(run, logdir)"
|
"# export_to_tensorboard(run, logdir)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"root_run.complete()"
|
"root_run.complete()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Start Tensorboard\n",
|
"## Start Tensorboard\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Or you can start the Tensorboard outside this notebook to view the result"
|
"Or you can start the Tensorboard outside this notebook to view the result"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.tensorboard import Tensorboard\n",
|
"from azureml.tensorboard import Tensorboard\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# The Tensorboard constructor takes an array of runs, so be sure and pass it in as a single-element array here\n",
|
"# The Tensorboard constructor takes an array of runs, so be sure and pass it in as a single-element array here\n",
|
||||||
"tb = Tensorboard([], local_root=logdir, port=6006)\n",
|
"tb = Tensorboard([], local_root=logdir, port=6006)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# If successful, start() returns a string with the URI of the instance.\n",
|
"# If successful, start() returns a string with the URI of the instance.\n",
|
||||||
"tb.start()"
|
"tb.start()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Stop Tensorboard\n",
|
"## Stop Tensorboard\n",
|
||||||
"\n",
|
"\n",
|
||||||
"When you're done, make sure to call the `stop()` method of the Tensorboard object."
|
"When you're done, make sure to call the `stop()` method of the Tensorboard object."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"tb.stop()"
|
"tb.stop()"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "roastala"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "roastala"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.5"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,57 +1,57 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import scrapbook as sb\n",
|
"import scrapbook as sb\n",
|
||||||
"sb.glue('Fibonacci numbers', [0, 1, 1, 2, 3, 5, 8, 13, 21, 34])"
|
"sb.glue('Fibonacci numbers', [0, 1, 1, 2, 3, 5, 8, 13, 21, 34])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": []
|
"source": []
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "jingywa"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "jingywa"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.8"
|
||||||
|
},
|
||||||
|
"msauthor": "jingywa"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.8"
|
|
||||||
},
|
|
||||||
"msauthor": "jingywa"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,432 +1,432 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
"Copyright (c) Microsoft Corporation. All rights reserved. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Train and hyperparameter tune with Chainer\n",
|
"# Train and hyperparameter tune with Chainer\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In this tutorial, we demonstrate how to use the Azure ML Python SDK to train a Convolutional Neural Network (CNN) on a single-node GPU with Chainer to perform handwritten digit recognition on the popular MNIST dataset. We will also demonstrate how to perform hyperparameter tuning of the model using Azure ML's HyperDrive service."
|
"In this tutorial, we demonstrate how to use the Azure ML Python SDK to train a Convolutional Neural Network (CNN) on a single-node GPU with Chainer to perform handwritten digit recognition on the popular MNIST dataset. We will also demonstrate how to perform hyperparameter tuning of the model using Azure ML's HyperDrive service."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
|
"* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics\n",
|
"## Diagnostics\n",
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"Diagnostics"
|
"Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace\n",
|
"## Initialize workspace\n",
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create or Attach existing AmlCompute\n",
|
"## Create or Attach existing AmlCompute\n",
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
"**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current cluster. \n",
|
"# use get_status() to get a detailed status for the current cluster. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
"The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute\n",
|
"## Train model on the remote compute\n",
|
||||||
"Now that you have your data and training script prepared, you are ready to train on your remote compute cluster. You can take advantage of Azure compute to leverage GPUs to cut down your training time. "
|
"Now that you have your data and training script prepared, you are ready to train on your remote compute cluster. You can take advantage of Azure compute to leverage GPUs to cut down your training time. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory\n",
|
"### Create a project directory\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './chainer-mnist'\n",
|
"project_folder = './chainer-mnist'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Prepare training script\n",
|
"### Prepare training script\n",
|
||||||
"Now you will need to create your training script. In this tutorial, the training script is already provided for you at `chainer_mnist.py`. In practice, you should be able to take any custom training script as is and run it with Azure ML without having to modify your code.\n",
|
"Now you will need to create your training script. In this tutorial, the training script is already provided for you at `chainer_mnist.py`. In practice, you should be able to take any custom training script as is and run it with Azure ML without having to modify your code.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"However, if you would like to use Azure ML's [tracking and metrics](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#metrics) capabilities, you will have to add a small amount of Azure ML code inside your training script. \n",
|
"However, if you would like to use Azure ML's [tracking and metrics](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#metrics) capabilities, you will have to add a small amount of Azure ML code inside your training script. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"In `chainer_mnist.py`, we will log some metrics to our Azure ML run. To do so, we will access the Azure ML `Run` object within the script:\n",
|
"In `chainer_mnist.py`, we will log some metrics to our Azure ML run. To do so, we will access the Azure ML `Run` object within the script:\n",
|
||||||
"```Python\n",
|
"```Python\n",
|
||||||
"from azureml.core.run import Run\n",
|
"from azureml.core.run import Run\n",
|
||||||
"run = Run.get_context()\n",
|
"run = Run.get_context()\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"Further within `chainer_mnist.py`, we log the batchsize and epochs parameters, and the highest accuracy the model achieves:\n",
|
"Further within `chainer_mnist.py`, we log the batchsize and epochs parameters, and the highest accuracy the model achieves:\n",
|
||||||
"```Python\n",
|
"```Python\n",
|
||||||
"run.log('Batch size', np.int(args.batchsize))\n",
|
"run.log('Batch size', np.int(args.batchsize))\n",
|
||||||
"run.log('Epochs', np.int(args.epochs))\n",
|
"run.log('Epochs', np.int(args.epochs))\n",
|
||||||
"\n",
|
"\n",
|
||||||
"run.log('Accuracy', np.float(val_accuracy))\n",
|
"run.log('Accuracy', np.float(val_accuracy))\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"These run metrics will become particularly important when we begin hyperparameter tuning our model in the \"Tune model hyperparameters\" section."
|
"These run metrics will become particularly important when we begin hyperparameter tuning our model in the \"Tune model hyperparameters\" section."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Once your script is ready, copy the training script `chainer_mnist.py` into your project directory."
|
"Once your script is ready, copy the training script `chainer_mnist.py` into your project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('chainer_mnist.py', project_folder)"
|
"shutil.copy('chainer_mnist.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment\n",
|
"### Create an experiment\n",
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this Chainer tutorial. "
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this Chainer tutorial. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'chainer-mnist'\n",
|
"experiment_name = 'chainer-mnist'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a Chainer estimator\n",
|
"### Create a Chainer estimator\n",
|
||||||
"The Azure ML SDK's Chainer estimator enables you to easily submit Chainer training jobs for both single-node and distributed runs. The following code will define a single-node Chainer job."
|
"The Azure ML SDK's Chainer estimator enables you to easily submit Chainer training jobs for both single-node and distributed runs. The following code will define a single-node Chainer job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.train.dnn import Chainer\n",
|
"from azureml.train.dnn import Chainer\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_params = {\n",
|
"script_params = {\n",
|
||||||
" '--epochs': 10,\n",
|
" '--epochs': 10,\n",
|
||||||
" '--batchsize': 128,\n",
|
" '--batchsize': 128,\n",
|
||||||
" '--output_dir': './outputs'\n",
|
" '--output_dir': './outputs'\n",
|
||||||
"}\n",
|
"}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = Chainer(source_directory=project_folder, \n",
|
"estimator = Chainer(source_directory=project_folder, \n",
|
||||||
" script_params=script_params,\n",
|
" script_params=script_params,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" pip_packages=['numpy', 'pytest'],\n",
|
" pip_packages=['numpy', 'pytest'],\n",
|
||||||
" entry_script='chainer_mnist.py',\n",
|
" entry_script='chainer_mnist.py',\n",
|
||||||
" use_gpu=True)"
|
" use_gpu=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The `script_params` parameter is a dictionary containing the command-line arguments to your training script `entry_script`. To leverage the Azure VM's GPU for training, we set `use_gpu=True`."
|
"The `script_params` parameter is a dictionary containing the command-line arguments to your training script `entry_script`. To leverage the Azure VM's GPU for training, we set `use_gpu=True`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job\n",
|
"### Submit job\n",
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)"
|
"run = experiment.submit(estimator)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor your run\n",
|
"### Monitor your run\n",
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# to get more details of your run\n",
|
"# to get more details of your run\n",
|
||||||
"print(run.get_details())"
|
"print(run.get_details())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Tune model hyperparameters\n",
|
"## Tune model hyperparameters\n",
|
||||||
"Now that we've seen how to do a simple Chainer training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's hyperparameter tuning capabilities."
|
"Now that we've seen how to do a simple Chainer training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's hyperparameter tuning capabilities."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Start a hyperparameter sweep\n",
|
"### Start a hyperparameter sweep\n",
|
||||||
"First, we will define the hyperparameter space to sweep over. Let's tune the batch size and epochs parameters. In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, accuracy.\n",
|
"First, we will define the hyperparameter space to sweep over. Let's tune the batch size and epochs parameters. In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, accuracy.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Then, we specify the early termination policy to use to early terminate poorly performing runs. Here we use the `BanditPolicy`, which will terminate any run that doesn't fall within the slack factor of our primary evaluation metric. In this tutorial, we will apply this policy every epoch (since we report our `Accuracy` metric every epoch and `evaluation_interval=1`). Notice we will delay the first policy evaluation until after the first `3` epochs (`delay_evaluation=3`).\n",
|
"Then, we specify the early termination policy to use to early terminate poorly performing runs. Here we use the `BanditPolicy`, which will terminate any run that doesn't fall within the slack factor of our primary evaluation metric. In this tutorial, we will apply this policy every epoch (since we report our `Accuracy` metric every epoch and `evaluation_interval=1`). Notice we will delay the first policy evaluation until after the first `3` epochs (`delay_evaluation=3`).\n",
|
||||||
"Refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-tune-hyperparameters#specify-an-early-termination-policy) for more information on the BanditPolicy and other policies available."
|
"Refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-tune-hyperparameters#specify-an-early-termination-policy) for more information on the BanditPolicy and other policies available."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.train.hyperdrive.runconfig import HyperDriveConfig\n",
|
"from azureml.train.hyperdrive.runconfig import HyperDriveConfig\n",
|
||||||
"from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
|
"from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
|
||||||
"from azureml.train.hyperdrive.policy import BanditPolicy\n",
|
"from azureml.train.hyperdrive.policy import BanditPolicy\n",
|
||||||
"from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
|
"from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
|
||||||
"from azureml.train.hyperdrive.parameter_expressions import choice\n",
|
"from azureml.train.hyperdrive.parameter_expressions import choice\n",
|
||||||
" \n",
|
" \n",
|
||||||
"\n",
|
"\n",
|
||||||
"param_sampling = RandomParameterSampling( {\n",
|
"param_sampling = RandomParameterSampling( {\n",
|
||||||
" \"--batchsize\": choice(128, 256),\n",
|
" \"--batchsize\": choice(128, 256),\n",
|
||||||
" \"--epochs\": choice(5, 10, 20, 40)\n",
|
" \"--epochs\": choice(5, 10, 20, 40)\n",
|
||||||
" }\n",
|
" }\n",
|
||||||
")\n",
|
")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"hyperdrive_config = HyperDriveConfig(estimator=estimator,\n",
|
"hyperdrive_config = HyperDriveConfig(estimator=estimator,\n",
|
||||||
" hyperparameter_sampling=param_sampling, \n",
|
" hyperparameter_sampling=param_sampling, \n",
|
||||||
" primary_metric_name='Accuracy',\n",
|
" primary_metric_name='Accuracy',\n",
|
||||||
" primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
|
" primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
|
||||||
" max_total_runs=8,\n",
|
" max_total_runs=8,\n",
|
||||||
" max_concurrent_runs=4)\n"
|
" max_concurrent_runs=4)\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Finally, lauch the hyperparameter tuning job."
|
"Finally, lauch the hyperparameter tuning job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# start the HyperDrive run\n",
|
"# start the HyperDrive run\n",
|
||||||
"hyperdrive_run = experiment.submit(hyperdrive_config)"
|
"hyperdrive_run = experiment.submit(hyperdrive_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Monitor HyperDrive runs\n",
|
"### Monitor HyperDrive runs\n",
|
||||||
"You can monitor the progress of the runs with the following Jupyter widget. "
|
"You can monitor the progress of the runs with the following Jupyter widget. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"RunDetails(hyperdrive_run).show()"
|
"RunDetails(hyperdrive_run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "minxia"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "minxia"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
},
|
||||||
|
"msauthor": "minxia"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
},
|
|
||||||
"msauthor": "minxia"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -5,10 +5,9 @@ Follow these sample notebooks to learn:
|
|||||||
1. [Train within notebook](train-within-notebook): train a simple scikit-learn model using the Jupyter kernel and deploy the model to Azure Container Service.
|
1. [Train within notebook](train-within-notebook): train a simple scikit-learn model using the Jupyter kernel and deploy the model to Azure Container Service.
|
||||||
2. [Train on local](train-on-local): train a model using local computer as compute target.
|
2. [Train on local](train-on-local): train a model using local computer as compute target.
|
||||||
3. [Train on remote VM](train-on-remote-vm): train a model using a remote Azure VM as compute target.
|
3. [Train on remote VM](train-on-remote-vm): train a model using a remote Azure VM as compute target.
|
||||||
4. [Train on ML Compute](train-on-amlcompute): train a model using an ML Compute cluster as compute target.
|
4. [Train on AmlCompute](train-on-amlcompute): train a model using an AmlCompute cluster as compute target.
|
||||||
5. [Train in an HDI Spark cluster](train-in-spark): train a Spark ML model using an HDInsight Spark cluster as compute target.
|
5. [Train in an HDI Spark cluster](train-in-spark): train a Spark ML model using an HDInsight Spark cluster as compute target.
|
||||||
6. [Logging API](logging-api): experiment with various logging functions to create runs and automatically generate graphs.
|
6. [Logging API](logging-api): experiment with various logging functions to create runs and automatically generate graphs.
|
||||||
7. [Manage runs](manage-runs): learn different ways how to start runs and child runs, monitor them, and cancel them.
|
7. [Train and hyperparameter tune on Iris Dataset with Scikit-learn](train-hyperparameter-tune-deploy-with-sklearn): train a model using the Scikit-learn estimator and tune hyperparameters with Hyperdrive.
|
||||||
8. [Train and hyperparameter tune on Iris Dataset with Scikit-learn](train-hyperparameter-tune-deploy-with-sklearn): train a model using the Scikit-learn estimator and tune hyperparameters with Hyperdrive.
|
|
||||||
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,501 +1,501 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Train and hyperparameter tune on Iris Dataset with Scikit-learn\n",
|
"# Train and hyperparameter tune on Iris Dataset with Scikit-learn\n",
|
||||||
"In this tutorial, we demonstrate how to use the Azure ML Python SDK to train a support vector machine (SVM) on a single-node CPU with Scikit-learn to perform classification on the popular [Iris dataset](https://archive.ics.uci.edu/ml/datasets/iris). We will also demonstrate how to perform hyperparameter tuning of the model using Azure ML's HyperDrive service."
|
"In this tutorial, we demonstrate how to use the Azure ML Python SDK to train a support vector machine (SVM) on a single-node CPU with Scikit-learn to perform classification on the popular [Iris dataset](https://archive.ics.uci.edu/ml/datasets/iris). We will also demonstrate how to perform hyperparameter tuning of the model using Azure ML's HyperDrive service."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites"
|
"## Prerequisites"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML Workspace"
|
"* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML Workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Diagnostics"
|
"## Diagnostics"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
"Opt-in diagnostics for better experience, quality, and security of future releases."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.telemetry import set_diagnostics_collection\n",
|
"from azureml.telemetry import set_diagnostics_collection\n",
|
||||||
"\n",
|
"\n",
|
||||||
"set_diagnostics_collection(send_diagnostics=True)"
|
"set_diagnostics_collection(send_diagnostics=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize workspace"
|
"## Initialize workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
"Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print('Workspace name: ' + ws.name, \n",
|
"print('Workspace name: ' + ws.name, \n",
|
||||||
" 'Azure region: ' + ws.location, \n",
|
" 'Azure region: ' + ws.location, \n",
|
||||||
" 'Subscription id: ' + ws.subscription_id, \n",
|
" 'Subscription id: ' + ws.subscription_id, \n",
|
||||||
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create AmlCompute"
|
"## Create AmlCompute"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
|
"You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
"As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# choose a name for your cluster\n",
|
"# choose a name for your cluster\n",
|
||||||
"cluster_name = \"gpu-cluster\"\n",
|
"cluster_name = \"gpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
" compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
|
||||||
" print('Found existing compute target.')\n",
|
" print('Found existing compute target.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" print('Creating a new compute target...')\n",
|
" print('Creating a new compute target...')\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" # create the cluster\n",
|
" # create the cluster\n",
|
||||||
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
" compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
" compute_target.wait_for_completion(show_output=True)\n",
|
" compute_target.wait_for_completion(show_output=True)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use get_status() to get a detailed status for the current cluster. \n",
|
"# use get_status() to get a detailed status for the current cluster. \n",
|
||||||
"print(compute_target.get_status().serialize())"
|
"print(compute_target.get_status().serialize())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
"The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Train model on the remote compute"
|
"## Train model on the remote compute"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Now that you have your data and training script prepared, you are ready to train on your remote compute cluster. You can take advantage of Azure compute to leverage GPUs to cut down your training time."
|
"Now that you have your data and training script prepared, you are ready to train on your remote compute cluster. You can take advantage of Azure compute to leverage GPUs to cut down your training time."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a project directory"
|
"### Create a project directory"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './sklearn-iris'\n",
|
"project_folder = './sklearn-iris'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)"
|
"os.makedirs(project_folder, exist_ok=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Prepare training script"
|
"### Prepare training script"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Now you will need to create your training script. In this tutorial, the training script is already provided for you at `train_iris`.py. In practice, you should be able to take any custom training script as is and run it with Azure ML without having to modify your code.\n",
|
"Now you will need to create your training script. In this tutorial, the training script is already provided for you at `train_iris`.py. In practice, you should be able to take any custom training script as is and run it with Azure ML without having to modify your code.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"However, if you would like to use Azure ML's [tracking and metrics](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#metrics) capabilities, you will have to add a small amount of Azure ML code inside your training script.\n",
|
"However, if you would like to use Azure ML's [tracking and metrics](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#metrics) capabilities, you will have to add a small amount of Azure ML code inside your training script.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In `train_iris.py`, we will log some metrics to our Azure ML run. To do so, we will access the Azure ML Run object within the script:\n",
|
"In `train_iris.py`, we will log some metrics to our Azure ML run. To do so, we will access the Azure ML Run object within the script:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"from azureml.core.run import Run\n",
|
"from azureml.core.run import Run\n",
|
||||||
"run = Run.get_context()\n",
|
"run = Run.get_context()\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Further within `train_iris.py`, we log the kernel and penalty parameters, and the highest accuracy the model achieves:\n",
|
"Further within `train_iris.py`, we log the kernel and penalty parameters, and the highest accuracy the model achieves:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"```python\n",
|
"```python\n",
|
||||||
"run.log('Kernel type', np.string(args.kernel))\n",
|
"run.log('Kernel type', np.string(args.kernel))\n",
|
||||||
"run.log('Penalty', np.float(args.penalty))\n",
|
"run.log('Penalty', np.float(args.penalty))\n",
|
||||||
"\n",
|
"\n",
|
||||||
"run.log('Accuracy', np.float(accuracy))\n",
|
"run.log('Accuracy', np.float(accuracy))\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"These run metrics will become particularly important when we begin hyperparameter tuning our model in the \"Tune model hyperparameters\" section.\n",
|
"These run metrics will become particularly important when we begin hyperparameter tuning our model in the \"Tune model hyperparameters\" section.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Once your script is ready, copy the training script `train_iris.py` into your project directory."
|
"Once your script is ready, copy the training script `train_iris.py` into your project directory."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"shutil.copy('train_iris.py', project_folder)"
|
"shutil.copy('train_iris.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create an experiment"
|
"### Create an experiment"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this Scikit-learn tutorial."
|
"Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this Scikit-learn tutorial."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"experiment_name = 'train_iris'\n",
|
"experiment_name = 'train_iris'\n",
|
||||||
"experiment = Experiment(ws, name=experiment_name)"
|
"experiment = Experiment(ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create a Scikit-learn estimator"
|
"### Create a Scikit-learn estimator"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The Azure ML SDK's Scikit-learn estimator enables you to easily submit Scikit-learn training jobs for single-node runs. The following code will define a single-node Scikit-learn job."
|
"The Azure ML SDK's Scikit-learn estimator enables you to easily submit Scikit-learn training jobs for single-node runs. The following code will define a single-node Scikit-learn job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.train.sklearn import SKLearn\n",
|
"from azureml.train.sklearn import SKLearn\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_params = {\n",
|
"script_params = {\n",
|
||||||
" '--kernel': 'linear',\n",
|
" '--kernel': 'linear',\n",
|
||||||
" '--penalty': 1.0,\n",
|
" '--penalty': 1.0,\n",
|
||||||
"}\n",
|
"}\n",
|
||||||
"\n",
|
"\n",
|
||||||
"estimator = SKLearn(source_directory=project_folder, \n",
|
"estimator = SKLearn(source_directory=project_folder, \n",
|
||||||
" script_params=script_params,\n",
|
" script_params=script_params,\n",
|
||||||
" compute_target=compute_target,\n",
|
" compute_target=compute_target,\n",
|
||||||
" entry_script='train_iris.py'\n",
|
" entry_script='train_iris.py'\n",
|
||||||
" )"
|
" )"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"The `script_params` parameter is a dictionary containing the command-line arguments to your training script `entry_script`. To leverage the Azure VM's GPU for training, we set `use_gpu=True`."
|
"The `script_params` parameter is a dictionary containing the command-line arguments to your training script `entry_script`. To leverage the Azure VM's GPU for training, we set `use_gpu=True`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit job"
|
"### Submit job"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
"Run your experiment by submitting your estimator object. Note that this call is asynchronous."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run = experiment.submit(estimator)"
|
"run = experiment.submit(estimator)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Monitor your run"
|
"## Monitor your run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
"You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"\n",
|
"\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.cancel()"
|
"run.cancel()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Tune model hyperparameters"
|
"## Tune model hyperparameters"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Now that we've seen how to do a simple Scikit-learn training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's hyperparameter tuning capabilities."
|
"Now that we've seen how to do a simple Scikit-learn training run using the SDK, let's see if we can further improve the accuracy of our model. We can optimize our model's hyperparameters using Azure Machine Learning's hyperparameter tuning capabilities."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Start a hyperparameter sweep"
|
"### Start a hyperparameter sweep"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"First, we will define the hyperparameter space to sweep over. Let's tune the `kernel` and `penalty` parameters. In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, `Accuracy`."
|
"First, we will define the hyperparameter space to sweep over. Let's tune the `kernel` and `penalty` parameters. In this example we will use random sampling to try different configuration sets of hyperparameters to maximize our primary metric, `Accuracy`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.train.hyperdrive.runconfig import HyperDriveRunConfig\n",
|
"from azureml.train.hyperdrive.runconfig import HyperDriveRunConfig\n",
|
||||||
"from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
|
"from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
|
||||||
"from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
|
"from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
|
||||||
"from azureml.train.hyperdrive.parameter_expressions import choice\n",
|
"from azureml.train.hyperdrive.parameter_expressions import choice\n",
|
||||||
" \n",
|
" \n",
|
||||||
"\n",
|
"\n",
|
||||||
"param_sampling = RandomParameterSampling( {\n",
|
"param_sampling = RandomParameterSampling( {\n",
|
||||||
" \"--kernel\": choice('linear', 'rbf', 'poly', 'sigmoid'),\n",
|
" \"--kernel\": choice('linear', 'rbf', 'poly', 'sigmoid'),\n",
|
||||||
" \"--penalty\": choice(0.5, 1, 1.5)\n",
|
" \"--penalty\": choice(0.5, 1, 1.5)\n",
|
||||||
" }\n",
|
" }\n",
|
||||||
")\n",
|
")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"hyperdrive_run_config = HyperDriveRunConfig(estimator=estimator,\n",
|
"hyperdrive_run_config = HyperDriveRunConfig(estimator=estimator,\n",
|
||||||
" hyperparameter_sampling=param_sampling, \n",
|
" hyperparameter_sampling=param_sampling, \n",
|
||||||
" primary_metric_name='Accuracy',\n",
|
" primary_metric_name='Accuracy',\n",
|
||||||
" primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
|
" primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
|
||||||
" max_total_runs=12,\n",
|
" max_total_runs=12,\n",
|
||||||
" max_concurrent_runs=4)"
|
" max_concurrent_runs=4)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Finally, lauch the hyperparameter tuning job."
|
"Finally, lauch the hyperparameter tuning job."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# start the HyperDrive run\n",
|
"# start the HyperDrive run\n",
|
||||||
"hyperdrive_run = experiment.submit(hyperdrive_run_config)"
|
"hyperdrive_run = experiment.submit(hyperdrive_run_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Monitor HyperDrive runs"
|
"## Monitor HyperDrive runs"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can monitor the progress of the runs with the following Jupyter widget."
|
"You can monitor the progress of the runs with the following Jupyter widget."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"RunDetails(hyperdrive_run).show()"
|
"RunDetails(hyperdrive_run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "dipeck"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "dipeck"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.5.2"
|
||||||
|
},
|
||||||
|
"msauthor": "dipeck"
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.5.2"
|
|
||||||
},
|
|
||||||
"msauthor": "dipeck"
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,285 +1,285 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# 05. Train in Spark\n",
|
"# 05. Train in Spark\n",
|
||||||
"* Create Workspace\n",
|
"* Create Workspace\n",
|
||||||
"* Create Experiment\n",
|
"* Create Experiment\n",
|
||||||
"* Copy relevant files to the script folder\n",
|
"* Copy relevant files to the script folder\n",
|
||||||
"* Configure and Run"
|
"* Configure and Run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
|
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize Workspace\n",
|
"## Initialize Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a workspace object from persisted configuration."
|
"Initialize a workspace object from persisted configuration."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create Experiment\n"
|
"## Create Experiment\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"experiment_name = 'train-on-spark'\n",
|
"experiment_name = 'train-on-spark'\n",
|
||||||
"\n",
|
"\n",
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"exp = Experiment(workspace=ws, name=experiment_name)"
|
"exp = Experiment(workspace=ws, name=experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## View `train-spark.py`\n",
|
"## View `train-spark.py`\n",
|
||||||
"\n",
|
"\n",
|
||||||
"For convenience, we created a training script for you. It is printed below as a text, but you can also run `%pfile ./train-spark.py` in a cell to show the file."
|
"For convenience, we created a training script for you. It is printed below as a text, but you can also run `%pfile ./train-spark.py` in a cell to show the file."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"with open('train-spark.py', 'r') as training_script:\n",
|
"with open('train-spark.py', 'r') as training_script:\n",
|
||||||
" print(training_script.read())"
|
" print(training_script.read())"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Configure & Run"
|
"## Configure & Run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"**Note** You can use Docker-based execution to run the Spark job in local computer or a remote VM. Please see the `train-in-remote-vm` notebook for example on how to configure and run in Docker mode in a VM. Make sure you choose a Docker image that has Spark installed, such as `microsoft/mmlspark:0.12`."
|
"**Note** You can use Docker-based execution to run the Spark job in local computer or a remote VM. Please see the `train-in-remote-vm` notebook for example on how to configure and run in Docker mode in a VM. Make sure you choose a Docker image that has Spark installed, such as `microsoft/mmlspark:0.12`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Attach an HDI cluster\n",
|
"### Attach an HDI cluster\n",
|
||||||
"Here we will use a actual Spark cluster, HDInsight for Spark, to run this job. To use HDI commpute target:\n",
|
"Here we will use a actual Spark cluster, HDInsight for Spark, to run this job. To use HDI commpute target:\n",
|
||||||
" 1. Create a Spark for HDI cluster in Azure. Here are some [quick instructions](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql). Make sure you use the Ubuntu flavor, NOT CentOS.\n",
|
" 1. Create a Spark for HDI cluster in Azure. Here are some [quick instructions](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql). Make sure you use the Ubuntu flavor, NOT CentOS.\n",
|
||||||
" 2. Enter the IP address, username and password below"
|
" 2. Enter the IP address, username and password below"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, HDInsightCompute\n",
|
"from azureml.core.compute import ComputeTarget, HDInsightCompute\n",
|
||||||
"from azureml.exceptions import ComputeTargetException\n",
|
"from azureml.exceptions import ComputeTargetException\n",
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"\n",
|
"\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" # if you want to connect using SSH key instead of username/password you can provide parameters private_key_file and private_key_passphrase\n",
|
" # if you want to connect using SSH key instead of username/password you can provide parameters private_key_file and private_key_passphrase\n",
|
||||||
" attach_config = HDInsightCompute.attach_configuration(address=os.environ.get('hdiservername', '<my_hdi_cluster_name>-ssh.azurehdinsight.net'), \n",
|
" attach_config = HDInsightCompute.attach_configuration(address=os.environ.get('hdiservername', '<my_hdi_cluster_name>-ssh.azurehdinsight.net'), \n",
|
||||||
" ssh_port=22, \n",
|
" ssh_port=22, \n",
|
||||||
" username=os.environ.get('hdiusername', '<ssh_username>'), \n",
|
" username=os.environ.get('hdiusername', '<ssh_username>'), \n",
|
||||||
" password=os.environ.get('hdipassword', '<my_password>'))\n",
|
" password=os.environ.get('hdipassword', '<my_password>'))\n",
|
||||||
" hdi_compute = ComputeTarget.attach(workspace=ws, \n",
|
" hdi_compute = ComputeTarget.attach(workspace=ws, \n",
|
||||||
" name='myhdi', \n",
|
" name='myhdi', \n",
|
||||||
" attach_configuration=attach_config)\n",
|
" attach_configuration=attach_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"except ComputeTargetException as e:\n",
|
"except ComputeTargetException as e:\n",
|
||||||
" print(\"Caught = {}\".format(e.message))\n",
|
" print(\"Caught = {}\".format(e.message))\n",
|
||||||
" \n",
|
" \n",
|
||||||
" \n",
|
" \n",
|
||||||
"hdi_compute.wait_for_completion(show_output=True)"
|
"hdi_compute.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Configure HDI run"
|
"### Configure HDI run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Configure an execution using the HDInsight cluster with a conda environment that has `numpy`."
|
"Configure an execution using the HDInsight cluster with a conda environment that has `numpy`."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.runconfig import RunConfiguration\n",
|
"from azureml.core.runconfig import RunConfiguration\n",
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies\n",
|
"from azureml.core.conda_dependencies import CondaDependencies\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# use pyspark framework\n",
|
"# use pyspark framework\n",
|
||||||
"hdi_run_config = RunConfiguration(framework=\"pyspark\")\n",
|
"hdi_run_config = RunConfiguration(framework=\"pyspark\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Set compute target to the HDI cluster\n",
|
"# Set compute target to the HDI cluster\n",
|
||||||
"hdi_run_config.target = hdi_compute.name\n",
|
"hdi_run_config.target = hdi_compute.name\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# specify CondaDependencies object to ask system installing numpy\n",
|
"# specify CondaDependencies object to ask system installing numpy\n",
|
||||||
"cd = CondaDependencies()\n",
|
"cd = CondaDependencies()\n",
|
||||||
"cd.add_conda_package('numpy')\n",
|
"cd.add_conda_package('numpy')\n",
|
||||||
"hdi_run_config.environment.python.conda_dependencies = cd"
|
"hdi_run_config.environment.python.conda_dependencies = cd"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Submit the script to HDI"
|
"### Submit the script to HDI"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import ScriptRunConfig\n",
|
"from azureml.core import ScriptRunConfig\n",
|
||||||
"\n",
|
"\n",
|
||||||
"script_run_config = ScriptRunConfig(source_directory = '.',\n",
|
"script_run_config = ScriptRunConfig(source_directory = '.',\n",
|
||||||
" script= 'train-spark.py',\n",
|
" script= 'train-spark.py',\n",
|
||||||
" run_config = hdi_run_config)\n",
|
" run_config = hdi_run_config)\n",
|
||||||
"run = exp.submit(config=script_run_config)"
|
"run = exp.submit(config=script_run_config)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Monitor the run using a Juypter widget"
|
"Monitor the run using a Juypter widget"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.widgets import RunDetails\n",
|
"from azureml.widgets import RunDetails\n",
|
||||||
"RunDetails(run).show()"
|
"RunDetails(run).show()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
|
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"After the run is succesfully finished, you can check the metrics logged."
|
"After the run is succesfully finished, you can check the metrics logged."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# get all metris logged in the run\n",
|
"# get all metris logged in the run\n",
|
||||||
"metrics = run.get_metrics()\n",
|
"metrics = run.get_metrics()\n",
|
||||||
"print(metrics)"
|
"print(metrics)"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "aashishb"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "aashishb"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.7"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.7"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
@@ -1,448 +1,448 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License."
|
"Licensed under the MIT License."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Train using Azure Machine Learning Compute\n",
|
"# Train using Azure Machine Learning Compute\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* Initialize a Workspace\n",
|
"* Initialize a Workspace\n",
|
||||||
"* Create an Experiment\n",
|
"* Create an Experiment\n",
|
||||||
"* Introduction to AmlCompute\n",
|
"* Introduction to AmlCompute\n",
|
||||||
"* Submit an AmlCompute run in a few different ways\n",
|
"* Submit an AmlCompute run in a few different ways\n",
|
||||||
" - Provision as a run based compute target \n",
|
" - Provision as a run based compute target \n",
|
||||||
" - Provision as a persistent compute target (Basic)\n",
|
" - Provision as a persistent compute target (Basic)\n",
|
||||||
" - Provision as a persistent compute target (Advanced)\n",
|
" - Provision as a persistent compute target (Advanced)\n",
|
||||||
"* Additional operations to perform on AmlCompute\n",
|
"* Additional operations to perform on AmlCompute\n",
|
||||||
"* Find the best model in the run"
|
"* Find the best model in the run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Prerequisites\n",
|
"## Prerequisites\n",
|
||||||
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
|
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Check core SDK version number\n",
|
"# Check core SDK version number\n",
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"SDK version:\", azureml.core.VERSION)"
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Initialize a Workspace\n",
|
"## Initialize a Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Initialize a workspace object from persisted configuration"
|
"Initialize a workspace object from persisted configuration"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"tags": [
|
"tags": [
|
||||||
"create workspace"
|
"create workspace"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Workspace\n",
|
"from azureml.core import Workspace\n",
|
||||||
"\n",
|
"\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create An Experiment\n",
|
"## Create An Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
|
"**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Experiment\n",
|
"from azureml.core import Experiment\n",
|
||||||
"experiment_name = 'train-on-amlcompute'\n",
|
"experiment_name = 'train-on-amlcompute'\n",
|
||||||
"experiment = Experiment(workspace = ws, name = experiment_name)"
|
"experiment = Experiment(workspace = ws, name = experiment_name)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Introduction to AmlCompute\n",
|
"## Introduction to AmlCompute\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Azure Machine Learning Compute is managed compute infrastructure that allows the user to easily create single to multi-node compute of the appropriate VM Family. It is created **within your workspace region** and is a resource that can be used by other users in your workspace. It autoscales by default to the max_nodes, when a job is submitted, and executes in a containerized environment packaging the dependencies as specified by the user. \n",
|
"Azure Machine Learning Compute is managed compute infrastructure that allows the user to easily create single to multi-node compute of the appropriate VM Family. It is created **within your workspace region** and is a resource that can be used by other users in your workspace. It autoscales by default to the max_nodes, when a job is submitted, and executes in a containerized environment packaging the dependencies as specified by the user. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"Since it is managed compute, job scheduling and cluster management are handled internally by Azure Machine Learning service. \n",
|
"Since it is managed compute, job scheduling and cluster management are handled internally by Azure Machine Learning service. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"For more information on Azure Machine Learning Compute, please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)\n",
|
"For more information on Azure Machine Learning Compute, please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you are an existing BatchAI customer who is migrating to Azure Machine Learning, please read [this article](https://aka.ms/batchai-retirement)\n",
|
"If you are an existing BatchAI customer who is migrating to Azure Machine Learning, please read [this article](https://aka.ms/batchai-retirement)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Note**: As with other Azure services, there are limits on certain resources (for eg. AmlCompute quota) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
|
"**Note**: As with other Azure services, there are limits on certain resources (for eg. AmlCompute quota) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The training script `train.py` is already created for you. Let's have a look."
|
"The training script `train.py` is already created for you. Let's have a look."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Submit an AmlCompute run in a few different ways\n",
|
"## Submit an AmlCompute run in a few different ways\n",
|
||||||
"\n",
|
"\n",
|
||||||
"First lets check which VM families are available in your region. Azure is a regional service and some specialized SKUs (especially GPUs) are only available in certain regions. Since AmlCompute is created in the region of your workspace, we will use the supported_vms () function to see if the VM family we want to use ('STANDARD_D2_V2') is supported.\n",
|
"First lets check which VM families are available in your region. Azure is a regional service and some specialized SKUs (especially GPUs) are only available in certain regions. Since AmlCompute is created in the region of your workspace, we will use the supported_vms () function to see if the VM family we want to use ('STANDARD_D2_V2') is supported.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can also pass a different region to check availability and then re-create your workspace in that region through the [configuration notebook](../../../configuration.ipynb)"
|
"You can also pass a different region to check availability and then re-create your workspace in that region through the [configuration notebook](../../../configuration.ipynb)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"\n",
|
"\n",
|
||||||
"AmlCompute.supported_vmsizes(workspace = ws)\n",
|
"AmlCompute.supported_vmsizes(workspace = ws)\n",
|
||||||
"#AmlCompute.supported_vmsizes(workspace = ws, location='southcentralus')"
|
"#AmlCompute.supported_vmsizes(workspace = ws, location='southcentralus')"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create project directory\n",
|
"### Create project directory\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on"
|
"Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script, and any additional files your training script depends on"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import os\n",
|
"import os\n",
|
||||||
"import shutil\n",
|
"import shutil\n",
|
||||||
"\n",
|
"\n",
|
||||||
"project_folder = './train-on-amlcompute'\n",
|
"project_folder = './train-on-amlcompute'\n",
|
||||||
"os.makedirs(project_folder, exist_ok=True)\n",
|
"os.makedirs(project_folder, exist_ok=True)\n",
|
||||||
"shutil.copy('train.py', project_folder)"
|
"shutil.copy('train.py', project_folder)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Create environment\n",
|
"### Create environment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Create Docker based environment with scikit-learn installed."
|
"Create Docker based environment with scikit-learn installed."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Environment\n",
|
"from azureml.core import Environment\n",
|
||||||
"from azureml.core.conda_dependencies import CondaDependencies\n",
|
"from azureml.core.conda_dependencies import CondaDependencies\n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = Environment(\"myenv\")\n",
|
"myenv = Environment(\"myenv\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv.docker.enabled = True\n",
|
"myenv.docker.enabled = True\n",
|
||||||
"myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
|
"myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Provision as a persistent compute target (Basic)\n",
|
"### Provision as a persistent compute target (Basic)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can provision a persistent AmlCompute resource by simply defining two parameters thanks to smart defaults. By default it autoscales from 0 nodes and provisions dedicated VMs to run your job in a container. This is useful when you want to continously re-use the same target, debug it between jobs or simply share the resource with other users of your workspace.\n",
|
"You can provision a persistent AmlCompute resource by simply defining two parameters thanks to smart defaults. By default it autoscales from 0 nodes and provisions dedicated VMs to run your job in a container. This is useful when you want to continously re-use the same target, debug it between jobs or simply share the resource with other users of your workspace.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* `vm_size`: VM family of the nodes provisioned by AmlCompute. Simply choose from the supported_vmsizes() above\n",
|
"* `vm_size`: VM family of the nodes provisioned by AmlCompute. Simply choose from the supported_vmsizes() above\n",
|
||||||
"* `max_nodes`: Maximum nodes to autoscale to while running a job on AmlCompute"
|
"* `max_nodes`: Maximum nodes to autoscale to while running a job on AmlCompute"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Choose a name for your CPU cluster\n",
|
"# Choose a name for your CPU cluster\n",
|
||||||
"cpu_cluster_name = \"cpu-cluster\"\n",
|
"cpu_cluster_name = \"cpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Verify that cluster does not exist already\n",
|
"# Verify that cluster does not exist already\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
||||||
" print('Found existing cluster, use it.')\n",
|
" print('Found existing cluster, use it.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
|
||||||
" max_nodes=4)\n",
|
" max_nodes=4)\n",
|
||||||
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"cpu_cluster.wait_for_completion(show_output=True)"
|
"cpu_cluster.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Configure & Run"
|
"### Configure & Run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import ScriptRunConfig\n",
|
"from azureml.core import ScriptRunConfig\n",
|
||||||
"from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
|
"from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
|
||||||
"\n",
|
"\n",
|
||||||
"src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
|
"src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Set compute target to the one created in previous step\n",
|
"# Set compute target to the one created in previous step\n",
|
||||||
"src.run_config.target = cpu_cluster.name\n",
|
"src.run_config.target = cpu_cluster.name\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Set environment\n",
|
"# Set environment\n",
|
||||||
"src.run_config.environment = myenv\n",
|
"src.run_config.environment = myenv\n",
|
||||||
" \n",
|
" \n",
|
||||||
"run = experiment.submit(config=src)\n",
|
"run = experiment.submit(config=src)\n",
|
||||||
"run"
|
"run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
|
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"# Shows output of the run on stdout.\n",
|
"# Shows output of the run on stdout.\n",
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.get_metrics()"
|
"run.get_metrics()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Provision as a persistent compute target (Advanced)\n",
|
"### Provision as a persistent compute target (Advanced)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can also specify additional properties or change defaults while provisioning AmlCompute using a more advanced configuration. This is useful when you want a dedicated cluster of 4 nodes (for example you can set the min_nodes and max_nodes to 4), or want the compute to be within an existing VNet in your subscription.\n",
|
"You can also specify additional properties or change defaults while provisioning AmlCompute using a more advanced configuration. This is useful when you want a dedicated cluster of 4 nodes (for example you can set the min_nodes and max_nodes to 4), or want the compute to be within an existing VNet in your subscription.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In addition to `vm_size` and `max_nodes`, you can specify:\n",
|
"In addition to `vm_size` and `max_nodes`, you can specify:\n",
|
||||||
"* `min_nodes`: Minimum nodes (default 0 nodes) to downscale to while running a job on AmlCompute\n",
|
"* `min_nodes`: Minimum nodes (default 0 nodes) to downscale to while running a job on AmlCompute\n",
|
||||||
"* `vm_priority`: Choose between 'dedicated' (default) and 'lowpriority' VMs when provisioning AmlCompute. Low Priority VMs use Azure's excess capacity and are thus cheaper but risk your run being pre-empted\n",
|
"* `vm_priority`: Choose between 'dedicated' (default) and 'lowpriority' VMs when provisioning AmlCompute. Low Priority VMs use Azure's excess capacity and are thus cheaper but risk your run being pre-empted\n",
|
||||||
"* `idle_seconds_before_scaledown`: Idle time (default 120 seconds) to wait after run completion before auto-scaling to min_nodes\n",
|
"* `idle_seconds_before_scaledown`: Idle time (default 120 seconds) to wait after run completion before auto-scaling to min_nodes\n",
|
||||||
"* `vnet_resourcegroup_name`: Resource group of the **existing** VNet within which AmlCompute should be provisioned\n",
|
"* `vnet_resourcegroup_name`: Resource group of the **existing** VNet within which AmlCompute should be provisioned\n",
|
||||||
"* `vnet_name`: Name of VNet\n",
|
"* `vnet_name`: Name of VNet\n",
|
||||||
"* `subnet_name`: Name of SubNet within the VNet"
|
"* `subnet_name`: Name of SubNet within the VNet"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
"from azureml.core.compute import ComputeTarget, AmlCompute\n",
|
||||||
"from azureml.core.compute_target import ComputeTargetException\n",
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Choose a name for your CPU cluster\n",
|
"# Choose a name for your CPU cluster\n",
|
||||||
"cpu_cluster_name = \"cpu-cluster\"\n",
|
"cpu_cluster_name = \"cpu-cluster\"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"# Verify that cluster does not exist already\n",
|
"# Verify that cluster does not exist already\n",
|
||||||
"try:\n",
|
"try:\n",
|
||||||
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
" cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
|
||||||
" print('Found existing cluster, use it.')\n",
|
" print('Found existing cluster, use it.')\n",
|
||||||
"except ComputeTargetException:\n",
|
"except ComputeTargetException:\n",
|
||||||
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
|
" compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
|
||||||
" vm_priority='lowpriority',\n",
|
" vm_priority='lowpriority',\n",
|
||||||
" min_nodes=2,\n",
|
" min_nodes=2,\n",
|
||||||
" max_nodes=4,\n",
|
" max_nodes=4,\n",
|
||||||
" idle_seconds_before_scaledown='300',\n",
|
" idle_seconds_before_scaledown='300',\n",
|
||||||
" vnet_resourcegroup_name='<my-resource-group>',\n",
|
" vnet_resourcegroup_name='<my-resource-group>',\n",
|
||||||
" vnet_name='<my-vnet-name>',\n",
|
" vnet_name='<my-vnet-name>',\n",
|
||||||
" subnet_name='<my-subnet-name>')\n",
|
" subnet_name='<my-subnet-name>')\n",
|
||||||
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
" cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"cpu_cluster.wait_for_completion(show_output=True)"
|
"cpu_cluster.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Configure & Run"
|
"### Configure & Run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"# Set compute target to the one created in previous step\n",
|
"# Set compute target to the one created in previous step\n",
|
||||||
"src.run_config.target = cpu_cluster.name\n",
|
"src.run_config.target = cpu_cluster.name\n",
|
||||||
" \n",
|
" \n",
|
||||||
"run = experiment.submit(config=src)\n",
|
"run = experiment.submit(config=src)\n",
|
||||||
"run"
|
"run"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"%%time\n",
|
"%%time\n",
|
||||||
"# Shows output of the run on stdout.\n",
|
"# Shows output of the run on stdout.\n",
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"run.get_metrics()"
|
"run.get_metrics()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Additional operations to perform on AmlCompute\n",
|
"## Additional operations to perform on AmlCompute\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can perform more operations on AmlCompute such as updating the node counts or deleting the compute. "
|
"You can perform more operations on AmlCompute such as updating the node counts or deleting the compute. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Get_status () gets the latest status of the AmlCompute target\n",
|
"#Get_status () gets the latest status of the AmlCompute target\n",
|
||||||
"cpu_cluster.get_status().serialize()\n"
|
"cpu_cluster.get_status().serialize()\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Update () takes in the min_nodes, max_nodes and idle_seconds_before_scaledown and updates the AmlCompute target\n",
|
"#Update () takes in the min_nodes, max_nodes and idle_seconds_before_scaledown and updates the AmlCompute target\n",
|
||||||
"#cpu_cluster.update(min_nodes=1)\n",
|
"#cpu_cluster.update(min_nodes=1)\n",
|
||||||
"#cpu_cluster.update(max_nodes=10)\n",
|
"#cpu_cluster.update(max_nodes=10)\n",
|
||||||
"cpu_cluster.update(idle_seconds_before_scaledown=300)\n",
|
"cpu_cluster.update(idle_seconds_before_scaledown=300)\n",
|
||||||
"#cpu_cluster.update(min_nodes=2, max_nodes=4, idle_seconds_before_scaledown=600)"
|
"#cpu_cluster.update(min_nodes=2, max_nodes=4, idle_seconds_before_scaledown=600)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"#Delete () is used to deprovision and delete the AmlCompute target. Useful if you want to re-use the compute name \n",
|
"#Delete () is used to deprovision and delete the AmlCompute target. Useful if you want to re-use the compute name \n",
|
||||||
"#'cpu-cluster' in this case but use a different VM family for instance.\n",
|
"#'cpu-cluster' in this case but use a different VM family for instance.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"#cpu_cluster.delete()"
|
"#cpu_cluster.delete()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Success!\n",
|
"## Success!\n",
|
||||||
"Great, you are ready to move on to the remaining notebooks."
|
"Great, you are ready to move on to the remaining notebooks."
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "nigup"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "nigup"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.6"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,372 +1,372 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Licensed under the MIT License"
|
"Licensed under the MIT License"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
""
|
""
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"# Using environments\n",
|
"# Using environments\n",
|
||||||
"\n",
|
"\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Contents\n",
|
"## Contents\n",
|
||||||
"\n",
|
"\n",
|
||||||
"1. [Introduction](#Introduction)\n",
|
"1. [Introduction](#Introduction)\n",
|
||||||
"1. [Setup](#Setup)\n",
|
"1. [Setup](#Setup)\n",
|
||||||
"1. [Create environment](#Create-environment)\n",
|
"1. [Create environment](#Create-environment)\n",
|
||||||
" 1. Add Python packages\n",
|
" 1. Add Python packages\n",
|
||||||
" 1. Specify environment variables\n",
|
" 1. Specify environment variables\n",
|
||||||
"1. [Submit run using environment](#Submit-run-using-environment)\n",
|
"1. [Submit run using environment](#Submit-run-using-environment)\n",
|
||||||
"1. [Register environment](#Register-environment)\n",
|
"1. [Register environment](#Register-environment)\n",
|
||||||
"1. [List and get existing environments](#List-and-get-existing-environments)\n",
|
"1. [List and get existing environments](#List-and-get-existing-environments)\n",
|
||||||
"1. [Other ways to create environments](#Other-ways-to-create-environments)\n",
|
"1. [Other ways to create environments](#Other-ways-to-create-environments)\n",
|
||||||
" 1. From existing Conda environment\n",
|
" 1. From existing Conda environment\n",
|
||||||
" 1. From Conda or pip files\n",
|
" 1. From Conda or pip files\n",
|
||||||
"1. [Docker settings](#Docker-settings)\n",
|
"1. [Docker settings](#Docker-settings)\n",
|
||||||
"1. [Spark and Azure Databricks settings](#Spark-and-Azure-Databricks-settings)\n",
|
"1. [Spark and Azure Databricks settings](#Spark-and-Azure-Databricks-settings)\n",
|
||||||
"1. [Next steps](#Next-steps)\n",
|
"1. [Next steps](#Next-steps)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Introduction\n",
|
"## Introduction\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Azure ML environments are an encapsulation of the environment where your machine learning training happens. They define Python packages, environment variables, Docker settings and other attributes in declarative fashion. Environments are versioned: you can update them and retrieve old versions to revist and review your work.\n",
|
"Azure ML environments are an encapsulation of the environment where your machine learning training happens. They define Python packages, environment variables, Docker settings and other attributes in declarative fashion. Environments are versioned: you can update them and retrieve old versions to revist and review your work.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Environments allow you to:\n",
|
"Environments allow you to:\n",
|
||||||
"* Encapsulate dependencies of your training process, such as Python packages and their versions.\n",
|
"* Encapsulate dependencies of your training process, such as Python packages and their versions.\n",
|
||||||
"* Reproduce the Python environment on your local computer in a remote run on VM or ML Compute cluster\n",
|
"* Reproduce the Python environment on your local computer in a remote run on VM or ML Compute cluster\n",
|
||||||
"* Reproduce your experimentation environment in production setting.\n",
|
"* Reproduce your experimentation environment in production setting.\n",
|
||||||
"* Revisit and audit the environment in which an existing model was trained.\n",
|
"* Revisit and audit the environment in which an existing model was trained.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Environment, compute target and training script together form run configuration: the full specification of training run.\n",
|
"Environment, compute target and training script together form run configuration: the full specification of training run.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"## Setup\n",
|
"## Setup\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't.\n",
|
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"First, let's validate Azure ML SDK version and connect to workspace."
|
"First, let's validate Azure ML SDK version and connect to workspace."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"import azureml.core\n",
|
"import azureml.core\n",
|
||||||
"print(azureml.core.VERSION)"
|
"print(azureml.core.VERSION)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core.workspace import Workspace\n",
|
"from azureml.core.workspace import Workspace\n",
|
||||||
"ws = Workspace.from_config()\n",
|
"ws = Workspace.from_config()\n",
|
||||||
"ws.get_details()"
|
"ws.get_details()"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Create environment\n",
|
"## Create environment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can create an environment by instantiating ```Environment``` object and then setting its attributes: set of Python packages, environment variables and others.\n",
|
"You can create an environment by instantiating ```Environment``` object and then setting its attributes: set of Python packages, environment variables and others.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### Add Python packages\n",
|
"### Add Python packages\n",
|
||||||
"\n",
|
"\n",
|
||||||
"The recommended way is to specify Conda packages, as they typically come with complete set of pre-built binaries."
|
"The recommended way is to specify Conda packages, as they typically come with complete set of pre-built binaries."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import Environment\n",
|
"from azureml.core import Environment\n",
|
||||||
"from azureml.core.environment import CondaDependencies\n",
|
"from azureml.core.environment import CondaDependencies\n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = Environment(name=\"myenv\")\n",
|
"myenv = Environment(name=\"myenv\")\n",
|
||||||
"conda_dep = CondaDependencies()\n",
|
"conda_dep = CondaDependencies()\n",
|
||||||
"conda_dep.add_conda_package(\"scikit-learn\")"
|
"conda_dep.add_conda_package(\"scikit-learn\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can also add pip packages, and specify the version of package"
|
"You can also add pip packages, and specify the version of package"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"conda_dep.add_pip_package(\"pillow==5.4.1\")\n",
|
"conda_dep.add_pip_package(\"pillow==5.4.1\")\n",
|
||||||
"myenv.python.conda_dependencies=conda_dep"
|
"myenv.python.conda_dependencies=conda_dep"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"### Specify environment variables\n",
|
"### Specify environment variables\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can add environment variables to your environment. These then become available using ```os.environ.get``` in your training script."
|
"You can add environment variables to your environment. These then become available using ```os.environ.get``` in your training script."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"myenv.environment_variables = {\"MESSAGE\":\"Hello from Azure Machine Learning\"}"
|
"myenv.environment_variables = {\"MESSAGE\":\"Hello from Azure Machine Learning\"}"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Submit run using environment\n",
|
"## Submit run using environment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"When you submit a run, you can specify which environment to use. \n",
|
"When you submit a run, you can specify which environment to use. \n",
|
||||||
"\n",
|
"\n",
|
||||||
"On the first run in given environment, Azure ML spends some time building the environment. On the subsequent runs, Azure ML keeps track of changes and uses the existing environment, resulting in faster run completion."
|
"On the first run in given environment, Azure ML spends some time building the environment. On the subsequent runs, Azure ML keeps track of changes and uses the existing environment, resulting in faster run completion."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"from azureml.core import ScriptRunConfig, Experiment\n",
|
"from azureml.core import ScriptRunConfig, Experiment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"myexp = Experiment(workspace=ws, name = \"environment-example\")"
|
"myexp = Experiment(workspace=ws, name = \"environment-example\")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"To submit a run, create a run configuration that combines the script file and environment, and pass it to ```Experiment.submit```. In this example, the script is submitted to local computer, but you can specify other compute targets such as remote clusters as well."
|
"To submit a run, create a run configuration that combines the script file and environment, and pass it to ```Experiment.submit```. In this example, the script is submitted to local computer, but you can specify other compute targets such as remote clusters as well."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"runconfig = ScriptRunConfig(source_directory=\".\", script=\"example.py\")\n",
|
"runconfig = ScriptRunConfig(source_directory=\".\", script=\"example.py\")\n",
|
||||||
"runconfig.run_config.target = \"local\"\n",
|
"runconfig.run_config.target = \"local\"\n",
|
||||||
"runconfig.run_config.environment = myenv\n",
|
"runconfig.run_config.environment = myenv\n",
|
||||||
"run = myexp.submit(config=runconfig)\n",
|
"run = myexp.submit(config=runconfig)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"run.wait_for_completion(show_output=True)"
|
"run.wait_for_completion(show_output=True)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Register environment\n",
|
"## Register environment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can manage environments by registering them. This allows you to track their versions, and reuse them in future runs. For example, once you've constructed an environment that meets your requirements, you can register it and use it in other experiments so as to standardize your workflow.\n",
|
"You can manage environments by registering them. This allows you to track their versions, and reuse them in future runs. For example, once you've constructed an environment that meets your requirements, you can register it and use it in other experiments so as to standardize your workflow.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"If you register the environment with same name, the version number is increased by one. Note that Azure ML keeps track of differences between the version, so if you re-register an identical version, the version number is not increased."
|
"If you register the environment with same name, the version number is increased by one. Note that Azure ML keeps track of differences between the version, so if you re-register an identical version, the version number is not increased."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"myenv.register(workspace=ws)"
|
"myenv.register(workspace=ws)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## List and get existing environments\n",
|
"## List and get existing environments\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Your workspace contains a dictionary of registered environments. You can then use ```Environment.get``` to retrieve a specific environment with specific version."
|
"Your workspace contains a dictionary of registered environments. You can then use ```Environment.get``` to retrieve a specific environment with specific version."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"for name,env in ws.environments.items():\n",
|
"for name,env in ws.environments.items():\n",
|
||||||
" print(\"Name {} \\t version {}\".format(name,env.version))\n",
|
" print(\"Name {} \\t version {}\".format(name,env.version))\n",
|
||||||
"\n",
|
"\n",
|
||||||
"restored_environment = Environment.get(workspace=ws,name=\"myenv\",version=\"1\")\n",
|
"restored_environment = Environment.get(workspace=ws,name=\"myenv\",version=\"1\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"print(\"Attributes of restored environment\")\n",
|
"print(\"Attributes of restored environment\")\n",
|
||||||
"restored_environment"
|
"restored_environment"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Other ways to create environments\n",
|
"## Other ways to create environments\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### From existing Conda environment\n",
|
"### From existing Conda environment\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can create an environment from existing conda environment. This make it easy to reuse your local interactive environment in Azure ML remote runs. For example, if you've created conda environment using\n",
|
"You can create an environment from existing conda environment. This make it easy to reuse your local interactive environment in Azure ML remote runs. For example, if you've created conda environment using\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"conda create -n mycondaenv\n",
|
"conda create -n mycondaenv\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"you can create Azure ML environment out of that conda environment using\n",
|
"you can create Azure ML environment out of that conda environment using\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"myenv = Environment.from_existing_conda_environment(name=\"myenv\",conda_environment_name=\"mycondaenv\")\n",
|
"myenv = Environment.from_existing_conda_environment(name=\"myenv\",conda_environment_name=\"mycondaenv\")\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"\n",
|
"\n",
|
||||||
"### From conda or pip files\n",
|
"### From conda or pip files\n",
|
||||||
"\n",
|
"\n",
|
||||||
"You can create environments from conda specification or pip requirements files using\n",
|
"You can create environments from conda specification or pip requirements files using\n",
|
||||||
"```\n",
|
"```\n",
|
||||||
"myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"path-to-conda-specification-file\")\n",
|
"myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"path-to-conda-specification-file\")\n",
|
||||||
"\n",
|
"\n",
|
||||||
"myenv = Environment.from_pip_requirements(name=\"myenv\", file_path=\"path-to-pip-requirements-file\")\n",
|
"myenv = Environment.from_pip_requirements(name=\"myenv\", file_path=\"path-to-pip-requirements-file\")\n",
|
||||||
"```\n"
|
"```\n"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Docker settings\n",
|
"## Docker settings\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Docker container provides an efficient way to encapsulate the dependencies. When you enable Docker, Azure ML builds a Docker image and creates a Python environment within that container, given your specifications. The Docker images are reused: the first run in a new environment typically takes longer as the image is build.\n",
|
"Docker container provides an efficient way to encapsulate the dependencies. When you enable Docker, Azure ML builds a Docker image and creates a Python environment within that container, given your specifications. The Docker images are reused: the first run in a new environment typically takes longer as the image is build.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"**Note:** For runs on local computer or attached virtual machine, that computer must have Docker installed and enabled. Machine Learning Compute has Docker pre-installed.\n",
|
"**Note:** For runs on local computer or attached virtual machine, that computer must have Docker installed and enabled. Machine Learning Compute has Docker pre-installed.\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Attribute ```docker.enabled``` controls whether to use Docker container or host OS for execution. "
|
"Attribute ```docker.enabled``` controls whether to use Docker container or host OS for execution. "
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"myenv.docker.enabled = True"
|
"myenv.docker.enabled = True"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can specify custom Docker base image and registry. This allows you to customize and control in detail the guest OS in which your training run executes. whether to use GPU, whether to use shared volumes, and shm size."
|
"You can specify custom Docker base image and registry. This allows you to customize and control in detail the guest OS in which your training run executes. whether to use GPU, whether to use shared volumes, and shm size."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"myenv.docker.base_image\n",
|
"myenv.docker.base_image\n",
|
||||||
"myenv.docker.base_image_registry"
|
"myenv.docker.base_image_registry"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"You can also specify whether to use GPU or shared volumes, and shm size."
|
"You can also specify whether to use GPU or shared volumes, and shm size."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": [
|
"source": [
|
||||||
"myenv.docker.gpu_support\n",
|
"myenv.docker.gpu_support\n",
|
||||||
"myenv.docker.shared_volumes\n",
|
"myenv.docker.shared_volumes\n",
|
||||||
"myenv.docker.shm_size"
|
"myenv.docker.shm_size"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Spark and Azure Databricks settings\n",
|
"## Spark and Azure Databricks settings\n",
|
||||||
"\n",
|
"\n",
|
||||||
"In addition to Python and Docker settings, Environment also contains attributes for Spark and Azure Databricks runs. These attributes become relevant when you submit runs on those compute targets."
|
"In addition to Python and Docker settings, Environment also contains attributes for Spark and Azure Databricks runs. These attributes become relevant when you submit runs on those compute targets."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"source": [
|
"source": [
|
||||||
"## Next steps\n",
|
"## Next steps\n",
|
||||||
"\n",
|
"\n",
|
||||||
"Learn more about remote runs on different compute targets:\n",
|
"Learn more about remote runs on different compute targets:\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* [Train on ML Compute](../../train-on-amlcompute)\n",
|
"* [Train on ML Compute](../../train-on-amlcompute)\n",
|
||||||
"\n",
|
"\n",
|
||||||
"* [Train on remote VM](../../train-on-remote-vm)"
|
"* [Train on remote VM](../../train-on-remote-vm)"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": null,
|
"execution_count": null,
|
||||||
"metadata": {},
|
"metadata": {},
|
||||||
"outputs": [],
|
"outputs": [],
|
||||||
"source": []
|
"source": []
|
||||||
}
|
}
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"name": "roastala"
|
|
||||||
}
|
|
||||||
],
|
],
|
||||||
"kernelspec": {
|
"metadata": {
|
||||||
"display_name": "Python 3.6",
|
"authors": [
|
||||||
"language": "python",
|
{
|
||||||
"name": "python36"
|
"name": "roastala"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.6",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python36"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.6.5"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 2
|
||||||
"name": "ipython",
|
}
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 2
|
|
||||||
}
|
|
||||||
Reference in New Issue
Block a user