Merge pull request #396 from rastala/master

version 1.0.41

Licenses/sdk-license/LICENSE

@@ -1,3 +1,4 @@
+    
 This software is made available to you on the condition that you agree to
 [your agreement][1] governing your use of Azure.
 If you do not have an existing agreement governing your use of Azure, you agree that 

NBSETUP.md

@@ -24,8 +24,8 @@ pip install azureml-sdk
 git clone https://github.com/Azure/MachineLearningNotebooks.git
 
 # below steps are optional
-# install the base SDK and a Jupyter notebook server
-pip install azureml-sdk[notebooks]
+# install the base SDK, Jupyter notebook server and tensorboard
+pip install azureml-sdk[notebooks,tensorboard]
 
 # install model explainability component
 pip install azureml-sdk[explain]

README.md

@@ -11,8 +11,7 @@ pip install azureml-sdk
 Read more detailed instructions on [how to set up your environment](./NBSETUP.md) using Azure Notebook service, your own Jupyter notebook server, or Docker.
 
 ## How to navigate and use the example notebooks?
-If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [Configuration](./configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. 
-It configures your notebook library to connect to an Azure Machine Learning workspace, and sets up your workspace and compute to be used by many of the other examples. 
+If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, you should always run the [Configuration](./configuration.ipynb) notebook first when setting up a notebook library on a new machine or in a new environment. It configures your notebook library to connect to an Azure Machine Learning workspace, and sets up your workspace and compute to be used by many of the other examples. 
 
 If you want to...
 
@@ -21,7 +20,7 @@ If you want to...
  * ...learn about experimentation and tracking run history, first [train within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then try [training on remote VM](./how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb) and [using logging APIs](./how-to-use-azureml/training/logging-api/logging-api.ipynb).
  * ...train deep learning models at scale, first learn about [Machine Learning Compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and then try [distributed hyperparameter tuning](./how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) and [distributed training](./how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb).
  * ...deploy models as a realtime scoring service, first learn the basics by [training within Notebook and deploying to Azure Container Instance](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then learn how to [register and manage models, and create Docker images](./how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb), and [production deploy models on Azure Kubernetes Cluster](./how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb).
- * ...deploy models as a batch scoring service, first [train a model within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), learn how to [register and manage models](./how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb), then [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and [use Machine Learning Pipelines to deploy your model](./how-to-use-azureml/machine-learning-pipelines/pipeline-mpi-batch-prediction.ipynb).
+ * ...deploy models as a batch scoring service, first [train a model within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), learn how to [register and manage models](./how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb), then [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and [use Machine Learning Pipelines to deploy your model](https://aka.ms/pl-batch-scoring).
  * ...monitor your deployed models, learn about using [App Insights](./how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb) and [model data collection](./how-to-use-azureml/deployment/enable-data-collection-for-models-in-aks/enable-data-collection-for-models-in-aks.ipynb).
 
 ## Tutorials
@@ -56,8 +55,24 @@ Visit following repos to see projects contributed by Azure ML users:
  - [Fine tune natural language processing models using Azure Machine Learning service](https://github.com/Microsoft/AzureML-BERT)
  - [Fashion MNIST with Azure ML SDK](https://github.com/amynic/azureml-sdk-fashion)
  
- 
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/README.png)
+## Data/Telemetry 
+This repository collects usage data and sends it to Mircosoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
 
- 
- 
+To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
+
+```sh
+    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)"
+```
+This URL will be slightly different depending on the file. 
+
+## Data/Telemetry 
+This repository collects usage data and sends it to Mircosoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
+
+To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
+
+```sh
+    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)"
+```
+This URL will be slightly different depending on the file. 
+
+ ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/README.png)

configuration.ipynb

@@ -1,376 +1,293 @@
 {
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Configuration\n",
-        "\n",
-        "_**Setting up your Azure Machine Learning services workspace and configuring your notebook library**_\n",
-        "\n",
-        "---\n",
-        "---\n",
-        "\n",
-        "## Table of Contents\n",
-        "\n",
-        "1. [Introduction](#Introduction)\n",
-        "    1. What is an Azure Machine Learning workspace\n",
-        "1. [Setup](#Setup)\n",
-        "    1. Azure subscription\n",
-        "    1. Azure ML SDK and other library installation\n",
-        "    1. Azure Container Instance registration\n",
-        "1. [Configure your Azure ML Workspace](#Configure%20your%20Azure%20ML%20workspace)\n",
-        "    1. Workspace parameters\n",
-        "    1. Access your workspace\n",
-        "    1. Create a new workspace\n",
-        "    1. Create compute resources\n",
-        "1. [Next steps](#Next%20steps)\n",
-        "\n",
-        "---\n",
-        "\n",
-        "## Introduction\n",
-        "\n",
-        "This notebook configures your library of notebooks to connect to an Azure Machine Learning (ML) workspace.  In this case, a library contains all of the notebooks in the current folder and any nested folders.  You can configure this notebook library to use an existing workspace or create a new workspace.\n",
-        "\n",
-        "Typically you will need to run this notebook only once per notebook library as all other notebooks will use connection information that is written here.  If you want to redirect your notebook library to work with a different workspace, then you should re-run this notebook.\n",
-        "\n",
-        "In this notebook you will\n",
-        "* Learn about getting an Azure subscription\n",
-        "* Specify your workspace parameters\n",
-        "* Access or create your workspace\n",
-        "* Add a default compute cluster for your workspace\n",
-        "\n",
-        "### What is an Azure Machine Learning workspace\n",
-        "\n",
-        "An Azure ML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows.  In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Setup\n",
-        "\n",
-        "This section describes activities required before you can access any Azure ML services functionality."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### 1. Azure Subscription\n",
-        "\n",
-        "In order to create an Azure ML Workspace, first you need access to an Azure subscription.  An Azure subscription allows you to manage storage, compute, and other assets in the Azure cloud.  You can [create a new subscription](https://azure.microsoft.com/en-us/free/) or access existing subscription information from the [Azure portal](https://portal.azure.com).  Later in this notebook you will need information such as your subscription ID in order to create and access AML workspaces.\n",
-        "\n",
-        "### 2. Azure ML SDK and other library installation\n",
-        "\n",
-        "If you are running in your own environment, follow [SDK installation instructions](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment).  If you are running in Azure Notebooks or another Microsoft managed environment, the SDK is already installed.\n",
-        "\n",
-        "Also install following libraries to your environment. Many of the example notebooks depend on them\n",
-        "\n",
-        "```\n",
-        "(myenv) $ conda install -y matplotlib tqdm scikit-learn\n",
-        "```\n",
-        "\n",
-        "Once installation is complete, the following cell checks the Azure ML SDK version:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "install"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "import azureml.core\n",
-        "\n",
-        "print(\"This notebook was created using version 1.0.23 of the Azure ML SDK\")\n",
-        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If you are using an older version of the SDK then this notebook was created using, you should upgrade your SDK.\n",
-        "\n",
-        "### 3. Azure Container Instance registration\n",
-        "Azure Machine Learning uses of [Azure Container Instance (ACI)](https://azure.microsoft.com/services/container-instances) to deploy dev/test web services. An Azure subscription needs to be registered to use ACI.  If you or the subscription owner have not yet registered ACI on your subscription, you will need to use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and execute the following commands.  Note that if you ran through the AML [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) you have already registered ACI. \n",
-        "\n",
-        "```shell\n",
-        "# check to see if ACI is already registered\n",
-        "(myenv) $ az provider show -n Microsoft.ContainerInstance -o table\n",
-        "\n",
-        "# if ACI is not registered, run this command.\n",
-        "# note you need to be the subscription owner in order to execute this command successfully.\n",
-        "(myenv) $ az provider register -n Microsoft.ContainerInstance\n",
-        "```\n",
-        "\n",
-        "---"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Configure your Azure ML workspace\n",
-        "\n",
-        "### Workspace parameters\n",
-        "\n",
-        "To use an AML Workspace, you will need to import the Azure ML SDK and supply the following information:\n",
-        "* Your subscription id\n",
-        "* A resource group name\n",
-        "* (optional) The region that will host your workspace\n",
-        "* A name for your workspace\n",
-        "\n",
-        "You can get your subscription ID from the [Azure portal](https://portal.azure.com).\n",
-        "\n",
-        "You will also need access to a [_resource group_](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), which organizes Azure resources and provides a default region for the resources in a group.  You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com).  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
-        "\n",
-        "The region to host your workspace will be used if you are creating a new workspace.  You do not need to specify this if you are using an existing workspace. You can find the list of supported regions [here](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=machine-learning-service).  You should pick a region that is close to your location or that contains your data.\n",
-        "\n",
-        "The name for your workspace is unique within the subscription and should be descriptive enough to discern among other AML Workspaces.  The subscription may be used only by you, or it may be used by your department or your entire enterprise, so choose a name that makes sense for your situation.\n",
-        "\n",
-        "The following cell allows you to specify your workspace parameters.  This cell uses the python method `os.getenv` to read values from environment variables which is useful for automation.  If no environment variable exists, the parameters will be set to the specified default values.  \n",
-        "\n",
-        "If you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace!  You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n",
-        "\n",
-        "Replace the default values in the cell below with your workspace parameters"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "subscription_id = os.getenv(\"SUBSCRIPTION_ID\", default=\"<my-subscription-id>\")\n",
-        "resource_group = os.getenv(\"RESOURCE_GROUP\", default=\"<my-resource-group>\")\n",
-        "workspace_name = os.getenv(\"WORKSPACE_NAME\", default=\"<my-workspace-name>\")\n",
-        "workspace_region = os.getenv(\"WORKSPACE_REGION\", default=\"eastus2\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Access your workspace\n",
-        "\n",
-        "The following cell uses the Azure ML SDK to attempt to load the workspace specified by your parameters.  If this cell succeeds, your notebook library will be configured to access the workspace from all notebooks using the `Workspace.from_config()` method.  The cell can fail if the specified workspace doesn't exist or you don't have permissions to access it. "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace\n",
-        "\n",
-        "try:\n",
-        "    ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n",
-        "    # write the details of the workspace to a configuration file to the notebook library\n",
-        "    ws.write_config()\n",
-        "    print(\"Workspace configuration succeeded. Skip the workspace creation steps below\")\n",
-        "except:\n",
-        "    print(\"Workspace not accessible. Change your parameters or create a new workspace below\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Create a new workspace\n",
-        "\n",
-        "If you don't have an existing workspace and are the owner of the subscription or resource group, you can create a new workspace.  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
-        "\n",
-        "**Note**: As with other Azure services, there are limits on certain resources (for example AmlCompute quota) associated with the Azure ML service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
-        "\n",
-        "This cell will create an Azure ML workspace for you in a subscription provided you have the correct permissions.\n",
-        "\n",
-        "This will fail if:\n",
-        "* You do not have permission to create a workspace in the resource group\n",
-        "* You do not have permission to create a resource group if it's non-existing.\n",
-        "* You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
-        "\n",
-        "If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "create workspace"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace\n",
-        "\n",
-        "# Create the workspace using the specified parameters\n",
-        "ws = Workspace.create(name = workspace_name,\n",
-        "                      subscription_id = subscription_id,\n",
-        "                      resource_group = resource_group, \n",
-        "                      location = workspace_region,\n",
-        "                      create_resource_group = True,\n",
-        "                      exist_ok = True)\n",
-        "ws.get_details()\n",
-        "\n",
-        "# write the details of the workspace to a configuration file to the notebook library\n",
-        "ws.write_config()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Create compute resources for your training experiments\n",
-        "\n",
-        "Many of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n",
-        "\n",
-        "To create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors.  Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n",
-        "\n",
-        "The cluster parameters are:\n",
-        "* vm_size - this describes the virtual machine type and size used in the cluster.  All machines in the cluster are the same type.  You can get the list of vm sizes available in your region by using the CLI command\n",
-        "\n",
-        "```shell\n",
-        "az vm list-skus -o tsv\n",
-        "```\n",
-        "* min_nodes - this sets the minimum size of the cluster.  If you set the minimum to 0 the cluster will shut down all nodes while note in use.  Setting this number to a value higher than 0 will allow for faster start-up times, but you will also be billed when the cluster is not in use.\n",
-        "* max_nodes - this sets the maximum size of the cluster.  Setting this to a larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n",
-        "\n",
-        "\n",
-        "To create a **CPU** cluster now, run the cell below. The autoscale settings mean that the cluster will scale down to 0 nodes when inactive and up to 4 nodes when busy."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpucluster\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
-        "    print(\"Found existing cpucluster\")\n",
-        "except ComputeTargetException:\n",
-        "    print(\"Creating new cpucluster\")\n",
-        "    \n",
-        "    # Specify the configuration for the new cluster\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n",
-        "                                                           min_nodes=0,\n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # Create the cluster with the specified name and configuration\n",
-        "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
-        "    \n",
-        "    # Wait for the cluster to complete, show the output log\n",
-        "    cpu_cluster.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "To create a **GPU** cluster, run the cell below. Note that your subscription must have sufficient quota for GPU VMs or the command will fail. To increase quota, see [these instructions](https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request). "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your GPU cluster\n",
-        "gpu_cluster_name = \"gpucluster\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
-        "    print(\"Found existing gpu cluster\")\n",
-        "except ComputeTargetException:\n",
-        "    print(\"Creating new gpucluster\")\n",
-        "    \n",
-        "    # Specify the configuration for the new cluster\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
-        "                                                           min_nodes=0,\n",
-        "                                                           max_nodes=4)\n",
-        "    # Create the cluster with the specified name and configuration\n",
-        "    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n",
-        "\n",
-        "    # Wait for the cluster to complete, show the output log\n",
-        "    gpu_cluster.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "---\n",
-        "\n",
-        "## Next steps\n",
-        "\n",
-        "In this notebook you configured this notebook library to connect easily to an Azure ML workspace.  You can copy this notebook to your own libraries to connect them to you workspace, or use it to bootstrap new workspaces completely.\n",
-        "\n",
-        "If you came here from another notebook, you can return there and complete that exercise, or you can try out the [Tutorials](./tutorials) or jump into \"how-to\" notebooks and start creating and deploying models.  A good place to start is the [train within notebook](./how-to-use-azureml/training/train-within-notebook) example that walks through a simplified but complete end to end machine learning process."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": []
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "roastala"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.5"
-    }
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+    "\n",
+    "Licensed under the MIT License."
+   ]
   },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/configuration.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Configuration\n",
+    "\n",
+    "_**Setting up your Azure Machine Learning services workspace and configuring your notebook library**_\n",
+    "\n",
+    "---\n",
+    "---\n",
+    "\n",
+    "## Table of Contents\n",
+    "\n",
+    "1. [Introduction](#Introduction)\n",
+    "    1. What is an Azure Machine Learning workspace\n",
+    "1. [Setup](#Setup)\n",
+    "    1. Azure subscription\n",
+    "    1. Azure ML SDK and other library installation\n",
+    "    1. Azure Container Instance registration\n",
+    "1. [Configure your Azure ML Workspace](#Configure%20your%20Azure%20ML%20workspace)\n",
+    "    1. Workspace parameters\n",
+    "    1. Access your workspace\n",
+    "    1. Create a new workspace\n",
+    "1. [Next steps](#Next%20steps)\n",
+    "\n",
+    "---\n",
+    "\n",
+    "## Introduction\n",
+    "\n",
+    "This notebook configures your library of notebooks to connect to an Azure Machine Learning (ML) workspace.  In this case, a library contains all of the notebooks in the current folder and any nested folders.  You can configure this notebook library to use an existing workspace or create a new workspace.\n",
+    "\n",
+    "Typically you will need to run this notebook only once per notebook library as all other notebooks will use connection information that is written here.  If you want to redirect your notebook library to work with a different workspace, then you should re-run this notebook.\n",
+    "\n",
+    "In this notebook you will\n",
+    "* Learn about getting an Azure subscription\n",
+    "* Specify your workspace parameters\n",
+    "* Access or create your workspace\n",
+    "* Add a default compute cluster for your workspace\n",
+    "\n",
+    "### What is an Azure Machine Learning workspace\n",
+    "\n",
+    "An Azure ML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows.  In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "This section describes activities required before you can access any Azure ML services functionality."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Azure Subscription\n",
+    "\n",
+    "In order to create an Azure ML Workspace, first you need access to an Azure subscription.  An Azure subscription allows you to manage storage, compute, and other assets in the Azure cloud.  You can [create a new subscription](https://azure.microsoft.com/en-us/free/) or access existing subscription information from the [Azure portal](https://portal.azure.com).  Later in this notebook you will need information such as your subscription ID in order to create and access AML workspaces.\n",
+    "\n",
+    "### 2. Azure ML SDK and other library installation\n",
+    "\n",
+    "If you are running in your own environment, follow [SDK installation instructions](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment).  If you are running in Azure Notebooks or another Microsoft managed environment, the SDK is already installed.\n",
+    "\n",
+    "Also install following libraries to your environment. Many of the example notebooks depend on them\n",
+    "\n",
+    "```\n",
+    "(myenv) $ conda install -y matplotlib tqdm scikit-learn\n",
+    "```\n",
+    "\n",
+    "Once installation is complete, the following cell checks the Azure ML SDK version:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "install"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import azureml.core\n",
+    "\n",
+    "print(\"This notebook was created using version 1.0.41 of the Azure ML SDK\")\n",
+    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you are using an older version of the SDK then this notebook was created using, you should upgrade your SDK.\n",
+    "\n",
+    "### 3. Azure Container Instance registration\n",
+    "Azure Machine Learning uses of [Azure Container Instance (ACI)](https://azure.microsoft.com/services/container-instances) to deploy dev/test web services. An Azure subscription needs to be registered to use ACI.  If you or the subscription owner have not yet registered ACI on your subscription, you will need to use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and execute the following commands.  Note that if you ran through the AML [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) you have already registered ACI. \n",
+    "\n",
+    "```shell\n",
+    "# check to see if ACI is already registered\n",
+    "(myenv) $ az provider show -n Microsoft.ContainerInstance -o table\n",
+    "\n",
+    "# if ACI is not registered, run this command.\n",
+    "# note you need to be the subscription owner in order to execute this command successfully.\n",
+    "(myenv) $ az provider register -n Microsoft.ContainerInstance\n",
+    "```\n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configure your Azure ML workspace\n",
+    "\n",
+    "### Workspace parameters\n",
+    "\n",
+    "To use an AML Workspace, you will need to import the Azure ML SDK and supply the following information:\n",
+    "* Your subscription id\n",
+    "* A resource group name\n",
+    "* (optional) The region that will host your workspace\n",
+    "* A name for your workspace\n",
+    "\n",
+    "You can get your subscription ID from the [Azure portal](https://portal.azure.com).\n",
+    "\n",
+    "You will also need access to a [_resource group_](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), which organizes Azure resources and provides a default region for the resources in a group.  You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com).  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
+    "\n",
+    "The region to host your workspace will be used if you are creating a new workspace.  You do not need to specify this if you are using an existing workspace. You can find the list of supported regions [here](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=machine-learning-service).  You should pick a region that is close to your location or that contains your data.\n",
+    "\n",
+    "The name for your workspace is unique within the subscription and should be descriptive enough to discern among other AML Workspaces.  The subscription may be used only by you, or it may be used by your department or your entire enterprise, so choose a name that makes sense for your situation.\n",
+    "\n",
+    "The following cell allows you to specify your workspace parameters.  This cell uses the python method `os.getenv` to read values from environment variables which is useful for automation.  If no environment variable exists, the parameters will be set to the specified default values.  \n",
+    "\n",
+    "If you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace!  You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n",
+    "\n",
+    "Replace the default values in the cell below with your workspace parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "subscription_id = os.getenv(\"SUBSCRIPTION_ID\", default=\"<my-subscription-id>\")\n",
+    "resource_group = os.getenv(\"RESOURCE_GROUP\", default=\"<my-resource-group>\")\n",
+    "workspace_name = os.getenv(\"WORKSPACE_NAME\", default=\"<my-workspace-name>\")\n",
+    "workspace_region = os.getenv(\"WORKSPACE_REGION\", default=\"eastus2\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Access your workspace\n",
+    "\n",
+    "The following cell uses the Azure ML SDK to attempt to load the workspace specified by your parameters.  If this cell succeeds, your notebook library will be configured to access the workspace from all notebooks using the `Workspace.from_config()` method.  The cell can fail if the specified workspace doesn't exist or you don't have permissions to access it. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Workspace\n",
+    "\n",
+    "try:\n",
+    "    ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n",
+    "    # write the details of the workspace to a configuration file to the notebook library\n",
+    "    ws.write_config()\n",
+    "    print(\"Workspace configuration succeeded. Skip the workspace creation steps below\")\n",
+    "except:\n",
+    "    print(\"Workspace not accessible. Change your parameters or create a new workspace below\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create a new workspace\n",
+    "\n",
+    "If you don't have an existing workspace and are the owner of the subscription or resource group, you can create a new workspace.  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
+    "\n",
+    "**Note**: The Workspace creation command will create default CPU and GPU compute clusters for you. As with other Azure services, there are limits on certain resources (for example AmlCompute quota) associated with the Azure ML service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n",
+    "\n",
+    "This cell will create an Azure ML workspace for you in a subscription provided you have the correct permissions.\n",
+    "\n",
+    "This will fail if:\n",
+    "* You do not have permission to create a workspace in the resource group\n",
+    "* You do not have permission to create a resource group if it's non-existing.\n",
+    "* You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
+    "\n",
+    "If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "create workspace"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "from azureml.core import Workspace\n",
+    "\n",
+    "# Create the workspace using the specified parameters\n",
+    "ws = Workspace.create(name = workspace_name,\n",
+    "                      subscription_id = subscription_id,\n",
+    "                      resource_group = resource_group, \n",
+    "                      location = workspace_region,\n",
+    "                      default_cpu_compute_target=Workspace.DEFAULT_CPU_CLUSTER_CONFIGURATION,\n",
+    "                      default_gpu_compute_target=Workspace.DEFAULT_GPU_CLUSTER_CONFIGURATION,\n",
+    "                      create_resource_group = True,\n",
+    "                      exist_ok = True)\n",
+    "ws.get_details()\n",
+    "\n",
+    "# write the details of the workspace to a configuration file to the notebook library\n",
+    "ws.write_config()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## Next steps\n",
+    "\n",
+    "In this notebook you configured this notebook library to connect easily to an Azure ML workspace.  You can copy this notebook to your own libraries to connect them to you workspace, or use it to bootstrap new workspaces completely.\n",
+    "\n",
+    "If you came here from another notebook, you can return there and complete that exercise, or you can try out the [Tutorials](./tutorials) or jump into \"how-to\" notebooks and start creating and deploying models.  A good place to start is the [train within notebook](./how-to-use-azureml/training/train-within-notebook) example that walks through a simplified but complete end to end machine learning process."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": "roastala"
+   }
+  ],
+  "kernelspec": {
+   "display_name": "Python 3.6",
+   "language": "python",
+   "name": "python36"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

how-to-use-azureml/README.md

@@ -2,7 +2,7 @@
 
 Learn how to use Azure Machine Learning services for experimentation and model management.
 
-If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration Notebook](../configuration.ipynb) first if you haven't already to establish your connection to the AzureML Workspace. Then, run the notebooks in following recommended order.
+As a pre-requisite, run the [configuration Notebook](../configuration.ipynb) notebook first to set up your Azure ML Workspace. Then, run the notebooks in following recommended order.
 
 * [train-within-notebook](./training/train-within-notebook): Train a model hile tracking run history, and learn how to deploy the model as web service to Azure Container Instance.
 * [train-on-local](./training/train-on-local): Learn how to submit a run to local computer and use Azure ML managed run configuration.
@@ -15,6 +15,3 @@ If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherw
 * [enable-app-insights-in-production-service](./deployment/enable-app-insights-in-production-service) Learn how to use App Insights with production web service.
  
 Find quickstarts, end-to-end tutorials, and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).
-
-
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)

how-to-use-azureml/automated-machine-learning/automl_env.yml

@@ -10,7 +10,7 @@ dependencies:
 - urllib3<1.24
 - scipy>=1.0.0,<=1.1.0
 - scikit-learn>=0.19.0,<=0.20.3
-- pandas>=0.22.0,<0.23.0
+- pandas>=0.22.0,<=0.23.4
 - py-xgboost<=0.80
 
 - pip:

how-to-use-azureml/automated-machine-learning/automl_env_mac.yml

@@ -0,0 +1,22 @@
+name: azure_automl
+dependencies:
+  # The python interpreter version.
+  # Currently Azure ML only supports 3.5.2 and later.
+- nomkl
+- python>=3.5.2,<3.6.8
+- nb_conda
+- matplotlib==2.1.0
+- numpy>=1.11.0,<=1.16.2
+- cython
+- urllib3<1.24
+- scipy>=1.0.0,<=1.1.0
+- scikit-learn>=0.19.0,<=0.20.3
+- pandas>=0.22.0,<0.23.0
+- py-xgboost<=0.80
+
+- pip:
+  # Required packages for AzureML execution, history, and data preparation.
+  - azureml-sdk[automl,explain]
+  - azureml-widgets
+  - pandas_ml
+  

how-to-use-azureml/automated-machine-learning/automl_setup_mac.sh

@@ -12,7 +12,7 @@ fi
 
 if [ "$AUTOML_ENV_FILE" == "" ]
 then
-  AUTOML_ENV_FILE="automl_env.yml"
+  AUTOML_ENV_FILE="automl_env_mac.yml"
 fi
 
 if [ ! -f $AUTOML_ENV_FILE ]; then

how-to-use-azureml/automated-machine-learning/classification-with-deployment/auto-ml-classification-with-deployment.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-deployment/auto-ml-classification-with-deployment.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/classification-with-onnx/auto-ml-classification-with-onnx.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-onnx/auto-ml-classification-with-onnx.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -66,11 +73,12 @@
         "import numpy as np\n",
         "import pandas as pd\n",
         "from sklearn import datasets\n",
+        "from sklearn.model_selection import train_test_split\n",
         "\n",
         "import azureml.core\n",
         "from azureml.core.experiment import Experiment\n",
         "from azureml.core.workspace import Workspace\n",
-        "from azureml.train.automl import AutoMLConfig"
+        "from azureml.train.automl import AutoMLConfig, constants"
       ]
     },
     {
@@ -106,7 +114,7 @@
       "source": [
         "## Data\n",
         "\n",
-        "This uses scikit-learn's [load_digits](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) method."
+        "This uses scikit-learn's [load_iris](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html) method."
       ]
     },
     {
@@ -115,11 +123,17 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "digits = datasets.load_digits()\n",
+        "iris = datasets.load_iris()\n",
+        "X_train, X_test, y_train, y_test = train_test_split(iris.data, \n",
+        "                                                    iris.target, \n",
+        "                                                    test_size=0.2, \n",
+        "                                                    random_state=0)\n",
         "\n",
-        "# Exclude the first 100 rows from training so that they can be used for test.\n",
-        "X_train = digits.data[100:,:]\n",
-        "y_train = digits.target[100:]"
+        "# Convert the X_train and X_test to pandas DataFrame and set column names,\n",
+        "# This is needed for initializing the input variable names of ONNX model, \n",
+        "# and the prediction with the ONNX model using the inference helper.\n",
+        "X_train = pd.DataFrame(X_train, columns=['c1', 'c2', 'c3', 'c4'])\n",
+        "X_test = pd.DataFrame(X_test, columns=['c1', 'c2', 'c3', 'c4'])"
       ]
     },
     {
@@ -155,9 +169,10 @@
         "                             primary_metric = 'AUC_weighted',\n",
         "                             iteration_timeout_minutes = 60,\n",
         "                             iterations = 10,\n",
-        "                             verbosity = logging.INFO,\n",
+        "                             verbosity = logging.INFO,                             \n",
         "                             X = X_train, \n",
         "                             y = y_train,\n",
+        "                             preprocess=True,\n",
         "                             enable_onnx_compatible_models=True,\n",
         "                             path = project_folder)"
       ]
@@ -249,10 +264,69 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.train.automl._vendor.automl.client.core.common.onnx_convert import OnnxConverter\n",
+        "from azureml.automl.core.onnx_convert import OnnxConverter\n",
         "onnx_fl_path = \"./best_model.onnx\"\n",
         "OnnxConverter.save_onnx_model(onnx_mdl, onnx_fl_path)"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Predict with the ONNX model, using onnxruntime package"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import sys\n",
+        "import json\n",
+        "from azureml.automl.core.onnx_convert import OnnxConvertConstants\n",
+        "\n",
+        "if sys.version_info < OnnxConvertConstants.OnnxIncompatiblePythonVersion:\n",
+        "    python_version_compatible = True\n",
+        "else:\n",
+        "    python_version_compatible = False\n",
+        "\n",
+        "try:\n",
+        "    import onnxruntime\n",
+        "    from azureml.automl.core.onnx_convert import OnnxInferenceHelper    \n",
+        "    onnxrt_present = True\n",
+        "except ImportError:\n",
+        "    onnxrt_present = False\n",
+        "\n",
+        "def get_onnx_res(run):\n",
+        "    res_path = '_debug_y_trans_converter.json'\n",
+        "    run.download_file(name=constants.MODEL_RESOURCE_PATH_ONNX, output_file_path=res_path)\n",
+        "    with open(res_path) as f:\n",
+        "        onnx_res = json.load(f)\n",
+        "    return onnx_res\n",
+        "\n",
+        "if onnxrt_present and python_version_compatible:    \n",
+        "    mdl_bytes = onnx_mdl.SerializeToString()\n",
+        "    onnx_res = get_onnx_res(best_run)\n",
+        "\n",
+        "    onnxrt_helper = OnnxInferenceHelper(mdl_bytes, onnx_res)\n",
+        "    pred_onnx, pred_prob_onnx = onnxrt_helper.predict(X_test)\n",
+        "\n",
+        "    print(pred_onnx)\n",
+        "    print(pred_prob_onnx)\n",
+        "else:\n",
+        "    if not python_version_compatible:\n",
+        "        print('Please use Python version 3.6 to run the inference helper.')    \n",
+        "    if not onnxrt_present:\n",
+        "        print('Please install the onnxruntime package to do the prediction with ONNX model.')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
     }
   ],
   "metadata": {

how-to-use-azureml/automated-machine-learning/classification-with-whitelisting/auto-ml-classification-with-whitelisting.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-whitelisting/auto-ml-classification-with-whitelisting.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/classification/auto-ml-classification.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification/auto-ml-classification.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -328,6 +335,12 @@
         "            print()\n",
         "            for estimator in step[1].estimators:\n",
         "                print_model(estimator[1], estimator[0]+ ' - ')\n",
+        "        elif hasattr(step[1], '_base_learners') and hasattr(step[1], '_meta_learner'):\n",
+        "            print(\"\\nMeta Learner\")\n",
+        "            pprint(step[1]._meta_learner)\n",
+        "            print()\n",
+        "            for estimator in step[1]._base_learners:\n",
+        "                print_model(estimator[1], estimator[0]+ ' - ')\n",
         "        else:\n",
         "            pprint(step[1].get_params())\n",
         "            print()\n",

how-to-use-azureml/automated-machine-learning/dataprep-remote-execution/auto-ml-dataprep-remote-execution.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/dataprep-remote-execution/auto-ml-dataprep-remote-execution.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -117,21 +124,12 @@
       "outputs": [],
       "source": [
         "# You can use `auto_read_file` which intelligently figures out delimiters and datatypes of a file.\n",
-        "# The data referenced here was pulled from `sklearn.datasets.load_digits()`.\n",
-        "simple_example_data_root = 'https://dprepdata.blob.core.windows.net/automl-notebook-data/'\n",
-        "X = dprep.auto_read_file(simple_example_data_root + 'X.csv').skip(1)  # Remove the header row.\n",
-        "\n",
+        "# The data referenced here was a 1MB simple random sample of the Chicago Crime data into a local temporary directory.\n",
         "# You can also use `read_csv` and `to_*` transformations to read (with overridable delimiter)\n",
         "# and convert column types manually.\n",
-        "# Here we read a comma delimited file and convert all columns to integers.\n",
-        "y = dprep.read_csv(simple_example_data_root + 'y.csv').to_long(dprep.ColumnSelector(term='.*', use_regex = True))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "You can peek the result of a Dataflow at any range using `skip(i)` and `head(j)`. Doing so evaluates only `j` records for all the steps in the Dataflow, which makes it fast even against large datasets."
+        "example_data = 'https://dprepdata.blob.core.windows.net/demo/crime0-random.csv'\n",
+        "dflow = dprep.auto_read_file(example_data).skip(1)  # Remove the header row.\n",
+        "dflow.get_profile()"
       ]
     },
     {
@@ -140,7 +138,30 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X.skip(1).head(5)"
+        "# As `Primary Type` is our y data, we need to drop the values those are null in this column.\n",
+        "dflow = dflow.drop_nulls('Primary Type')\n",
+        "dflow.head(5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Review the Data Preparation Result\n",
+        "\n",
+        "You can peek the result of a Dataflow at any range using `skip(i)` and `head(j)`. Doing so evaluates only `j` records for all the steps in the Dataflow, which makes it fast even against large datasets.\n",
+        "\n",
+        "`Dataflow` objects are immutable and are composed of a list of data preparation steps. A `Dataflow` object can be branched at any point for further usage."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X = dflow.drop_columns(columns=['Primary Type', 'FBI Code'])\n",
+        "y = dflow.keep_columns(columns=['Primary Type'], validate_column_exists=True)"
       ]
     },
     {
@@ -162,9 +183,8 @@
         "    \"iteration_timeout_minutes\" : 10,\n",
         "    \"iterations\" : 2,\n",
         "    \"primary_metric\" : 'AUC_weighted',\n",
-        "    \"preprocess\" : False,\n",
-        "    \"verbosity\" : logging.INFO,\n",
-        "    \"n_cross_validations\": 3\n",
+        "    \"preprocess\" : True,\n",
+        "    \"verbosity\" : logging.INFO\n",
         "}"
       ]
     },
@@ -181,7 +201,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "dsvm_name = 'mydsvmc'\n",
+        "dsvm_name = 'mydsvmb'\n",
         "\n",
         "try:\n",
         "    while ws.compute_targets[dsvm_name].provisioning_state == 'Creating':\n",
@@ -257,6 +277,23 @@
         "remote_run"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Pre-process cache cleanup\n",
+        "The preprocess data gets cache at user default file store. When the run is completed the cache can be cleaned by running below cell"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "remote_run.clean_preprocessor_cache()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -376,7 +413,8 @@
       "source": [
         "## Test\n",
         "\n",
-        "#### Load Test Data"
+        "#### Load Test Data\n",
+        "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
       ]
     },
     {
@@ -385,12 +423,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from sklearn import datasets\n",
-        "\n",
-        "digits = datasets.load_digits()\n",
-        "X_test = digits.data[:10, :]\n",
-        "y_test = digits.target[:10]\n",
-        "images = digits.images[:10]"
+        "dflow_test = dprep.auto_read_file(path='https://dprepdata.blob.core.windows.net/demo/crime0-test.csv').skip(1)\n",
+        "dflow_test = dflow_test.drop_nulls('Primary Type')"
       ]
     },
     {
@@ -398,7 +432,7 @@
       "metadata": {},
       "source": [
         "#### Testing Our Best Fitted Model\n",
-        "We will try to predict 2 digits and see how our model works."
+        "We will use confusion matrix to see how our model works."
       ]
     },
     {
@@ -407,65 +441,19 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "#Randomly select digits and test\n",
-        "from matplotlib import pyplot as plt\n",
-        "import numpy as np\n",
+        "from pandas_ml import ConfusionMatrix\n",
         "\n",
-        "for index in np.random.choice(len(y_test), 2, replace = False):\n",
-        "    print(index)\n",
-        "    predicted = fitted_model.predict(X_test[index:index + 1])[0]\n",
-        "    label = y_test[index]\n",
-        "    title = \"Label value = %d  Predicted value = %d \" % (label, predicted)\n",
-        "    fig = plt.figure(1, figsize=(3,3))\n",
-        "    ax1 = fig.add_axes((0,0,.8,.8))\n",
-        "    ax1.set_title(title)\n",
-        "    plt.imshow(images[index], cmap = plt.cm.gray_r, interpolation = 'nearest')\n",
-        "    plt.show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Appendix"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Capture the `Dataflow` Objects for Later Use in AutoML\n",
+        "y_test = dflow_test.keep_columns(columns=['Primary Type']).to_pandas_dataframe()\n",
+        "X_test = dflow_test.drop_columns(columns=['Primary Type', 'FBI Code']).to_pandas_dataframe()\n",
         "\n",
-        "`Dataflow` objects are immutable and are composed of a list of data preparation steps. A `Dataflow` object can be branched at any point for further usage."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# sklearn.digits.data + target\n",
-        "digits_complete = dprep.auto_read_file('https://dprepdata.blob.core.windows.net/automl-notebook-data/digits-complete.csv')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "`digits_complete` (sourced from `sklearn.datasets.load_digits()`) is forked into `dflow_X` to capture all the feature columns and `dflow_y` to capture the label column."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print(digits_complete.to_pandas_dataframe().shape)\n",
-        "labels_column = 'Column64'\n",
-        "dflow_X = digits_complete.drop_columns(columns = [labels_column])\n",
-        "dflow_y = digits_complete.keep_columns(columns = [labels_column])"
+        "\n",
+        "ypred = fitted_model.predict(X_test)\n",
+        "\n",
+        "cm = ConfusionMatrix(y_test['Primary Type'], ypred)\n",
+        "\n",
+        "print(cm)\n",
+        "\n",
+        "cm.plot()"
       ]
     }
   ],

how-to-use-azureml/automated-machine-learning/dataprep/auto-ml-dataprep.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/dataprep/auto-ml-dataprep.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -115,23 +122,12 @@
       "outputs": [],
       "source": [
         "# You can use `auto_read_file` which intelligently figures out delimiters and datatypes of a file.\n",
-        "# The data referenced here was pulled from `sklearn.datasets.load_digits()`.\n",
-        "simple_example_data_root = 'https://dprepdata.blob.core.windows.net/automl-notebook-data/'\n",
-        "X = dprep.auto_read_file(simple_example_data_root + 'X.csv').skip(1)  # Remove the header row.\n",
-        "\n",
+        "# The data referenced here was a 1MB simple random sample of the Chicago Crime data into a local temporary directory.\n",
         "# You can also use `read_csv` and `to_*` transformations to read (with overridable delimiter)\n",
         "# and convert column types manually.\n",
-        "# Here we read a comma delimited file and convert all columns to integers.\n",
-        "y = dprep.read_csv(simple_example_data_root + 'y.csv').to_long(dprep.ColumnSelector(term='.*', use_regex = True))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Review the Data Preparation Result\n",
-        "\n",
-        "You can peek the result of a Dataflow at any range using `skip(i)` and `head(j)`. Doing so evaluates only `j` records for all the steps in the Dataflow, which makes it fast even against large datasets."
+        "example_data = 'https://dprepdata.blob.core.windows.net/demo/crime0-random.csv'\n",
+        "dflow = dprep.auto_read_file(example_data).skip(1)  # Remove the header row.\n",
+        "dflow.get_profile()"
       ]
     },
     {
@@ -140,7 +136,30 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X.skip(1).head(5)"
+        "# As `Primary Type` is our y data, we need to drop the values those are null in this column.\n",
+        "dflow = dflow.drop_nulls('Primary Type')\n",
+        "dflow.head(5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Review the Data Preparation Result\n",
+        "\n",
+        "You can peek the result of a Dataflow at any range using `skip(i)` and `head(j)`. Doing so evaluates only `j` records for all the steps in the Dataflow, which makes it fast even against large datasets.\n",
+        "\n",
+        "`Dataflow` objects are immutable and are composed of a list of data preparation steps. A `Dataflow` object can be branched at any point for further usage."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X = dflow.drop_columns(columns=['Primary Type', 'FBI Code'])\n",
+        "y = dflow.keep_columns(columns=['Primary Type'], validate_column_exists=True)"
       ]
     },
     {
@@ -162,7 +181,7 @@
         "    \"iteration_timeout_minutes\" : 10,\n",
         "    \"iterations\" : 2,\n",
         "    \"primary_metric\" : 'AUC_weighted',\n",
-        "    \"preprocess\" : False,\n",
+        "    \"preprocess\" : True,\n",
         "    \"verbosity\" : logging.INFO\n",
         "}"
       ]
@@ -326,7 +345,8 @@
       "source": [
         "## Test\n",
         "\n",
-        "#### Load Test Data"
+        "#### Load Test Data\n",
+        "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
       ]
     },
     {
@@ -335,12 +355,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from sklearn import datasets\n",
-        "\n",
-        "digits = datasets.load_digits()\n",
-        "X_test = digits.data[:10, :]\n",
-        "y_test = digits.target[:10]\n",
-        "images = digits.images[:10]"
+        "dflow_test = dprep.auto_read_file(path='https://dprepdata.blob.core.windows.net/demo/crime0-test.csv').skip(1)\n",
+        "dflow_test = dflow_test.drop_nulls('Primary Type')"
       ]
     },
     {
@@ -348,7 +364,7 @@
       "metadata": {},
       "source": [
         "#### Testing Our Best Fitted Model\n",
-        "We will try to predict 2 digits and see how our model works."
+        "We will use confusion matrix to see how our model works."
       ]
     },
     {
@@ -357,65 +373,18 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "#Randomly select digits and test\n",
-        "from matplotlib import pyplot as plt\n",
-        "import numpy as np\n",
+        "from pandas_ml import ConfusionMatrix\n",
         "\n",
-        "for index in np.random.choice(len(y_test), 2, replace = False):\n",
-        "    print(index)\n",
-        "    predicted = fitted_model.predict(X_test[index:index + 1])[0]\n",
-        "    label = y_test[index]\n",
-        "    title = \"Label value = %d  Predicted value = %d \" % (label, predicted)\n",
-        "    fig = plt.figure(1, figsize=(3,3))\n",
-        "    ax1 = fig.add_axes((0,0,.8,.8))\n",
-        "    ax1.set_title(title)\n",
-        "    plt.imshow(images[index], cmap = plt.cm.gray_r, interpolation = 'nearest')\n",
-        "    plt.show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Appendix"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Capture the `Dataflow` Objects for Later Use in AutoML\n",
+        "y_test = dflow_test.keep_columns(columns=['Primary Type']).to_pandas_dataframe()\n",
+        "X_test = dflow_test.drop_columns(columns=['Primary Type', 'FBI Code']).to_pandas_dataframe()\n",
         "\n",
-        "`Dataflow` objects are immutable and are composed of a list of data preparation steps. A `Dataflow` object can be branched at any point for further usage."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# sklearn.digits.data + target\n",
-        "digits_complete = dprep.auto_read_file('https://dprepdata.blob.core.windows.net/automl-notebook-data/digits-complete.csv')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "`digits_complete` (sourced from `sklearn.datasets.load_digits()`) is forked into `dflow_X` to capture all the feature columns and `dflow_y` to capture the label column."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print(digits_complete.to_pandas_dataframe().shape)\n",
-        "labels_column = 'Column64'\n",
-        "dflow_X = digits_complete.drop_columns(columns = [labels_column])\n",
-        "dflow_y = digits_complete.keep_columns(columns = [labels_column])"
+        "ypred = fitted_model.predict(X_test)\n",
+        "\n",
+        "cm = ConfusionMatrix(y_test['Primary Type'], ypred)\n",
+        "\n",
+        "print(cm)\n",
+        "\n",
+        "cm.plot()"
       ]
     }
   ],

how-to-use-azureml/automated-machine-learning/exploring-previous-runs/auto-ml-exploring-previous-runs.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/exploring-previous-runs/auto-ml-exploring-previous-runs.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -220,7 +227,7 @@
         "|**X**|(sparse) array-like, shape = [n_samples, n_features]|\n",
         "|**y**|(sparse) array-like, shape = [n_samples, ], targets values.|\n",
         "|**n_cross_validations**|Number of cross validation splits.|\n",
-        "|**country**|The country used to generate holiday features. These should be ISO 3166 two-letter country codes (i.e. 'US', 'GB').|\n",
+        "|**country_or_region**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
         "|**path**|Relative path to the project folder.  AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder. "
       ]
     },
@@ -235,8 +242,8 @@
         "    \"time_column_name\": time_column_name,\n",
         "    # these columns are a breakdown of the total and therefore a leak\n",
         "    \"drop_column_names\": ['casual', 'registered'],\n",
-        "    # knowing the country allows Automated ML to bring in holidays\n",
-        "    \"country\" : 'US',\n",
+        "    # knowing the country/region allows Automated ML to bring in holidays\n",
+        "    \"country_or_region\" : 'US',\n",
         "    \"max_horizon\" : max_horizon,\n",
         "    \"target_lags\": 1    \n",
         "}\n",

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/missing-data-blacklist-early-termination/auto-ml-missing-data-blacklist-early-termination.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/missing-data-blacklist-early-termination/auto-ml-missing-data-blacklist-early-termination.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/model-explanation/auto-ml-model-explanation.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/model-explanation/auto-ml-model-explanation.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/remote-amlcompute/auto-ml-remote-amlcompute.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-amlcompute/auto-ml-remote-amlcompute.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -112,9 +119,7 @@
       "metadata": {},
       "source": [
         "### Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create an AmlCompute as your training compute resource.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -129,39 +134,34 @@
         "from azureml.core.compute import ComputeTarget\n",
         "\n",
         "# Choose a name for your cluster.\n",
-        "amlcompute_cluster_name = \"automlcl\"\n",
+        "amlcompute_cluster_name = \"cpucluster\"\n",
         "\n",
         "found = False\n",
+        "\n",
         "# Check if this compute target already exists in the workspace.\n",
+        "\n",
         "cts = ws.compute_targets\n",
         "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
         "    found = True\n",
         "    print('Found existing compute target.')\n",
         "    compute_target = cts[amlcompute_cluster_name]\n",
-        "    \n",
+        "\n",
         "if not found:\n",
         "    print('Creating a new compute target...')\n",
         "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\", # for GPU, use \"STANDARD_NC6\"\n",
         "                                                                #vm_priority = 'lowpriority', # optional\n",
         "                                                                max_nodes = 6)\n",
         "\n",
-        "    # Create the cluster.\n",
+        "    # Create the cluster.\\n\",\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
-        "    \n",
+        "\n",
         "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
         "    # If no min_node_count is provided, it will use the scale settings for the cluster.\n",
         "    compute_target.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)\n",
-        "    \n",
+        "\n",
         "     # For a more detailed view of current AmlCompute status, use get_status()."
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": []
-    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/remote-attach/auto-ml-remote-attach.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-attach/auto-ml-remote-attach.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/remote-execution/auto-ml-remote-execution.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-execution/auto-ml-remote-execution.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -130,7 +137,7 @@
         "    print('Found an existing DSVM.')\n",
         "except:\n",
         "    print('Creating a new DSVM.')\n",
-        "    dsvm_config = DsvmCompute.provisioning_configuration(vm_size = \"Standard_D2s_v3\")\n",
+        "    dsvm_config = DsvmCompute.provisioning_configuration(vm_size = \"Standard_D2_v2\")\n",
         "    dsvm_compute = DsvmCompute.create(ws, name = dsvm_name, provisioning_configuration = dsvm_config)\n",
         "    dsvm_compute.wait_for_completion(show_output = True)\n",
         "    print(\"Waiting one minute for ssh to be accessible\")\n",

how-to-use-azureml/automated-machine-learning/sample-weight/auto-ml-sample-weight.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/sample-weight/auto-ml-sample-weight.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/sparse-data-train-test-split/auto-ml-sparse-data-train-test-split.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/sparse-data-train-test-split/auto-ml-sparse-data-train-test-split.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/subsampling/auto-ml-subsampling-local.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/subsampling/auto-ml-subsampling-local.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/azure-databricks/README.md

@@ -26,4 +26,8 @@ You can use Azure Databricks as a compute target from [Azure Machine Learning Pi
 
 For more on SDK concepts, please refer to [notebooks](https://github.com/Azure/MachineLearningNotebooks).
 
-**Please let us know your feedback.**
+**Please let us know your feedback.**
+
+ 
+
+![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/README.png) 

how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.ipynb

@@ -11,6 +11,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/build-model-run-history-03.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -333,6 +340,13 @@
       "source": [
         "dbutils.notebook.exit(\"success\")"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.ipynb

@@ -11,6 +11,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/deploy-to-aci-04.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -277,6 +284,13 @@
         "#comment to not delete the web service\n",
         "myservice.delete()"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.ipynb

@@ -11,6 +11,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -203,6 +210,13 @@
         "#model.delete()\n",
         "aks_target.delete() "
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.ipynb

@@ -11,6 +11,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/ingest-data-02.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -139,6 +146,13 @@
       "metadata": {},
       "outputs": [],
       "source": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.ipynb

@@ -11,6 +11,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/installation-and-configuration-01.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -143,6 +150,13 @@
         "      'Subscription id: ' + ws.subscription_id, \n",
         "      'Resource group: ' + ws.resource_group, sep = '\\n')"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/automl/automl-databricks-local-01.ipynb

@@ -660,6 +660,13 @@
       "metadata": {},
       "outputs": [],
       "source": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/automl/automl-databricks-local-01.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/automl/automl-databricks-local-with-deployment.ipynb

@@ -796,6 +796,13 @@
       "source": [
         "myservice.delete()"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/automl/automl-databricks-local-with-deployment.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.ipynb

@@ -677,6 +677,13 @@
         "# Next: ADLA as a Compute Target\n",
         "To use ADLA as a compute target from Azure Machine Learning Pipeline, a AdlaStep is used. This [notebook](./aml-pipelines-use-adla-as-compute-target.ipynb) demonstrates the use of AdlaStep in Azure Machine Learning Pipeline."
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.png)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/deploy-to-cloud/model-register-and-deploy.ipynb

@@ -9,13 +9,20 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-cloud/model-register-and-deploy.png)"
-         ] 
-    },  
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/deploy-to-cloud/model-register-and-deploy.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-cloud/model-register-and-deploy.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -33,7 +40,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration](../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
       ]
     },
     {
@@ -279,4 +286,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deploy-to-local/register-model-deploy-local-advanced.ipynb

@@ -13,8 +13,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-local/register-model-deploy-local-advanced.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-local/register-model-deploy-local-advanced.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -35,7 +35,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise,make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
       ]
     },
     {
@@ -491,4 +491,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deploy-to-local/register-model-deploy-local.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-local/register-model-deploy-local.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deploy-to-local/register-model-deploy-local.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -346,4 +346,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/accelerated-models/README.md

@@ -13,8 +13,21 @@ To learn more about the azureml-accel-model classes, see the section [Model Clas
 
 ### Step 1: Create an Azure ML workspace
 Follow [these instructions](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python) to install the Azure ML SDK on your local machine, create an Azure ML workspace, and set up your notebook environment, which is required for the next step.
-  
-### Step 2: Install the Azure ML Accelerated Models SDK
+
+### Step 2: Check your FPGA quota
+Use the Azure CLI to check whether you have quota.
+
+```shell
+az vm list-usage --location "eastus" -o table
+```
+
+The other locations are ``southeastasia``, ``westeurope``, and ``westus2``.
+
+Under the "Name" column, look for "Standard PBS Family vCPUs" and ensure you have at least 6 vCPUs under "CurrentValue."
+
+If you do not have quota, then submit a request form [here](https://aka.ms/accelerateAI).
+
+### Step 3: Install the Azure ML Accelerated Models SDK
 Once you have set up your environment, install the Azure ML Accel Models SDK. This package requires tensorflow >= 1.6,<2.0 to be installed. 
 
 If you already have tensorflow >= 1.6,<2.0 installed in your development environment, you can install the SDK package using: 
@@ -35,7 +48,7 @@ If your machine supports GPU (for example, on an [Azure DSVM](https://docs.micro
 pip install azureml-accel-models[gpu]
 ```
 
-### Step 3: Follow our notebooks
+### Step 4: Follow our notebooks
 
 The notebooks in this repo walk through the following scenarios: 
 * [Quickstart](accelerated-models-quickstart.ipynb), deploy and inference a ResNet50 model trained on ImageNet

how-to-use-azureml/deployment/accelerated-models/accelerated-models-object-detection.ipynb

@@ -273,11 +273,12 @@
     "from azureml.core.compute import AksCompute, ComputeTarget\n",
     "\n",
     "# Uses the specific FPGA enabled VM (sku: Standard_PB6s)\n",
-    "# Authentication is enabled by default, but for testing we specify False\n",
+    "# Standard_PB6s are available in: eastus, westus2, westeurope, southeastasia\n",
     "prov_config = AksCompute.provisioning_configuration(vm_size = \"Standard_PB6s\",\n",
-    "                                                    agent_count = 1)\n",
+    "                                                    agent_count = 1, \n",
+    "                                                    location = \"eastus\")\n",
     "\n",
-    "aks_name = 'my-aks-pb6-ssd-vgg'\n",
+    "aks_name = 'aks-pb6-obj'\n",
     "# Create the cluster\n",
     "aks_target = ComputeTarget.create(workspace = ws, \n",
     "                                  name = aks_name, \n",
@@ -318,6 +319,7 @@
     "from azureml.core.webservice import Webservice, AksWebservice\n",
     "\n",
     "# Set the web service configuration (for creating a test service, we don't want autoscale enabled)\n",
+    "# Authentication is enabled by default, but for testing we specify False\n",
     "aks_config = AksWebservice.deploy_configuration(autoscale_enabled=False,\n",
     "                                                num_replicas=1,\n",
     "                                                auth_enabled = False)\n",

how-to-use-azureml/deployment/accelerated-models/accelerated-models-quickstart.ipynb

@@ -341,9 +341,10 @@
     "from azureml.core.compute import AksCompute, ComputeTarget\n",
     "\n",
     "# Uses the specific FPGA enabled VM (sku: Standard_PB6s)\n",
-    "# Authentication is enabled by default, but for testing we specify False\n",
+    "# Standard_PB6s are available in: eastus, westus2, westeurope, southeastasia\n",
     "prov_config = AksCompute.provisioning_configuration(vm_size = \"Standard_PB6s\",\n",
-    "                                                    agent_count = 1)\n",
+    "                                                    agent_count = 1, \n",
+    "                                                    location = \"eastus\")\n",
     "\n",
     "aks_name = 'my-aks-pb6'\n",
     "# Create the cluster\n",
@@ -386,6 +387,7 @@
     "from azureml.core.webservice import Webservice, AksWebservice\n",
     "\n",
     "#Set the web service configuration (for creating a test service, we don't want autoscale enabled)\n",
+    "# Authentication is enabled by default, but for testing we specify False\n",
     "aks_config = AksWebservice.deploy_configuration(autoscale_enabled=False,\n",
     "                                                num_replicas=1,\n",
     "                                                auth_enabled = False)\n",

how-to-use-azureml/deployment/accelerated-models/accelerated-models-training.ipynb

@@ -47,7 +47,7 @@
     "    * [Transfer Learning](#transfer-learning)\n",
     "    * [Transfer Learning with Custom Weights](#custom-weights)\n",
     "* [Create Image](#create-image)\n",
-    "* [Deploy Model](#deploy-model)\n",
+    "* [Deploy Image](#deploy-image)\n",
     "* [Test the service](#test-service)\n",
     "* [Clean-up](#cleanup)\n",
     "* [Appendix](#appendix)"
@@ -630,11 +630,12 @@
     "from azureml.core.compute import AksCompute, ComputeTarget\n",
     "\n",
     "# Uses the specific FPGA enabled VM (sku: Standard_PB6s)\n",
-    "# Authentication is enabled by default, but for testing we specify False\n",
+    "# Standard_PB6s are available in: eastus, westus2, westeurope, southeastasia\n",
     "prov_config = AksCompute.provisioning_configuration(vm_size = \"Standard_PB6s\",\n",
-    "                                                    agent_count = 1)\n",
+    "                                                    agent_count = 1,\n",
+    "                                                    location = \"eastus\")\n",
     "\n",
-    "aks_name = 'my-aks-pb6-training'\n",
+    "aks_name = 'aks-pb6-tl'\n",
     "# Create the cluster\n",
     "aks_target = ComputeTarget.create(workspace = ws, \n",
     "                                  name = aks_name, \n",
@@ -675,6 +676,7 @@
     "from azureml.core.webservice import Webservice, AksWebservice\n",
     "\n",
     "# Set the web service configuration (for creating a test service, we don't want autoscale enabled)\n",
+    "# Authentication is enabled by default, but for testing we specify False\n",
     "aks_config = AksWebservice.deploy_configuration(autoscale_enabled=False,\n",
     "                                                num_replicas=1,\n",
     "                                                auth_enabled = False)\n",

how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb

@@ -29,8 +29,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -495,4 +495,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/enable-data-collection-for-models-in-aks/enable-data-collection-for-models-in-aks.ipynb

@@ -1,5 +1,12 @@
 {
   "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/enable-data-collection-for-models-in-aks/enable-data-collection-for-models-in-aks.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -24,13 +31,6 @@
         "4. Build new image and deploy it. "
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/enable-data-collection-for-models-in-aks/enable-data-collection-for-models-in-aks.png)"
-         ] 
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -475,4 +475,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/onnx/README.md

@@ -1,5 +1,4 @@
 # ONNX on Azure Machine Learning
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/README.png)
 
 These tutorials show how to create and deploy Open Neural Network eXchange ([ONNX](http://onnx.ai)) models in Azure Machine Learning environments using [ONNX Runtime](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-build-deploy-onnx) for inference. Once deployed as a web service, you can ping the model with your own set of images to be analyzed!
 
@@ -37,3 +36,4 @@ Licensed under the MIT License.
 These tutorials were developed by Vinitra Swamy and Prasanth Pulavarthi of the Microsoft AI Frameworks team and adapted for presentation at Microsoft Ignite 2018.
 
 
+ ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/README.png)

how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb

@@ -13,8 +13,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -255,7 +255,7 @@
       "source": [
         "from azureml.core.conda_dependencies import CondaDependencies \n",
         "\n",
-        "myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime\",\"azureml-core\"])\n",
+        "myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime==0.4.0\",\"azureml-core\"])\n",
         "\n",
         "with open(\"myenv.yml\",\"w\") as f:\n",
         "    f.write(myenv.serialize_to_string())"
@@ -440,4 +440,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.ipynb

@@ -8,12 +8,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -813,4 +813,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -817,4 +817,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.ipynb

@@ -13,8 +13,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -40,7 +40,7 @@
         "To make the best use of your time, make sure you have done the following:\n",
         "\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (config.json)"
       ]
@@ -424,4 +424,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb

@@ -13,8 +13,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -670,4 +670,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-       {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -474,4 +474,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-       {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -450,4 +450,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-binary-classification.ipynb

@@ -1,265 +1,279 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Breast cancer diagnosis classification with scikit-learn (run model explainer locally)"
-   ]
-  },
-  {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-binary-classification.png)"
-         ] 
+        "# Breast cancer diagnosis classification with scikit-learn (run model explainer locally)"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package\n",
-    "\n",
-    "1. Train a SVM classification model using Scikit-learn\n",
-    "2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
-    "3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
-    "4. Visualize the global and local explanations with the visualization dashboard."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.datasets import load_breast_cancer\n",
-    "from sklearn import svm\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 1. Run model explainer locally with full data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load the breast cancer diagnosis data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "breast_cancer_data = load_breast_cancer()\n",
-    "classes = breast_cancer_data.target_names.tolist()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Split data into train and test\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a SVM classification model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
-    "model = clf.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions (global explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
-    "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Sorted SHAP values\n",
-    "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
-    "# Corresponding feature names\n",
-    "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
-    "# feature ranks (based on original order of features)\n",
-    "print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
-    "# per class feature names\n",
-    "print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
-    "# per class feature importance values\n",
-    "print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions as a collection of local (instance-level) explanations"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# feature shap values for all features and all data points in the training data\n",
-    "print('local importance values: {}'.format(global_explanation.local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain local data points (individual instances)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# explain the first member of the test set\n",
-    "instance_num = 0\n",
-    "local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# get the prediction for the first member of the test set and explain why model made that prediction\n",
-    "prediction_value = clf.predict(x_test)[instance_num]\n",
-    "\n",
-    "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
-    "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
-    "\n",
-    "\n",
-    "dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 2. Load visualization dashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ExplanationDashboard(global_explanation, model, x_test)"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-binary-classification.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package\n",
+        "\n",
+        "1. Train a SVM classification model using Scikit-learn\n",
+        "2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
+        "3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
+        "4. Visualize the global and local explanations with the visualization dashboard."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.datasets import load_breast_cancer\n",
+        "from sklearn import svm\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Run model explainer locally with full data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load the breast cancer diagnosis data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "breast_cancer_data = load_breast_cancer()\n",
+        "classes = breast_cancer_data.target_names.tolist()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Split data into train and test\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a SVM classification model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
+        "model = clf.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions (global explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
+        "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Sorted SHAP values\n",
+        "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
+        "# Corresponding feature names\n",
+        "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
+        "# feature ranks (based on original order of features)\n",
+        "print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
+        "# per class feature names\n",
+        "print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
+        "# per class feature importance values\n",
+        "print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions as a collection of local (instance-level) explanations"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# feature shap values for all features and all data points in the training data\n",
+        "print('local importance values: {}'.format(global_explanation.local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain local data points (individual instances)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# explain the first member of the test set\n",
+        "instance_num = 0\n",
+        "local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# get the prediction for the first member of the test set and explain why model made that prediction\n",
+        "prediction_value = clf.predict(x_test)[instance_num]\n",
+        "\n",
+        "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
+        "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
+        "\n",
+        "\n",
+        "dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 2. Load visualization dashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
+        "!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "# Or, in Jupyter Labs, uncomment below\n",
+        "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
+        "# jupyter labextension install microsoft-mli-widget"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ExplanationDashboard(global_explanation, model, x_test)"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-multiclass-classification.ipynb

@@ -1,266 +1,280 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Iris flower classification with scikit-learn (run model explainer locally)"
-   ]
-  },
-  {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-multiclass-classification.png)"
-         ] 
+        "# Iris flower classification with scikit-learn (run model explainer locally)"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package\n",
-    "\n",
-    "1. Train a SVM classification model using Scikit-learn\n",
-    "2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
-    "3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
-    "4. Visualize the global and local explanations with the visualization dashboard."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn import svm\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 1. Run model explainer locally with full data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load the breast cancer diagnosis data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "iris = load_iris()\n",
-    "X = iris['data']\n",
-    "y = iris['target']\n",
-    "classes = iris['target_names']\n",
-    "feature_names = iris['feature_names']"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Split data into train and test\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a SVM classification model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
-    "model = clf.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(model, x_train, features = feature_names, classes=classes)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions (global explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Sorted SHAP values\n",
-    "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
-    "# Corresponding feature names\n",
-    "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
-    "# feature ranks (based on original order of features)\n",
-    "print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
-    "# per class feature names\n",
-    "print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
-    "# per class feature importance values\n",
-    "print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions as a collection of local (instance-level) explanations"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# feature shap values for all features and all data points in the training data\n",
-    "print('local importance values: {}'.format(global_explanation.local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain local data points (individual instances)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# explain the first member of the test set\n",
-    "instance_num = 0\n",
-    "local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# get the prediction for the first member of the test set and explain why model made that prediction\n",
-    "prediction_value = clf.predict(x_test)[instance_num]\n",
-    "\n",
-    "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
-    "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
-    "\n",
-    "\n",
-    "dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load visualization dashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ExplanationDashboard(global_explanation, model, x_test)"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-multiclass-classification.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package\n",
+        "\n",
+        "1. Train a SVM classification model using Scikit-learn\n",
+        "2. Run 'explain_model' with full data in local mode, which doesn't contact any Azure services\n",
+        "3. Run 'explain_model' with summarized data in local mode, which doesn't contact any Azure services\n",
+        "4. Visualize the global and local explanations with the visualization dashboard."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.datasets import load_iris\n",
+        "from sklearn import svm\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Run model explainer locally with full data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load the breast cancer diagnosis data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "iris = load_iris()\n",
+        "X = iris['data']\n",
+        "y = iris['target']\n",
+        "classes = iris['target_names']\n",
+        "feature_names = iris['feature_names']"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Split data into train and test\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a SVM classification model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
+        "model = clf.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(model, x_train, features = feature_names, classes=classes)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions (global explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Sorted SHAP values\n",
+        "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
+        "# Corresponding feature names\n",
+        "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
+        "# feature ranks (based on original order of features)\n",
+        "print('global importance rank: {}'.format(global_explanation.global_importance_rank))\n",
+        "# per class feature names\n",
+        "print('ranked per class feature names: {}'.format(global_explanation.get_ranked_per_class_names()))\n",
+        "# per class feature importance values\n",
+        "print('ranked per class feature values: {}'.format(global_explanation.get_ranked_per_class_values()))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions as a collection of local (instance-level) explanations"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# feature shap values for all features and all data points in the training data\n",
+        "print('local importance values: {}'.format(global_explanation.local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain local data points (individual instances)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# explain the first member of the test set\n",
+        "instance_num = 0\n",
+        "local_explanation = tabular_explainer.explain_local(x_test[instance_num,:])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# get the prediction for the first member of the test set and explain why model made that prediction\n",
+        "prediction_value = clf.predict(x_test)[instance_num]\n",
+        "\n",
+        "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
+        "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
+        "\n",
+        "\n",
+        "dict(zip(sorted_local_importance_names, sorted_local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load visualization dashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
+        "!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "# Or, in Jupyter Labs, uncomment below\n",
+        "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
+        "# jupyter labextension install microsoft-mli-widget"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ExplanationDashboard(global_explanation, model, x_test)"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-regression.ipynb

@@ -1,258 +1,272 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Boston Housing Price Prediction with scikit-learn (run model explainer locally)"
-   ]
-  },
-  {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-regression.png)"
-         ] 
+        "# Boston Housing Price Prediction with scikit-learn (run model explainer locally)"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package\n",
-    "\n",
-    "1. Train a GradientBoosting regression model using Scikit-learn\n",
-    "2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
-    "3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
-    "4. Visualize the global and local explanations with the visualization dashboard."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn import datasets\n",
-    "from sklearn.ensemble import GradientBoostingRegressor\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 1. Run model explainer locally with full data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load the Boston house price data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "boston_data = datasets.load_boston()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Split data into train and test\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a GradientBoosting Regression model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "reg = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
-    "                                learning_rate=0.1, loss='huber',\n",
-    "                                random_state=1)\n",
-    "model = reg.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(model, x_train, features = boston_data.feature_names)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions (global explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
-    "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Sorted SHAP values \n",
-    "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
-    "# Corresponding feature names\n",
-    "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
-    "# feature ranks (based on original order of features)\n",
-    "print('global importance rank: {}'.format(global_explanation.global_importance_rank))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions as a collection of local (instance-level) explanations"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# feature shap values for all features and all data points in the training data\n",
-    "print('local importance values: {}'.format(global_explanation.local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain local data points (individual instances)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "local_explanation = tabular_explainer.explain_local(x_test[0,:])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# sorted local feature importance information; reflects the original feature order\n",
-    "sorted_local_importance_names = local_explanation.get_ranked_local_names()\n",
-    "sorted_local_importance_values = local_explanation.get_ranked_local_values()\n",
-    "\n",
-    "print('sorted local importance names: {}'.format(sorted_local_importance_names))\n",
-    "print('sorted local importance values: {}'.format(sorted_local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load visualization dashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ExplanationDashboard(global_explanation, model, x_test)"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-local/explain-local-sklearn-regression.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package\n",
+        "\n",
+        "1. Train a GradientBoosting regression model using Scikit-learn\n",
+        "2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
+        "3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
+        "4. Visualize the global and local explanations with the visualization dashboard."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn import datasets\n",
+        "from sklearn.ensemble import GradientBoostingRegressor\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Run model explainer locally with full data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load the Boston house price data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "boston_data = datasets.load_boston()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Split data into train and test\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a GradientBoosting Regression model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "reg = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
+        "                                learning_rate=0.1, loss='huber',\n",
+        "                                random_state=1)\n",
+        "model = reg.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(model, x_train, features = boston_data.feature_names)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions (global explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
+        "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Sorted SHAP values \n",
+        "print('ranked global importance values: {}'.format(global_explanation.get_ranked_global_values()))\n",
+        "# Corresponding feature names\n",
+        "print('ranked global importance names: {}'.format(global_explanation.get_ranked_global_names()))\n",
+        "# feature ranks (based on original order of features)\n",
+        "print('global importance rank: {}'.format(global_explanation.global_importance_rank))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dict(zip(global_explanation.get_ranked_global_names(), global_explanation.get_ranked_global_values()))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions as a collection of local (instance-level) explanations"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# feature shap values for all features and all data points in the training data\n",
+        "print('local importance values: {}'.format(global_explanation.local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain local data points (individual instances)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "local_explanation = tabular_explainer.explain_local(x_test[0,:])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# sorted local feature importance information; reflects the original feature order\n",
+        "sorted_local_importance_names = local_explanation.get_ranked_local_names()\n",
+        "sorted_local_importance_values = local_explanation.get_ranked_local_values()\n",
+        "\n",
+        "print('sorted local importance names: {}'.format(sorted_local_importance_names))\n",
+        "print('sorted local importance values: {}'.format(sorted_local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load visualization dashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
+        "!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "# Or, in Jupyter Labs, uncomment below\n",
+        "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
+        "# jupyter labextension install microsoft-mli-widget"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ExplanationDashboard(global_explanation, model, x_test)"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/explain-model/explain-tabular-data-raw-features/explain-sklearn-raw-features.ipynb

@@ -1,288 +1,302 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Summary\n",
-    "From raw data that is a mixture of categoricals and numeric, featurize the categoricals using one hot encoding. Use tabular explainer to get explain object and then get raw feature importances"
-   ]
-  },
-  {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-raw-features/explain-sklearn-raw-features.png)"
-         ] 
+        "# Summary\n",
+        "From raw data that is a mixture of categoricals and numeric, featurize the categoricals using one hot encoding. Use tabular explainer to get explain object and then get raw feature importances"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package on raw features\n",
-    "\n",
-    "1. Train a Logistic Regression model using Scikit-learn\n",
-    "2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
-    "3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
-    "4. Visualize the global and local explanations with the visualization dashboard."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "This example needs sklearn-pandas. If it is not installed, uncomment and run the following line."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#!pip install sklearn-pandas"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.pipeline import Pipeline\n",
-    "from sklearn.impute import SimpleImputer\n",
-    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
-    "from sklearn.linear_model import LogisticRegression\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
-    "from sklearn_pandas import DataFrameMapper\n",
-    "import pandas as pd\n",
-    "import numpy as np"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "titanic_url = ('https://raw.githubusercontent.com/amueller/'\n",
-    "               'scipy-2017-sklearn/091d371/notebooks/datasets/titanic3.csv')\n",
-    "data = pd.read_csv(titanic_url)\n",
-    "# fill missing values\n",
-    "data = data.fillna(method=\"ffill\")\n",
-    "data = data.fillna(method=\"bfill\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 1. Run model explainer locally with full data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Similar to example [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py), use a subset of columns"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.model_selection import train_test_split\n",
-    "\n",
-    "numeric_features = ['age', 'fare']\n",
-    "categorical_features = ['embarked', 'sex', 'pclass']\n",
-    "\n",
-    "y = data['survived'].values\n",
-    "X = data[categorical_features + numeric_features]\n",
-    "\n",
-    "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.pipeline import Pipeline\n",
-    "from sklearn.impute import SimpleImputer\n",
-    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
-    "from sklearn_pandas import DataFrameMapper\n",
-    "\n",
-    "# Impute, standardize the numeric features and one-hot encode the categorical features.    \n",
-    "\n",
-    "transformations = [\n",
-    "    ([\"age\", \"fare\"], Pipeline(steps=[\n",
-    "        ('imputer', SimpleImputer(strategy='median')),\n",
-    "        ('scaler', StandardScaler())\n",
-    "    ])),\n",
-    "    ([\"embarked\"], Pipeline(steps=[\n",
-    "        (\"imputer\", SimpleImputer(strategy='constant', fill_value='missing')), \n",
-    "        (\"encoder\", OneHotEncoder(sparse=False))])),\n",
-    "    ([\"sex\", \"pclass\"], OneHotEncoder(sparse=False))    \n",
-    "]\n",
-    "\n",
-    "\n",
-    "# Append classifier to preprocessing pipeline.\n",
-    "# Now we have a full prediction pipeline.\n",
-    "clf = Pipeline(steps=[('preprocessor', DataFrameMapper(transformations)),\n",
-    "                      ('classifier', LogisticRegression(solver='lbfgs'))])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a Logistic Regression model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = clf.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(clf.steps[-1][1], initialization_examples=x_train, features=x_train.columns, transformations=transformations)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
-    "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "sorted_global_importance_values = global_explanation.get_ranked_global_values()\n",
-    "sorted_global_importance_names = global_explanation.get_ranked_global_names()\n",
-    "dict(zip(sorted_global_importance_names, sorted_global_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions as a collection of local (instance-level) explanations"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# explain the first member of the test set\n",
-    "local_explanation = tabular_explainer.explain_local(x_test[:1])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# get the prediction for the first member of the test set and explain why model made that prediction\n",
-    "prediction_value = clf.predict(x_test)[0]\n",
-    "\n",
-    "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
-    "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
-    "\n",
-    "# Sorted local SHAP values\n",
-    "print('ranked local importance values: {}'.format(sorted_local_importance_values))\n",
-    "# Corresponding feature names\n",
-    "print('ranked local importance names: {}'.format(sorted_local_importance_names))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 2. Load visualization dashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ExplanationDashboard(global_explanation, model, x_test)"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-raw-features/explain-sklearn-raw-features.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package on raw features\n",
+        "\n",
+        "1. Train a Logistic Regression model using Scikit-learn\n",
+        "2. Run 'explain_model' with full dataset in local mode, which doesn't contact any Azure services.\n",
+        "3. Run 'explain_model' with summarized dataset in local mode, which doesn't contact any Azure services.\n",
+        "4. Visualize the global and local explanations with the visualization dashboard."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This example needs sklearn-pandas. If it is not installed, uncomment and run the following line."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "#!pip install sklearn-pandas"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.pipeline import Pipeline\n",
+        "from sklearn.impute import SimpleImputer\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn.linear_model import LogisticRegression\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
+        "from sklearn_pandas import DataFrameMapper\n",
+        "import pandas as pd\n",
+        "import numpy as np"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "titanic_url = ('https://raw.githubusercontent.com/amueller/'\n",
+        "               'scipy-2017-sklearn/091d371/notebooks/datasets/titanic3.csv')\n",
+        "data = pd.read_csv(titanic_url)\n",
+        "# fill missing values\n",
+        "data = data.fillna(method=\"ffill\")\n",
+        "data = data.fillna(method=\"bfill\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Run model explainer locally with full data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Similar to example [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py), use a subset of columns"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.model_selection import train_test_split\n",
+        "\n",
+        "numeric_features = ['age', 'fare']\n",
+        "categorical_features = ['embarked', 'sex', 'pclass']\n",
+        "\n",
+        "y = data['survived'].values\n",
+        "X = data[categorical_features + numeric_features]\n",
+        "\n",
+        "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.pipeline import Pipeline\n",
+        "from sklearn.impute import SimpleImputer\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn_pandas import DataFrameMapper\n",
+        "\n",
+        "# Impute, standardize the numeric features and one-hot encode the categorical features.    \n",
+        "\n",
+        "transformations = [\n",
+        "    ([\"age\", \"fare\"], Pipeline(steps=[\n",
+        "        ('imputer', SimpleImputer(strategy='median')),\n",
+        "        ('scaler', StandardScaler())\n",
+        "    ])),\n",
+        "    ([\"embarked\"], Pipeline(steps=[\n",
+        "        (\"imputer\", SimpleImputer(strategy='constant', fill_value='missing')), \n",
+        "        (\"encoder\", OneHotEncoder(sparse=False))])),\n",
+        "    ([\"sex\", \"pclass\"], OneHotEncoder(sparse=False))    \n",
+        "]\n",
+        "\n",
+        "\n",
+        "# Append classifier to preprocessing pipeline.\n",
+        "# Now we have a full prediction pipeline.\n",
+        "clf = Pipeline(steps=[('preprocessor', DataFrameMapper(transformations)),\n",
+        "                      ('classifier', LogisticRegression(solver='lbfgs'))])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a Logistic Regression model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "model = clf.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(clf.steps[-1][1], initialization_examples=x_train, features=x_train.columns, transformations=transformations)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
+        "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sorted_global_importance_values = global_explanation.get_ranked_global_values()\n",
+        "sorted_global_importance_names = global_explanation.get_ranked_global_names()\n",
+        "dict(zip(sorted_global_importance_names, sorted_global_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions as a collection of local (instance-level) explanations"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# explain the first member of the test set\n",
+        "local_explanation = tabular_explainer.explain_local(x_test[:1])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# get the prediction for the first member of the test set and explain why model made that prediction\n",
+        "prediction_value = clf.predict(x_test)[0]\n",
+        "\n",
+        "sorted_local_importance_values = local_explanation.get_ranked_local_values()[prediction_value]\n",
+        "sorted_local_importance_names = local_explanation.get_ranked_local_names()[prediction_value]\n",
+        "\n",
+        "# Sorted local SHAP values\n",
+        "print('ranked local importance values: {}'.format(sorted_local_importance_values))\n",
+        "# Corresponding feature names\n",
+        "print('ranked local importance names: {}'.format(sorted_local_importance_names))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 2. Load visualization dashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Note you will need to have extensions enabled prior to jupyter kernel starting\n",
+        "!jupyter nbextension install --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "!jupyter nbextension enable --py --sys-prefix azureml.contrib.explain.model.visualize\n",
+        "# Or, in Jupyter Labs, uncomment below\n",
+        "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
+        "# jupyter labextension install microsoft-mli-widget"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.explain.model.visualize import ExplanationDashboard"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ExplanationDashboard(global_explanation, model, x_test)"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-classification.ipynb

@@ -1,262 +1,262 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Breast cancer diagnosis classification with scikit-learn (save model explanations via AML Run History)"
-   ]
-  },
-  {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-classification.png)"
-         ] 
+        "# Breast cancer diagnosis classification with scikit-learn (save model explanations via AML Run History)"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package\n",
-    "\n",
-    "1. Train a SVM classification model using Scikit-learn\n",
-    "2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from sklearn.datasets import load_breast_cancer\n",
-    "from sklearn import svm\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 1. Run model explainer locally with full data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load the breast cancer diagnosis data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "breast_cancer_data = load_breast_cancer()\n",
-    "classes = breast_cancer_data.target_names.tolist()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Split data into train and test\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a SVM classification model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
-    "model = clf.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions (global explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
-    "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# 2. Save Model Explanation With AML Run History"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import azureml.core\n",
-    "from azureml.core import Workspace, Experiment, Run\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
-    "from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
-    "# Check core SDK version number\n",
-    "print(\"SDK version:\", azureml.core.VERSION)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ws = Workspace.from_config()\n",
-    "print('Workspace name: ' + ws.name, \n",
-    "      'Azure region: ' + ws.location, \n",
-    "      'Subscription id: ' + ws.subscription_id, \n",
-    "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "experiment_name = 'explain_model'\n",
-    "experiment = Experiment(ws, experiment_name)\n",
-    "run = experiment.start_logging()\n",
-    "client = ExplanationClient.from_run(run)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Uploading model explanation data for storage or visualization in webUX\n",
-    "# The explanation can then be downloaded on any compute\n",
-    "client.upload_model_explanation(global_explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Get model explanation data\n",
-    "explanation = client.download_model_explanation()\n",
-    "local_importance_values = explanation.local_importance_values\n",
-    "expected_values = explanation.expected_values"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Get the top k (e.g., 4) most important features with their importance values\n",
-    "explanation = client.download_model_explanation(top_k=4)\n",
-    "global_importance_values = explanation.get_ranked_global_values()\n",
-    "global_importance_names = explanation.get_ranked_global_names()\n",
-    "per_class_names = explanation.get_ranked_per_class_names()[0]\n",
-    "per_class_values = explanation.get_ranked_per_class_values()[0]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "print('per class feature importance values: {}'.format(per_class_values))\n",
-    "print('per class feature importance names: {}'.format(per_class_names))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "dict(zip(per_class_names, per_class_values))"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-classification.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package\n",
+        "\n",
+        "1. Train a SVM classification model using Scikit-learn\n",
+        "2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.datasets import load_breast_cancer\n",
+        "from sklearn import svm\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Run model explainer locally with full data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load the breast cancer diagnosis data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "breast_cancer_data = load_breast_cancer()\n",
+        "classes = breast_cancer_data.target_names.tolist()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Split data into train and test\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "x_train, x_test, y_train, y_test = train_test_split(breast_cancer_data.data, breast_cancer_data.target, test_size=0.2, random_state=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a SVM classification model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "clf = svm.SVC(gamma=0.001, C=100., probability=True)\n",
+        "model = clf.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(model, x_train, features=breast_cancer_data.feature_names, classes=classes)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions (global explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
+        "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 2. Save Model Explanation With AML Run History"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import azureml.core\n",
+        "from azureml.core import Workspace, Experiment, Run\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
+        "from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
+        "# Check core SDK version number\n",
+        "print(\"SDK version:\", azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "print('Workspace name: ' + ws.name, \n",
+        "      'Azure region: ' + ws.location, \n",
+        "      'Subscription id: ' + ws.subscription_id, \n",
+        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "experiment_name = 'explain_model'\n",
+        "experiment = Experiment(ws, experiment_name)\n",
+        "run = experiment.start_logging()\n",
+        "client = ExplanationClient.from_run(run)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Uploading model explanation data for storage or visualization in webUX\n",
+        "# The explanation can then be downloaded on any compute\n",
+        "client.upload_model_explanation(global_explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Get model explanation data\n",
+        "explanation = client.download_model_explanation()\n",
+        "local_importance_values = explanation.local_importance_values\n",
+        "expected_values = explanation.expected_values"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Get the top k (e.g., 4) most important features with their importance values\n",
+        "explanation = client.download_model_explanation(top_k=4)\n",
+        "global_importance_values = explanation.get_ranked_global_values()\n",
+        "global_importance_names = explanation.get_ranked_global_names()\n",
+        "per_class_names = explanation.get_ranked_per_class_names()[0]\n",
+        "per_class_values = explanation.get_ranked_per_class_values()[0]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print('per class feature importance values: {}'.format(per_class_values))\n",
+        "print('per class feature importance names: {}'.format(per_class_names))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dict(zip(per_class_names, per_class_values))"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-regression.ipynb

@@ -1,276 +1,276 @@
 {
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Boston Housing Price Prediction with scikit-learn (save model explanations via AML Run History)"
-   ]
-  },
-   {
+  "cells": [
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-regression.png)"
-         ] 
+        "# Boston Housing Price Prediction with scikit-learn (save model explanations via AML Run History)"
+      ]
     },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-    "\n",
-    "Licensed under the MIT License."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Explain a model with the AML explain-model package\n",
-    "\n",
-    "1. Train a GradientBoosting regression model using Scikit-learn\n",
-    "2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Save Model Explanation With AML Run History"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "#Import Iris dataset\n",
-    "from sklearn import datasets\n",
-    "from sklearn.ensemble import GradientBoostingRegressor\n",
-    "\n",
-    "import azureml.core\n",
-    "from azureml.core import Workspace, Experiment, Run\n",
-    "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
-    "from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
-    "# Check core SDK version number\n",
-    "print(\"SDK version:\", azureml.core.VERSION)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ws = Workspace.from_config()\n",
-    "print('Workspace name: ' + ws.name, \n",
-    "      'Azure region: ' + ws.location, \n",
-    "      'Subscription id: ' + ws.subscription_id, \n",
-    "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "experiment_name = 'explain_model'\n",
-    "experiment = Experiment(ws, experiment_name)\n",
-    "run = experiment.start_logging()\n",
-    "client = ExplanationClient.from_run(run)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Load the Boston house price data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "boston_data = datasets.load_boston()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Split data into train and test\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Train a GradientBoosting Regression model, which you want to explain"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "clf = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
-    "                                learning_rate=0.1, loss='huber',\n",
-    "                                random_state=1)\n",
-    "model = clf.fit(x_train, y_train)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain predictions on your local machine"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "tabular_explainer = TabularExplainer(model, x_train, features=boston_data.feature_names)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain overall model predictions (global explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
-    "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
-    "global_explanation = tabular_explainer.explain_global(x_test)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Uploading model explanation data for storage or visualization in webUX\n",
-    "# The explanation can then be downloaded on any compute\n",
-    "client.upload_model_explanation(global_explanation)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Get model explanation data\n",
-    "explanation = client.download_model_explanation()\n",
-    "local_importance_values = explanation.local_importance_values\n",
-    "expected_values = explanation.expected_values"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Print the values\n",
-    "print('expected values: {}'.format(expected_values))\n",
-    "print('local importance values: {}'.format(local_importance_values))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Get the top k (e.g., 4) most important features with their importance values\n",
-    "explanation = client.download_model_explanation(top_k=4)\n",
-    "global_importance_values = explanation.get_ranked_global_values()\n",
-    "global_importance_names = explanation.get_ranked_global_names()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "print('global importance values: {}'.format(global_importance_values))\n",
-    "print('global importance names: {}'.format(global_importance_names))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Explain individual instance predictions (local explanation) ##### needs to get updated with the new build"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "local_explanation = tabular_explainer.explain_local(x_test[0,:])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# local feature importance information\n",
-    "local_importance_values = local_explanation.local_importance_values\n",
-    "print('local importance values: {}'.format(local_importance_values))"
-   ]
-  }
- ],
- "metadata": {
-  "authors": [
-   {
-    "name": "mesameki"
-   }
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/explain-tabular-data-run-history/explain-run-history-sklearn-regression.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Explain a model with the AML explain-model package\n",
+        "\n",
+        "1. Train a GradientBoosting regression model using Scikit-learn\n",
+        "2. Run 'explain_model' with AML Run History, which leverages run history service to store and manage the explanation data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Save Model Explanation With AML Run History"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "#Import Iris dataset\n",
+        "from sklearn import datasets\n",
+        "from sklearn.ensemble import GradientBoostingRegressor\n",
+        "\n",
+        "import azureml.core\n",
+        "from azureml.core import Workspace, Experiment, Run\n",
+        "from azureml.explain.model.tabular_explainer import TabularExplainer\n",
+        "from azureml.contrib.explain.model.explanation.explanation_client import ExplanationClient\n",
+        "# Check core SDK version number\n",
+        "print(\"SDK version:\", azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "print('Workspace name: ' + ws.name, \n",
+        "      'Azure region: ' + ws.location, \n",
+        "      'Subscription id: ' + ws.subscription_id, \n",
+        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "experiment_name = 'explain_model'\n",
+        "experiment = Experiment(ws, experiment_name)\n",
+        "run = experiment.start_logging()\n",
+        "client = ExplanationClient.from_run(run)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load the Boston house price data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "boston_data = datasets.load_boston()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Split data into train and test\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "x_train, x_test, y_train, y_test = train_test_split(boston_data.data, boston_data.target, test_size=0.2, random_state=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train a GradientBoosting Regression model, which you want to explain"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "clf = GradientBoostingRegressor(n_estimators=100, max_depth=4,\n",
+        "                                learning_rate=0.1, loss='huber',\n",
+        "                                random_state=1)\n",
+        "model = clf.fit(x_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain predictions on your local machine"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "tabular_explainer = TabularExplainer(model, x_train, features=boston_data.feature_names)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain overall model predictions (global explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
+        "# x_train can be passed as well, but with more examples explanations will take longer although they may be more accurate\n",
+        "global_explanation = tabular_explainer.explain_global(x_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Uploading model explanation data for storage or visualization in webUX\n",
+        "# The explanation can then be downloaded on any compute\n",
+        "client.upload_model_explanation(global_explanation)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Get model explanation data\n",
+        "explanation = client.download_model_explanation()\n",
+        "local_importance_values = explanation.local_importance_values\n",
+        "expected_values = explanation.expected_values"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Print the values\n",
+        "print('expected values: {}'.format(expected_values))\n",
+        "print('local importance values: {}'.format(local_importance_values))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Get the top k (e.g., 4) most important features with their importance values\n",
+        "explanation = client.download_model_explanation(top_k=4)\n",
+        "global_importance_values = explanation.get_ranked_global_values()\n",
+        "global_importance_names = explanation.get_ranked_global_names()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print('global importance values: {}'.format(global_importance_values))\n",
+        "print('global importance names: {}'.format(global_importance_names))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explain individual instance predictions (local explanation) ##### needs to get updated with the new build"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "local_explanation = tabular_explainer.explain_local(x_test[0,:])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# local feature importance information\n",
+        "local_importance_values = local_explanation.local_importance_values\n",
+        "print('local importance values: {}'.format(local_importance_values))"
+      ]
+    }
   ],
-  "kernelspec": {
-   "display_name": "Python 3.6",
-   "language": "python",
-   "name": "python36"
+  "metadata": {
+    "authors": [
+      {
+        "name": "mesameki"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.8"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/machine-learning-pipelines/README.md

@@ -37,23 +37,11 @@ Azure Machine Learning Pipelines optimize for simplicity, speed, and efficiency.
 In this directory, there are two types of notebooks: 
 
 * The first type of notebooks will introduce you to core Azure Machine Learning Pipelines features. These notebooks below belong in this category, and are designed to go in sequence; they're all located in the "intro-to-pipelines" folder:
-
-1. [aml-pipelines-getting-started.ipynb](https://aka.ms/pl-get-started): Start with this notebook to understand the concepts of using Azure Machine Learning Pipelines. This notebook will show you how to runs steps in parallel and in sequence.
-2. [aml-pipelines-with-data-dependency-steps.ipynb](https://aka.ms/pl-data-dep): This notebooks shows how to connect steps in your pipeline using data. Data produced by one step is used by subsequent steps to force an explicit dependency between steps. 
-3. [aml-pipelines-publish-and-run-using-rest-endpoint.ipynb](https://aka.ms/pl-pub-rep): Once you are satisfied with your iterative runs in, you could publish your pipeline to get a REST endpoint which could be invoked from non-Pythons clients as well. 
-4. [aml-pipelines-data-transfer.ipynb](https://aka.ms/pl-data-trans): This notebook shows how you transfer data between supported datastores.
-5. [aml-pipelines-use-databricks-as-compute-target.ipynb](https://aka.ms/pl-databricks): This notebooks shows how you can use Pipelines to send your compute payload to Azure Databricks.
-6. [aml-pipelines-use-adla-as-compute-target.ipynb](https://aka.ms/pl-adla): This notebook shows how you can use Azure Data Lake Analytics (ADLA) as a compute target.
-7. [aml-pipelines-how-to-use-estimatorstep.ipynb](https://aka.ms/pl-estimator): This notebook shows how to use the EstimatorStep.
-7. [aml-pipelines-parameter-tuning-with-hyperdrive.ipynb](https://aka.ms/pl-hyperdrive): HyperDriveStep in Pipelines shows how you can do hyper parameter tuning using Pipelines.
-8. [aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.ipynb](https://aka.ms/pl-azbatch): AzureBatchStep can be used to run your custom code in AzureBatch cluster.
-9. [aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb](https://aka.ms/pl-schedule): Once you publish a Pipeline, you can schedule it to trigger based on an interval or on data change in a defined datastore. 
-10. [aml-pipelines-with-automated-machine-learning-step.ipynb](https://aka.ms/pl-automl): AutoMLStep in Pipelines shows how you can do automated machine learning using Pipelines.
+Take a look at [intro-to-pipelines](./intro-to-pipelines/) for the list of notebooks that introduce Azure Machine Learning concepts for you.
 
 * The second type of notebooks illustrate more sophisticated scenarios, and are independent of each other. These notebooks include:
 
 1. [pipeline-batch-scoring.ipynb](https://aka.ms/pl-batch-score): This notebook demonstrates how to run a batch scoring job using Azure Machine Learning pipelines.
-2. [pipeline-style-transfer.ipynb](https://aka.ms/pl-style-trans)
-
+2. [pipeline-style-transfer.ipynb](https://aka.ms/pl-style-trans): This notebook demonstrates a multi-step pipeline that uses GPU compute.
 
  ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/README.png)

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-data-transfer.ipynb

@@ -8,15 +8,13 @@
         "Licensed under the MIT License."
       ]
     },
-    
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-data-transfer.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-data-transfer.png)"
+      ]
     },
-     
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -62,7 +60,7 @@
       "source": [
         "## Initialize Workspace\n",
         "\n",
-        "Initialize a workspace object from persisted configuration.If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure the config file is present at .\\config.json\n",
+        "Initialize a workspace object from persisted configuration. If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure the config file is present at .\\config.json\n",
         "\n",
         "If you don't have a config.json file, please go through the configuration Notebook located here:\n",
         "https://github.com/Azure/MachineLearningNotebooks. \n",
@@ -475,4 +473,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-getting-started.ipynb

@@ -8,15 +8,13 @@
         "Licensed under the MIT License."
       ]
     },
-    
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-getting-started.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-getting-started.png)"
+      ]
     },
-        
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -46,7 +44,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites and Azure Machine Learning Basics\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the configuration Notebook located at https://github.com/Azure/MachineLearningNotebooks first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc. \n"
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) located at https://github.com/Azure/MachineLearningNotebooks first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc. \n"
       ]
     },
     {
@@ -155,7 +153,7 @@
       "metadata": {},
       "source": [
         "### Datastore concepts\n",
-        "A [Datastore](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore(class)?view=azure-ml-py) is a place where data can be stored that is then made accessible to a compute either by means of mounting or copying the data to the compute target. \n",
+        "A [Datastore](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore.datastore?view=azure-ml-py) is a place where data can be stored that is then made accessible to a compute either by means of mounting or copying the data to the compute target. \n",
         "\n",
         "A Datastore can either be backed by an Azure File Storage (default) or by an Azure Blob Storage.\n",
         "\n",
@@ -198,7 +196,7 @@
       "metadata": {},
       "source": [
         "#### (Optional) See your files using Azure Portal\n",
-        "Once you successfully uploaded the files, you can browse to them (or upload more files) using [Azure Portal](https://portal.azure.com). At the portal, make sure you have selected **AzureML Nursery** as your subscription (click *Resource Groups* and then select the subscription). Then look for your **Machine Learning Workspace** (it has your *alias* as the name). It has a link to your storage. Click on the storage link. It will take you to a page where you can see [Blobs](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction), [Files](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction), [Tables](https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-overview), and [Queues](https://docs.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction). We have just uploaded a file to the Blob storage and another one to the File storage. You should be able to see both of these files in their respective locations. "
+        "Once you successfully uploaded the files, you can browse to them (or upload more files) using [Azure Portal](https://portal.azure.com). At the portal, make sure you have selected your subscription (click *Resource Groups* and then select the subscription). Then look for your **Machine Learning Workspace** name. It has a link to your storage. Click on the storage link. It will take you to a page where you can see [Blobs](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction), [Files](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction), [Tables](https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-overview), and [Queues](https://docs.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction). We have uploaded a file each to the Blob storage and to the File storage in the above step. You should be able to see both of these files in their respective locations. "
       ]
     },
     {
@@ -235,15 +233,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Retrieve or create a Azure Machine Learning compute\n",
-        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's create a new Azure Machine Learning Compute in the current workspace, if it doesn't already exist. We will then run the training script on this compute target.\n",
-        "\n",
-        "If we could not find the compute with the given name in the previous cell, then we will create a new compute here. We will create an Azure Machine Learning Compute containing **STANDARD_D2_V2 CPU VMs**. This process is broken down into the following steps:\n",
-        "\n",
-        "1. Create the configuration\n",
-        "2. Create the Azure Machine Learning compute\n",
-        "\n",
-        "**This process will take about 3 minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell.**"
+        "#### Retrieve default Azure Machine Learning compute\n",
+        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's get the default Azure Machine Learning Compute in the current workspace. We will then run the training script on this compute target."
       ]
     },
     {
@@ -252,22 +243,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "aml_compute_target = \"aml-compute\"\n",
-        "try:\n",
-        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
-        "    print(\"found existing compute target.\")\n",
-        "except ComputeTargetException:\n",
-        "    print(\"creating new compute target\")\n",
-        "    \n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
-        "                                                                min_nodes = 1, \n",
-        "                                                                max_nodes = 4)    \n",
-        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
-        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
-        "    \n",
-        "print(\"Azure Machine Learning Compute attached\")\n"
+        "aml_compute = ws.get_default_compute_target(\"CPU\")"
       ]
     },
     {
@@ -304,11 +280,15 @@
         "## Creating a Step in a Pipeline\n",
         "A Step is a unit of execution. Step typically needs a target of execution (compute target), a script to execute, and may require script arguments and inputs, and can produce outputs. The step also could take a number of other parameters. Azure Machine Learning Pipelines provides the following built-in Steps:\n",
         "\n",
-        "- [**PythonScriptStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py): Add a step to run a Python script in a Pipeline.\n",
+        "- [**PythonScriptStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py): Adds a step to run a Python script in a Pipeline.\n",
         "- [**AdlaStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.adla_step.adlastep?view=azure-ml-py): Adds a step to run U-SQL script using Azure Data Lake Analytics.\n",
         "- [**DataTransferStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.data_transfer_step.datatransferstep?view=azure-ml-py): Transfers data between Azure Blob and Data Lake accounts.\n",
         "- [**DatabricksStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py): Adds a DataBricks notebook as a step in a Pipeline.\n",
         "- [**HyperDriveStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.hyper_drive_step.hyperdrivestep?view=azure-ml-py): Creates a Hyper Drive step for Hyper Parameter Tuning in a Pipeline.\n",
+        "- [**AzureBatchStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.azurebatch_step.azurebatchstep?view=azure-ml-py): Creates a step for submitting jobs to Azure Batch\n",
+        "- [**EstimatorStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.estimator_step.estimatorstep?view=azure-ml-py): Adds a step to run Estimator in a Pipeline.\n",
+        "- [**MpiStep**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.mpi_step.mpistep?view=azure-ml-py): Adds a step to run a MPI job in a Pipeline.\n",
+        "- [**AutoMLStep**](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlstep?view=azure-ml-py): Creates a AutoML step in a Pipeline.\n",
         "\n",
         "The following code will create a PythonScriptStep to be executed in the Azure Machine Learning Compute we created above using train.py, one of the files already made available in the project folder.\n",
         "\n",
@@ -395,9 +375,6 @@
         "# use conda_dependencies.yml to create a conda environment in the Docker image for execution\n",
         "run_config.environment.python.user_managed_dependencies = False\n",
         "\n",
-        "# auto-prepare the Docker image when used for execution (if it is not already prepared)\n",
-        "run_config.auto_prepare_environment = True\n",
-        "\n",
         "# specify CondaDependencies obj\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
         "\n",
@@ -476,8 +453,8 @@
       "source": [
         "# Submit syntax\n",
         "# submit(experiment_name, \n",
-        "#        pipeline_parameters=None, \n",
-        "#        continue_on_node_failure=False, \n",
+        "#        pipeline_params=None, \n",
+        "#        continue_on_step_failure=False, \n",
         "#        regenerate_outputs=False)\n",
         "\n",
         "pipeline_run1 = Experiment(ws, 'Hello_World1').submit(pipeline1, regenerate_outputs=False)\n",
@@ -643,4 +620,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -74,7 +74,7 @@
       "source": [
         "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json\n",
         "\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, if you don't have a config.json file, please go through the configuration Notebook located [here](https://github.com/Azure/MachineLearningNotebooks).  \n",
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, If you don't have a config.json file, please go through the configuration Notebook located [here](https://github.com/Azure/MachineLearningNotebooks).  \n",
         "\n",
         "This sets you up with a working config file that has information on your workspace, subscription id, etc. "
       ]
@@ -113,25 +113,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "batch_compute_name = 'mybatchcompute' # Name to associate with new compute in workspace\n",
-        "\n",
-        "# Batch account details needed to attach as compute to workspace\n",
-        "batch_account_name = \"<batch_account_name>\" # Name of the Batch account\n",
-        "batch_resource_group = \"<batch_resource_group>\" # Name of the resource group which contains this account\n",
-        "\n",
-        "try:\n",
-        "    # check if already attached\n",
-        "    batch_compute = BatchCompute(ws, batch_compute_name)\n",
-        "except ComputeTargetException:\n",
-        "    print('Attaching Batch compute...')\n",
-        "    provisioning_config = BatchCompute.attach_configuration(resource_group=batch_resource_group, \n",
-        "                                                            account_name=batch_account_name)\n",
-        "    batch_compute = ComputeTarget.attach(ws, batch_compute_name, provisioning_config)\n",
-        "    batch_compute.wait_for_completion()\n",
-        "    print(\"Provisioning state:{}\".format(batch_compute.provisioning_state))\n",
-        "    print(\"Provisioning errors:{}\".format(batch_compute.provisioning_errors))\n",
-        "\n",
-        "print(\"Using Batch compute:{}\".format(batch_compute.cluster_resource_id))"
+        "batch_compute = ws.get_default_compute_target(\"CPU\")"
       ]
     },
     {
@@ -196,8 +178,8 @@
         "\n",
         "\n",
         "def upload_file_to_datastore(datastore, file_name, content):\n",
-        "    dir = create_local_file(content=content, file_name=file_name)\n",
-        "    datastore.upload(src_dir=dir, overwrite=True, show_progress=True)"
+        "    src_dir = create_local_file(content=content, file_name=file_name)\n",
+        "    datastore.upload(src_dir=src_dir, overwrite=True, show_progress=True)"
       ]
     },
     {
@@ -252,7 +234,7 @@
         "\n",
         "file_name=\"azurebatch.cmd\"\n",
         "with open(path.join(binaries_folder, file_name), 'w') as f:\n",
-        "        f.write(\"copy \\\"%1\\\" \\\"%2\\\"\")"
+        "    f.write(\"copy \\\"%1\\\" \\\"%2\\\"\")"
       ]
     },
     {
@@ -380,4 +362,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-estimatorstep.ipynb

@@ -13,8 +13,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-estimatorstep.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-estimatorstep.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -27,7 +27,7 @@
         "\n",
         "## Prerequisite:\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise,go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -76,18 +76,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
-        "1. create the configuration (this step is local and only takes a second)\n",
-        "2. create the cluster (this step will take about **20 seconds**)\n",
-        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
       ]
     },
     {
@@ -96,25 +86,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"cpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    cpu_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    cpu_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it uses the scale settings for the cluster\n",
-        "    cpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "cpu_cluster = ws.get_default_compute_target(\"CPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(cpu_cluster.get_status().serialize())"
@@ -285,4 +257,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -135,14 +135,7 @@
       "metadata": {},
       "source": [
         "## Retrieve or create a Azure Machine Learning compute\n",
-        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's create a new Azure Machine Learning Compute in the current workspace, if it doesn't already exist. We will then run the training script on this compute target.\n",
-        "\n",
-        "If we could not find the compute with the given name in the previous cell, then we will create a new compute here. This process is broken down into the following steps:\n",
-        "\n",
-        "1. Create the configuration\n",
-        "2. Create the Azure Machine Learning compute\n",
-        "\n",
-        "**This process will take a few minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell.**\n"
+        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's get the default Azure Machine Learning Compute in the current workspace. We will then run the training script on this compute target."
       ]
     },
     {
@@ -151,20 +144,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target {}.'.format(cluster_name))\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
-        "                                                               max_nodes=4)\n",
-        "\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "    compute_target.wait_for_completion(show_output=True, timeout_in_minutes=20)\n",
-        "\n",
-        "print(\"Azure Machine Learning Compute attached\")"
+        "compute_target = ws.get_default_compute_target(\"GPU\")"
       ]
     },
     {
@@ -266,13 +246,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "hd_config = HyperDriveRunConfig(estimator=est, \n",
-        "                                hyperparameter_sampling=ps,\n",
-        "                                policy=early_termination_policy,\n",
-        "                                primary_metric_name='validation_acc', \n",
-        "                                primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
-        "                                max_total_runs=1,\n",
-        "                                max_concurrent_runs=1)"
+        "hd_config = HyperDriveConfig(estimator=est, \n",
+        "                             hyperparameter_sampling=ps,\n",
+        "                             policy=early_termination_policy,\n",
+        "                             primary_metric_name='validation_acc', \n",
+        "                             primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
+        "                             max_total_runs=1,\n",
+        "                             max_concurrent_runs=1)"
       ]
     },
     {
@@ -302,7 +282,7 @@
         "### HyperDriveStep\n",
         "HyperDriveStep can be used to run HyperDrive job as a step in pipeline.\n",
         "- **name:** Name of the step\n",
-        "- **hyperdrive_run_config:** A HyperDriveRunConfig that defines the configuration for this HyperDrive run\n",
+        "- **hyperdrive_config:** A HyperDriveConfig that defines the configuration for this HyperDrive run\n",
         "- **estimator_entry_script_arguments:** List of command-line arguments for estimator entry script\n",
         "- **inputs:** List of input port bindings\n",
         "- **outputs:** List of output port bindings\n",
@@ -324,7 +304,7 @@
         "\n",
         "hd_step = HyperDriveStep(\n",
         "    name=\"hyperdrive_module\",\n",
-        "    hyperdrive_run_config=hd_config,\n",
+        "    hyperdrive_config=hd_config,\n",
         "    estimator_entry_script_arguments=['--data-folder', data_folder],\n",
         "    inputs=[data_folder],\n",
         "    metrics_output=metirics_data)"
@@ -441,4 +421,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-publish-and-run-using-rest-endpoint.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-publish-and-run-using-rest-endpoint.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-publish-and-run-using-rest-endpoint.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -79,20 +79,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "aml_compute_target = \"cpucluster\"\n",
-        "try:\n",
-        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
-        "    print(\"found existing compute target.\")\n",
-        "except ComputeTargetException:\n",
-        "    print(\"creating new compute target\")\n",
-        "    \n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
-        "                                                                min_nodes = 1, \n",
-        "                                                                max_nodes = 4)    \n",
-        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
-        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n"
+        "aml_compute = ws.get_default_compute_target(\"CPU\")"
       ]
     },
     {
@@ -420,4 +407,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -54,7 +54,7 @@
       "metadata": {},
       "source": [
         "### Compute Targets\n",
-        "#### Retrieve an already attached Azure Machine Learning Compute"
+        "#### Retrieve the default Azure Machine Learning Compute"
       ]
     },
     {
@@ -63,31 +63,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core import Run, Experiment, Datastore\n",
-        "\n",
-        "from azureml.widgets import RunDetails\n",
-        "\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import AmlCompute, ComputeTarget\n",
-        "aml_compute_target = \"aml-compute\"\n",
-        "try:\n",
-        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
-        "    print(\"Found existing compute target: {}\".format(aml_compute_target))\n",
-        "except:\n",
-        "    print(\"Creating new compute target: {}\".format(aml_compute_target))\n",
-        "    \n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
-        "                                                                min_nodes = 1, \n",
-        "                                                                max_nodes = 4)    \n",
-        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
-        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)"
+        "aml_compute_target = ws.get_default_compute_target(\"CPU\")"
       ]
     },
     {
@@ -308,7 +284,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Disable the schedule"
+        "### Disable the schedule\n",
+        "It is important to note the best practice of disabling schedules when not in use.\n",
+        "The number of schedule triggers allowed per month per region per subscription is 100,000.\n",
+        "This is calculated using the project trigger counts for all active schedules."
       ]
     },
     {
@@ -383,7 +362,7 @@
       "source": [
         "### Create a schedule for the pipeline using a Datastore\n",
         "This schedule will run when additions or modifications are made to Blobs in the Datastore.\n",
-        "By default, the Datastore container is monitored for changes. Use the path_on_datastore parameter to instead specify a path on the Datastore to monitor for changes. Changes made to subfolders in the container/path will not trigger the schedule.\n",
+        "By default, the Datastore container is monitored for changes. Use the path_on_datastore parameter to instead specify a path on the Datastore to monitor for changes. Note: the path_on_datastore will be under the container for the datastore, so the actual path monitored will be container/path_on_datastore. Changes made to subfolders in the container/path will not trigger the schedule.\n",
         "Note: Only Blob Datastores are supported."
       ]
     },
@@ -403,6 +382,7 @@
         "                           datastore=datastore,\n",
         "                           wait_for_provisioning=True,\n",
         "                           description=\"Schedule Run\")\n",
+        "                          #polling_interval=5, use polling_interval to specify how often to poll for blob additions/modifications. Default value is 5 minutes.\n",
         "                          #path_on_datastore=\"file/path\") use path_on_datastore to specify a specific folder to monitor for changes.\n",
         "\n",
         "# You may want to make sure that the schedule is provisioned properly\n",
@@ -451,4 +431,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-versioned-pipeline-endpoints.ipynb

@@ -0,0 +1,553 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-versioned-pipeline-endpoints.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n",
+        "# How to Setup a PipelineEndpoint and Submit a Pipeline Using the PipelineEndpoint.\n",
+        "In this notebook, we will see how to setup a PipelineEndpoint and run specific pipeline version.\n",
+        "\n",
+        "PipelineEndpoint can be used to update a published pipeline while maintaining same endpoint.\n",
+        "PipelineEndpoint, provides a way to keep track of [PublishedPipelines](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.publishedpipeline) using versions. PipelineEndpoint uses endpoint with version information to trigger underlying published pipeline. Pipeline endpoints are uniquely named within a workspace.  \n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Prerequisites and AML Basics\n",
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration Notebook](https://github.com/Azure/MachineLearningNotebooks) first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Notebook Overview\n",
+        "In this notebook, we provide an introduction to Azure machine learning PipelineEndpoints. It covers:\n",
+        "* [Create PipelineEndpoint](#Create-PipelineEndpoint), How to create PipelineEndpoint.\n",
+        "* [Retrieving PipelineEndpoint](#Retrieving-PipelineEndpoint), How to get specific PipelineEndpoint from worskpace by name/Id and get all [PipelineEndpoints](#Get-all-PipelineEndpoints-in-workspace) within workspace.\n",
+        "* [PipelineEndpoint Properties](#PipelineEndpoint-properties). How to get and set PipelineEndpoint properties, such as default version of PipelineEndpoint.\n",
+        "* [PipelineEndpoint Submission](#PipelineEndpoint-Submission). How to run a Pipeline using PipelineEndpoint."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "###  Create PipelineEndpoint\n",
+        "Following are required input parameters to create PipelineEndpoint:\n",
+        "\n",
+        "* *workspace*: AML workspace.\n",
+        "* *name*: name of PipelineEndpoint, it is unique within workspace.\n",
+        "* *description*: description details for PipelineEndpoint.\n",
+        "* *pipeline*: A [Pipeline](#Steps-to-create-simple-Pipeline) or [PublishedPipeline](#Publish-Pipeline), to set default version of PipelineEndpoint.                                                       "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "####  Initialization, Steps to create a Pipeline"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.pipeline.steps import PythonScriptStep\n",
+        "from azureml.pipeline.core import Pipeline\n",
+        "\n",
+        "aml_compute = ws.get_default_compute_target(\"CPU\")\n",
+        "\n",
+        "# source_directory\n",
+        "source_directory = '.'\n",
+        "# define a single step pipeline for demonstration purpose.\n",
+        "trainStep = PythonScriptStep(\n",
+        "    name=\"Training_Step\",\n",
+        "    script_name=\"train.py\", \n",
+        "    compute_target=aml_compute_target, \n",
+        "    source_directory=source_directory\n",
+        ")\n",
+        "print(\"TrainStep created\")\n",
+        "# build and validate Pipeline\n",
+        "pipeline = Pipeline(workspace=ws, steps=[trainStep])\n",
+        "print(\"Pipeline is built\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Publish Pipeline"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from datetime import datetime\n",
+        "\n",
+        "timenow = datetime.now().strftime('%m-%d-%Y-%H-%M')\n",
+        "\n",
+        "pipeline_name = timenow + \"-Pipeline\"\n",
+        "print(pipeline_name)\n",
+        "\n",
+        "published_pipeline = pipeline.publish(\n",
+        "    name=pipeline_name, \n",
+        "    description=pipeline_name)\n",
+        "print(\"Newly published pipeline id: {}\".format(published_pipeline.id))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Publishing PipelineEndpoint\n",
+        "Create PipelineEndpoint with required parameters: workspace, name, description and pipeline"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.pipeline.core import PipelineEndpoint\n",
+        "\n",
+        "pipeline_endpoint = PipelineEndpoint.publish(workspace=ws, name=\"PipelineEndpointTest\",\n",
+        "                                            pipeline=pipeline, description=\"Test description Notebook\")\n",
+        "pipeline_endpoint"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Retrieving PipelineEndpoint\n",
+        "\n",
+        "PipelineEndpoint is uniquely defined by name and id within workspace. PipelineEndpoint in workspace can be retrived by Id or by name."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get PipelineEndpoint by Name\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name = PipelineEndpoint.get(workspace=ws, name=\"PipelineEndpointTest\")\n",
+        "pipeline_endpoint_by_name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get PipelineEndpoint by Id\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "#get the PipelineEndpoint Id\n",
+        "pipeline_endpoint_by_name = PipelineEndpoint.get(workspace=ws, name=\"PipelineEndpointTest\")\n",
+        "endpoint_id = pipeline_endpoint_by_name.id\n",
+        "\n",
+        "pipeline_endpoint_by_id = PipelineEndpoint.get(workspace=ws, id=endpoint_id)\n",
+        "pipeline_endpoint_by_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get all PipelineEndpoints in workspace\n",
+        "Returns all PipelineEndpoints within workspace"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "endpoint_list = PipelineEndpoint.get_all(workspace=ws, active_only=True)\n",
+        "endpoint_list"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### PipelineEndpoint properties"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Default Version of PipelineEndpoint\n",
+        "Default version of PipelineEndpoint starts from \"0\" and increments on addition of pipelines.\n",
+        "\n",
+        "##### Get the Default Version"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "default_version = pipeline_endpoint_by_name.get_default_version()\n",
+        "default_version"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#####  Set default version \n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name.set_default_version(\"0\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get the Published Pipeline corresponds to specific version of PipelineEndpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline = pipeline_endpoint_by_name.get_pipeline(\"0\")\n",
+        "pipeline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get default version Published Pipeline"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline = pipeline_endpoint_by_name.get_pipeline()\n",
+        "pipeline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Set Published Pipeline to default version"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Set Published Pipeline to PipelineEndpoint, if exists\n",
+        "pipeline_endpoint_by_name.set_default(published_pipeline)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get all Versions in PipelineEndpoint\n",
+        "Returns list of published pipelines and its versions"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "versions = pipeline_endpoint_by_name.get_all_versions()\n",
+        "\n",
+        "for ve in versions:\n",
+        "    print(ve.version)\n",
+        "    print(ve.pipeline.id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Get all Published Pipelines in PipelineEndpoint\n",
+        "Returns all active pipelines in PipelineEnpoint, if active_only flag is set to True."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipelines = pipeline_endpoint_by_name.get_all_pipelines(active_only=True)\n",
+        "pipelines"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Name property of PipelineEndpoint\n",
+        "PipelineEndpoint is uniquely identified by name"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "##### Set Name PipelineEndpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name.set_name(name=\"NewName\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Add Published Pipeline to PipelineEndpoint, \n",
+        "Adding published pipeline, if its not present in PipelineEndpoint."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name.add(published_pipeline)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Add Published pipeline to PipelineEndpoint and set it to default version\n",
+        "Adding published pipeline to PipelineEndpoint if not present and set it to default"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name.add_default(published_pipeline)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### PipelineEndpoint Submission\n",
+        "PipelineEndpoint triggers specific versioned pipeline or default pipeline by:\n",
+        "* Rest Endpoint \n",
+        "* Submit call."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Run Pipeline by endpoint property of PipelineEndpoint\n",
+        "Run specific pipeline using endpoint property of PipelineEndpoint and executing http post."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_endpoint_by_name = PipelineEndpoint.get(workspace=ws, name=\"PipelineEndpointTest\")\n",
+        "\n",
+        "# endpoint with id \n",
+        "rest_endpoint_id =  pipeline_endpoint_by_name.endpoint\n",
+        "\n",
+        "# for default version pipeline\n",
+        "rest_endpoint_id_without_version_with_id = rest_endpoint_id\n",
+        "\n",
+        "# for specific version pipeline just append version info\n",
+        "version=\"0\"\n",
+        "rest_endpoint_id_with_version = rest_endpoint_id_without_version_with_id+\"/\"+ version\n",
+        "print(rest_endpoint_id_with_version)\n",
+        "pipeline_endpoint_by_name"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# endpoint with name\n",
+        "rest_endpoint_name = rest_endpoint_id.split(\"Id\", 1)[0] + \"Name?name=\" + pipeline_endpoint_by_name.name\n",
+        "\n",
+        "# for default version pipeline\n",
+        "rest_endpoint_name_without_version = rest_endpoint_name\n",
+        "\n",
+        "# for specific version pipeline just append version info\n",
+        "version=\"0\"\n",
+        "rest_endpoint_name_with_version = rest_endpoint_name_without_version+\"&pipelineVersion=\"+ version\n",
+        "print(rest_endpoint_name_with_version)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "[This notebook](https://aka.ms/pl-restep-auth) shows how to authenticate to AML workspace."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
+        "import requests\n",
+        "\n",
+        "auth = InteractiveLoginAuthentication()\n",
+        "aad_token = auth.get_authentication_header()\n",
+        "\n",
+        "#endpoint = pipeline_endpoint_by_name.url\n",
+        "\n",
+        "print(\"You can perform HTTP POST on URL {} to trigger this pipeline\".format(rest_endpoint_name_with_version))\n",
+        "\n",
+        "# specify the param when running the pipeline\n",
+        "response = requests.post(rest_endpoint_name_with_version, \n",
+        "                         headers=aad_token, \n",
+        "                         json={\"ExperimentName\": \"default_pipeline\",\n",
+        "                               \"RunSource\": \"SDK\",\n",
+        "                               \"ParameterAssignments\": {\"1\": \"united\", \"2\":\"city\"}})\n",
+        "\n",
+        "run_id = response.json()[\"Id\"]\n",
+        "print(run_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Run Pipeline by Submit call of PipelineEndpoint \n",
+        "Run specific pipeline using Submit api of PipelineEndpoint"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# submit pipeline with specific version\n",
+        "run_id = pipeline_endpoint_by_name.submit(\"TestPipelineEndpoint\", pipeline_version=\"0\")\n",
+        "print(run_id)\n",
+        "\n",
+        "# submit pipeline with default version\n",
+        "run_id = pipeline_endpoint_by_name.submit(\"TestPipelineEndpoint\")\n",
+        "print(run_id)"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "mameghwa"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.2"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-adla-as-compute-target.ipynb

@@ -12,8 +12,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-adla-as-compute-target.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-adla-as-compute-target.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -371,4 +371,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-databricks-as-compute-target.ipynb

@@ -8,12 +8,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-databricks-as-compute-target.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-databricks-as-compute-target.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -404,7 +404,9 @@
       "metadata": {},
       "source": [
         "### 1. Running the demo notebook already added to the Databricks workspace\n",
-        "Create a notebook in the Azure Databricks workspace, and provide the path to that notebook as the value associated with the environment variable \"DATABRICKS_NOTEBOOK_PATH\". This will then set the variable\u00c2\u00a0notebook_path\u00c2\u00a0when you run the code cell below:"
+        "Create a notebook in the Azure Databricks workspace, and provide the path to that notebook as the value associated with the environment variable \"DATABRICKS_NOTEBOOK_PATH\". This will then set the variable\u00c2\u00a0notebook_path\u00c2\u00a0when you run the code cell below:\n",
+        "\n",
+        "your notebook's path in Azure Databricks UI by hovering over to notebook's title. A typical path of notebook looks like this `/Users/example@databricks.com/example`. See [Databricks Workspace](https://docs.azuredatabricks.net/user-guide/workspace.html) to learn about the folder structure."
       ]
     },
     {
@@ -712,4 +714,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.ipynb

@@ -8,12 +8,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -130,9 +130,7 @@
       "metadata": {},
       "source": [
         "### Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process."
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource."
       ]
     },
     {
@@ -141,31 +139,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Choose a name for your cluster.\n",
-        "amlcompute_cluster_name = \"cpucluster\"\n",
-        "\n",
-        "found = False\n",
-        "# Check if this compute target already exists in the workspace.\n",
-        "cts = ws.compute_targets\n",
-        "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
-        "    found = True\n",
-        "    print('Found existing compute target.')\n",
-        "    compute_target = cts[amlcompute_cluster_name]\n",
-        "    \n",
-        "if not found:\n",
-        "    print('Creating a new compute target...')\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\", # for GPU, use \"STANDARD_NC6\"\n",
-        "                                                                #vm_priority = 'lowpriority', # optional\n",
-        "                                                                max_nodes = 4)\n",
-        "\n",
-        "    # Create the cluster.\n",
-        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
-        "    \n",
-        "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
-        "    # If no min_node_count is provided, it will use the scale settings for the cluster.\n",
-        "    compute_target.wait_for_completion(show_output = True, min_node_count = 1, timeout_in_minutes = 10)\n",
-        "    \n",
-        "     # For a more detailed view of current AmlCompute status, use get_status()."
+        "compute_target = ws.get_default_compute_target(\"CPU\")"
       ]
     },
     {
@@ -244,7 +218,7 @@
         "    X_train = pd.read_csv(\"/tmp/azureml_runs/bai_data/X_train.tsv\", delimiter=\"\\t\", header=None, quotechar='\"')\n",
         "    y_train = pd.read_csv(\"/tmp/azureml_runs/bai_data/y_train.tsv\", delimiter=\"\\t\", header=None, quotechar='\"')\n",
         "\n",
-        "    return { \"X\" : X_train.values, \"y\" : y_train[0].values }\n"
+        "    return { \"X\" : X_train.values, \"y\" : y_train.values.flatten() }\n"
       ]
     },
     {
@@ -521,4 +495,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.ipynb

@@ -8,12 +8,12 @@
         "Licensed under the MIT License."
       ]
     },
-      {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -128,7 +128,7 @@
       "metadata": {},
       "source": [
         "#### Retrieve or create a Aml compute\n",
-        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's create a new Aml Compute in the current workspace, if it doesn't already exist. We will then run the training script on this compute target."
+        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's get the default Aml Compute in the current workspace. We will then run the training script on this compute target."
       ]
     },
     {
@@ -137,22 +137,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "aml_compute_target = \"aml-compute\"\n",
-        "try:\n",
-        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
-        "    print(\"found existing compute target.\")\n",
-        "except ComputeTargetException:\n",
-        "    print(\"creating new compute target\")\n",
-        "    \n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
-        "                                                                min_nodes = 1, \n",
-        "                                                                max_nodes = 4)    \n",
-        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
-        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
-        "    \n",
-        "print(\"Aml Compute attached\")\n"
+        "aml_compute = ws.get_default_compute_target(\"CPU\")\n"
       ]
     },
     {
@@ -297,9 +282,6 @@
         "# use conda_dependencies.yml to create a conda environment in the Docker image for execution\n",
         "run_config.environment.python.user_managed_dependencies = False\n",
         "\n",
-        "# auto-prepare the Docker image when used for execution (if it is not already prepared)\n",
-        "run_config.auto_prepare_environment = True\n",
-        "\n",
         "# specify CondaDependencies obj\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
       ]
@@ -471,4 +453,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/pipeline-batch-scoring/pipeline-batch-scoring.ipynb

@@ -8,12 +8,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/pipeline-batch-scoring/pipeline-batch-scoring.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/pipeline-batch-scoring/pipeline-batch-scoring.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -162,7 +162,7 @@
       "metadata": {},
       "source": [
         "### Create and attach Compute targets\n",
-        "Use the below code to create and attach Compute targets. "
+        "Use the below code to get the default Compute target. "
       ]
     },
     {
@@ -171,33 +171,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# choose a name for your cluster\n",
-        "aml_compute_name = os.environ.get(\"AML_COMPUTE_NAME\", \"gpucluster\")\n",
-        "cluster_min_nodes = os.environ.get(\"AML_COMPUTE_MIN_NODES\", 0)\n",
-        "cluster_max_nodes = os.environ.get(\"AML_COMPUTE_MAX_NODES\", 1)\n",
-        "vm_size = os.environ.get(\"AML_COMPUTE_SKU\", \"STANDARD_NC6\")\n",
+        "cluster_type = os.environ.get(\"AML_CLUSTER_TYPE\", \"GPU\")\n",
         "\n",
-        "\n",
-        "if aml_compute_name in ws.compute_targets:\n",
-        "    compute_target = ws.compute_targets[aml_compute_name]\n",
-        "    if compute_target and type(compute_target) is AmlCompute:\n",
-        "        print('found compute target. just use it. ' + aml_compute_name)\n",
-        "else:\n",
-        "    print('creating a new compute target...')\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = vm_size, # NC6 is GPU-enabled\n",
-        "                                                                vm_priority = 'lowpriority', # optional\n",
-        "                                                                min_nodes = cluster_min_nodes, \n",
-        "                                                                max_nodes = cluster_max_nodes)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, aml_compute_name, provisioning_config)\n",
-        "    \n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it will use the scale settings for the cluster\n",
-        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
-        "    \n",
-        "     # For a more detailed view of current Azure Machine Learning Compute  status, use get_status()\n",
-        "    print(compute_target.get_status().serialize())"
+        "compute_target = ws.get_default_compute_target(cluster_type)"
       ]
     },
     {
@@ -600,4 +576,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -32,7 +32,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwsie, make sure you go through the configuration Notebook located at https://github.com/Azure/MachineLearningNotebooks first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc. "
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the configuration Notebook located at https://github.com/Azure/MachineLearningNotebooks first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc. "
       ]
     },
     {
@@ -650,4 +650,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb

@@ -0,0 +1,260 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License.\n",
+        "\n",
+        "## Authentication in Azure Machine Learning\n",
+        "\n",
+        "This notebook shows you how to authenticate to your Azure ML Workspace using\n",
+        "\n",
+        " 1. Interactive Login Authentication\n",
+        " 2. Azure CLI Authentication\n",
+        " 3. Service Principal Authentication\n",
+        " \n",
+        "The interactive authentication is suitable for local experimentation on your own computer. Azure CLI authentication is suitable if you are already using Azure CLI for managing Azure resources, and want to sign in only once. The Service Principal authentication is suitable for automated workflows, for example as part of Azure Devops build."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.png)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Interactive Authentication\n",
+        "\n",
+        "Interactive authentication is the default mode when using Azure ML SDK.\n",
+        "\n",
+        "When you connect to your workspace using workspace.from_config, you will get an interactive login dialog."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Also, if you explicitly specify the subscription ID, resource group and resource group, you will get the dialog."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace(subscription_id=\"my-subscription-id\",\n",
+        "               resource_group=\"my-ml-rg\",\n",
+        "               workspace_name=\"my-ml-workspace\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Note the user you're authenticated as must have access to the subscription and resource group. If you receive an error\n",
+        "\n",
+        "```\n",
+        "AuthenticationException: You don't have access to xxxxxx-xxxx-xxx-xxx-xxxxxxxxxx subscription. All the subscriptions that you have access to = ...\n",
+        "```\n",
+        "\n",
+        "check that the you used correct login and entered the correct subscription ID."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "In some cases, you may see a version of the error message containing text: ```All the subscriptions that you have access to = []```\n",
+        "\n",
+        "In such a case, you may have to specify the tenant ID of the Azure Active Directory you're using. An example would be accessing a subscription as a guest to a tenant that is not your default. You specify the tenant by explicitly instantiating _InteractiveLoginAuthentication_ with tenant ID as argument ([see instructions how to obtain tenant Id](#get-tenant-id))."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
+        "\n",
+        "interactive_auth = InteractiveLoginAuthentication(tenant_id=\"my-tenant-id\")\n",
+        "\n",
+        "ws = Workspace(subscription_id=\"my-subscription-id\",\n",
+        "               resource_group=\"my-ml-rg\",\n",
+        "               workspace_name=\"my-ml-workspace\",\n",
+        "               auth=interactive_auth)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Azure CLI Authentication\n",
+        "\n",
+        "If you have installed azure-cli package, and used ```az login``` command to log in to your Azure Subscription, you can use _AzureCliAuthentication_ class.\n",
+        "\n",
+        "Note that interactive authentication described above won't use existing Azure CLI auth tokens. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.authentication import AzureCliAuthentication\n",
+        "\n",
+        "cli_auth = AzureCliAuthentication()\n",
+        "\n",
+        "ws = Workspace(subscription_id=\"my-subscription-id\",\n",
+        "               resource_group=\"my-ml-rg\",\n",
+        "               workspace_name=\"my-ml-workspace\",\n",
+        "               auth=cli_auth)\n",
+        "\n",
+        "print(\"Found workspace {} at location {}\".format(ws.name, ws.location))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Service Principal Authentication\n",
+        "\n",
+        "When setting up a machine learning workflow as an automated process, we recommend using Service Principal Authentication. This approach decouples the authentication from any specific user login, and allows managed access control.\n",
+        "\n",
+        "Note that you must have administrator privileges over the Azure subscription to complete these steps.\n",
+        "\n",
+        "The first step is to create a service principal. First, go to [Azure Portal](https://portal.azure.com), select **Azure Active Directory** and **App Registrations**. Then select **+New application registration**, give your service principal a name, for example _my-svc-principal_. You can leave application type as is, and specify a dummy value for Sign-on URL, such as _https://invalid_.\n",
+        "\n",
+        "Then click **Create**.\n",
+        "\n",
+        "![service principal creation]<img src=\"images/svc-pr-1.PNG\">"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The next step is to obtain the _Application ID_ (also called username) and create _password_ for the service principal.\n",
+        "\n",
+        "From the page for your newly created service principal, copy the _Application ID_. Then select **Settings** and **Keys**, write a description for your key, and select duration. Then click **Save**, and copy the _password_ to a secure location.\n",
+        "\n",
+        "![application id and password](images/svc-pr-2.PNG)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id =\"get-tenant-id\"></a>\n",
+        "\n",
+        "Also, you need to obtain the tenant ID of your Azure subscription. Go back to **Azure Active Directory**, select **Properties** and copy _Directory ID_.\n",
+        "\n",
+        "![tenant id](images/svc-pr-3.PNG)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, you need to give the service principal permissions to access your workspace. Navigate to **Resource Groups**, to the resource group for your Machine Learning Workspace. \n",
+        "\n",
+        "Then select **Access Control (IAM)** and **Add a role assignment**. For _Role_, specify which level of access you need to grant, for example _Contributor_. Start entering your service principal name and once it is found, select it, and click **Save**.\n",
+        "\n",
+        "![add role](images/svc-pr-4.PNG)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now you are ready to use the service principal authentication. For example, to connect to your Workspace, see code below and enter your own values for tenant ID, application ID, subscription ID, resource group and workspace.\n",
+        "\n",
+        "**We strongly recommended that you do not insert the secret password to code**. Instead, you can use environment variables to pass it to your code, for example through Azure Key Vault, or through secret build variables in Azure DevOps. For local testing, you can for example use following PowerShell command to set the environment variable.\n",
+        "\n",
+        "```\n",
+        "$env:AZUREML_PASSWORD = \"my-password\"\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "from azureml.core.authentication import ServicePrincipalAuthentication\n",
+        "\n",
+        "svc_pr_password = os.environ.get(\"AZUREML_PASSWORD\")\n",
+        "\n",
+        "svc_pr = ServicePrincipalAuthentication(\n",
+        "    tenant_id=\"my-tenant-id\",\n",
+        "    service_principal_id=\"my-application-id\",\n",
+        "    service_principal_password=svc_pr_password)\n",
+        "\n",
+        "\n",
+        "ws = Workspace(\n",
+        "    subscription_id=\"my-subscription-id\",\n",
+        "    resource_group=\"my-ml-rg\",\n",
+        "    workspace_name=\"my-ml-workspace\",\n",
+        "    auth=svc_pr\n",
+        "    )\n",
+        "\n",
+        "print(\"Found workspace {} at location {}\".format(ws.name, ws.location))"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "roastala"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.5"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/training-with-deep-learning/README.md

@@ -16,3 +16,6 @@ These examples show you:
 12. [Use TensorBoard to monitor training execution](tensorboard)
 
 Learn more about how to use `Estimator` class to  [train deep neural networks with Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-ml-models).
+
+ ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/README.png)
+

how-to-use-azureml/training-with-deep-learning/distributed-chainer/distributed-chainer.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/distributed-chainer/distributed-chainer.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -22,7 +29,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
       ]
     },
     {
@@ -88,10 +95,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code gets the default compute cluster.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -102,24 +107,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target.')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute. \n",
         "print(compute_target.get_status().serialize())"
@@ -129,7 +117,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
+        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-cntk-with-custom-docker/distributed-cntk-with-custom-docker.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/distributed-cntk-with-custom-docker/distributed-cntk-with-custom-docker.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -23,7 +30,7 @@
       "source": [
         "## Prerequisites\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -91,10 +98,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use default `AmlCompute` as the training compute resource.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -105,24 +110,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target.')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute\n",
         "print(compute_target.get_status().serialize())"

how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -22,7 +29,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`\n",
         "* Review the [tutorial](../train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) on single-node PyTorch training using Azure Machine Learning"
       ]
     },
@@ -89,10 +96,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code uses the default compute in the workspace.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -103,24 +108,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target.')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute. \n",
         "print(compute_target.get_status().serialize())"
@@ -130,7 +118,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
+        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-tensorflow-with-horovod/distributed-tensorflow-with-horovod.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/manage-runs/manage-runs.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -23,7 +30,7 @@
       "source": [
         "## Prerequisites\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)\n",
         "* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
@@ -91,10 +98,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -105,24 +110,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -132,7 +120,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
+        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-tensorflow-with-parameter-server/distributed-tensorflow-with-parameter-server.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/distributed-tensorflow-with-parameter-server/distributed-tensorflow-with-parameter-server.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -23,7 +30,7 @@
       "source": [
         "## Prerequisites\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)\n",
         "* Review the [tutorial](../train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb) on single-node TensorFlow training using the SDK"
@@ -91,10 +98,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -105,24 +110,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target.')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"

how-to-use-azureml/training-with-deep-learning/export-run-history-to-tensorboard/export-run-history-to-tensorboard.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/export-run-history-to-tensorboard/export-run-history-to-tensorboard.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -26,7 +33,7 @@
       "source": [
         "## Prerequisites\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -43,22 +50,6 @@
         "print(\"SDK version:\", azureml.core.VERSION)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Install the Azure ML TensorBoard integration package if you haven't already."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "!pip install azureml-tensorboard"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/training-with-deep-learning/how-to-use-estimator/dummy_train.py

@@ -1,14 +1,31 @@
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # Licensed under the MIT License.
 
+import argparse
+
 print("*********************************************************")
 print("Hello Azure ML!")
 
+parser = argparse.ArgumentParser()
+parser.add_argument('--numbers-in-sequence', type=int, dest='num_in_sequence', default=10,
+                    help='number of fibonacci numbers in sequence')
+args = parser.parse_args()
+num = args.num_in_sequence
+
+
+def fibo(n):
+    if n < 2:
+        return n
+    else:
+        return fibo(n - 1) + fibo(n - 2)
+
+
 try:
     from azureml.core import Run
     run = Run.get_context()
     print("Log Fibonacci numbers.")
-    run.log_list('Fibonacci numbers', [0, 1, 1, 2, 3, 5, 8, 13, 21, 34])
+    for i in range(0, num - 1):
+        run.log('Fibonacci numbers', fibo(i))
     run.complete()
 except:
     print("Warning: you need to install Azure ML SDK in order to log metrics.")

how-to-use-azureml/training-with-deep-learning/how-to-use-estimator/how-to-use-estimator.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/how-to-use-estimator/how-to-use-estimator.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -25,7 +32,7 @@
         "\n",
         "## Prerequisite:\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -106,18 +113,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
-        "1. create the configuration (this step is local and only takes a second)\n",
-        "2. create the cluster (this step will take about **20 seconds**)\n",
-        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
       ]
     },
     {
@@ -126,25 +123,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"cpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    cpu_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    cpu_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it uses the scale settings for the cluster\n",
-        "    cpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "cpu_cluster = ws.get_default_compute_target(\"CPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(cpu_cluster.get_status().serialize())"
@@ -154,7 +133,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'cpucluster' of type `AmlCompute`."
+        "Now that you have retrieved the compute target, let's see what the workspace's `compute_targets` property returns."
       ]
     },
     {
@@ -177,7 +156,7 @@
       },
       "source": [
         "## Use a simple script\n",
-        "We have already created a simple \"hello world\" script. This is the script that we will submit through the estimator pattern. It prints a hello-world message, and if Azure ML SDK is installed, it will also logs an array of values ([Fibonacci numbers](https://en.wikipedia.org/wiki/Fibonacci_number))."
+        "We have already created a simple \"hello world\" script. This is the script that we will submit through the estimator pattern. It prints a hello-world message, and if Azure ML SDK is installed, it will also logs an array of values ([Fibonacci numbers](https://en.wikipedia.org/wiki/Fibonacci_number)). The script takes as input the number of Fibonacci numbers in the sequence to log."
       ]
     },
     {
@@ -228,7 +207,10 @@
       "outputs": [],
       "source": [
         "# use a conda environment, don't use Docker, on local computer\n",
-        "est = Estimator(source_directory='.', compute_target='local', entry_script='dummy_train.py', use_docker=False)\n",
+        "script_params = {\n",
+        "    '--numbers-in-sequence': 10\n",
+        "}\n",
+        "est = Estimator(source_directory='.', script_params=script_params, compute_target='local', entry_script='dummy_train.py', use_docker=False)\n",
         "run = exp.submit(est)\n",
         "RunDetails(run).show()"
       ]
@@ -247,7 +229,10 @@
       "outputs": [],
       "source": [
         "# use a conda environment on default Docker image in an AmlCompute cluster\n",
-        "est = Estimator(source_directory='.', compute_target=cpu_cluster, entry_script='dummy_train.py', use_docker=True)\n",
+        "script_params = {\n",
+        "    '--numbers-in-sequence': 10\n",
+        "}\n",
+        "est = Estimator(source_directory='.', script_params=script_params, compute_target=cpu_cluster, entry_script='dummy_train.py', use_docker=True)\n",
         "run = exp.submit(est)\n",
         "RunDetails(run).show()"
       ]
@@ -266,7 +251,11 @@
       "outputs": [],
       "source": [
         "# add a conda package\n",
+        "script_params = {\n",
+        "    '--numbers-in-sequence': 10\n",
+        "}\n",
         "est = Estimator(source_directory='.', \n",
+        "                script_params=script_params, \n",
         "                compute_target='local', \n",
         "                entry_script='dummy_train.py', \n",
         "                use_docker=False, \n",
@@ -306,7 +295,12 @@
         "user_managed_dependencies = True\n",
         "\n",
         "# submit to a local Docker container. if you don't have Docker engine running locally, you can set compute_target to cpu_cluster.\n",
-        "est = Estimator(source_directory='.', compute_target='local', \n",
+        "script_params = {\n",
+        "    '--numbers-in-sequence': 10\n",
+        "}\n",
+        "est = Estimator(source_directory='.', \n",
+        "                script_params=script_params, \n",
+        "                compute_target='local', \n",
         "                entry_script='dummy_train.py',\n",
         "                custom_docker_image=image_name,\n",
         "                # uncomment below line to use your private ACR\n",
@@ -325,6 +319,130 @@
         "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Intelligent hyperparameter tuning\n",
+        "\n",
+        "The simple \"hello world\" script above lets the user fix the value of a parameter for the number of Fibonacci numbers in the sequence to log. Similarly, when training models, you can fix values of parameters of the training algorithm itself. E.g. the learning rate, the number of layers, the number of nodes in each layer in a neural network, etc. These adjustable parameters that govern the training process are referred to as the hyperparameters of the model. The goal of hyperparameter tuning is to search across various hyperparameter configurations and find the configuration that results in the best performance.\n",
+        "\n",
+        "\n",
+        "To demonstrate how Azure Machine Learning can help you automate the process of hyperarameter tuning, we will launch multiple runs with different values for numbers in the sequence. First let's define the parameter space using random sampling."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveConfig, PrimaryMetricGoal\n",
+        "from azureml.train.hyperdrive import choice\n",
+        "\n",
+        "ps = RandomParameterSampling(\n",
+        "    {\n",
+        "        '--numbers-in-sequence': choice(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)\n",
+        "    }\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Next, we will create a new estimator without the above numbers-in-sequence parameter since that will be passed in later. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "est = Estimator(source_directory='.', script_params={}, compute_target=cpu_cluster, entry_script='dummy_train.py', use_docker=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Next, we will look at training metrics and early termination policies. When training a model, users are interested in logging and optimizing certain metrics of the model e.g. maximize the accuracy of the model, or minimize loss. This metric is logged by the training script for each run. In our simple script above, we are logging Fibonacci numbers in a sequence. But a training script could just as easily log other metrics like accuracy or loss, which can be used to evaluate the performance of a given training run.\n",
+        "\n",
+        "The intelligent hyperparameter tuning capability in Azure Machine Learning automatically terminates poorly performing runs using an early termination policy. Early termination reduces wastage of compute resources and instead uses these resources for exploring other hyperparameter configurations. In this example, we use the BanditPolicy. This basically states to check the job every 2 iterations. If the primary metric (defined later) falls outside of the top 10% range, Azure ML will terminate the training run. This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "policy = BanditPolicy(evaluation_interval=2, slack_factor=0.1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we are ready to configure a run configuration object for hyperparameter tuning. We need to call out the primary metric that we want the experiment to optimize. The name of the primary metric needs to exactly match the name of the metric logged by the training script and we specify that we are looking to maximize this value. Next, we control the resource budget for the experiment by setting the maximum total number of training runs to 10. We also set the maximum number of training runs to run concurrently at 4, which is the same as the number of nodes in our computer cluster."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "hdc = HyperDriveConfig(estimator=est, \n",
+        "                          hyperparameter_sampling=ps, \n",
+        "                          policy=policy, \n",
+        "                          primary_metric_name='Fibonacci numbers', \n",
+        "                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
+        "                          max_total_runs=10,\n",
+        "                          max_concurrent_runs=4)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, let's launch the hyperparameter tuning job."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "hdr = exp.submit(config=hdc)\n",
+        "RunDetails(hdr).show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "When all the runs complete, we can find the run with the best performance"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "best_run = hdr.get_best_run_by_primary_metric()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can register the model from the best run and use it to deploy a web service that can be used for Inferencing. Details on how how you can do this can be found in the sample folders for the ohter types of estimators.\n"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/training-with-deep-learning/tensorboard/tensorboard.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/tensorboard/tensorboard.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -27,7 +34,7 @@
       "source": [
         "## Prerequisites\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) notebook to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -44,22 +51,6 @@
         "print(\"SDK version:\", azureml.core.VERSION)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Install the Azure ML TensorBoard package."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "!pip install azureml-tensorboard"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-chainer/train-hyperparameter-tune-deploy-with-chainer.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-chainer/train-hyperparameter-tune-deploy-with-chainer.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -23,7 +30,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
       ]
     },
     {
@@ -88,10 +95,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
-        "\n",
-        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -102,24 +107,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target.')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    compute_target.wait_for_completion(show_output=True)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -129,7 +117,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
+        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
       ]
     },
     {
@@ -330,7 +318,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.train.hyperdrive.runconfig import HyperDriveRunConfig\n",
+        "from azureml.train.hyperdrive.runconfig import HyperDriveConfig\n",
         "from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
         "from azureml.train.hyperdrive.policy import BanditPolicy\n",
         "from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
@@ -343,12 +331,12 @@
         "    }\n",
         ")\n",
         "\n",
-        "hyperdrive_run_config = HyperDriveRunConfig(estimator=estimator,\n",
-        "                                            hyperparameter_sampling=param_sampling, \n",
-        "                                            primary_metric_name='Accuracy',\n",
-        "                                            primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
-        "                                            max_total_runs=8,\n",
-        "                                            max_concurrent_runs=4)\n"
+        "hyperdrive_config = HyperDriveConfig(estimator=estimator,\n",
+        "                                     hyperparameter_sampling=param_sampling, \n",
+        "                                     primary_metric_name='Accuracy',\n",
+        "                                     primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
+        "                                     max_total_runs=8,\n",
+        "                                     max_concurrent_runs=4)\n"
       ]
     },
     {
@@ -365,7 +353,7 @@
       "outputs": [],
       "source": [
         "# start the HyperDrive run\n",
-        "hyperdrive_run = experiment.submit(hyperdrive_run_config)"
+        "hyperdrive_run = experiment.submit(hyperdrive_config)"
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-keras/train-hyperparameter-tune-deploy-with-keras.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-keras/train-hyperparameter-tune-deploy-with-keras.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -26,7 +33,7 @@
         "\n",
         "## Prerequisite:\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)\n",
         "* For local scoring test, you will also need to have `tensorflow` and `keras` installed in the current Jupyter kernel."
@@ -232,18 +239,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
-        "1. create the configuration (this step is local and only takes a second)\n",
-        "2. create the cluster (this step will take about **20 seconds**)\n",
-        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
       ]
     },
     {
@@ -252,26 +249,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it uses the scale settings for the cluster\n",
-        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -281,7 +259,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named \"gpucluster\" of type `AmlCompute`."
+        "Now that you have retrtieved the compute target, let's see what the workspace's `compute_targets` property returns."
       ]
     },
     {
@@ -664,7 +642,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveRunConfig, PrimaryMetricGoal\n",
+        "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveConfig, PrimaryMetricGoal\n",
         "from azureml.train.hyperdrive import choice, loguniform\n",
         "\n",
         "ps = RandomParameterSampling(\n",
@@ -727,13 +705,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "hdc = HyperDriveRunConfig(estimator=est, \n",
-        "                          hyperparameter_sampling=ps, \n",
-        "                          policy=policy, \n",
-        "                          primary_metric_name='Accuracy', \n",
-        "                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
-        "                          max_total_runs=20,\n",
-        "                          max_concurrent_runs=4)"
+        "hdc = HyperDriveConfig(estimator=est, \n",
+        "                       hyperparameter_sampling=ps, \n",
+        "                       policy=policy, \n",
+        "                       primary_metric_name='Accuracy', \n",
+        "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
+        "                       max_total_runs=20,\n",
+        "                       max_concurrent_runs=4)"
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb

@@ -8,6 +8,13 @@
         "\n",
         "Licensed under the MIT License."
       ]
+    },
+     {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.png)"
+         ] 
     },
     {
       "cell_type": "markdown",
@@ -25,7 +32,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "* Go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [Configuration](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`"
       ]
     },
     {
@@ -735,4 +742,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -26,7 +33,7 @@
         "\n",
         "## Prerequisite:\n",
         "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning\n",
-        "* Go through the [configuration notebook](../../../configuration.ipynb) to:\n",
+        "* If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) to:\n",
         "    * install the AML SDK\n",
         "    * create a workspace and its configuration file (`config.json`)"
       ]
@@ -254,18 +261,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
-        "1. create the configuration (this step is local and only takes a second)\n",
-        "2. create the cluster (this step will take about **20 seconds**)\n",
-        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
+        "## Get default AmlCompute\n",
+        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
       ]
     },
     {
@@ -274,26 +271,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
-        "\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
-        "    print('Found existing compute target')\n",
-        "except ComputeTargetException:\n",
-        "    print('Creating a new compute target...')\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
-        "                                                           max_nodes=4)\n",
-        "\n",
-        "    # create the cluster\n",
-        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
-        "\n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it uses the scale settings for the cluster\n",
-        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -303,7 +281,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'gpucluster' of type `AmlCompute`."
+        "Now that you have retrieved the compute target, let's see what the workspace's `compute_targets` property returns."
       ]
     },
     {
@@ -684,7 +662,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveRunConfig, PrimaryMetricGoal\n",
+        "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveConfig, PrimaryMetricGoal\n",
         "from azureml.train.hyperdrive import choice, loguniform\n",
         "\n",
         "ps = RandomParameterSampling(\n",
@@ -746,13 +724,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "htc = HyperDriveRunConfig(estimator=est, \n",
-        "                          hyperparameter_sampling=ps, \n",
-        "                          policy=policy, \n",
-        "                          primary_metric_name='validation_acc', \n",
-        "                          primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
-        "                          max_total_runs=8,\n",
-        "                          max_concurrent_runs=4)"
+        "htc = HyperDriveConfig(estimator=est, \n",
+        "                       hyperparameter_sampling=ps, \n",
+        "                       policy=policy, \n",
+        "                       primary_metric_name='validation_acc', \n",
+        "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
+        "                       max_total_runs=8,\n",
+        "                       max_concurrent_runs=4)"
       ]
     },
     {

how-to-use-azureml/training/README.md

@@ -8,5 +8,6 @@ Follow these sample notebooks to learn:
 4. [Train on AmlCompute](train-on-amlcompute): train a model using an AmlCompute cluster as compute target.
 5. [Train in an HDI Spark cluster](train-in-spark): train a Spark ML model using an HDInsight Spark cluster as compute target.
 6. [Logging API](logging-api): experiment with various logging functions to create runs and automatically generate graphs.
+7. [Train and hyperparameter tune on Iris Dataset with Scikit-learn](train-hyperparameter-tune-deploy-with-sklearn): train a model using the Scikit-learn estimator and tune hyperparameters with Hyperdrive.
 
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/README.png)
+ ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/README.png)

how-to-use-azureml/training/logging-api/logging-api.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/logging-api/logging-api.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/logging-api/logging-api.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -100,7 +100,7 @@
         "\n",
         "# Check core SDK version number\n",
         "\n",
-        "print(\"This notebook was created using SDK version 1.0.23, you are currently running version\", azureml.core.VERSION)"
+        "print(\"This notebook was created using SDK version 1.0.41, you are currently running version\", azureml.core.VERSION)"
       ]
     },
     {
@@ -271,9 +271,11 @@
       "metadata": {},
       "source": [
         "### Logging vectors\n",
-        "Vectors are good for recording information such as loss curves.  You can log a vector by create a list of numbers and call ``log_list()`` and supply a name and the list, or by repeatedly logging a value using the same name.\n",
+        "Vectors are good for recording information such as loss curves. You can log a vector by creating a list of numbers, calling ``log_list()`` and supplying a name and the list, or by repeatedly logging a value using the same name.\n",
         "\n",
-        "Vectors are presented in Run Details as a chart, and are directly comparable in experiment reports when placed in a chart.  **Note:** vectors logged into the run are expected to be relatively small.  Logging very large vectors into Azure ML can result in reduced performance.  If you need to store large amounts of data associated with the run, you can write the data to file that will be uploaded."
+        "Vectors are presented in Run Details as a chart, and are directly comparable in experiment reports when placed in a chart. \n",
+        "\n",
+        "**Note:** vectors logged into the run are expected to be relatively small. Logging very large vectors into Azure ML can result in reduced performance. If you need to store large amounts of data associated with the run, you can write the data to file that will be uploaded."
       ]
     },
     {
@@ -304,7 +306,9 @@
         "* Create a dictionary of lists where each list represents a column in the table and call ``log_table()``\n",
         "* Repeatedly call ``log_row()`` providing the same table name with a consistent set of named args as the column values\n",
         "\n",
-        "Tables are presented in Run Details as a chart using the first two columns of the table  **Note:** tables logged into the run are expected to be relatively small.  Logging very large tables into Azure ML can result in reduced performance.  If you need to store large amounts of data associated with the run, you can write the data to file that will be uploaded."
+        "Tables are presented in Run Details as a chart using the first two columns of the table  \n",
+        "\n",
+        "**Note:** tables logged into the run are expected to be relatively small.  Logging very large tables into Azure ML can result in reduced performance.  If you need to store large amounts of data associated with the run, you can write the data to file that will be uploaded."
       ]
     },
     {
@@ -365,7 +369,7 @@
       "source": [
         "### Uploading files\n",
         "\n",
-        "Any files that are placed in the ``.\\outputs`` directory are automatically uploaded when the run is completed.  These files are also visible in the *Outputs* tab of the Run Details page.  Files can also be uploaded explicitly and stored as artifacts along with the run record.\n"
+        "Files can also be uploaded explicitly and stored as artifacts along with the run record. These files are also visible in the *Outputs* tab of the Run Details page.\n"
       ]
     },
     {
@@ -374,9 +378,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "%%writefile .\\outputs\\myfile.txt\n",
+        "file_name = 'outputs/myfile.txt'\n",
         "\n",
-        "This is an output file that will be automatically uploaded."
+        "with open(file_name, \"w\") as f:\n",
+        "    f.write('This is an output file that will be uploaded.\\n')\n",
+        "\n",
+        "# Upload the file explicitly into artifacts \n",
+        "run.upload_file(name = file_name, path_or_stream = file_name)"
       ]
     },
     {
@@ -504,7 +512,7 @@
         "## Next steps\n",
         "To experiment more with logging and to understand how metrics can be visualized, go back to the *Start a run* section, try changing the category and scale_factor values and going through the notebook several times.  Play with the KPI, charting, and column selection options on the experiment's Run History reports page to see how the various metrics can be combined and visualized.\n",
         "\n",
-        "After learning about all of the logging options, go to the [train on remote vm](..\\train_on_remote_vm\\train_on_remote_vm.ipnyb) notebook and experiment with logging from remote compute contexts."
+        "After learning about all of the logging options, go to the [train on remote vm](..\\train-on-remote-vm\\train-on-remote-vm.ipynb) notebook and experiment with logging from remote compute contexts."
       ]
     }
   ],
@@ -534,4 +542,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training/manage-runs/manage-runs.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/manage-runs/manage-runs.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/manage-runs/manage-runs.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -52,7 +52,7 @@
       "source": [
         "## Setup\n",
         "\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't. Also, if you're new to Azure ML, we recommend that you go through [the tutorial](https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-train-models-with-aml) first to learn the basic concepts.\n",
+        "If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace. Also, if you're new to Azure ML, we recommend that you go through [the tutorial](https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-train-models-with-aml) first to learn the basic concepts.\n",
         "\n",
         "Let's first import required packages, check Azure ML SDK version, connect to your workspace and create an Experiment to hold the runs."
       ]
@@ -599,4 +599,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training/train-in-spark/train-in-spark.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-in-spark/train-in-spark.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-in-spark/train-in-spark.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -32,7 +32,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't."
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
       ]
     },
     {
@@ -117,7 +117,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "**Note** You can use Docker-based execution to run the Spark job in local computer or a remote VM. Please see the `train-in-remote-vm` notebook for example on how to configure and run in Docker mode in a VM. Make sure you choose a Docker image that has Spark installed, such as `azureml.core.runconfig.DEFAULT_MMLSPARK_CPU_IMAGE`."
+        "**Note** You can use Docker-based execution to run the Spark job in local computer or a remote VM. Please see the `train-in-remote-vm` notebook for example on how to configure and run in Docker mode in a VM. Make sure you choose a Docker image that has Spark installed, such as `microsoft/mmlspark:0.12`."
       ]
     },
     {
@@ -282,4 +282,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb

@@ -9,12 +9,12 @@
         "Licensed under the MIT License."
       ]
     },
-     {
+    {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.png)"
-         ] 
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.png)"
+      ]
     },
     {
       "cell_type": "markdown",
@@ -38,7 +38,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't."
+        "If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace."
       ]
     },
     {
@@ -168,9 +168,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Provision as a run based compute target\n",
+        "### Create environment\n",
         "\n",
-        "You can provision AmlCompute as a compute target at run-time. In this case, the compute is auto-created for your run, scales up to max_nodes that you specify, and then **deleted automatically** after the run completes."
+        "Create Docker based environment with scikit-learn installed."
       ]
     },
     {
@@ -179,43 +179,56 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import RunConfiguration\n",
+        "from azureml.core import Environment\n",
         "from azureml.core.conda_dependencies import CondaDependencies\n",
+        "\n",
+        "myenv = Environment(\"myenv\")\n",
+        "\n",
+        "myenv.docker.enabled = True\n",
+        "myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Get the default compute target\n",
+        "\n",
+        "In this case, we use the default `AmlCompute`target from the workspace."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import ScriptRunConfig\n",
         "from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
         "\n",
-        "# create a new runconfig object\n",
-        "run_config = RunConfiguration()\n",
+        "src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
         "\n",
-        "# signal that you want to use AmlCompute to execute script.\n",
-        "run_config.target = \"amlcompute\"\n",
+        "# Use default compute target\n",
+        "src.run_config.target = ws.get_default_compute_target(type=\"CPU\").name\n",
         "\n",
-        "# AmlCompute will be created in the same region as workspace\n",
-        "# Set vm size for AmlCompute\n",
-        "run_config.amlcompute.vm_size = 'STANDARD_D2_V2'\n",
-        "\n",
-        "# enable Docker \n",
-        "run_config.environment.docker.enabled = True\n",
-        "\n",
-        "# set Docker base image to the default CPU-based image\n",
-        "run_config.environment.docker.base_image = DEFAULT_CPU_IMAGE\n",
-        "\n",
-        "# use conda_dependencies.yml to create a conda environment in the Docker image for execution\n",
-        "run_config.environment.python.user_managed_dependencies = False\n",
-        "\n",
-        "# auto-prepare the Docker image when used for execution (if it is not already prepared)\n",
-        "run_config.auto_prepare_environment = True\n",
-        "\n",
-        "# specify CondaDependencies obj\n",
-        "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
-        "\n",
-        "# Now submit a run on AmlCompute\n",
-        "from azureml.core.script_run_config import ScriptRunConfig\n",
-        "\n",
-        "script_run_config = ScriptRunConfig(source_directory=project_folder,\n",
-        "                                    script='train.py',\n",
-        "                                    run_config=run_config)\n",
-        "\n",
-        "run = experiment.submit(script_run_config)\n",
+        "# Set environment\n",
+        "src.run_config.environment = myenv"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Submit run"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "run = experiment.submit(src)\n",
         "\n",
         "# Show run details\n",
         "run"
@@ -297,27 +310,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import RunConfiguration\n",
-        "from azureml.core.conda_dependencies import CondaDependencies\n",
-        "\n",
-        "# create a new RunConfig object\n",
-        "run_config = RunConfiguration(framework=\"python\")\n",
-        "\n",
-        "# Set compute target to AmlCompute target created in previous step\n",
-        "run_config.target = cpu_cluster.name\n",
-        "\n",
-        "# enable Docker \n",
-        "run_config.environment.docker.enabled = True\n",
-        "\n",
-        "# specify CondaDependencies obj\n",
-        "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
-        "\n",
-        "from azureml.core import Run\n",
-        "from azureml.core import ScriptRunConfig\n",
-        "\n",
-        "src = ScriptRunConfig(source_directory=project_folder, \n",
-        "                      script='train.py', \n",
-        "                      run_config=run_config) \n",
+        "# Set compute target to the one created in previous step\n",
+        "src.run_config.target = cpu_cluster.name\n",
+        " \n",
         "run = experiment.submit(config=src)\n",
         "run"
       ]
@@ -402,27 +397,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import RunConfiguration\n",
-        "from azureml.core.conda_dependencies import CondaDependencies\n",
-        "\n",
-        "# create a new RunConfig object\n",
-        "run_config = RunConfiguration(framework=\"python\")\n",
-        "\n",
-        "# Set compute target to AmlCompute target created in previous step\n",
-        "run_config.target = cpu_cluster.name\n",
-        "\n",
-        "# enable Docker \n",
-        "run_config.environment.docker.enabled = True\n",
-        "\n",
-        "# specify CondaDependencies obj\n",
-        "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
-        "\n",
-        "from azureml.core import Run\n",
-        "from azureml.core import ScriptRunConfig\n",
-        "\n",
-        "src = ScriptRunConfig(source_directory=project_folder, \n",
-        "                      script='train.py', \n",
-        "                      run_config=run_config) \n",
+        "# Set compute target to the one created in previous step\n",
+        "src.run_config.target = cpu_cluster.name\n",
+        " \n",
         "run = experiment.submit(config=src)\n",
         "run"
       ]
@@ -526,4 +503,4 @@
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training/train-on-local/train-on-local.ipynb

@@ -8,35 +8,68 @@
         "\n",
         "Licensed under the MIT License."
       ]
-    },
-     {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-          "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-local/train-on-local.png)"
-         ] 
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# 02. Train locally\n",
-        "* Create or load workspace.\n",
-        "* Create scripts locally.\n",
-        "* Create `train.py` in a folder, along with a `my.lib` file.\n",
-        "* Configure & execute a local run in a user-managed Python environment.\n",
-        "* Configure & execute a local run in a system-managed Python environment.\n",
-        "* Configure & execute a local run in a Docker environment.\n",
-        "* Query run metrics to find the best model\n",
-        "* Register model for operationalization."
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-local/train-on-local.png)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Prerequisites\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't."
+        "# 02. Train locally\n",
+        "_**Train a model locally: Directly on your machine and within a Docker container**_\n",
+        "\n",
+        "---\n",
+        "\n",
+        "\n",
+        "## Table of contents\n",
+        "1. [Introduction](#intro)\n",
+        "1. [Pre-requisites](#pre-reqs)\n",
+        "1. [Initialize Workspace](#init)\n",
+        "1. [Create An Experiment](#exp)\n",
+        "1. [View training and auxiliary scripts](#view)\n",
+        "1. [Configure & Run](#config-run)\n",
+        "    1. User-managed environment\n",
+        "        1. Set the environment up\n",
+        "        1. Submit the script to run in the user-managed environment\n",
+        "        1. Get run history details\n",
+        "    1. System-managed environment\n",
+        "        1. Set the environment up\n",
+        "        1. Submit the script to run in the system-managed environment\n",
+        "        1. Get run history details\n",
+        "    1. Docker-based execution\n",
+        "        1. Set the environment up\n",
+        "        1. Submit the script to run in the system-managed environment\n",
+        "        1. Get run history details\n",
+        "        1. Use a custom Docker image\n",
+        "1. [Query run metrics](#query)\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## 1. Introduction <a id='intro'></a>\n",
+        "\n",
+        "In this notebook, we will learn how to:\n",
+        "\n",
+        "* Connect to our AML workspace\n",
+        "* Create or load a workspace\n",
+        "* Configure & execute a local run in:\n",
+        "    - a user-managed Python environment\n",
+        "    - a system-managed Python environment\n",
+        "    - a Docker environment\n",
+        "* Query run metrics to find the best model trained in the run\n",
+        "* Register that model for operationalization"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 2. Pre-requisites <a id='pre-reqs'></a>\n",
+        "In this notebook, we assume that you have set your Azure Machine Learning workspace. If you have not, make sure you go through the [configuration notebook](../../../configuration.ipynb) first. In the end, you should have configuration file that contains the subscription ID, resource group and name of your workspace."
       ]
     },
     {
@@ -55,9 +88,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Initialize Workspace\n",
+        "## 3. Initialize Workspace <a id='init'></a>\n",
         "\n",
-        "Initialize a workspace object from persisted configuration."
+        "Initialize your workspace object from configuration file"
       ]
     },
     {
@@ -76,8 +109,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Create An Experiment\n",
-        "**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
+        "## 4. Create An Experiment <a id='exp'></a>\n",
+        "An experiment is a logical container in an Azure ML Workspace. It contains a series of trials called `Runs`. As such, it hosts run records such as run metrics, logs, and other output artifacts from your experiments."
       ]
     },
     {
@@ -95,9 +128,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## View `train.py`\n",
+        "## 5. View training and auxiliary scripts <a id='view'></a>\n",
         "\n",
-        "`train.py` is already created for you."
+        "For convenience, we already created the training (`train.py`) script and supportive libraries (`mylib.py`) for you. Take a few minutes to examine both files."
       ]
     },
     {
@@ -110,13 +143,6 @@
         "    print(f.read())"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Note `train.py` also references a `mylib.py` file."
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -131,9 +157,11 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Configure & Run\n",
-        "### User-managed environment\n",
-        "Below, we use a user-managed run, which means you are responsible to ensure all the necessary packages are available in the Python environment you choose to run the script."
+        "## 6. Configure & Run <a id='config-run'></a>\n",
+        "### 6.A User-managed environment\n",
+        "\n",
+        "#### 6.A.a Set the environment up\n",
+        "When using a user-managed environment, you are responsible for ensuring that all the necessary packages are available in the Python environment you choose to run the script in."
       ]
     },
     {
@@ -142,23 +170,23 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import RunConfiguration\n",
+        "from azureml.core import Environment\n",
         "\n",
         "# Editing a run configuration property on-fly.\n",
-        "run_config_user_managed = RunConfiguration()\n",
+        "user_managed_env = Environment(\"user-managed-env\")\n",
         "\n",
-        "run_config_user_managed.environment.python.user_managed_dependencies = True\n",
+        "user_managed_env.python.user_managed_dependencies = True\n",
         "\n",
         "# You can choose a specific Python environment by pointing to a Python path \n",
-        "#run_config.environment.python.interpreter_path = '/home/johndoe/miniconda3/envs/myenv/bin/python'"
+        "#user_managed_env.python.interpreter_path = '/home/johndoe/miniconda3/envs/myenv/bin/python'"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Submit script to run in the user-managed environment\n",
-        "Note whole script folder is submitted for execution, including the `mylib.py` file."
+        "#### 6.A.b Submit the script to run in the user-managed environment\n",
+        "Whatever the way you manage your environment, you need to use the `ScriptRunConfig` class. It allows you to further configure your run by pointing to the `train.py` script and to the working directory, which also contains the `mylib.py` file. These inputs indeed provide the commands to execute in the run. Once the run is configured, you submit it to your experiment."
       ]
     },
     {
@@ -169,7 +197,16 @@
       "source": [
         "from azureml.core import ScriptRunConfig\n",
         "\n",
-        "src = ScriptRunConfig(source_directory='./', script='train.py', run_config=run_config_user_managed)\n",
+        "src = ScriptRunConfig(source_directory='./', script='train.py')\n",
+        "src.run_config.environment = user_managed_env"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
         "run = exp.submit(src)"
       ]
     },
@@ -177,7 +214,11 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Get run history details"
+        "#### 6.A.c Get run history details\n",
+        "\n",
+        "While all calculations were run on your machine (cf. below), by using a `run` you also captured the results of your calculations into your run and experiment. You can then see them on the Azure portal, through the link displayed as output of the following cell.\n",
+        "\n",
+        "**Note**: The recording of the computation results into your run was made possible by the `run.log()` commands in the `train.py` file."
       ]
     },
     {
@@ -200,7 +241,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Block to wait till run finishes."
+        "Block any execution to wait until the run finishes."
       ]
     },
     {
@@ -216,8 +257,25 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### System-managed environment\n",
-        "You can also ask the system to build a new conda environment and execute your scripts in it. The environment is built once and will be reused in subsequent executions as long as the conda dependencies remain unchanged. "
+        "**Note:** All these calculations were run on your local machine, in the conda environment you defined above. You can find the results in:\n",
+        "- `~/.azureml/envs/azureml_xxxx` for the conda environment you just created\n",
+        "- `~/AppData/Local/Temp/azureml_runs/train-on-local_xxxx` for the machine learning models you trained (this path may differ depending on the platform you use). This folder also contains\n",
+        "  - Logs (under azureml_logs/)\n",
+        "  - Output pickled files (under outputs/)\n",
+        "  - The configuration files (credentials, local and docker image setups)\n",
+        "  - The train.py and mylib.py scripts\n",
+        "  - The current notebook\n",
+        "\n",
+        "Take a few minutes to examine the output of the cell above. It shows the content of some of the log files, and extra information on the conda environment used."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### 6.B System-managed environment\n",
+        "#### 6.B.a Set the environment up\n",
+        "Now, instead of managing the setup of the environment yourself, you can ask the system to build a new conda environment for you. The environment is built once, and will be reused in subsequent executions as long as the conda dependencies remain unchanged."
       ]
     },
     {
@@ -228,22 +286,23 @@
       "source": [
         "from azureml.core.conda_dependencies import CondaDependencies\n",
         "\n",
-        "run_config_system_managed = RunConfiguration()\n",
+        "system_managed_env = Environment(\"system-managed-env\")\n",
         "\n",
-        "run_config_system_managed.environment.python.user_managed_dependencies = False\n",
-        "run_config_system_managed.auto_prepare_environment = True\n",
+        "system_managed_env.python.user_managed_dependencies = False\n",
         "\n",
         "# Specify conda dependencies with scikit-learn\n",
         "cd = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
-        "run_config_system_managed.environment.python.conda_dependencies = cd"
+        "system_managed_env.python.conda_dependencies = cd"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Submit script to run in the system-managed environment\n",
-        "A new conda environment is built based on the conda dependencies object. If you are running this for the first time,  this might take up to 5 mninutes. But this conda environment is reused so long as you don't change the conda dependencies."
+        "#### 6.B.b Submit the script to run in the system-managed environment\n",
+        "A new conda environment is built based on the conda dependencies object. If you are running this for the first time, this might take up to 5 minutes.\n",
+        "\n",
+        "The commands used to execute the run are then the same as the ones you used above."
       ]
     },
     {
@@ -252,7 +311,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "src = ScriptRunConfig(source_directory=\"./\", script='train.py', run_config=run_config_system_managed)\n",
+        "src.run_config.environment = system_managed_env\n",
         "run = exp.submit(src)"
       ]
     },
@@ -260,7 +319,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Get run history details"
+        "#### 6.B.c Get run history details"
       ]
     },
     {
@@ -272,13 +331,6 @@
         "run"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Block and wait till run finishes."
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -292,12 +344,34 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Docker-based execution\n",
-        "**IMPORTANT**: You must have Docker engine installed locally in order to use this execution mode. If your kernel is already running in a Docker container, such as **Azure Notebooks**, this mode will **NOT** work.\n",
+        "### 6.C Docker-based execution\n",
+        "In this section, you will train the same models, but you will do so in a Docker container, on your local machine. For this, you then need to have the Docker engine installed locally. If you don't have it yet, please follow the instructions below.\n",
         "\n",
-        "NOTE: The GPU base image must be used on Microsoft Azure Services only such as ACI, AML Compute, Azure VMs, and AKS.\n",
+        "#### How to install Docker\n",
         "\n",
-        "You can also ask the system to pull down a Docker image and execute your scripts in it."
+        "- [Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/)\n",
+        "- [MacOs](https://docs.docker.com/docker-for-mac/install/)\n",
+        "- [Windows](https://docs.docker.com/docker-for-windows/install/)\n",
+        "\n",
+        "    In case of issues, troubleshooting documentation can be found [here](https://docs.docker.com/docker-for-windows/troubleshoot/#running-docker-for-windows-in-nested-virtualization-scenarios). Additionally, you can follow the steps below, if Virtualization is not enabled on your machine:\n",
+        "    - Go to Task Manager > Performance\n",
+        "    - Check that Virtualization is enabled\n",
+        "    - If it is not, go to `Start > Settings > Update and security > Recovery > Advanced Startup - Restart now > Troubleshoot > Advanced options > UEFI firmware settings - restart`\n",
+        "    - In the BIOS, go to `Advanced > System options > Click the \"Virtualization Technology (VTx)\" only > Save > Exit > Save all changes` -- This will restart the machine\n",
+        "\n",
+        "**Notes**: \n",
+        "- If your kernel is already running in a Docker container, such as **Azure Notebooks**, this mode will **NOT** work.\n",
+        "- If you use a GPU base image, it needs to be used on Microsoft Azure Services such as ACI, AML Compute, Azure VMs, or AKS.\n",
+        "\n",
+        "You can also ask the system to pull down a Docker image and execute your scripts in it.\n",
+        "\n",
+        "#### 6.C.a Set the environment up\n",
+        "\n",
+        "In the cell below, you will configure your run to execute in a Docker container. It will:\n",
+        "- run on a CPU\n",
+        "- contain a conda environment in which the scikit-learn library will be installed.\n",
+        "\n",
+        "As before, you will finish configuring your run by pointing to the `train.py` and `mylib.py` files."
       ]
     },
     {
@@ -306,27 +380,25 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "run_config_docker = RunConfiguration()\n",
-        "run_config_docker.environment.python.user_managed_dependencies = False\n",
-        "run_config_docker.auto_prepare_environment = True\n",
-        "run_config_docker.environment.docker.enabled = True\n",
+        "docker_env = Environment(\"docker-env\")\n",
+        "\n",
+        "docker_env.python.user_managed_dependencies = False\n",
+        "docker_env.docker.enabled = True\n",
         "\n",
         "# use the default CPU-based Docker image from Azure ML\n",
-        "run_config_docker.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE\n",
+        "print(docker_env.docker.base_image)\n",
         "\n",
         "# Specify conda dependencies with scikit-learn\n",
-        "cd = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
-        "run_config_docker.environment.python.conda_dependencies = cd\n",
-        "\n",
-        "src = ScriptRunConfig(source_directory=\"./\", script='train.py', run_config=run_config_docker)"
+        "docker_env.python.conda_dependencies = cd"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Submit script to run in the system-managed environment\n",
-        "A new conda environment is built based on the conda dependencies object. If you are running this for the first time, this might take up to 5 minutes. But this conda environment is reused so long as you don't change the conda dependencies."
+        "####  6.C.b Submit the script to run in the system-managed environment\n",
+        "\n",
+        "The run is now configured and ready to be executed in a Docker container. If you are running this for the first time, the Docker container will get created, as well as the conda environment inside it. This will take several minutes. Once all this is generated, however, this conda environment will be reused as long as you don't change the conda dependencies."
       ]
     },
     {
@@ -337,6 +409,8 @@
       "source": [
         "import subprocess\n",
         "\n",
+        "src.run_config.environment = docker_env\n",
+        "\n",
         "# Check if Docker is installed and Linux containers are enabled\n",
         "if subprocess.run(\"docker -v\", shell=True).returncode == 0:\n",
         "    out = subprocess.check_output(\"docker system info\", shell=True).decode('ascii')\n",
@@ -345,7 +419,34 @@
         "    else:\n",
         "        run = exp.submit(src)\n",
         "else:\n",
-        "    print(\"Docker engine not installed.\")"
+        "    print(\"Docker engine is not installed.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "##### Potential issue on Windows and how to solve it\n",
+        "\n",
+        "If you are using a Windows machine, the creation of the Docker image may fail, and you may see the following error message\n",
+        "`docker: Error response from daemon: Drive has not been shared. Failed to launch docker container. Check that docker is running and that C:\\ on Windows and /tmp elsewhere is shared.`\n",
+        "\n",
+        "This is because the process above tries to create a linux-based, i.e. non-windows-based, Docker image. To fix this, you can:\n",
+        "- Open the Docker user interface\n",
+        "- Navigate to Settings > Shared drives\n",
+        "- Select C (or both C and D, if you have one)\n",
+        "- Apply\n",
+        "\n",
+        "When this is done, you can try and re-run the command above.\n",
+        "\n",
+        "<img src=\"./docker_settings.png\" width=\"500\" align=\"left\">"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 6.C.c Get run history details"
       ]
     },
     {
@@ -354,7 +455,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "#Get run history details\n",
+        "# Get run history details\n",
         "run"
       ]
     },
@@ -371,22 +472,51 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Use a custom Docker image\n",
+        "The results obtained here should be the same as those obtained before. However, take a look at the \"Execution summary\" section in the output of the cell above. Look for \"docker\". There, you should see the \"enabled\" field set to True. Compare this to the 2 prior runs (\"enabled\" was then set to False)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 6.C.d Use a custom Docker image\n",
         "\n",
-        "You can also specify a custom Docker image if you don't want to use the default image provided by Azure ML.\n",
+        "You can also specify a custom Docker image, if you don't want to use the default image provided by Azure ML.\n",
         "\n",
+        "You can either pull an image directly from Anaconda:\n",
         "```python\n",
-        "# use an image available in Docker Hub without authentication\n",
+        "# Use an image available in Docker Hub without authentication\n",
         "run_config_docker.environment.docker.base_image = \"continuumio/miniconda3\"\n",
+        "```\n",
         "\n",
-        "# or, use an image available in a private Azure Container Registry\n",
+        "Or one of the images you may already have created:\n",
+        "```python\n",
+        "# or, use an image available in your private Azure Container Registry\n",
         "run_config_docker.environment.docker.base_image = \"mycustomimage:1.0\"\n",
         "run_config_docker.environment.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
         "run_config_docker.environment.docker.base_image_registry.username = \"username\"\n",
         "run_config_docker.environment.docker.base_image_registry.password = \"password\"\n",
         "```\n",
         "\n",
-        "When you are using a custom Docker image, you might already have your environment setup properly in a Python environment in the Docker image. In that case, you can skip specifying conda dependencies, and just use `user_managed_dependencies` option instead:\n",
+        "##### Where to find my Docker image name and registry credentials\n",
+        "    If you do not know what the name of your Docker image or container registry is, or if you don't know how to access the username and password needed above, proceed as follows:\n",
+        "    - Docker image name:\n",
+        "        - In the portal, under your resource group, click on your current workspace\n",
+        "        - Click on Experiments\n",
+        "        - Click on Images\n",
+        "        - Click on the image of your choice\n",
+        "        - Copy the \"ID\" string\n",
+        "        - In this notebook, replace \"mycustomimage:1/0\" with that ID string\n",
+        "    - Username and password:\n",
+        "        - In the portal, under your resource group, click on the container registry associated with your workspace\n",
+        "            - If you have several and don't know which one you need, click on your workspace, go to Overview and click on the \"Registry\" name on the upper right of the screen\n",
+        "        - There, go to \"Access keys\"\n",
+        "        - Copy the username and one of the passwords\n",
+        "        - In this notebook, replace \"username\" and \"password\" by these values\n",
+        "\n",
+        "In any case, you will need to use the lines above in place of the line marked as `# Reference Docker image` in section 6.C.a. \n",
+        "\n",
+        "When you are using your custom Docker image, you might already have your Python environment properly set up. In that case, you can skip specifying conda dependencies, and just use the `user_managed_dependencies` option instead:\n",
         "```python\n",
         "run_config_docker.environment.python.user_managed_dependencies = True\n",
         "# path to the Python environment in the custom Docker image\n",
@@ -398,7 +528,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Query run metrics"
+        "## 7. Query run metrics <a id='query'></a>\n",
+        "\n",
+        "Once your run has completed, you can now extract the metrics you captured by using the `get_metrics` method. As shown in the `train.py` file, these metrics are \"alpha\" and \"mse\"."
       ]
     },
     {
@@ -412,7 +544,7 @@
       },
       "outputs": [],
       "source": [
-        "# get all metris logged in the run\n",
+        "# Get all metris logged in the run\n",
         "run.get_metrics()\n",
         "metrics = run.get_metrics()"
       ]
@@ -483,7 +615,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "We know the model `ridge_0.40.pkl` is the best performing model from the earlier queries. So let's register it with the workspace."
+        "From the results obtained above, `ridge_0.40.pkl` is the best performing model. You can now register that particular model with the workspace. Once you have done so, go back to the portal and click on \"Models\". You should see it there."
       ]
     },
     {
@@ -492,7 +624,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# supply a model name, and the full path to the serialized model file.\n",
+        "# Supply a model name, and the full path to the serialized model file.\n",
         "model = run.register_model(model_name='best_ridge_model', model_path='./outputs/ridge_0.40.pkl')"
       ]
     },
@@ -502,14 +634,14 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(model.name, model.version, model.url)"
+        "print(\"Registered model:\\n --> Name: {}\\n --> Version: {}\\n --> URL: {}\".format(model.name, model.version, model.url))"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now you can deploy this model following the example in the 01 notebook."
+        "You can now deploy your model by following [this example](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb)."
       ]
     }
   ],
@@ -534,9 +666,9 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.8"
+      "version": "3.6.5"
     }
   },
   "nbformat": 4,
   "nbformat_minor": 2
-}
+}

how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -30,7 +37,7 @@
       "metadata": {},
       "source": [
         "## Prerequisites\n",
-        "Make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't."
+        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) first if you haven't already to establish your connection to the AzureML Workspace."
       ]
     },
     {
@@ -273,20 +280,11 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import RunConfiguration\n",
+        "from azureml.core import Environment\n",
         "from azureml.core.conda_dependencies import CondaDependencies\n",
         "\n",
-        "# create a new RunConfig object\n",
-        "conda_run_config = RunConfiguration(framework=\"python\")\n",
-        "\n",
-        "# Set compute target to the Linux DSVM\n",
-        "conda_run_config.target = attached_dsvm_compute.name\n",
-        "\n",
-        "# set the data reference of the run configuration\n",
-        "conda_run_config.data_references = {ds.name: dr}\n",
-        "\n",
-        "# specify CondaDependencies obj\n",
-        "conda_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
+        "conda_env = Environment(\"conda-env\")\n",
+        "conda_env.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
       ]
     },
     {
@@ -299,18 +297,14 @@
         "\n",
         "src = ScriptRunConfig(source_directory=script_folder, \n",
         "                      script='train.py', \n",
-        "                      run_config=conda_run_config, \n",
         "                      # pass the datastore reference as a parameter to the training script\n",
         "                      arguments=['--data-folder', str(ds.as_download())] \n",
         "                     ) \n",
-        "run = exp.submit(config=src)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
+        "\n",
+        "src.run_config.framework = \"python\"\n",
+        "src.run_config.environment = conda_env\n",
+        "src.run_config.target = attached_dsvm_compute.name\n",
+        "src.run_config.data_references = {ds.name: dr}"
       ]
     },
     {
@@ -319,9 +313,18 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "run = exp.submit(config=src)\n",
+        "\n",
         "run.wait_for_completion(show_output=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -352,17 +355,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# create a new RunConfig object\n",
-        "vm_run_config = RunConfiguration(framework=\"python\")\n",
-        "\n",
-        "# Set compute target to the Linux DSVM\n",
-        "vm_run_config.target = attached_dsvm_compute.name\n",
-        "\n",
-        "# set the data reference of the run coonfiguration\n",
-        "conda_run_config.data_references = {ds.name: dr}\n",
-        "\n",
-        "# Let system know that you will configure the Python environment yourself.\n",
-        "vm_run_config.environment.python.user_managed_dependencies = True"
+        "conda_env.python.user_managed_dependencies = True"
       ]
     },
     {
@@ -378,11 +371,6 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "src = ScriptRunConfig(source_directory=script_folder, \n",
-        "                      script='train.py', \n",
-        "                      run_config=vm_run_config,\n",
-        "                      # pass the datastore reference as a parameter to the training script\n",
-        "                      arguments=['--data-folder', str(ds.as_download())])\n",
         "run = exp.submit(config=src)\n",
         "run.wait_for_completion(show_output=True)"
       ]
@@ -391,7 +379,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script `train2.py` that doesn't have azureml dependencies, and submit it instead."
+        "You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script `train2.py` that doesn't have azureml or data store dependencies, and submit it instead."
       ]
     },
     {
@@ -404,7 +392,10 @@
         "shutil.copy('./train2.py', os.path.join(script_folder, 'train2.py'))\n",
         "\n",
         "with open(os.path.join(script_folder, './train2.py'), 'r') as training_script:\n",
-        "    print(training_script.read())"
+        "    print(training_script.read())\n",
+        "    \n",
+        "src.run_config.data_references = {}\n",
+        "src.script = \"train2.py\""
       ]
     },
     {
@@ -420,10 +411,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "src = ScriptRunConfig(source_directory=script_folder, \n",
-        "                      script='train2.py', \n",
-        "                      run_config=vm_run_config)\n",
         "run = exp.submit(config=src)\n",
+        "\n",
         "run.wait_for_completion(show_output=True)"
       ]
     },
@@ -457,24 +446,10 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Load the \"cpu-dsvm.runconfig\" file (created by the above attach operation) in memory\n",
-        "docker_run_config = RunConfiguration(framework=\"python\")\n",
+        "conda_env.docker.enabled = True\n",
+        "conda_env.python.user_managed_dependencies = False\n",
         "\n",
-        "# Set compute target to the Linux DSVM\n",
-        "docker_run_config.target = attached_dsvm_compute.name\n",
-        "\n",
-        "# Use Docker in the remote VM\n",
-        "docker_run_config.environment.docker.enabled = True\n",
-        "\n",
-        "# Use CPU base image from DockerHub\n",
-        "docker_run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE\n",
-        "print('Base Docker image is:', docker_run_config.environment.docker.base_image)\n",
-        "\n",
-        "# set the data reference of the run coonfiguration\n",
-        "docker_run_config.data_references = {ds.name: dr}\n",
-        "\n",
-        "# specify CondaDependencies obj\n",
-        "docker_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
+        "print('Base Docker image is:', conda_env.docker.base_image)"
       ]
     },
     {
@@ -491,20 +466,11 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "src = ScriptRunConfig(source_directory=script_folder, \n",
-        "                      script='train.py', \n",
-        "                      run_config=docker_run_config,\n",
-        "                      # pass the datastore reference as a parameter to the training script\n",
-        "                      arguments=['--data-folder', str(ds.as_download())])\n",
-        "run = exp.submit(config=src)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
+        "src.script = \"train.py\"\n",
+        "src.run_config.data_references = {ds.name: dr}\n",
+        "\n",
+        "run = exp.submit(config=src)\n",
+        "\n",
         "run.wait_for_completion(show_output=True)"
       ]
     },
@@ -518,20 +484,20 @@
         "\n",
         "```python\n",
         "# use an image available in Docker Hub without authentication\n",
-        "run_config_docker.environment.docker.base_image = \"continuumio/miniconda3\"\n",
+        "conda_env.docker.base_image = \"continuumio/miniconda3\"\n",
         "\n",
         "# or, use an image available in a private Azure Container Registry\n",
-        "run_config_docker.environment.docker.base_image = \"mycustomimage:1.0\"\n",
-        "run_config_docker.environment.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
-        "run_config_docker.environment.docker.base_image_registry.username = \"username\"\n",
-        "run_config_docker.environment.docker.base_image_registry.password = \"password\"\n",
+        "conda_env.docker.base_image = \"mycustomimage:1.0\"\n",
+        "conda_env.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
+        "conda_env.docker.base_image_registry.username = \"username\"\n",
+        "conda_env.docker.base_image_registry.password = \"password\"\n",
         "```\n",
         "\n",
         "When you are using a custom Docker image, you might already have your environment setup properly in a Python environment in the Docker image. In that case, you can skip specifying conda dependencies, and just use `user_managed_dependencies` option instead:\n",
         "```python\n",
-        "run_config_docker.environment.python.user_managed_dependencies = True\n",
+        "conda_env.python.user_managed_dependencies = True\n",
         "# path to the Python environment in the custom Docker image\n",
-        "run_config.environment.python.interpreter_path = '/opt/conda/bin/python'\n",
+        "conda_env.python.interpreter_path = '/opt/conda/bin/python'\n",
         "```"
       ]
     },
@@ -633,7 +599,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.6"
+      "version": "3.6.5"
     }
   },
   "nbformat": 4,

how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb

@@ -9,6 +9,13 @@
         "Licensed under the MIT License."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-within-notebook/train-within-notebook.png)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -36,7 +43,7 @@
         "    1. Deploy your webservice\n",
         "    1. Test your webservice\n",
         "    1. Clean up\n",
-        "1. [Next Steps](#Next%20Steps)\n",
+        "1. [Next Steps](#nextsteps)\n",
         "\n",
         "---\n",
         "\n",
@@ -57,7 +64,7 @@
         "---\n",
         "\n",
         "## Setup\n",
-        "Make sure you have completed the [Configuration](../../../configuration.ipnyb) notebook to set up your Azure Machine Learning workspace and ensure other common prerequisites are met.  From the configuration, the important sections are the workspace configuration and ACI regristration.\n",
+        "If you are using an Azure Machine Learning Notebook VM, you are all set.  Otherwise, go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't already to establish your connection to the AzureML Workspace. From the configuration, the important sections are the workspace configuration and ACI regristration.\n",
         "\n",
         "We will also need the following libraries install to our conda environment.  If these are not installed, use the following command to do so and restart the notebook.\n",
         "```shell\n",
@@ -156,7 +163,7 @@
         "experiment = Experiment(workspace=ws, name=\"train-within-notebook\")\n",
         "\n",
         "# Create a run object in the experiment\n",
-        "run = experiment.start_logging()\n",
+        "run =  experiment.start_logging()\n",
         "# Log the algorithm parameter alpha to the run\n",
         "run.log('alpha', 0.03)\n",
         "\n",
@@ -170,7 +177,12 @@
         "run.log('mse', mean_squared_error(data['test']['y'], preds))\n",
         "\n",
         "# Save the model to the outputs directory for capture\n",
-        "joblib.dump(value=regression_model, filename='outputs/model.pkl')\n",
+        "model_file_name = 'outputs/model.pkl'\n",
+        "\n",
+        "joblib.dump(value = regression_model, filename = model_file_name)\n",
+        "\n",
+        "# upload the model file explicitly into artifacts \n",
+        "run.upload_file(name = model_file_name, path_or_stream = model_file_name)\n",
         "\n",
         "# Complete the run\n",
         "run.complete()"
@@ -214,8 +226,6 @@
         "import numpy as np\n",
         "from tqdm import tqdm\n",
         "\n",
-        "model_name = \"model.pkl\"\n",
-        "\n",
         "# list of numbers from 0 to 1.0 with a 0.05 interval\n",
         "alphas = np.arange(0.0, 1.0, 0.05)\n",
         "\n",
@@ -414,7 +424,7 @@
         "### Describe your target compute\n",
         "In addition to the container, we also need to describe the type of compute we want to allocate for our webservice.  In in this example we are using an [Azure Container Instance](https://azure.microsoft.com/en-us/services/container-instances/) which is a good choice for quick and cost-effective dev/test deployment scenarios.  ACI instances require the number of cores you want to run and memory you need.  Tags and descriptions are available for you to identify the instances in AML when viewing the Compute tab in the AML Portal.\n",
         "\n",
-        "For production workloads, it is better to use [Azure Kubernentes Service (AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/) instead. Try [this notebook](11.production-deploy-to-aks.ipynb) to see how that can be done from Azure ML.\n"
+        "For production workloads, it is better to use [Azure Kubernentes Service (AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/) instead. Try [this notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb) to see how that can be done from Azure ML.\n"
       ]
     },
     {
@@ -510,6 +520,9 @@
       "outputs": [],
       "source": [
         "import json\n",
+        "\n",
+        "service = ws.webservices['my-aci-svc']\n",
+        "\n",
         "# scrape the first row from the test set.\n",
         "test_samples = json.dumps({\"data\": X_test[0:1, :].tolist()})\n",
         "\n",
@@ -643,7 +656,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "---\n",
+        "<a id='nextsteps'></a>\n",
         "## Next Steps"
       ]
     },
@@ -661,13 +674,6 @@
         "If you want to deploy models to a production cluster try the [production-deploy-to-aks](../../deployment/production-deploy-to-aks\n",
         ") notebook."
       ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": []
     }
   ],
   "metadata": {
@@ -691,7 +697,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.6"
+      "version": "3.6.5"
     }
   },
   "nbformat": 4,