update samples - test

add pipeline scripts

Dockerfiles/1.0.43/Dockerfile

@@ -0,0 +1,29 @@
+FROM continuumio/miniconda:4.5.11
+
+# install git
+RUN apt-get update && apt-get upgrade -y && apt-get install -y git
+
+# create a new conda environment named azureml
+RUN conda create -n azureml -y -q Python=3.6
+
+# install additional packages used by sample notebooks. this is optional
+RUN ["/bin/bash", "-c", "source activate azureml && conda install -y tqdm cython matplotlib scikit-learn"]
+
+# install azurmel-sdk components
+RUN ["/bin/bash", "-c", "source activate azureml && pip install azureml-sdk[notebooks]==1.0.43"]
+
+# clone Azure ML GitHub sample notebooks
+RUN cd /home && git clone -b "azureml-sdk-1.0.43" --single-branch https://github.com/Azure/MachineLearningNotebooks.git
+
+# generate jupyter configuration file
+RUN ["/bin/bash", "-c", "source activate azureml && mkdir ~/.jupyter && cd ~/.jupyter && jupyter notebook --generate-config"]
+
+# set an emtpy token for Jupyter to remove authentication. 
+# this is NOT recommended for production environment
+RUN echo "c.NotebookApp.token = ''" >> ~/.jupyter/jupyter_notebook_config.py
+
+# open up port 8887 on the container
+EXPOSE 8887
+
+# start Jupyter notebook server on port 8887 when the container starts
+CMD /bin/bash -c "cd /home/MachineLearningNotebooks && source activate azureml && jupyter notebook --port 8887 --no-browser --ip 0.0.0.0 --allow-root"

README.md

@@ -4,6 +4,10 @@ This repository contains example notebooks demonstrating the [Azure Machine Lear
 
 ![Azure ML workflow](https://raw.githubusercontent.com/MicrosoftDocs/azure-docs/master/articles/machine-learning/service/media/overview-what-is-azure-ml/aml.png)
 
+## News
+
+ * [Try Azure Machine Learning with MLflow](./how-to-use-azureml/using-mlflow)
+
 ## Quick installation
 ```sh
 pip install azureml-sdk

configuration.ipynb

@@ -39,6 +39,7 @@
         "    1. Workspace parameters\n",
         "    1. Access your workspace\n",
         "    1. Create a new workspace\n",
+        "    1. Create compute resources\n",
         "1. [Next steps](#Next%20steps)\n",
         "\n",
         "---\n",
@@ -241,6 +242,97 @@
         "ws.write_config()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Create compute resources for your training experiments\n",
+        "\n",
+        "Many of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n",
+        "\n",
+        "To create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors.  Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n",
+        "\n",
+        "The cluster parameters are:\n",
+        "* vm_size - this describes the virtual machine type and size used in the cluster.  All machines in the cluster are the same type.  You can get the list of vm sizes available in your region by using the CLI command\n",
+        "\n",
+        "```shell\n",
+        "az vm list-skus -o tsv\n",
+        "```\n",
+        "* min_nodes - this sets the minimum size of the cluster.  If you set the minimum to 0 the cluster will shut down all nodes while note in use.  Setting this number to a value higher than 0 will allow for faster start-up times, but you will also be billed when the cluster is not in use.\n",
+        "* max_nodes - this sets the maximum size of the cluster.  Setting this to a larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n",
+        "\n",
+        "\n",
+        "To create a **CPU** cluster now, run the cell below. The autoscale settings mean that the cluster will scale down to 0 nodes when inactive and up to 4 nodes when busy."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your CPU cluster\n",
+        "cpu_cluster_name = \"cpu-cluster\"\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
+        "    print(\"Found existing cpu-cluster\")\n",
+        "except ComputeTargetException:\n",
+        "    print(\"Creating new cpu-cluster\")\n",
+        "    \n",
+        "    # Specify the configuration for the new cluster\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n",
+        "                                                           min_nodes=0,\n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # Create the cluster with the specified name and configuration\n",
+        "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
+        "    \n",
+        "    # Wait for the cluster to complete, show the output log\n",
+        "    cpu_cluster.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "To create a **GPU** cluster, run the cell below. Note that your subscription must have sufficient quota for GPU VMs or the command will fail. To increase quota, see [these instructions](https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request). "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your GPU cluster\n",
+        "gpu_cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
+        "    print(\"Found existing gpu cluster\")\n",
+        "except ComputeTargetException:\n",
+        "    print(\"Creating new gpu-cluster\")\n",
+        "    \n",
+        "    # Specify the configuration for the new cluster\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
+        "                                                           min_nodes=0,\n",
+        "                                                           max_nodes=4)\n",
+        "    # Create the cluster with the specified name and configuration\n",
+        "    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n",
+        "\n",
+        "    # Wait for the cluster to complete, show the output log\n",
+        "    gpu_cluster.wait_for_completion(show_output=True)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/automated-machine-learning/README.md

@@ -115,16 +115,7 @@ jupyter notebook
     - Simple example of using automated ML for regression
     - Uses local compute for training
 
-- [auto-ml-remote-execution.ipynb](remote-execution/auto-ml-remote-execution.ipynb)
-    - Dataset: scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)
-    - Example of using automated ML for classification using a remote linux DSVM for training
-    - Parallel execution of iterations
-    - Async tracking of progress
-    - Cancelling individual iterations or entire run
-    - Retrieving models for any iteration or logged metric
-    - Specify automated ML settings as kwargs
-
-- [auto-ml-remote-amlcompute.ipynb](remote-batchai/auto-ml-remote-amlcompute.ipynb)
+- [auto-ml-remote-amlcompute.ipynb](remote-amlcompute/auto-ml-remote-amlcompute.ipynb)
     - Dataset: scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)
     - Example of using automated ML for classification using remote AmlCompute for training
     - Parallel execution of iterations
@@ -133,12 +124,6 @@ jupyter notebook
     - Retrieving models for any iteration or logged metric
     - Specify automated ML settings as kwargs
 
-- [auto-ml-remote-attach.ipynb](remote-attach/auto-ml-remote-attach.ipynb)
-    - Dataset: Scikit learn's [20newsgroup](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html)
-    - handling text data with preprocess flag
-    - Reading data from a blob store for remote executions
-    - using pandas dataframes for reading data
-
 - [auto-ml-missing-data-blacklist-early-termination.ipynb](missing-data-blacklist-early-termination/auto-ml-missing-data-blacklist-early-termination.ipynb)
     - Dataset: scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)
     - Blacklist certain pipelines
@@ -156,10 +141,6 @@ jupyter notebook
     - Get details for a automated ML Run. (automated ML settings, run widget & all metrics)
     - Download fitted pipeline for any iteration
 
-- [auto-ml-remote-execution-with-datastore.ipynb](remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.ipynb)
-    - Dataset: Scikit learn's [20newsgroup](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html)
-    - Download the data and store it in DataStore.
-
 - [auto-ml-classification-with-deployment.ipynb](classification-with-deployment/auto-ml-classification-with-deployment.ipynb)
     - Dataset: scikit learn's [digit dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)
     - Simple example of using automated ML for classification

how-to-use-azureml/automated-machine-learning/dataprep-remote-execution/auto-ml-dataprep-remote-execution.ipynb

@@ -205,7 +205,7 @@
         "from azureml.core.compute import ComputeTarget\n",
         "\n",
         "# Choose a name for your cluster.\n",
-        "amlcompute_cluster_name = \"cpucluster\"\n",
+        "amlcompute_cluster_name = \"cpu-cluster\"\n",
         "\n",
         "found = False\n",
         "\n",

how-to-use-azureml/automated-machine-learning/remote-amlcompute/auto-ml-remote-amlcompute.ipynb

@@ -119,7 +119,9 @@
       "metadata": {},
       "source": [
         "### Create or Attach existing AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create an AmlCompute as your training compute resource.\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -134,12 +136,10 @@
         "from azureml.core.compute import ComputeTarget\n",
         "\n",
         "# Choose a name for your cluster.\n",
-        "amlcompute_cluster_name = \"cpucluster\"\n",
+        "amlcompute_cluster_name = \"cpu-cluster\"\n",
         "\n",
         "found = False\n",
-        "\n",
         "# Check if this compute target already exists in the workspace.\n",
-        "\n",
         "cts = ws.compute_targets\n",
         "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
         "    found = True\n",

how-to-use-azureml/data-drift/azure-ml-datadrift.ipynb

@@ -0,0 +1,709 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Track Data Drift between Training and Inference Data in Production \n",
+    "\n",
+    "With this notebook, you will learn how to enable the DataDrift service to automatically track and determine whether your inference data is drifting from the data your model was initially trained on. The DataDrift service provides metrics and visualizations to help stakeholders identify which specific features cause the concept drift to occur.\n",
+    "\n",
+    "Please email driftfeedback@microsoft.com with any issues. A member from the DataDrift team will respond shortly. \n",
+    "\n",
+    "The DataDrift Public Preview API can be found [here](https://docs.microsoft.com/en-us/python/api/azureml-contrib-datadrift/?view=azure-ml-py). "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/contrib/datadrift/azureml-datadrift.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prerequisites and Setup"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Install the DataDrift package\n",
+    "\n",
+    "Install the azureml-contrib-datadrift, azureml-contrib-opendatasets and lightgbm packages before running this notebook.\n",
+    "```\n",
+    "pip install azureml-contrib-datadrift\n",
+    "pip install azureml-contrib-datasets\n",
+    "pip install lightgbm\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Import Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "import time\n",
+    "from datetime import datetime, timedelta\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import requests\n",
+    "from azureml.contrib.datadrift import DataDriftDetector, AlertConfiguration\n",
+    "from azureml.contrib.opendatasets import NoaaIsdWeather\n",
+    "from azureml.core import Dataset, Workspace, Run\n",
+    "from azureml.core.compute import AksCompute, ComputeTarget\n",
+    "from azureml.core.conda_dependencies import CondaDependencies\n",
+    "from azureml.core.experiment import Experiment\n",
+    "from azureml.core.image import ContainerImage\n",
+    "from azureml.core.model import Model\n",
+    "from azureml.core.webservice import Webservice, AksWebservice\n",
+    "from azureml.widgets import RunDetails\n",
+    "from sklearn.externals import joblib\n",
+    "from sklearn.model_selection import train_test_split\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set up Configuraton and Create Azure ML Workspace\n",
+    "\n",
+    "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration notebook](../../../configuration.ipynb) first if you haven't already to establish your connection to the AzureML Workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Please type in your initials/alias. The prefix is prepended to the names of resources created by this notebook. \n",
+    "prefix = \"dd\"\n",
+    "\n",
+    "# NOTE: Please do not change the model_name, as it's required by the score.py file\n",
+    "model_name = \"driftmodel\"\n",
+    "image_name = \"{}driftimage\".format(prefix)\n",
+    "service_name = \"{}driftservice\".format(prefix)\n",
+    "\n",
+    "# optionally, set email address to receive an email alert for DataDrift\n",
+    "email_address = \"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ws = Workspace.from_config()\n",
+    "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generate Train/Testing Data\n",
+    "\n",
+    "For this demo, we will use NOAA weather data from [Azure Open Datasets](https://azure.microsoft.com/services/open-datasets/). You may replace this step with your own dataset. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "usaf_list = ['725724', '722149', '723090', '722159', '723910', '720279',\n",
+    "             '725513', '725254', '726430', '720381', '723074', '726682',\n",
+    "             '725486', '727883', '723177', '722075', '723086', '724053',\n",
+    "             '725070', '722073', '726060', '725224', '725260', '724520',\n",
+    "             '720305', '724020', '726510', '725126', '722523', '703333',\n",
+    "             '722249', '722728', '725483', '722972', '724975', '742079',\n",
+    "             '727468', '722193', '725624', '722030', '726380', '720309',\n",
+    "             '722071', '720326', '725415', '724504', '725665', '725424',\n",
+    "             '725066']\n",
+    "\n",
+    "columns = ['usaf', 'wban', 'datetime', 'latitude', 'longitude', 'elevation', 'windAngle', 'windSpeed', 'temperature', 'stationName', 'p_k']\n",
+    "\n",
+    "\n",
+    "def enrich_weather_noaa_data(noaa_df):\n",
+    "    hours_in_day = 23\n",
+    "    week_in_year = 52\n",
+    "    \n",
+    "    noaa_df[\"hour\"] = noaa_df[\"datetime\"].dt.hour\n",
+    "    noaa_df[\"weekofyear\"] = noaa_df[\"datetime\"].dt.week\n",
+    "    \n",
+    "    noaa_df[\"sine_weekofyear\"] = noaa_df['datetime'].transform(lambda x: np.sin((2*np.pi*x.dt.week-1)/week_in_year))\n",
+    "    noaa_df[\"cosine_weekofyear\"] = noaa_df['datetime'].transform(lambda x: np.cos((2*np.pi*x.dt.week-1)/week_in_year))\n",
+    "\n",
+    "    noaa_df[\"sine_hourofday\"] = noaa_df['datetime'].transform(lambda x: np.sin(2*np.pi*x.dt.hour/hours_in_day))\n",
+    "    noaa_df[\"cosine_hourofday\"] = noaa_df['datetime'].transform(lambda x: np.cos(2*np.pi*x.dt.hour/hours_in_day))\n",
+    "    \n",
+    "    return noaa_df\n",
+    "\n",
+    "def add_window_col(input_df):\n",
+    "    shift_interval = pd.Timedelta('-7 days') # your X days interval\n",
+    "    df_shifted = input_df.copy()\n",
+    "    df_shifted['datetime'] = df_shifted['datetime'] - shift_interval\n",
+    "    df_shifted.drop(list(input_df.columns.difference(['datetime', 'usaf', 'wban', 'sine_hourofday', 'temperature'])), axis=1, inplace=True)\n",
+    "\n",
+    "    # merge, keeping only observations where -1 lag is present\n",
+    "    df2 = pd.merge(input_df,\n",
+    "                   df_shifted,\n",
+    "                   on=['datetime', 'usaf', 'wban', 'sine_hourofday'],\n",
+    "                   how='inner',  # use 'left' to keep observations without lags\n",
+    "                   suffixes=['', '-7'])\n",
+    "    return df2\n",
+    "\n",
+    "def get_noaa_data(start_time, end_time, cols, station_list):\n",
+    "    isd = NoaaIsdWeather(start_time, end_time, cols=cols)\n",
+    "    # Read into Pandas data frame.\n",
+    "    noaa_df = isd.to_pandas_dataframe()\n",
+    "    noaa_df = noaa_df.rename(columns={\"stationName\": \"station_name\"})\n",
+    "    \n",
+    "    df_filtered = noaa_df[noaa_df[\"usaf\"].isin(station_list)]\n",
+    "    df_filtered.reset_index(drop=True)\n",
+    "    \n",
+    "    # Enrich with time features\n",
+    "    df_enriched = enrich_weather_noaa_data(df_filtered)\n",
+    "    \n",
+    "    return df_enriched\n",
+    "\n",
+    "def get_featurized_noaa_df(start_time, end_time, cols, station_list):\n",
+    "    df_1 = get_noaa_data(start_time - timedelta(days=7), start_time - timedelta(seconds=1), cols, station_list)\n",
+    "    df_2 = get_noaa_data(start_time, end_time, cols, station_list)\n",
+    "    noaa_df = pd.concat([df_1, df_2])\n",
+    "    \n",
+    "    print(\"Adding window feature\")\n",
+    "    df_window = add_window_col(noaa_df)\n",
+    "    \n",
+    "    cat_columns = df_window.dtypes == object\n",
+    "    cat_columns = cat_columns[cat_columns == True]\n",
+    "    \n",
+    "    print(\"Encoding categorical columns\")\n",
+    "    df_encoded = pd.get_dummies(df_window, columns=cat_columns.keys().tolist())\n",
+    "    \n",
+    "    print(\"Dropping unnecessary columns\")\n",
+    "    df_featurized = df_encoded.drop(['windAngle', 'windSpeed', 'datetime', 'elevation'], axis=1).dropna().drop_duplicates()\n",
+    "    \n",
+    "    return df_featurized"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Train model on Jan 1 - 14, 2009 data\n",
+    "df = get_featurized_noaa_df(datetime(2009, 1, 1), datetime(2009, 1, 14, 23, 59, 59), columns, usaf_list)\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "label = \"temperature\"\n",
+    "x_df = df.drop(label, axis=1)\n",
+    "y_df = df[[label]]\n",
+    "x_train, x_test, y_train, y_test = train_test_split(df, y_df, test_size=0.2, random_state=223)\n",
+    "print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)\n",
+    "\n",
+    "training_dir = 'outputs/training'\n",
+    "training_file = \"training.csv\"\n",
+    "\n",
+    "# Generate training dataframe to register as Training Dataset\n",
+    "os.makedirs(training_dir, exist_ok=True)\n",
+    "training_df = pd.merge(x_train.drop(label, axis=1), y_train, left_index=True, right_index=True)\n",
+    "training_df.to_csv(training_dir + \"/\" + training_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create/Register Training Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_name = \"dataset\"\n",
+    "name_suffix = datetime.utcnow().strftime(\"%Y-%m-%d-%H-%M-%S\")\n",
+    "snapshot_name = \"snapshot-{}\".format(name_suffix)\n",
+    "\n",
+    "dstore = ws.get_default_datastore()\n",
+    "dstore.upload(training_dir, \"data/training\", show_progress=True)\n",
+    "dpath = dstore.path(\"data/training/training.csv\")\n",
+    "trainingDataset = Dataset.auto_read_files(dpath, include_path=True)\n",
+    "trainingDataset = trainingDataset.register(workspace=ws, name=dataset_name, description=\"dset\", exist_ok=True)\n",
+    "\n",
+    "trainingDataSnapshot = trainingDataset.create_snapshot(snapshot_name=snapshot_name, compute_target=None, create_data_snapshot=True)\n",
+    "datasets = [(Dataset.Scenario.TRAINING, trainingDataSnapshot)]\n",
+    "print(\"dataset registration done.\\n\")\n",
+    "datasets"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Train and Save Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lightgbm as lgb\n",
+    "\n",
+    "train = lgb.Dataset(data=x_train, \n",
+    "                    label=y_train)\n",
+    "\n",
+    "test = lgb.Dataset(data=x_test, \n",
+    "                   label=y_test,\n",
+    "                   reference=train)\n",
+    "\n",
+    "params = {'learning_rate'    : 0.1,\n",
+    "          'boosting'         : 'gbdt',\n",
+    "          'metric'           : 'rmse',\n",
+    "          'feature_fraction' : 1,\n",
+    "          'bagging_fraction' : 1,\n",
+    "          'max_depth': 6,\n",
+    "          'num_leaves'       : 31,\n",
+    "          'objective'        : 'regression',\n",
+    "          'bagging_freq'     : 1,\n",
+    "          \"verbose\": -1,\n",
+    "          'min_data_per_leaf': 100}\n",
+    "\n",
+    "model = lgb.train(params, \n",
+    "                  num_boost_round=500,\n",
+    "                  train_set=train,\n",
+    "                  valid_sets=[train, test],\n",
+    "                  verbose_eval=50,\n",
+    "                  early_stopping_rounds=25)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_file = 'outputs/{}.pkl'.format(model_name)\n",
+    "\n",
+    "os.makedirs('outputs', exist_ok=True)\n",
+    "joblib.dump(model, model_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Register Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = Model.register(model_path=model_file,\n",
+    "                       model_name=model_name,\n",
+    "                       workspace=ws,\n",
+    "                       datasets=datasets)\n",
+    "\n",
+    "print(model_name, image_name, service_name, model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Deploy Model To AKS"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prepare Environment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn', 'joblib', 'lightgbm', 'pandas'],\n",
+    "                                 pip_packages=['azureml-monitoring', 'azureml-sdk[automl]'])\n",
+    "\n",
+    "with open(\"myenv.yml\",\"w\") as f:\n",
+    "    f.write(myenv.serialize_to_string())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Image creation may take up to 15 minutes.\n",
+    "\n",
+    "image_name = image_name + str(model.version)\n",
+    "\n",
+    "if not image_name in ws.images:\n",
+    "    # Use the score.py defined in this directory as the execution script\n",
+    "    # NOTE: The Model Data Collector must be enabled in the execution script for DataDrift to run correctly\n",
+    "    image_config = ContainerImage.image_configuration(execution_script=\"score.py\",\n",
+    "                                                      runtime=\"python\",\n",
+    "                                                      conda_file=\"myenv.yml\",\n",
+    "                                                      description=\"Image with weather dataset model\")\n",
+    "    image = ContainerImage.create(name=image_name,\n",
+    "                                  models=[model],\n",
+    "                                  image_config=image_config,\n",
+    "                                  workspace=ws)\n",
+    "\n",
+    "    image.wait_for_creation(show_output=True)\n",
+    "else:\n",
+    "    image = ws.images[image_name]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Compute Target"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "aks_name = 'dd-demo-e2e'\n",
+    "prov_config = AksCompute.provisioning_configuration()\n",
+    "\n",
+    "if not aks_name in ws.compute_targets:\n",
+    "    aks_target = ComputeTarget.create(workspace=ws,\n",
+    "                                      name=aks_name,\n",
+    "                                      provisioning_configuration=prov_config)\n",
+    "\n",
+    "    aks_target.wait_for_completion(show_output=True)\n",
+    "    print(aks_target.provisioning_state)\n",
+    "    print(aks_target.provisioning_errors)\n",
+    "else:\n",
+    "    aks_target=ws.compute_targets[aks_name]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Deploy Service"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "aks_service_name = service_name\n",
+    "\n",
+    "if not aks_service_name in ws.webservices:\n",
+    "    aks_config = AksWebservice.deploy_configuration(collect_model_data=True, enable_app_insights=True)\n",
+    "    aks_service = Webservice.deploy_from_image(workspace=ws,\n",
+    "                                               name=aks_service_name,\n",
+    "                                               image=image,\n",
+    "                                               deployment_config=aks_config,\n",
+    "                                               deployment_target=aks_target)\n",
+    "    aks_service.wait_for_deployment(show_output=True)\n",
+    "    print(aks_service.state)\n",
+    "else:\n",
+    "    aks_service = ws.webservices[aks_service_name]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Run DataDrift Analysis"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Send Scoring Data to Service"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Download Scoring Data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Score Model on March 15, 2016 data\n",
+    "scoring_df = get_noaa_data(datetime(2016, 3, 15) - timedelta(days=7), datetime(2016, 3, 16),  columns, usaf_list)\n",
+    "# Add the window feature column\n",
+    "scoring_df = add_window_col(scoring_df)\n",
+    "\n",
+    "# Drop features not used by the model\n",
+    "print(\"Dropping unnecessary columns\")\n",
+    "scoring_df = scoring_df.drop(['windAngle', 'windSpeed', 'datetime', 'elevation'], axis=1).dropna()\n",
+    "scoring_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# One Hot Encode the scoring dataset to match the training dataset schema\n",
+    "columns_dict = model.datasets[\"training\"][0].get_profile().columns\n",
+    "extra_cols = ('Path', 'Column1')\n",
+    "for k in extra_cols:\n",
+    "    columns_dict.pop(k, None)\n",
+    "training_columns = list(columns_dict.keys())\n",
+    "\n",
+    "categorical_columns = scoring_df.dtypes == object\n",
+    "categorical_columns = categorical_columns[categorical_columns == True]\n",
+    "\n",
+    "test_df = pd.get_dummies(scoring_df[categorical_columns.keys().tolist()])\n",
+    "encoded_df = scoring_df.join(test_df)\n",
+    "\n",
+    "# Populate missing OHE columns with 0 values to match traning dataset schema\n",
+    "difference = list(set(training_columns) - set(encoded_df.columns.tolist()))\n",
+    "for col in difference:\n",
+    "    encoded_df[col] = 0\n",
+    "encoded_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Serialize dataframe to list of row dictionaries\n",
+    "encoded_dict = encoded_df.to_dict('records')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Submit Scoring Data to Service"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%time\n",
+    "\n",
+    "# retreive the API keys. AML generates two keys.\n",
+    "key1, key2 = aks_service.get_keys()\n",
+    "\n",
+    "total_count = len(scoring_df)\n",
+    "i = 0\n",
+    "load = []\n",
+    "for row in encoded_dict:\n",
+    "    load.append(row)\n",
+    "    i = i + 1\n",
+    "    if i % 100 == 0:\n",
+    "        payload = json.dumps({\"data\": load})\n",
+    "        \n",
+    "        # construct raw HTTP request and send to the service\n",
+    "        payload_binary = bytes(payload,encoding = 'utf8')\n",
+    "        headers = {'Content-Type':'application/json', 'Authorization': 'Bearer ' + key1}\n",
+    "        resp = requests.post(aks_service.scoring_uri, payload_binary, headers=headers)\n",
+    "        \n",
+    "        print(\"prediction:\", resp.content, \"Progress: {}/{}\".format(i, total_count))   \n",
+    "\n",
+    "        load = []\n",
+    "        time.sleep(3)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configure DataDrift"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "services = [service_name]\n",
+    "start = datetime.now() - timedelta(days=2)\n",
+    "end = datetime(year=2020, month=1, day=22, hour=15, minute=16)\n",
+    "feature_list = ['usaf', 'wban', 'latitude', 'longitude', 'station_name', 'p_k',  'sine_hourofday', 'cosine_hourofday', 'temperature-7']\n",
+    "alert_config = AlertConfiguration([email_address]) if email_address else None\n",
+    "\n",
+    "# there will be an exception indicating using get() method if DataDrift object already exist\n",
+    "try:\n",
+    "    datadrift = DataDriftDetector.create(ws, model.name, model.version, services, frequency=\"Day\", alert_config=alert_config)\n",
+    "except KeyError:\n",
+    "    datadrift = DataDriftDetector.get(ws, model.name, model.version)\n",
+    "    \n",
+    "print(\"Details of DataDrift Object:\\n{}\".format(datadrift))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run an Adhoc DataDriftDetector Run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "target_date = datetime.today()\n",
+    "run = datadrift.run(target_date, services, feature_list=feature_list, create_compute_target=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "exp = Experiment(ws, datadrift._id)\n",
+    "dd_run = Run(experiment=exp, run_id=run)\n",
+    "RunDetails(dd_run).show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Get Drift Analysis Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "children = list(dd_run.get_children())\n",
+    "for child in children:\n",
+    "    child.wait_for_completion()\n",
+    "\n",
+    "drift_metrics = datadrift.get_output(start_time=start, end_time=end)\n",
+    "drift_metrics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Show all drift figures, one per serivice.\n",
+    "# If setting with_details is False (by default), only drift will be shown; if it's True, all details will be shown.\n",
+    "\n",
+    "drift_figures = datadrift.show(with_details=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Enable DataDrift Schedule"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "datadrift.enable_schedule()"
+   ]
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": "rafarmah"
+   }
+  ],
+  "kernelspec": {
+   "display_name": "Python 3.6",
+   "language": "python",
+   "name": "python36"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

how-to-use-azureml/data-drift/readme.md

@@ -1 +1,3 @@
-Under contruction...please visit again soon!
+## Using data drift APIs
+
+1. [Detect data drift for a model](azure-ml-datadrift.ipynb): Detect data drift for a deployed model.

how-to-use-azureml/data-drift/score.py

@@ -0,0 +1,58 @@
+import pickle
+import json
+import numpy
+import azureml.train.automl
+from sklearn.externals import joblib
+from sklearn.linear_model import Ridge
+from azureml.core.model import Model
+from azureml.core.run import Run
+from azureml.monitoring import ModelDataCollector
+import time
+import pandas as pd
+
+
+def init():
+    global model, inputs_dc, prediction_dc, feature_names, categorical_features
+
+    print("Model is initialized" + time.strftime("%H:%M:%S"))
+    model_path = Model.get_model_path(model_name="driftmodel")
+    model = joblib.load(model_path)
+
+    feature_names = ["usaf", "wban", "latitude", "longitude", "station_name", "p_k",
+                     "sine_weekofyear", "cosine_weekofyear", "sine_hourofday", "cosine_hourofday",
+                     "temperature-7"]
+
+    categorical_features = ["usaf", "wban", "p_k", "station_name"]
+
+    inputs_dc = ModelDataCollector(model_name="driftmodel",
+                                   identifier="inputs",
+                                   feature_names=feature_names)
+
+    prediction_dc = ModelDataCollector("driftmodel",
+                                       identifier="predictions",
+                                       feature_names=["temperature"])
+
+
+def run(raw_data):
+    global inputs_dc, prediction_dc
+
+    try:
+        data = json.loads(raw_data)["data"]
+        data = pd.DataFrame(data)
+
+        # Remove the categorical features as the model expects OHE values
+        input_data = data.drop(categorical_features, axis=1)
+
+        result = model.predict(input_data)
+
+        # Collect the non-OHE dataframe
+        collected_df = data[feature_names]
+
+        inputs_dc.collect(collected_df.values)
+        prediction_dc.collect(result)
+        return result.tolist()
+    except Exception as e:
+        error = str(e)
+
+        print(error + time.strftime("%H:%M:%S"))
+        return error

how-to-use-azureml/deployment/accelerated-models/accelerated-models-object-detection.ipynb

@@ -344,7 +344,9 @@
     "### 5.a. Create Client\n",
     "The image supports gRPC and the TensorFlow Serving \"predict\" API. We have a client that can call into the docker image to get predictions. \n",
     "\n",
-    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice.deploy_configuration(), see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key)."
+    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice.deploy_configuration(), see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key).",
+    "\n",
+    "**WARNING:** If you are running on Azure Notebooks free compute, you will not be able to make outgoing calls to your service. Try locating your client on a different machine to consume it."
    ]
   },
   {

how-to-use-azureml/deployment/accelerated-models/accelerated-models-quickstart.ipynb

@@ -417,7 +417,9 @@
     "### 7.a. Create Client\n",
     "The image supports gRPC and the TensorFlow Serving \"predict\" API. We have a client that can call into the docker image to get predictions.\n",
     "\n",
-    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice, see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key)."
+    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice, see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key).",
+    "\n",
+    "**WARNING:** If you are running on Azure Notebooks free compute, you will not be able to make outgoing calls to your service. Try locating your client on a different machine to consume it."
    ]
   },
   {

how-to-use-azureml/deployment/accelerated-models/accelerated-models-training.ipynb

@@ -702,7 +702,9 @@
     "### 9.a. Create Client\n",
     "The image supports gRPC and the TensorFlow Serving \"predict\" API. We have a client that can call into the docker image to get predictions. \n",
     "\n",
-    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice.deploy_configuration(), see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key)."
+    "**Note:** If you chose to use auth_enabled=True when creating your AksWebservice.deploy_configuration(), see documentation [here](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.webservice(class)?view=azure-ml-py#get-keys--) on how to retrieve your keys and use either key as an argument to PredictionClient(...,access_token=key).",
+    "\n",
+    "**WARNING:** If you are running on Azure Notebooks free compute, you will not be able to make outgoing calls to your service. Try locating your client on a different machine to consume it."
    ]
   },
   {

how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb

@@ -98,7 +98,7 @@
         "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
         "# choose a name for your cluster\n",
-        "cluster_name = \"gpucluster\"\n",
+        "cluster_name = \"gpu-cluster\"\n",
         "\n",
         "try:\n",
         "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-getting-started.ipynb

@@ -206,8 +206,15 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "#### Retrieve default Azure Machine Learning compute\n",
-        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's get the default Azure Machine Learning Compute in the current workspace. We will then run the training script on this compute target."
+        "#### Retrieve or create a Azure Machine Learning compute\n",
+        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's create a new Azure Machine Learning Compute in the current workspace, if it doesn't already exist. We will then run the training script on this compute target.\n",
+        "\n",
+        "If we could not find the compute with the given name in the previous cell, then we will create a new compute here. We will create an Azure Machine Learning Compute containing **STANDARD_D2_V2 CPU VMs**. This process is broken down into the following steps:\n",
+        "\n",
+        "1. Create the configuration\n",
+        "2. Create the Azure Machine Learning compute\n",
+        "\n",
+        "**This process will take about 3 minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell.**"
       ]
     },
     {
@@ -216,7 +223,23 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "aml_compute = ws.get_default_compute_target(\"CPU\")"
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "aml_compute_target = \"cpu-cluster\"\n",
+        "try:\n",
+        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
+        "    print(\"found existing compute target.\")\n",
+        "except ComputeTargetException:\n",
+        "    print(\"creating new compute target\")\n",
+        "    \n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
+        "                                                                min_nodes = 1, \n",
+        "                                                                max_nodes = 4)    \n",
+        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
+        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "    \n",
+        "print(\"Azure Machine Learning Compute attached\")\n"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.ipynb

@@ -113,7 +113,25 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "batch_compute = ws.get_default_compute_target(\"CPU\")"
+        "batch_compute_name = 'mybatchcompute' # Name to associate with new compute in workspace\n",
+        "\n",
+        "# Batch account details needed to attach as compute to workspace\n",
+        "batch_account_name = \"<batch_account_name>\" # Name of the Batch account\n",
+        "batch_resource_group = \"<batch_resource_group>\" # Name of the resource group which contains this account\n",
+        "\n",
+        "try:\n",
+        "    # check if already attached\n",
+        "    batch_compute = BatchCompute(ws, batch_compute_name)\n",
+        "except ComputeTargetException:\n",
+        "    print('Attaching Batch compute...')\n",
+        "    provisioning_config = BatchCompute.attach_configuration(resource_group=batch_resource_group, \n",
+        "                                                            account_name=batch_account_name)\n",
+        "    batch_compute = ComputeTarget.attach(ws, batch_compute_name, provisioning_config)\n",
+        "    batch_compute.wait_for_completion()\n",
+        "    print(\"Provisioning state:{}\".format(batch_compute.provisioning_state))\n",
+        "    print(\"Provisioning errors:{}\".format(batch_compute.provisioning_errors))\n",
+        "\n",
+        "print(\"Using Batch compute:{}\".format(batch_compute.cluster_resource_id))"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-estimatorstep.ipynb

@@ -76,8 +76,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
+        "1. create the configuration (this step is local and only takes a second)\n",
+        "2. create the cluster (this step will take about **20 seconds**)\n",
+        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
       ]
     },
     {
@@ -86,7 +96,25 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "cpu_cluster = ws.get_default_compute_target(\"CPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"cpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    cpu_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    cpu_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it uses the scale settings for the cluster\n",
+        "    cpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(cpu_cluster.get_status().serialize())"
@@ -96,7 +124,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'cpucluster' of type `AmlCompute`."
+        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'cpu-cluster' of type `AmlCompute`."
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.ipynb

@@ -184,37 +184,36 @@
       "metadata": {},
       "source": [
         "## Retrieve or create a Azure Machine Learning compute\n",
-        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads.\n",
-        "Let's check the available computes first."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "cts = ws.compute_targets\n",
-        "for name, ct in cts.items():\n",
-        "    print(name, ct.type, ct.provisioning_state)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Now let's get the default Azure Machine Learning Compute in the current workspace. We will then run the training script on this compute target."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "compute_target = ws.get_default_compute_target(\"GPU\")\n",
+        "Azure Machine Learning Compute is a service for provisioning and managing clusters of Azure virtual machines for running machine learning workloads. Let's create a new Azure Machine Learning Compute in the current workspace, if it doesn't already exist. We will then run the training script on this compute target.\n",
         "\n",
-        "print(compute_target.get_status().serialize())"
+        "If we could not find the compute with the given name in the previous cell, then we will create a new compute here. This process is broken down into the following steps:\n",
+        "\n",
+        "1. Create the configuration\n",
+        "2. Create the Azure Machine Learning compute\n",
+        "\n",
+        "**This process will take a few minutes and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell.**\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target {}.'.format(cluster_name))\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
+        "                                                               max_nodes=4)\n",
+        "\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "    compute_target.wait_for_completion(show_output=True, timeout_in_minutes=20)\n",
+        "\n",
+        "print(\"Azure Machine Learning Compute attached\")"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-publish-and-run-using-rest-endpoint.ipynb

@@ -79,7 +79,20 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "aml_compute = ws.get_default_compute_target(\"CPU\")"
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "aml_compute_target = \"cpu-cluster\"\n",
+        "try:\n",
+        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
+        "    print(\"found existing compute target.\")\n",
+        "except ComputeTargetException:\n",
+        "    print(\"creating new compute target\")\n",
+        "    \n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
+        "                                                                min_nodes = 1, \n",
+        "                                                                max_nodes = 4)    \n",
+        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
+        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb

@@ -54,7 +54,7 @@
       "metadata": {},
       "source": [
         "### Compute Targets\n",
-        "#### Retrieve the default Azure Machine Learning Compute"
+        "#### Retrieve an already attached Azure Machine Learning Compute"
       ]
     },
     {
@@ -63,7 +63,31 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "aml_compute_target = ws.get_default_compute_target(\"CPU\")"
+        "from azureml.core import Run, Experiment, Datastore\n",
+        "\n",
+        "from azureml.widgets import RunDetails\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import AmlCompute, ComputeTarget\n",
+        "aml_compute_target = \"cpu-cluster\"\n",
+        "try:\n",
+        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
+        "    print(\"Found existing compute target: {}\".format(aml_compute_target))\n",
+        "except:\n",
+        "    print(\"Creating new compute target: {}\".format(aml_compute_target))\n",
+        "    \n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
+        "                                                                min_nodes = 1, \n",
+        "                                                                max_nodes = 4)    \n",
+        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
+        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-versioned-pipeline-endpoints.ipynb

@@ -85,10 +85,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "from azureml.core import Run, Experiment, Datastore\n",
+        "from azureml.core.compute import AmlCompute, ComputeTarget\n",
         "from azureml.pipeline.steps import PythonScriptStep\n",
         "from azureml.pipeline.core import Pipeline\n",
         "\n",
-        "aml_compute = ws.get_default_compute_target(\"CPU\")\n",
+        "#Retrieve an already attached Azure Machine Learning Compute\n",
+        "aml_compute_target = \"cpu-cluster\"\n",
+        "try:\n",
+        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
+        "    print(\"Found existing compute target: {}\".format(aml_compute_target))\n",
+        "except:\n",
+        "    print(\"Creating new compute target: {}\".format(aml_compute_target))\n",
+        "    \n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
+        "                                                                min_nodes = 1, \n",
+        "                                                                max_nodes = 4)    \n",
+        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
+        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "\n",
         "# source_directory\n",
         "source_directory = '.'\n",

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.ipynb

@@ -139,7 +139,31 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(\"CPU\")"
+        "# Choose a name for your cluster.\n",
+        "amlcompute_cluster_name = \"cpu-cluster\"\n",
+        "\n",
+        "found = False\n",
+        "# Check if this compute target already exists in the workspace.\n",
+        "cts = ws.compute_targets\n",
+        "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
+        "    found = True\n",
+        "    print('Found existing compute target.')\n",
+        "    compute_target = cts[amlcompute_cluster_name]\n",
+        "    \n",
+        "if not found:\n",
+        "    print('Creating a new compute target...')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\", # for GPU, use \"STANDARD_NC6\"\n",
+        "                                                                #vm_priority = 'lowpriority', # optional\n",
+        "                                                                max_nodes = 4)\n",
+        "\n",
+        "    # Create the cluster.\n",
+        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
+        "    \n",
+        "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
+        "    # If no min_node_count is provided, it will use the scale settings for the cluster.\n",
+        "    compute_target.wait_for_completion(show_output = True, min_node_count = 1, timeout_in_minutes = 10)\n",
+        "    \n",
+        "     # For a more detailed view of current AmlCompute status, use get_status()."
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.ipynb

@@ -137,7 +137,22 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "aml_compute = ws.get_default_compute_target(\"CPU\")\n"
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "aml_compute_target = \"cpu-cluster\"\n",
+        "try:\n",
+        "    aml_compute = AmlCompute(ws, aml_compute_target)\n",
+        "    print(\"found existing compute target.\")\n",
+        "except ComputeTargetException:\n",
+        "    print(\"creating new compute target\")\n",
+        "    \n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
+        "                                                                min_nodes = 1, \n",
+        "                                                                max_nodes = 4)    \n",
+        "    aml_compute = ComputeTarget.create(ws, aml_compute_target, provisioning_config)\n",
+        "    aml_compute.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "    \n",
+        "print(\"Aml Compute attached\")\n"
       ]
     },
     {
@@ -418,13 +433,31 @@
         "RunDetails(pipeline_run1).show()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Wait for pipeline run to complete"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run1.wait_for_completion(show_output=True)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
         "### See Outputs\n",
         "\n",
-        "See where outputs of each pipeline step are located on your datastore."
+        "See where outputs of each pipeline step are located on your datastore.\n",
+        "\n",
+        "***Wait for pipeline run to complete, to make sure all the outputs are ready***"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/prepdata/cleanse.py

@@ -0,0 +1,58 @@
+# Copyright (c) Microsoft. All rights reserved.
+# Licensed under the MIT license.
+
+import argparse
+import os
+import pandas as pd
+import azureml.dataprep as dprep
+
+
+def get_dict(dict_str):
+    pairs = dict_str.strip("{}").split("\;")
+    new_dict = {}
+    for pair in pairs:
+        key, value = pair.strip('\\').split(":")
+        new_dict[key.strip().strip("'")] = value.strip().strip("'")
+
+    return new_dict
+
+
+print("Cleans the input data")
+
+parser = argparse.ArgumentParser("cleanse")
+parser.add_argument("--input_cleanse", type=str, help="raw taxi data")
+parser.add_argument("--output_cleanse", type=str, help="cleaned taxi data directory")
+parser.add_argument("--useful_columns", type=str, help="useful columns to keep")
+parser.add_argument("--columns", type=str, help="rename column pattern")
+
+args = parser.parse_args()
+
+print("Argument 1(input taxi data path): %s" % args.input_cleanse)
+print("Argument 2(columns to keep): %s" % str(args.useful_columns.strip("[]").split("\;")))
+print("Argument 3(columns renaming mapping): %s" % str(args.columns.strip("{}").split("\;")))
+print("Argument 4(output cleansed taxi data path): %s" % args.output_cleanse)
+
+raw_df = dprep.read_csv(path=args.input_cleanse, header=dprep.PromoteHeadersMode.GROUPED)
+
+# These functions ensure that null data is removed from the data set,
+# which will help increase machine learning model accuracy.
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-data-prep
+# for more details
+
+useful_columns = [s.strip().strip("'") for s in args.useful_columns.strip("[]").split("\;")]
+columns = get_dict(args.columns)
+
+all_columns = dprep.ColumnSelector(term=".*", use_regex=True)
+drop_if_all_null = [all_columns, dprep.ColumnRelationship(dprep.ColumnRelationship.ALL)]
+
+new_df = (raw_df
+          .replace_na(columns=all_columns)
+          .drop_nulls(*drop_if_all_null)
+          .rename_columns(column_pairs=columns)
+          .keep_columns(columns=useful_columns))
+
+if not (args.output_cleanse is None):
+    os.makedirs(args.output_cleanse, exist_ok=True)
+    print("%s created" % args.output_cleanse)
+    write_df = new_df.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_cleanse))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/prepdata/filter.py

@@ -0,0 +1,55 @@
+import argparse
+import os
+import azureml.dataprep as dprep
+
+print("Filters out coordinates for locations that are outside the city border.",
+      "Chain the column filter commands within the filter() function",
+      "and define the minimum and maximum bounds for each field.")
+
+parser = argparse.ArgumentParser("filter")
+parser.add_argument("--input_filter", type=str, help="merged taxi data directory")
+parser.add_argument("--output_filter", type=str, help="filter out out of city locations")
+
+args = parser.parse_args()
+
+print("Argument 1(input taxi data path): %s" % args.input_filter)
+print("Argument 2(output filtered taxi data path): %s" % args.output_filter)
+
+combined_df = dprep.read_csv(args.input_filter + '/part-*')
+
+# These functions filter out coordinates for locations that are outside the city border.
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-data-prep for more details
+
+# Create a condensed view of the dataflow to just show the lat/long fields,
+# which makes it easier to evaluate missing or out-of-scope coordinates
+decimal_type = dprep.TypeConverter(data_type=dprep.FieldType.DECIMAL)
+combined_df = combined_df.set_column_types(type_conversions={
+    "pickup_longitude": decimal_type,
+    "pickup_latitude": decimal_type,
+    "dropoff_longitude": decimal_type,
+    "dropoff_latitude": decimal_type
+})
+
+# Filter out coordinates for locations that are outside the city border.
+# Chain the column filter commands within the filter() function
+# and define the minimum and maximum bounds for each field
+latlong_filtered_df = (combined_df
+                       .drop_nulls(columns=["pickup_longitude",
+                                            "pickup_latitude",
+                                            "dropoff_longitude",
+                                            "dropoff_latitude"],
+                                   column_relationship=dprep.ColumnRelationship(dprep.ColumnRelationship.ANY))
+                       .filter(dprep.f_and(dprep.col("pickup_longitude") <= -73.72,
+                                           dprep.col("pickup_longitude") >= -74.09,
+                                           dprep.col("pickup_latitude") <= 40.88,
+                                           dprep.col("pickup_latitude") >= 40.53,
+                                           dprep.col("dropoff_longitude") <= -73.72,
+                                           dprep.col("dropoff_longitude") >= -74.09,
+                                           dprep.col("dropoff_latitude") <= 40.88,
+                                           dprep.col("dropoff_latitude") >= 40.53)))
+
+if not (args.output_filter is None):
+    os.makedirs(args.output_filter, exist_ok=True)
+    print("%s created" % args.output_filter)
+    write_df = latlong_filtered_df.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_filter))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/prepdata/merge.py

@@ -0,0 +1,29 @@
+
+import argparse
+import os
+import azureml.dataprep as dprep
+
+print("Merge Green and Yellow taxi data")
+
+parser = argparse.ArgumentParser("merge")
+parser.add_argument("--input_green_merge", type=str, help="cleaned green taxi data directory")
+parser.add_argument("--input_yellow_merge", type=str, help="cleaned yellow taxi data directory")
+parser.add_argument("--output_merge", type=str, help="green and yellow taxi data merged")
+
+args = parser.parse_args()
+
+print("Argument 1(input green taxi data path): %s" % args.input_green_merge)
+print("Argument 2(input yellow taxi data path): %s" % args.input_yellow_merge)
+print("Argument 3(output merge taxi data path): %s" % args.output_merge)
+
+green_df = dprep.read_csv(args.input_green_merge + '/part-*')
+yellow_df = dprep.read_csv(args.input_yellow_merge + '/part-*')
+
+# Appending yellow data to green data
+combined_df = green_df.append_rows([yellow_df])
+
+if not (args.output_merge is None):
+    os.makedirs(args.output_merge, exist_ok=True)
+    print("%s created" % args.output_merge)
+    write_df = combined_df.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_merge))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/prepdata/normalize.py

@@ -0,0 +1,47 @@
+import argparse
+import os
+import azureml.dataprep as dprep
+
+print("Replace undefined values to relavant values and rename columns to meaningful names")
+
+parser = argparse.ArgumentParser("normalize")
+parser.add_argument("--input_normalize", type=str, help="combined and converted taxi data")
+parser.add_argument("--output_normalize", type=str, help="replaced undefined values and renamed columns")
+
+args = parser.parse_args()
+
+print("Argument 1(input taxi data path): %s" % args.input_normalize)
+print("Argument 2(output normalized taxi data path): %s" % args.output_normalize)
+
+combined_converted_df = dprep.read_csv(args.input_normalize + '/part-*')
+
+# These functions replace undefined values and rename to use meaningful names.
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-data-prep for more details
+
+replaced_stfor_vals_df = combined_converted_df.replace(columns="store_forward",
+                                                       find="0",
+                                                       replace_with="N").fill_nulls("store_forward", "N")
+
+replaced_distance_vals_df = replaced_stfor_vals_df.replace(columns="distance",
+                                                           find=".00",
+                                                           replace_with=0).fill_nulls("distance", 0)
+
+replaced_distance_vals_df = replaced_distance_vals_df.to_number(["distance"])
+
+time_split_df = (replaced_distance_vals_df
+                 .split_column_by_example(source_column="pickup_datetime")
+                 .split_column_by_example(source_column="dropoff_datetime"))
+
+# Split the pickup and dropoff datetime values into the respective date and time columns
+renamed_col_df = (time_split_df
+                  .rename_columns(column_pairs={
+                      "pickup_datetime_1": "pickup_date",
+                      "pickup_datetime_2": "pickup_time",
+                      "dropoff_datetime_1": "dropoff_date",
+                      "dropoff_datetime_2": "dropoff_time"}))
+
+if not (args.output_normalize is None):
+    os.makedirs(args.output_normalize, exist_ok=True)
+    print("%s created" % args.output_normalize)
+    write_df = renamed_col_df.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_normalize))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/prepdata/transform.py

@@ -0,0 +1,88 @@
+import argparse
+import os
+import azureml.dataprep as dprep
+
+print("Transforms the renamed taxi data to the required format")
+
+parser = argparse.ArgumentParser("transform")
+parser.add_argument("--input_transform", type=str, help="renamed taxi data")
+parser.add_argument("--output_transform", type=str, help="transformed taxi data")
+
+args = parser.parse_args()
+
+print("Argument 1(input taxi data path): %s" % args.input_transform)
+print("Argument 2(output final transformed taxi data): %s" % args.output_transform)
+
+renamed_df = dprep.read_csv(args.input_transform + '/part-*')
+
+# These functions transform the renamed data to be used finally for training.
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-data-prep for more details
+
+# Split the pickup and dropoff date further into the day of the week, day of the month, and month values.
+# To get the day of the week value, use the derive_column_by_example() function.
+# The function takes an array parameter of example objects that define the input data,
+# and the preferred output. The function automatically determines your preferred transformation.
+# For the pickup and dropoff time columns, split the time into the hour, minute, and second by using
+# the split_column_by_example() function with no example parameter. After you generate the new features,
+# use the drop_columns() function to delete the original fields as the newly generated features are preferred.
+# Rename the rest of the fields to use meaningful descriptions.
+
+transformed_features_df = (renamed_df
+                           .derive_column_by_example(
+                               source_columns="pickup_date",
+                               new_column_name="pickup_weekday",
+                               example_data=[("2009-01-04", "Sunday"), ("2013-08-22", "Thursday")])
+                           .derive_column_by_example(
+                               source_columns="dropoff_date",
+                               new_column_name="dropoff_weekday",
+                               example_data=[("2013-08-22", "Thursday"), ("2013-11-03", "Sunday")])
+
+                           .split_column_by_example(source_column="pickup_time")
+                           .split_column_by_example(source_column="dropoff_time")
+
+                           .split_column_by_example(source_column="pickup_time_1")
+                           .split_column_by_example(source_column="dropoff_time_1")
+                           .drop_columns(columns=[
+                               "pickup_date", "pickup_time", "dropoff_date", "dropoff_time",
+                               "pickup_date_1", "dropoff_date_1", "pickup_time_1", "dropoff_time_1"])
+
+                           .rename_columns(column_pairs={
+                               "pickup_date_2": "pickup_month",
+                               "pickup_date_3": "pickup_monthday",
+                               "pickup_time_1_1": "pickup_hour",
+                               "pickup_time_1_2": "pickup_minute",
+                               "pickup_time_2": "pickup_second",
+                               "dropoff_date_2": "dropoff_month",
+                               "dropoff_date_3": "dropoff_monthday",
+                               "dropoff_time_1_1": "dropoff_hour",
+                               "dropoff_time_1_2": "dropoff_minute",
+                               "dropoff_time_2": "dropoff_second"}))
+
+# Drop the pickup_datetime and dropoff_datetime columns because they're
+# no longer needed (granular time features like hour,
+# minute and second are more useful for model training).
+processed_df = transformed_features_df.drop_columns(columns=["pickup_datetime", "dropoff_datetime"])
+
+# Use the type inference functionality to automatically check the data type of each field,
+# and display the inference results.
+type_infer = processed_df.builders.set_column_types()
+type_infer.learn()
+
+# The inference results look correct based on the data. Now apply the type conversions to the dataflow.
+type_converted_df = type_infer.to_dataflow()
+
+# Before you package the dataflow, run two final filters on the data set.
+# To eliminate incorrectly captured data points,
+# filter the dataflow on records where both the cost and distance variable values are greater than zero.
+# This step will significantly improve machine learning model accuracy,
+# because data points with a zero cost or distance represent major outliers that throw off prediction accuracy.
+
+final_df = type_converted_df.filter(dprep.col("distance") > 0)
+final_df = final_df.filter(dprep.col("cost") > 0)
+
+# Writing the final dataframe to use for training in the following steps
+if not (args.output_transform is None):
+    os.makedirs(args.output_transform, exist_ok=True)
+    print("%s created" % args.output_transform)
+    write_df = final_df.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_transform))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/trainmodel/featurization.py

@@ -0,0 +1,31 @@
+import argparse
+import os
+import azureml.dataprep as dprep
+import azureml.core
+
+print("Extracts important features from prepared data")
+
+parser = argparse.ArgumentParser("featurization")
+parser.add_argument("--input_featurization", type=str, help="input featurization")
+parser.add_argument("--useful_columns", type=str, help="columns to use")
+parser.add_argument("--output_featurization", type=str, help="output featurization")
+
+args = parser.parse_args()
+
+print("Argument 1(input training data path): %s" % args.input_featurization)
+print("Argument 2(column features to use): %s" % str(args.useful_columns.strip("[]").split("\;")))
+print("Argument 3:(output featurized training data path) %s" % args.output_featurization)
+
+dflow_prepared = dprep.read_csv(args.input_featurization + '/part-*')
+
+# These functions extracts useful features for training
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-auto-train-models for more detail
+
+useful_columns = [s.strip().strip("'") for s in args.useful_columns.strip("[]").split("\;")]
+dflow = dflow_prepared.keep_columns(useful_columns)
+
+if not (args.output_featurization is None):
+    os.makedirs(args.output_featurization, exist_ok=True)
+    print("%s created" % args.output_featurization)
+    write_df = dflow.write_to_csv(directory_path=dprep.LocalFileOutput(args.output_featurization))
+    write_df.run_local()

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/trainmodel/get_data.py

@@ -0,0 +1,12 @@
+
+import os
+import pandas as pd
+
+
+def get_data():
+    print("In get_data")
+    print(os.environ['AZUREML_DATAREFERENCE_output_split_train_x'])
+    X_train = pd.read_csv(os.environ['AZUREML_DATAREFERENCE_output_split_train_x'] + "/part-00000", header=0)
+    y_train = pd.read_csv(os.environ['AZUREML_DATAREFERENCE_output_split_train_y'] + "/part-00000", header=0)
+
+    return {"X": X_train.values, "y": y_train.values.flatten()}

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/scripts/trainmodel/train_test_split.py

@@ -0,0 +1,48 @@
+import argparse
+import os
+import azureml.dataprep as dprep
+import azureml.core
+from sklearn.model_selection import train_test_split
+
+
+def write_output(df, path):
+    os.makedirs(path, exist_ok=True)
+    print("%s created" % path)
+    df.to_csv(path + "/part-00000", index=False)
+
+
+print("Split the data into train and test")
+
+parser = argparse.ArgumentParser("split")
+parser.add_argument("--input_split_features", type=str, help="input split features")
+parser.add_argument("--input_split_labels", type=str, help="input split labels")
+parser.add_argument("--output_split_train_x", type=str, help="output split train features")
+parser.add_argument("--output_split_train_y", type=str, help="output split train labels")
+parser.add_argument("--output_split_test_x", type=str, help="output split test features")
+parser.add_argument("--output_split_test_y", type=str, help="output split test labels")
+
+args = parser.parse_args()
+
+print("Argument 1(input taxi data features path): %s" % args.input_split_features)
+print("Argument 2(input taxi data labels path): %s" % args.input_split_labels)
+print("Argument 3(output training features split path): %s" % args.output_split_train_x)
+print("Argument 4(output training labels split path): %s" % args.output_split_train_y)
+print("Argument 5(output test features split path): %s" % args.output_split_test_x)
+print("Argument 6(output test labels split path): %s" % args.output_split_test_y)
+
+x_df = dprep.read_csv(path=args.input_split_features, header=dprep.PromoteHeadersMode.GROUPED).to_pandas_dataframe()
+y_df = dprep.read_csv(path=args.input_split_labels, header=dprep.PromoteHeadersMode.GROUPED).to_pandas_dataframe()
+
+# These functions splits the input features and labels into test and train data
+# Visit https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-auto-train-models for more detail
+
+x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=223)
+
+if not (args.output_split_train_x is None and
+        args.output_split_test_x is None and
+        args.output_split_train_y is None and
+        args.output_split_test_y is None):
+    write_output(x_train, args.output_split_train_x)
+    write_output(y_train, args.output_split_train_y)
+    write_output(x_test, args.output_split_test_x)
+    write_output(y_test, args.output_split_test_y)

how-to-use-azureml/machine-learning-pipelines/pipeline-batch-scoring/pipeline-batch-scoring.ipynb

@@ -162,7 +162,7 @@
       "metadata": {},
       "source": [
         "### Create and attach Compute targets\n",
-        "Use the below code to get the default Compute target. "
+        "Use the below code to create and attach Compute targets. "
       ]
     },
     {
@@ -171,9 +171,33 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "cluster_type = os.environ.get(\"AML_CLUSTER_TYPE\", \"GPU\")\n",
+        "# choose a name for your cluster\n",
+        "aml_compute_name = os.environ.get(\"AML_COMPUTE_NAME\", \"gpu-cluster\")\n",
+        "cluster_min_nodes = os.environ.get(\"AML_COMPUTE_MIN_NODES\", 0)\n",
+        "cluster_max_nodes = os.environ.get(\"AML_COMPUTE_MAX_NODES\", 1)\n",
+        "vm_size = os.environ.get(\"AML_COMPUTE_SKU\", \"STANDARD_NC6\")\n",
         "\n",
-        "compute_target = ws.get_default_compute_target(cluster_type)"
+        "\n",
+        "if aml_compute_name in ws.compute_targets:\n",
+        "    compute_target = ws.compute_targets[aml_compute_name]\n",
+        "    if compute_target and type(compute_target) is AmlCompute:\n",
+        "        print('found compute target. just use it. ' + aml_compute_name)\n",
+        "else:\n",
+        "    print('creating a new compute target...')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = vm_size, # NC6 is GPU-enabled\n",
+        "                                                                vm_priority = 'lowpriority', # optional\n",
+        "                                                                min_nodes = cluster_min_nodes, \n",
+        "                                                                max_nodes = cluster_max_nodes)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, aml_compute_name, provisioning_config)\n",
+        "    \n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it will use the scale settings for the cluster\n",
+        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "    \n",
+        "     # For a more detailed view of current Azure Machine Learning Compute  status, use get_status()\n",
+        "    print(compute_target.get_status().serialize())"
       ]
     },
     {

how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.ipynb

@@ -94,7 +94,7 @@
       "outputs": [],
       "source": [
         "# AmlCompute\n",
-        "cpu_cluster_name = \"cpucluster\"\n",
+        "cpu_cluster_name = \"cpu-cluster\"\n",
         "try:\n",
         "    cpu_cluster = AmlCompute(ws, cpu_cluster_name)\n",
         "    print(\"found existing cluster.\")\n",
@@ -108,7 +108,7 @@
         "    cpu_cluster.wait_for_completion(show_output=True)\n",
         "    \n",
         "# AmlCompute\n",
-        "gpu_cluster_name = \"gpucluster\"\n",
+        "gpu_cluster_name = \"gpu-cluster\"\n",
         "try:\n",
         "    gpu_cluster = AmlCompute(ws, gpu_cluster_name)\n",
         "    print(\"found existing cluster.\")\n",
@@ -351,7 +351,6 @@
         "    inputs=[models_dir, ffmpeg_images],\n",
         "    outputs=[processed_images],\n",
         "    pip_packages=[\"mpi4py\", \"torch\", \"torchvision\"],\n",
-        "    runconfig=amlcompute_run_config,\n",
         "    use_gpu=True,\n",
         "    source_directory=scripts_folder\n",
         ")\n",

how-to-use-azureml/training-with-deep-learning/distributed-chainer/distributed-chainer.ipynb

@@ -95,8 +95,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code gets the default compute cluster.\n",
+        "## Create or attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -107,7 +109,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute. \n",
         "print(compute_target.get_status().serialize())"
@@ -117,7 +136,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
+        "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
       ]
     },
     {
@@ -223,7 +242,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI, you must provide the argument `distributed_training=MpiConfiguration()`. Using this estimator with these settings, Chainer and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `Chainer` constructor's `pip_packages` or `conda_packages` parameters."
+        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, Chainer and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `Chainer` constructor's `pip_packages` or `conda_packages` parameters."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-cntk-with-custom-docker/distributed-cntk-with-custom-docker.ipynb

@@ -98,8 +98,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use default `AmlCompute` as the training compute resource.\n",
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -110,7 +112,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute\n",
         "print(compute_target.get_status().serialize())"
@@ -270,7 +289,6 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import MpiConfiguration\n",
         "from azureml.train.estimator import Estimator\n",
         "\n",
         "script_params = {\n",
@@ -284,7 +302,8 @@
         "                      entry_script='cntk_distr_mnist.py',\n",
         "                      script_params=script_params,\n",
         "                      node_count=2,\n",
-        "                      distributed_training=MpiConfiguration(),\n",
+        "                      process_count_per_node=1,\n",
+        "                      distributed_backend='mpi',\n",
         "                      pip_packages=['cntk-gpu==2.6'],\n",
         "                      custom_docker_image='microsoft/mmlspark:gpu-0.12',\n",
         "                      use_gpu=True)"
@@ -296,7 +315,7 @@
       "source": [
         "We would like to train our model using a [pre-built Docker container](https://hub.docker.com/r/microsoft/mmlspark/). To do so, specify the name of the docker image to the argument `custom_docker_image`. Finally, we provide the `cntk` package to `pip_packages` to install CNTK 2.6 on our custom image.\n",
         "\n",
-        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to run distributed CNTK, which uses MPI, you must provide the argument `distributed_training=MpiConfiguration()`."
+        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to run distributed CNTK, which uses MPI, you must provide the argument `distributed_backend='mpi'`."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb

@@ -96,8 +96,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code uses the default compute in the workspace.\n",
+        "## Create or attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -108,7 +110,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current AmlCompute. \n",
         "print(compute_target.get_status().serialize())"
@@ -118,7 +137,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
+        "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
       ]
     },
     {
@@ -236,7 +255,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_training=MpiConfiguration()`. Using this estimator with these settings, PyTorch, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `PyTorch` constructor's `pip_packages` or `conda_packages` parameters."
+        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, PyTorch, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `PyTorch` constructor's `pip_packages` or `conda_packages` parameters."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/distributed-tensorflow-with-horovod/distributed-tensorflow-with-horovod.ipynb

@@ -98,8 +98,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource.\n",
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -110,7 +112,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -120,7 +139,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
+        "The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
       ]
     },
     {
@@ -304,7 +323,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_training=MpiConfiguration()`. Using this estimator with these settings, TensorFlow, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters.\n",
+        "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, TensorFlow, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters.\n",
         "\n",
         "Note that we passed our training data reference `ds_data` to our script's `--input_data` argument. This will 1) mount our datastore on the remote compute and 2) provide the path to the data zip file on our datastore."
       ]

how-to-use-azureml/training-with-deep-learning/distributed-tensorflow-with-parameter-server/distributed-tensorflow-with-parameter-server.ipynb

@@ -98,8 +98,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource.\n",
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -110,7 +112,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -220,7 +239,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code specifies that we will run our training script on `2` nodes, with two workers and one parameter server. In order to execute a native distributed TensorFlow run, you must provide the argument `distributed_training=TensorflowConfiguration()`. Using this estimator with these settings, TensorFlow and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters."
+        "The above code specifies that we will run our training script on `2` nodes, with two workers and one parameter server. In order to execute a native distributed TensorFlow run, you must provide the argument `distributed_backend='ps'`. Using this estimator with these settings, TensorFlow and its dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `TensorFlow` constructor's `pip_packages` or `conda_packages` parameters."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/how-to-use-estimator/dummy_train.py

@@ -9,8 +9,16 @@ print("Hello Azure ML!")
 parser = argparse.ArgumentParser()
 parser.add_argument('--numbers-in-sequence', type=int, dest='num_in_sequence', default=10,
                     help='number of fibonacci numbers in sequence')
+
+# This is how you can use a bool argument in Python. If you want the 'my_bool_var' to be True, just pass it
+# in Estimator's script_param as script+params:{'my_bool_var': ''}.
+# And, if you want to use it as False, then do not pass it in the Estimator's script_params.
+# You can reverse the behavior by setting action='store_false' in the next line.
+parser.add_argument("--my_bool_var", action='store_true')
+
 args = parser.parse_args()
 num = args.num_in_sequence
+my_bool_var = args.my_bool_var
 
 
 def fibo(n):
@@ -23,6 +31,7 @@ def fibo(n):
 try:
     from azureml.core import Run
     run = Run.get_context()
+    print("The value of boolean parameter 'my_bool_var' is {}".format(my_bool_var))
     print("Log Fibonacci numbers.")
     for i in range(0, num - 1):
         run.log('Fibonacci numbers', fibo(i))

how-to-use-azureml/training-with-deep-learning/how-to-use-estimator/how-to-use-estimator.ipynb

@@ -113,8 +113,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
+        "1. create the configuration (this step is local and only takes a second)\n",
+        "2. create the cluster (this step will take about **20 seconds**)\n",
+        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
       ]
     },
     {
@@ -123,7 +133,25 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "cpu_cluster = ws.get_default_compute_target(\"CPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"cpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    cpu_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    cpu_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it uses the scale settings for the cluster\n",
+        "    cpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(cpu_cluster.get_status().serialize())"
@@ -207,8 +235,11 @@
       "outputs": [],
       "source": [
         "# use a conda environment, don't use Docker, on local computer\n",
+        "# Let's see how you can pass bool arguments in the script_params. Passing `'--my_bool_var': ''` will set my_bool_var as True and\n",
+        "# if you want it to be False, just do not pass it in the script_params.\n",
         "script_params = {\n",
-        "    '--numbers-in-sequence': 10\n",
+        "    '--numbers-in-sequence': 10,\n",
+        "    '--my_bool_var': ''\n",
         "}\n",
         "est = Estimator(source_directory='.', script_params=script_params, compute_target='local', entry_script='dummy_train.py', use_docker=False)\n",
         "run = exp.submit(est)\n",

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-chainer/train-hyperparameter-tune-deploy-with-chainer.ipynb

@@ -95,8 +95,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource.\n",
+        "\n",
+        "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
@@ -107,7 +109,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -117,7 +136,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code retrieves the default GPU compute. If you instead want to use default CPU compute, provide type=\"CPU\"."
+        "The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-keras/train-hyperparameter-tune-deploy-with-keras.ipynb

@@ -239,8 +239,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
+        "1. create the configuration (this step is local and only takes a second)\n",
+        "2. create the cluster (this step will take about **20 seconds**)\n",
+        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
       ]
     },
     {
@@ -249,7 +259,26 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it uses the scale settings for the cluster\n",
+        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -259,7 +288,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have retrtieved the compute target, let's see what the workspace's `compute_targets` property returns."
+        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named \"gpu-cluster\" of type `AmlCompute`."
       ]
     },
     {
@@ -351,7 +380,7 @@
       "metadata": {},
       "source": [
         "## Create TensorFlow estimator & add Keras\n",
-        "Next, we construct an `azureml.train.dnn.TensorFlow` estimator object, use the `gpucluster` as compute target, and pass the mount-point of the datastore to the training code as a parameter.\n",
+        "Next, we construct an `azureml.train.dnn.TensorFlow` estimator object, use the `gpu-cluster` as compute target, and pass the mount-point of the datastore to the training code as a parameter.\n",
         "The TensorFlow estimator is providing a simple way of launching a TensorFlow training job on a compute target. It will automatically provide a docker image that has TensorFlow installed. In this case, we add `keras` package (for the Keras framework obviously), and `matplotlib` package for plotting a \"Loss vs. Accuracy\" chart and record it in run history."
       ]
     },
@@ -524,7 +553,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "In the training script, the Keras model is saved into two files, `model.json` and `model.h5`, in the `outputs/models` folder on the gpucluster AmlCompute node. Azure ML automatically uploaded anything written in the `./outputs` folder into run history file store. Subsequently, we can use the `run` object to download the model files. They are under the the `outputs/model` folder in the run history file store, and are downloaded into a local folder named `model`."
+        "In the training script, the Keras model is saved into two files, `model.json` and `model.h5`, in the `outputs/models` folder on the gpu-cluster AmlCompute node. Azure ML automatically uploaded anything written in the `./outputs` folder into run history file store. Subsequently, we can use the `run` object to download the model files. They are under the the `outputs/model` folder in the run history file store, and are downloaded into a local folder named `model`."
       ]
     },
     {

how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb

@@ -261,8 +261,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute\n",
-        "You can create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you use default `AmlCompute` as your training compute resource."
+        "## Create or Attach existing AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
+        "1. create the configuration (this step is local and only takes a second)\n",
+        "2. create the cluster (this step will take about **20 seconds**)\n",
+        "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
       ]
     },
     {
@@ -271,7 +281,26 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"GPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it uses the scale settings for the cluster\n",
+        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -281,7 +310,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Now that you have retrieved the compute target, let's see what the workspace's `compute_targets` property returns."
+        "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'gpu-cluster' of type `AmlCompute`."
       ]
     },
     {

how-to-use-azureml/training/train-hyperparameter-tune-deploy-with-sklearn/train-hyperparameter-tune-deploy-with-sklearn.ipynb

@@ -108,7 +108,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Get default AmlCompute"
+        "## Create AmlCompute"
       ]
     },
     {
@@ -126,7 +126,24 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "compute_target = ws.get_default_compute_target(type=\"CPU\")\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# choose a name for your cluster\n",
+        "cluster_name = \"gpu-cluster\"\n",
+        "\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
+        "    print('Found existing compute target.')\n",
+        "except ComputeTargetException:\n",
+        "    print('Creating a new compute target...')\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
+        "                                                           max_nodes=4)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+        "\n",
+        "    compute_target.wait_for_completion(show_output=True)\n",
         "\n",
         "# use get_status() to get a detailed status for the current cluster. \n",
         "print(compute_target.get_status().serialize())"
@@ -136,7 +153,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The above code retrieves the default CPU compute."
+        "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
       ]
     },
     {

how-to-use-azureml/training/train-hyperparameter-tune-deploy-with-sklearn/train_iris.py

@@ -22,7 +22,7 @@ def main():
                         help='Penalty parameter of the error term')
 
     args = parser.parse_args()
-    run.log('Kernel type', np.string(args.kernel))
+    run.log('Kernel type', np.str(args.kernel))
     run.log('Penalty', np.float(args.penalty))
 
     # loading the iris dataset

how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb

@@ -188,79 +188,6 @@
         "myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Get the default compute target\n",
-        "\n",
-        "In this case, we use the default `AmlCompute`target from the workspace."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import ScriptRunConfig\n",
-        "from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
-        "\n",
-        "src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
-        "\n",
-        "# Use default compute target\n",
-        "src.run_config.target = ws.get_default_compute_target(type=\"CPU\").name\n",
-        "\n",
-        "# Set environment\n",
-        "src.run_config.environment = myenv"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Submit run"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "run = experiment.submit(src)\n",
-        "\n",
-        "# Show run details\n",
-        "run"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "%%time\n",
-        "# Shows output of the run on stdout.\n",
-        "run.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "run.get_metrics()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -283,7 +210,7 @@
         "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
         "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpucluster\"\n",
+        "cpu_cluster_name = \"cpu-cluster\"\n",
         "\n",
         "# Verify that cluster does not exist already\n",
         "try:\n",
@@ -310,13 +237,28 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "from azureml.core import ScriptRunConfig\n",
+        "from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
+        "\n",
+        "src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
+        "\n",
         "# Set compute target to the one created in previous step\n",
         "src.run_config.target = cpu_cluster.name\n",
+        "\n",
+        "# Set environment\n",
+        "src.run_config.environment = myenv\n",
         " \n",
         "run = experiment.submit(config=src)\n",
         "run"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -364,7 +306,7 @@
         "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
         "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpucluster\"\n",
+        "cpu_cluster_name = \"cpu-cluster\"\n",
         "\n",
         "# Verify that cluster does not exist already\n",
         "try:\n",
@@ -463,7 +405,7 @@
       "outputs": [],
       "source": [
         "#Delete () is used to deprovision and delete the AmlCompute target. Useful if you want to re-use the compute name \n",
-        "#'cpucluster' in this case but use a different VM family for instance.\n",
+        "#'cpu-cluster' in this case but use a different VM family for instance.\n",
         "\n",
         "#cpu_cluster.delete()"
       ]

how-to-use-azureml/using-mlflow/README.md

@@ -4,8 +4,9 @@
 
 Try out the sample notebooks:
 
-* [Use MLflow with Azure Machine Learning for local training run](./train-local/train-local.ipynb)
-* [Use MLflow with Azure Machine Learning for remote training run](./train-remote/train-remote.ipynb)
-* [Deploy Model as Azure Machine Learning web service using MLflow](./deploy-model/deploy-model.ipynb)
+* [Use MLflow with Azure Machine Learning for Local Training Run](./train-local/train-local.ipynb)
+* [Use MLflow with Azure Machine Learning for Remote Training Run](./train-remote/train-remote.ipynb)
+* [Deploy Model as Azure Machine Learning Web Service using MLflow](./deploy-model/deploy-model.ipynb)
+* [Train and Deploy PyTorch Image Classifier](./train-deploy-pytorch/train-deploy-pytorch.ipynb)
 
 ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/using-mlflow/README..png) 

how-to-use-azureml/using-mlflow/train-deploy-pytorch/scripts/train.py

@@ -0,0 +1,150 @@
+# Copyright (c) 2017, PyTorch Team
+# All rights reserved
+# Licensed under BSD 3-Clause License.
+
+# This example is based on PyTorch MNIST example:
+# https://github.com/pytorch/examples/blob/master/mnist/main.py
+
+import mlflow
+import mlflow.pytorch
+from mlflow.utils.environment import _mlflow_conda_env
+import warnings
+import cloudpickle
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+import torchvision
+from torchvision import datasets, transforms
+
+
+class Net(nn.Module):
+    def __init__(self):
+        super(Net, self).__init__()
+        self.conv1 = nn.Conv2d(1, 20, 5, 1)
+        self.conv2 = nn.Conv2d(20, 50, 5, 1)
+        self.fc1 = nn.Linear(4 * 4 * 50, 500)
+        self.fc2 = nn.Linear(500, 10)
+
+    def forward(self, x):
+        # Added the view for reshaping score requests
+        x = x.view(-1, 1, 28, 28)
+        x = F.relu(self.conv1(x))
+        x = F.max_pool2d(x, 2, 2)
+        x = F.relu(self.conv2(x))
+        x = F.max_pool2d(x, 2, 2)
+        x = x.view(-1, 4 * 4 * 50)
+        x = F.relu(self.fc1(x))
+        x = self.fc2(x)
+        return F.log_softmax(x, dim=1)
+
+
+def train(args, model, device, train_loader, optimizer, epoch):
+    model.train()
+    for batch_idx, (data, target) in enumerate(train_loader):
+        data, target = data.to(device), target.to(device)
+        optimizer.zero_grad()
+        output = model(data)
+        loss = F.nll_loss(output, target)
+        loss.backward()
+        optimizer.step()
+        if batch_idx % args.log_interval == 0:
+            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
+                epoch, batch_idx * len(data), len(train_loader.dataset),
+                100. * batch_idx / len(train_loader), loss.item()))
+            # Use MLflow logging
+            mlflow.log_metric("epoch_loss", loss.item())
+
+
+def test(args, model, device, test_loader):
+    model.eval()
+    test_loss = 0
+    correct = 0
+    with torch.no_grad():
+        for data, target in test_loader:
+            data, target = data.to(device), target.to(device)
+            output = model(data)
+            # sum up batch loss
+            test_loss += F.nll_loss(output, target, reduction="sum").item()
+            # get the index of the max log-probability
+            pred = output.argmax(dim=1, keepdim=True)
+            correct += pred.eq(target.view_as(pred)).sum().item()
+
+    test_loss /= len(test_loader.dataset)
+    print("\n")
+    print("Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n".format(
+        test_loss, correct, len(test_loader.dataset),
+        100. * correct / len(test_loader.dataset)))
+    # Use MLflow logging
+    mlflow.log_metric("average_loss", test_loss)
+
+
+class Args(object):
+    pass
+
+
+# Training settings
+args = Args()
+setattr(args, 'batch_size', 64)
+setattr(args, 'test_batch_size', 1000)
+setattr(args, 'epochs', 3)  # Higher number for better convergence
+setattr(args, 'lr', 0.01)
+setattr(args, 'momentum', 0.5)
+setattr(args, 'no_cuda', True)
+setattr(args, 'seed', 1)
+setattr(args, 'log_interval', 10)
+setattr(args, 'save_model', True)
+
+use_cuda = not args.no_cuda and torch.cuda.is_available()
+
+torch.manual_seed(args.seed)
+
+device = torch.device("cuda" if use_cuda else "cpu")
+
+kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
+train_loader = torch.utils.data.DataLoader(
+    datasets.MNIST('../data', train=True, download=True,
+                   transform=transforms.Compose([
+                       transforms.ToTensor(),
+                       transforms.Normalize((0.1307,), (0.3081,))
+                   ])),
+    batch_size=args.batch_size, shuffle=True, **kwargs)
+test_loader = torch.utils.data.DataLoader(
+    datasets.MNIST(
+        '../data',
+        train=False,
+        transform=transforms.Compose([
+            transforms.ToTensor(),
+            transforms.Normalize((0.1307,), (0.3081,))])),
+    batch_size=args.test_batch_size, shuffle=True, **kwargs)
+
+
+def driver():
+    warnings.filterwarnings("ignore")
+    # Dependencies for deploying the model
+    pytorch_index = "https://download.pytorch.org/whl/"
+    pytorch_version = "cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl"
+    deps = [
+        "cloudpickle=={}".format(cloudpickle.__version__),
+        pytorch_index + pytorch_version,
+        "torchvision=={}".format(torchvision.__version__),
+        "Pillow=={}".format("6.0.0")
+    ]
+    with mlflow.start_run() as run:
+        model = Net().to(device)
+        optimizer = optim.SGD(
+            model.parameters(),
+            lr=args.lr,
+            momentum=args.momentum)
+        for epoch in range(1, args.epochs + 1):
+            train(args, model, device, train_loader, optimizer, epoch)
+            test(args, model, device, test_loader)
+        # Log model to run history using MLflow
+        if args.save_model:
+            model_env = _mlflow_conda_env(additional_pip_deps=deps)
+            mlflow.pytorch.log_model(model, "model", conda_env=model_env)
+    return run
+
+
+if __name__ == "__main__":
+    driver()

how-to-use-azureml/using-mlflow/train-deploy-pytorch/train-and-deploy-pytorch.ipynb

@@ -0,0 +1,481 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+    "\n",
+    "Licensed under the MIT License."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/using-mlflow/train-deploy-pytorch/train-deploy-pytorch.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Use MLflow with Azure Machine Learning to Train and Deploy PyTorch Image Classifier\n",
+    "\n",
+    "This example shows you how to use MLflow together with Azure Machine Learning services for tracking the metrics and artifacts while training a PyTorch model to classify MNIST digit images, and then deploy the model  as a web service. You'll learn how to:\n",
+    "\n",
+    " 1. Set up MLflow tracking URI so as to use Azure ML\n",
+    " 2. Create experiment\n",
+    " 3. Instrument your model with MLflow tracking\n",
+    " 4. Train a PyTorch model locally\n",
+    " 5. Train a model on GPU compute on Azure\n",
+    " 6. View your experiment within your Azure ML Workspace in Azure Portal\n",
+    " 7. Create a Docker image from the trained model\n",
+    " 8. Deploy the model as a web service on Azure Container Instance\n",
+    " 9. Call the model to make predictions\n",
+    " \n",
+    "### Pre-requisites\n",
+    " \n",
+    "Make sure you have completed the [Configuration](../../../configuration.ipnyb) notebook to set up your Azure Machine Learning workspace and ensure other common prerequisites are met.\n",
+    "\n",
+    "Also, install mlflow-azureml package using ```pip install mlflow-azureml```. Note that mlflow-azureml installs mlflow package itself as a dependency, if you haven't done so previously.\n",
+    "\n",
+    "### Set-up\n",
+    "\n",
+    "Import packages and check versions of Azure ML SDK and MLflow installed on your computer. Then connect to your Workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys, os\n",
+    "import mlflow\n",
+    "import mlflow.azureml\n",
+    "import mlflow.sklearn\n",
+    "\n",
+    "import azureml.core\n",
+    "from azureml.core import Workspace\n",
+    "\n",
+    "\n",
+    "print(\"SDK version:\", azureml.core.VERSION)\n",
+    "print(\"MLflow version:\", mlflow.version.VERSION)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ws = Workspace.from_config()\n",
+    "ws.get_details()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set tracking URI\n",
+    "\n",
+    "Set the MLFlow tracking URI to point to your Azure ML Workspace. The subsequent logging calls from MLFlow APIs will go to Azure ML services and will be tracked under your Workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create Experiment\n",
+    "\n",
+    "In both MLflow and Azure ML, training runs are grouped into experiments. Let's create one for our experimentation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "experiment_name = \"pytorch-with-mlflow\"\n",
+    "mlflow.set_experiment(experiment_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Train model locally while logging metrics and artifacts\n",
+    "\n",
+    "The ```scripts/train.py``` program contains the code to load the image dataset, and train and test the model. Within this program, the train.driver function wraps the end-to-end workflow.\n",
+    "\n",
+    "Within the driver, the ```mlflow.start_run``` starts MLflow tracking. Then, ```mlflow.log_metric``` functions are used to track the convergence of the neural network training iterations. Finally ```mlflow.pytorch.save_model``` is used to save the trained model in framework-aware manner.\n",
+    "\n",
+    "Let's add the program to search path, import it as a module, and then invoke the driver function. Note that the training can take few minutes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lib_path = os.path.abspath(\"scripts\")\n",
+    "sys.path.append(lib_path)\n",
+    "\n",
+    "import train"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run = train.driver()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can view the metrics of the run at Azure Portal"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(azureml.mlflow.get_portal_url(run))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Train model on GPU compute on Azure\n",
+    "\n",
+    "Next, let's run the same script on GPU-enabled compute for faster training. If you've completed the the [Configuration](../../../configuration.ipnyb) notebook, you should have a GPU cluster named \"gpu-cluster\" available in your workspace. Otherwise, follow the instructions in the notebook to create one. For simplicity, this example uses single process on single VM to train the model.\n",
+    "\n",
+    "Create a PyTorch estimator to specify the training configuration: script, compute as well as additional packages needed. To enable MLflow tracking, include ```azureml-mlflow``` as pip package. The low-level specifications for the training run are encapsulated in the estimator instance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.train.dnn import PyTorch\n",
+    "\n",
+    "pt = PyTorch(source_directory=\"./scripts\", \n",
+    "             entry_script = \"train.py\", \n",
+    "             compute_target = \"gpu-cluster\", \n",
+    "             node_count = 1, \n",
+    "             process_count_per_node = 1, \n",
+    "             use_gpu=True,\n",
+    "             pip_packages = [\"azureml-mlflow\", \"Pillow==6.0.0\"])\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Get a reference to the experiment you created previously, but this time, as Azure Machine Learning experiment object.\n",
+    "\n",
+    "Then, use ```Experiment.submit``` method to start the remote training run. Note that the first training run often takes longer as Azure Machine Learning service builds the Docker image for executing the script. Subsequent runs will be faster as cached image is used."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Experiment\n",
+    "\n",
+    "exp = Experiment(ws, experiment_name)\n",
+    "run = exp.submit(pt)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can monitor the run and its metrics on Azure Portal."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Also, you can wait for run to complete."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run.wait_for_completion(show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Deploy model as web service\n",
+    "\n",
+    "To deploy a web service, first create a Docker image, and then deploy that Docker image on inferencing compute.\n",
+    "\n",
+    "The ```mlflow.azureml.build_image``` function builds a Docker image from saved PyTorch model in a framework-aware manner. It automatically creates the PyTorch-specific inferencing wrapper code and specififies package dependencies for you."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run.get_file_names()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then build a docker image using *runs:/<run.id>/model* as the model_uri path.\n",
+    "\n",
+    "Note that the image building can take several minutes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_path = \"model\"\n",
+    "\n",
+    "\n",
+    "azure_image, azure_model = mlflow.azureml.build_image(model_uri='runs:/{}/{}'.format(run.id, model_path),\n",
+    "                                                      workspace=ws,\n",
+    "                                                      model_name='pytorch_mnist',\n",
+    "                                                      image_name='pytorch-mnist-img',\n",
+    "                                                      synchronous=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then, deploy the Docker image to Azure Container Instance: a serverless compute capable of running a single container. You can tag and add descriptions to help keep track of your web service. \n",
+    "\n",
+    "[Other inferencing compute choices](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where) include Azure Kubernetes Service which provides scalable endpoint suitable for production use.\n",
+    "\n",
+    "Note that the service deployment can take several minutes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.webservice import AciWebservice, Webservice\n",
+    "\n",
+    "aci_config = AciWebservice.deploy_configuration(cpu_cores=2, \n",
+    "                                                memory_gb=5, \n",
+    "                                                tags={\"data\": \"MNIST\",  \"method\" : \"pytorch\"}, \n",
+    "                                                description=\"Predict using webservice\")\n",
+    "\n",
+    "\n",
+    "# Deploy the image to Azure Container Instances (ACI) for real-time serving\n",
+    "webservice = Webservice.deploy_from_image(\n",
+    "    image=azure_image, workspace=ws, name=\"pytorch-mnist-1\", deployment_config=aci_config)\n",
+    "\n",
+    "\n",
+    "webservice.wait_for_deployment()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once the deployment has completed you can check the scoring URI of the web service."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Scoring URI is: {}\".format(webservice.scoring_uri))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In case of a service creation issue, you can use ```webservice.get_logs()``` to get logs to debug."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Make predictions using web service\n",
+    "\n",
+    "To make the web service, create a test data set as normalized PyTorch tensors. \n",
+    "\n",
+    "Then, let's define a utility function that takes a random image and converts it into format and shape suitable for as input to PyTorch inferencing end-point. The conversion is done by: \n",
+    "\n",
+    " 1. Select a random (image, label) tuple\n",
+    " 2. Take the image and converting the tensor to NumPy array \n",
+    " 3. Reshape array into 1 x 1 x N array\n",
+    "    * 1 image in batch, 1 color channel, N = 784 pixels for MNIST images\n",
+    "    * Note also ```x = x.view(-1, 1, 28, 28)``` in net definition in ```train.py``` program to shape incoming scoring requests.\n",
+    " 4. Convert the NumPy array to list to make it into a built-in type.\n",
+    " 5. Create a dictionary {\"data\", <list>} that can be converted to JSON string for web service requests."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from torchvision import datasets, transforms\n",
+    "import random\n",
+    "import numpy as np\n",
+    "\n",
+    "test_data = datasets.MNIST('../data', train=False, transform=transforms.Compose([\n",
+    "                       transforms.ToTensor(),\n",
+    "                       transforms.Normalize((0.1307,), (0.3081,))]))\n",
+    "\n",
+    "\n",
+    "def get_random_image():\n",
+    "    image_idx = random.randint(0,len(test_data))\n",
+    "    image_as_tensor = test_data[image_idx][0]\n",
+    "    return {\"data\": elem for elem in image_as_tensor.numpy().reshape(1,1,-1).tolist()}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then, invoke the web service using a random test image. Convert the dictionary containing the image to JSON string before passing it to web service.\n",
+    "\n",
+    "The response contains the raw scores for each label, with greater value indicating higher probability. Sort the labels and select the one with greatest score to get the prediction. Let's also plot the image sent to web service for comparison purposes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "import json\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "test_image = get_random_image()\n",
+    "\n",
+    "response = webservice.run(json.dumps(test_image))\n",
+    "\n",
+    "response = sorted(response[0].items(), key = lambda x: x[1], reverse = True)\n",
+    "\n",
+    "\n",
+    "print(\"Predicted label:\", response[0][0])\n",
+    "plt.imshow(np.array(test_image[\"data\"]).reshape(28,28), cmap = \"gray\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also call the web service using a raw POST method against the web service"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(url=webservice.scoring_uri, data=json.dumps(test_image),headers={\"Content-type\": \"application/json\"})\n",
+    "print(response.text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": "roastala"
+   }
+  ],
+  "celltoolbar": "Edit Metadata",
+  "kernelspec": {
+   "display_name": "Python 3.6",
+   "language": "python",
+   "name": "python36"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.3"
+  },
+  "name": "mlflow-sparksummit-pytorch",
+  "notebookId": 2495374963457641
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

tutorials/img-classification-part1-training.ipynb

@@ -126,7 +126,9 @@
       "metadata": {},
       "source": [
         "### Create or Attach existing compute resource\n",
-        "By using Azure Machine Learning Compute, a managed service, data scientists can train machine learning models on clusters of Azure virtual machines. Examples include VMs with GPU support. In this tutorial, you use default Azure Machine Learning Compute as your training environment."
+        "By using Azure Machine Learning Compute, a managed service, data scientists can train machine learning models on clusters of Azure virtual machines. Examples include VMs with GPU support. In this tutorial, you create Azure Machine Learning Compute as your training environment. The code below creates the compute clusters for you if they don't already exist in your workspace.\n",
+        "\n",
+        "**Creation of compute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace the code will skip the creation process."
       ]
     },
     {
@@ -140,10 +142,38 @@
       },
       "outputs": [],
       "source": [
+        "from azureml.core.compute import AmlCompute\n",
+        "from azureml.core.compute import ComputeTarget\n",
         "import os\n",
         "\n",
-        "cluster_type = os.environ.get(\"AML_COMPUTE_CLUSTER_TYPE\", \"CPU\")\n",
-        "compute_target = ws.get_default_compute_target(cluster_type)"
+        "# choose a name for your cluster\n",
+        "compute_name = os.environ.get(\"AML_COMPUTE_CLUSTER_NAME\", \"cpu-cluster\")\n",
+        "compute_min_nodes = os.environ.get(\"AML_COMPUTE_CLUSTER_MIN_NODES\", 0)\n",
+        "compute_max_nodes = os.environ.get(\"AML_COMPUTE_CLUSTER_MAX_NODES\", 4)\n",
+        "\n",
+        "# This example uses CPU VM. For using GPU VM, set SKU to STANDARD_NC6\n",
+        "vm_size = os.environ.get(\"AML_COMPUTE_CLUSTER_SKU\", \"STANDARD_D2_V2\")\n",
+        "\n",
+        "\n",
+        "if compute_name in ws.compute_targets:\n",
+        "    compute_target = ws.compute_targets[compute_name]\n",
+        "    if compute_target and type(compute_target) is AmlCompute:\n",
+        "        print('found compute target. just use it. ' + compute_name)\n",
+        "else:\n",
+        "    print('creating a new compute target...')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = vm_size,\n",
+        "                                                                min_nodes = compute_min_nodes, \n",
+        "                                                                max_nodes = compute_max_nodes)\n",
+        "\n",
+        "    # create the cluster\n",
+        "    compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)\n",
+        "    \n",
+        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
+        "    # if no min node count is provided it will use the scale settings for the cluster\n",
+        "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
+        "    \n",
+        "     # For a more detailed view of current AmlCompute status, use get_status()\n",
+        "    print(compute_target.get_status().serialize())"
       ]
     },
     {
@@ -324,8 +354,8 @@
         "# get hold of the current run\n",
         "run = Run.get_context()\n",
         "\n",
-        "print('Train a logistic regression model with regularizaion rate of', args.reg)\n",
-        "clf = LogisticRegression(C=1.0/args.reg, random_state=42)\n",
+        "print('Train a logistic regression model with regularization rate of', args.reg)\n",
+        "clf = LogisticRegression(C=1.0/args.reg, solver=\"liblinear\", multi_class=\"auto\", random_state=42)\n",
         "clf.fit(X_train, y_train)\n",
         "\n",
         "print('Predict the test set')\n",
@@ -386,14 +416,13 @@
       "source": [
         "### Create an estimator\n",
         "\n",
-        "An estimator object is used to submit the run.  Create your estimator by running the following code to define:\n",
+        "An estimator object is used to submit the run. Azure Machine Learning has pre-configured estimators for common machine learning frameworks, as well as generic Estimator. Create SKLearn estimator for scikit-learn model, by specifying\n",
         "\n",
         "* The name of the estimator object, `est`\n",
         "* The directory that contains your scripts. All the files in this directory are uploaded into the cluster nodes for execution. \n",
         "* The compute target.  In this case you will use the AmlCompute you created\n",
         "* The training script name, train.py\n",
         "* Parameters required from the training script \n",
-        "* Python packages needed for training\n",
         "\n",
         "In this tutorial, this target is AmlCompute. All files in the script folder are uploaded into the cluster nodes for execution. The data_folder is set to use the datastore (`ds.path('mnist').as_mount()`)."
       ]
@@ -408,18 +437,17 @@
       },
       "outputs": [],
       "source": [
-        "from azureml.train.estimator import Estimator\n",
+        "from azureml.train.sklearn import SKLearn\n",
         "\n",
         "script_params = {\n",
         "    '--data-folder': ds.path('mnist').as_mount(),\n",
-        "    '--regularization': 0.05\n",
+        "    '--regularization': 0.5\n",
         "}\n",
         "\n",
-        "est = Estimator(source_directory=script_folder,\n",
+        "est = SKLearn(source_directory=script_folder,\n",
         "                script_params=script_params,\n",
         "                compute_target=compute_target,\n",
-        "                entry_script='train.py',\n",
-        "                conda_packages=['scikit-learn'])"
+        "                entry_script='train.py')"
       ]
     },
     {
@@ -646,18 +674,6 @@
       "language": "python",
       "name": "python36"
     },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.8"
-    },
     "msauthor": "roastala"
   },
   "nbformat": 4,

update.md

@@ -0,0 +1 @@
+##test logic