update samples from Release-56 as a part of SDK release

update samples from Release-55 as a part of  SDK release

README.md

@@ -40,6 +40,7 @@ The [How to use Azure ML](./how-to-use-azureml) folder contains specific example
 - [Deployment](./how-to-use-azureml/deployment) - Examples showing how to deploy and manage machine learning models and solutions
 - [Azure Databricks](./how-to-use-azureml/azure-databricks) - Examples showing how to use Azure ML with Azure Databricks
 - [Monitor Models](./how-to-use-azureml/monitor-models) - Examples showing how to enable model monitoring services such as DataDrift
+- [Reinforcement Learning](./how-to-use-azureml/reinforcement-learning) - Examples showing how to train reinforcement learning agents
 
 ---
 ## Documentation

configuration.ipynb

@@ -103,7 +103,7 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

contrib/fairness/fairlearn-azureml-mitigation.ipynb

@@ -0,0 +1,549 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/contrib/fairness/fairlearn-azureml-mitigation.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Unfairness Mitigation with Fairlearn and Azure Machine Learning\n",
+        "**This notebook shows how to upload results from Fairlearn's GridSearch mitigation algorithm into a dashboard in Azure Machine Learning Studio**\n",
+        "\n",
+        "## Table of Contents\n",
+        "\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Loading the Data](#LoadingData)\n",
+        "1. [Training an Unmitigated Model](#UnmitigatedModel)\n",
+        "1. [Mitigation with GridSearch](#Mitigation)\n",
+        "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
+        "    1. Registering models\n",
+        "    1. Computing Fairness Metrics\n",
+        "    1. Uploading to Azure\n",
+        "1. [Conclusion](#Conclusion)\n",
+        "\n",
+        "<a id=\"Introduction\"></a>\n",
+        "## Introduction\n",
+        "This notebook shows how to use [Fairlearn (an open source fairness assessment and unfairness mitigation package)](http://fairlearn.github.io) and Azure Machine Learning Studio for a binary classification problem. This example uses the well-known adult census dataset. For the purposes of this notebook, we shall treat this as a loan decision problem. We will pretend that the label indicates whether or not each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan. Its purpose is purely illustrative of a workflow including a fairness dashboard - in particular, we do **not** include a full discussion of the detailed issues which arise when considering fairness in machine learning. For such discussions, please [refer to the Fairlearn website](http://fairlearn.github.io/).\n",
+        "\n",
+        "We will apply the [grid search algorithm](https://fairlearn.github.io/api_reference/fairlearn.reductions.html#fairlearn.reductions.GridSearch) from the Fairlearn package using a specific notion of fairness called Demographic Parity. This produces a set of models, and we will view these in a dashboard both locally and in the Azure Machine Learning Studio.\n",
+        "\n",
+        "### Setup\n",
+        "\n",
+        "To use this notebook, an Azure Machine Learning workspace is required.\n",
+        "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
+        "This notebook also requires the following packages:\n",
+        "* `azureml-contrib-fairness`\n",
+        "* `fairlearn==0.4.6`\n",
+        "* `joblib`\n",
+        "* `shap`\n",
+        "\n",
+        "\n",
+        "<a id=\"LoadingData\"></a>\n",
+        "## Loading the Data\n",
+        "We use the well-known `adult` census dataset, which we load using `shap` (for convenience). We start with a fairly unremarkable set of imports:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairlearn.reductions import GridSearch, DemographicParity, ErrorRate\n",
+        "from fairlearn.widget import FairlearnDashboard\n",
+        "from sklearn import svm\n",
+        "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
+        "from sklearn.linear_model import LogisticRegression\n",
+        "import pandas as pd\n",
+        "import shap"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can now load and inspect the data from the `shap` package:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_raw, Y = shap.datasets.adult()\n",
+        "X_raw[\"Race\"].value_counts().to_dict()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We are going to treat the sex of each individual as a protected attribute (where 0 indicates female and 1 indicates male), and in this particular case we are going separate this attribute out and drop it from the main data (this is not always the best option - see the [Fairlearn website](http://fairlearn.github.io/) for further discussion). We also separate out the Race column, but we will not perform any mitigation based on it. Finally, we perform some standard data preprocessing steps to convert the data into a format suitable for the ML algorithms"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "A = X_raw[['Sex','Race']]\n",
+        "X = X_raw.drop(labels=['Sex', 'Race'],axis = 1)\n",
+        "X = pd.get_dummies(X)\n",
+        "\n",
+        "\n",
+        "le = LabelEncoder()\n",
+        "Y = le.fit_transform(Y)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "With our data prepared, we can make the conventional split in to 'test' and 'train' subsets:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.model_selection import train_test_split\n",
+        "X_train, X_test, Y_train, Y_test, A_train, A_test = train_test_split(X_raw, \n",
+        "                                                    Y, \n",
+        "                                                    A,\n",
+        "                                                    test_size = 0.2,\n",
+        "                                                    random_state=0,\n",
+        "                                                    stratify=Y)\n",
+        "\n",
+        "# Work around indexing issue\n",
+        "X_train = X_train.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "X_test = X_test.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)\n",
+        "\n",
+        "# Improve labels\n",
+        "A_test.Sex.loc[(A_test['Sex'] == 0)] = 'female'\n",
+        "A_test.Sex.loc[(A_test['Sex'] == 1)] = 'male'\n",
+        "\n",
+        "\n",
+        "A_test.Race.loc[(A_test['Race'] == 0)] = 'Amer-Indian-Eskimo'\n",
+        "A_test.Race.loc[(A_test['Race'] == 1)] = 'Asian-Pac-Islander'\n",
+        "A_test.Race.loc[(A_test['Race'] == 2)] = 'Black'\n",
+        "A_test.Race.loc[(A_test['Race'] == 3)] = 'Other'\n",
+        "A_test.Race.loc[(A_test['Race'] == 4)] = 'White'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"UnmitigatedModel\"></a>\n",
+        "## Training an Unmitigated Model\n",
+        "\n",
+        "So we have a point of comparison, we first train a model (specifically, logistic regression from scikit-learn) on the raw data, without applying any mitigation algorithm:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "unmitigated_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
+        "\n",
+        "unmitigated_predictor.fit(X_train, Y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can view this model in the fairness dashboard, and see the disparities which appear:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "FairlearnDashboard(sensitive_features=A_test, sensitive_feature_names=['Sex', 'Race'],\n",
+        "                   y_true=Y_test,\n",
+        "                   y_pred={\"unmitigated\": unmitigated_predictor.predict(X_test)})"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Looking at the disparity in accuracy when we select 'Sex' as the sensitive feature, we see that males have an error rate about three times greater than the females. More interesting is the disparity in opportunitiy - males are offered loans at three times the rate of females.\n",
+        "\n",
+        "Despite the fact that we removed the feature from the training data, our predictor still discriminates based on sex. This demonstrates that simply ignoring a protected attribute when fitting a predictor rarely eliminates unfairness. There will generally be enough other features correlated with the removed attribute to lead to disparate impact."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Mitigation\"></a>\n",
+        "## Mitigation with GridSearch\n",
+        "\n",
+        "The `GridSearch` class in `Fairlearn` implements a simplified version of the exponentiated gradient reduction of [Agarwal et al. 2018](https://arxiv.org/abs/1803.02453). The user supplies a standard ML estimator, which is treated as a blackbox - for this simple example, we shall use the logistic regression estimator from scikit-learn. `GridSearch` works by generating a sequence of relabellings and reweightings, and trains a predictor for each.\n",
+        "\n",
+        "For this example, we specify demographic parity (on the protected attribute of sex) as the fairness metric. Demographic parity requires that individuals are offered the opportunity (a loan in this example) independent of membership in the protected class (i.e., females and males should be offered loans at the same rate). *We are using this metric for the sake of simplicity* in this example; the appropriate fairness metric can only be selected after *careful examination of the broader context* in which the model is to be used."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sweep = GridSearch(LogisticRegression(solver='liblinear', fit_intercept=True),\n",
+        "                   constraints=DemographicParity(),\n",
+        "                   grid_size=71)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "With our estimator created, we can fit it to the data. After `fit()` completes, we extract the full set of predictors from the `GridSearch` object.\n",
+        "\n",
+        "The following cell trains a many copies of the underlying estimator, and may take a minute or two to run:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sweep.fit(X_train, Y_train,\n",
+        "          sensitive_features=A_train.Sex)\n",
+        "\n",
+        "predictors = sweep._predictors"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We could load these predictors into the Fairness dashboard now. However, the plot would be somewhat confusing due to their number. In this case, we are going to remove the predictors which are dominated in the error-disparity space by others from the sweep (note that the disparity will only be calculated for the protected attribute; other potentially protected attributes will *not* be mitigated). In general, one might not want to do this, since there may be other considerations beyond the strict optimisation of error and disparity (of the given protected attribute)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "errors, disparities = [], []\n",
+        "for m in predictors:\n",
+        "    classifier = lambda X: m.predict(X)\n",
+        "    \n",
+        "    error = ErrorRate()\n",
+        "    error.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
+        "    disparity = DemographicParity()\n",
+        "    disparity.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
+        "    \n",
+        "    errors.append(error.gamma(classifier)[0])\n",
+        "    disparities.append(disparity.gamma(classifier).max())\n",
+        "    \n",
+        "all_results = pd.DataFrame( {\"predictor\": predictors, \"error\": errors, \"disparity\": disparities})\n",
+        "\n",
+        "dominant_models_dict = dict()\n",
+        "base_name_format = \"census_gs_model_{0}\"\n",
+        "row_id = 0\n",
+        "for row in all_results.itertuples():\n",
+        "    model_name = base_name_format.format(row_id)\n",
+        "    errors_for_lower_or_eq_disparity = all_results[\"error\"][all_results[\"disparity\"]<=row.disparity]\n",
+        "    if row.error <= errors_for_lower_or_eq_disparity.min():\n",
+        "        dominant_models_dict[model_name] = row.predictor\n",
+        "    row_id = row_id + 1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can construct predictions for the dominant models (we include the unmitigated predictor as well, for comparison):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "predictions_dominant = {\"census_unmitigated\": unmitigated_predictor.predict(X_test)}\n",
+        "models_dominant = {\"census_unmitigated\": unmitigated_predictor}\n",
+        "for name, predictor in dominant_models_dict.items():\n",
+        "    value = predictor.predict(X_test)\n",
+        "    predictions_dominant[name] = value\n",
+        "    models_dominant[name] = predictor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "These predictions may then be viewed in the fairness dashboard. We include the race column from the dataset, as an alternative basis for assessing the models. However, since we have not based our mitigation on it, the variation in the models with respect to race can be large."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "FairlearnDashboard(sensitive_features=A_test, \n",
+        "                   sensitive_feature_names=['Sex', 'Race'],\n",
+        "                   y_true=Y_test.tolist(),\n",
+        "                   y_pred=predictions_dominant)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "When using sex as the sensitive feature, we see a Pareto front forming - the set of predictors which represent optimal tradeoffs between accuracy and disparity in predictions. In the ideal case, we would have a predictor at (1,0) - perfectly accurate and without any unfairness under demographic parity (with respect to the protected attribute \"sex\"). The Pareto front represents the closest we can come to this ideal based on our data and choice of estimator. Note the range of the axes - the disparity axis covers more values than the accuracy, so we can reduce disparity substantially for a small loss in accuracy. Finally, we also see that the unmitigated model is towards the top right of the plot, with high accuracy, but worst disparity.\n",
+        "\n",
+        "By clicking on individual models on the plot, we can inspect their metrics for disparity and accuracy in greater detail. In a real example, we would then pick the model which represented the best trade-off between accuracy and disparity given the relevant business constraints."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"AzureUpload\"></a>\n",
+        "## Uploading a Fairness Dashboard to Azure\n",
+        "\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairlearnDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. By default, the dashboard in Azure Machine Learning Studio also requires the models to be registered. The required stages are therefore:\n",
+        "1. Register the dominant models\n",
+        "1. Precompute all the required metrics\n",
+        "1. Upload to Azure\n",
+        "\n",
+        "Before that, we need to connect to Azure Machine Learning Studio:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace, Experiment, Model\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "ws.get_details()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"RegisterModels\"></a>\n",
+        "### Registering Models\n",
+        "\n",
+        "The fairness dashboard is designed to integrate with registered models, so we need to do this for the models we want in the Studio portal. The assumption is that the names of the models specified in the dashboard dictionary correspond to the `id`s (i.e. `<name>:<version>` pairs) of registered models in the workspace. We register each of the models in the `models_dominant` dictionary into the workspace. For this, we have to save each model to a file, and then register that file:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import joblib\n",
+        "import os\n",
+        "\n",
+        "os.makedirs('models', exist_ok=True)\n",
+        "def register_model(name, model):\n",
+        "    print(\"Registering \", name)\n",
+        "    model_path = \"models/{0}.pkl\".format(name)\n",
+        "    joblib.dump(value=model, filename=model_path)\n",
+        "    registered_model = Model.register(model_path=model_path,\n",
+        "                                      model_name=name,\n",
+        "                                      workspace=ws)\n",
+        "    print(\"Registered \", registered_model.id)\n",
+        "    return registered_model.id\n",
+        "\n",
+        "model_name_id_mapping = dict()\n",
+        "for name, model in models_dominant.items():\n",
+        "    m_id = register_model(name, model)\n",
+        "    model_name_id_mapping[name] = m_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, produce new predictions dictionaries, with the updated names:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "predictions_dominant_ids = dict()\n",
+        "for name, y_pred in predictions_dominant.items():\n",
+        "    predictions_dominant_ids[model_name_id_mapping[name]] = y_pred"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"PrecomputeMetrics\"></a>\n",
+        "### Precomputing Metrics\n",
+        "\n",
+        "We create a _dashboard dictionary_ using Fairlearn's `metrics` package. The `_create_group_metric_set` method has arguments similar to the Dashboard constructor, except that the sensitive features are passed as a dictionary (to ensure that names are available), and we must specify the type of prediction. Note that we use the `predictions_dominant_ids` dictionary we just created:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sf = { 'sex': A_test.Sex, 'race': A_test.Race }\n",
+        "\n",
+        "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
+        "\n",
+        "\n",
+        "dash_dict = _create_group_metric_set(y_true=Y_test,\n",
+        "                                     predictions=predictions_dominant_ids,\n",
+        "                                     sensitive_features=sf,\n",
+        "                                     prediction_type='binary_classification')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"DashboardUpload\"></a>\n",
+        "### Uploading the Dashboard\n",
+        "\n",
+        "Now, we import our `contrib` package which contains the routine to perform the upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we can create an Experiment, then a Run, and upload our dashboard to it:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "exp = Experiment(ws, \"Test_Fairlearn_GridSearch_Census_Demo\")\n",
+        "print(exp)\n",
+        "\n",
+        "run = exp.start_logging()\n",
+        "try:\n",
+        "    dashboard_title = \"Dominant Models from GridSearch\"\n",
+        "    upload_id = upload_dashboard_dictionary(run,\n",
+        "                                            dash_dict,\n",
+        "                                            dashboard_name=dashboard_title)\n",
+        "    print(\"\\nUploaded to id: {0}\\n\".format(upload_id))\n",
+        "\n",
+        "    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)\n",
+        "finally:\n",
+        "    run.complete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The dashboard can be viewed in the Run Details page.\n",
+        "\n",
+        "Finally, we can verify that the dashboard dictionary which we downloaded matches our upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(dash_dict == downloaded_dict)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Conclusion\"></a>\n",
+        "## Conclusion\n",
+        "\n",
+        "In this notebook we have demonstrated how to use the `GridSearch` algorithm from Fairlearn to generate a collection of models, and then present them in the fairness dashboard in Azure Machine Learning Studio. Please remember that this notebook has not attempted to discuss the many considerations which should be part of any approach to unfairness mitigation. The [Fairlearn website](http://fairlearn.github.io/) provides that discussion"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "riedgar"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

contrib/fairness/fairlearn-azureml-mitigation.yml

@@ -0,0 +1,8 @@
+name: fairlearn-azureml-mitigation
+dependencies:
+- pip:
+  - azureml-sdk
+  - azureml-contrib-fairness
+  - fairlearn==0.4.6
+  - joblib
+  - shap

contrib/fairness/upload-fairness-dashboard.ipynb

@@ -0,0 +1,494 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/contrib/fairness/upload-fairness-dashboard.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Upload a Fairness Dashboard to Azure Machine Learning Studio\n",
+        "**This notebook shows how to generate and upload a fairness assessment dashboard from Fairlearn to AzureML Studio**\n",
+        "\n",
+        "## Table of Contents\n",
+        "\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Loading the Data](#LoadingData)\n",
+        "1. [Processing the Data](#ProcessingData)\n",
+        "1. [Training Models](#TrainingModels)\n",
+        "1. [Logging in to AzureML](#LoginAzureML)\n",
+        "1. [Registering the Models](#RegisterModels)\n",
+        "1. [Using the Fairlearn Dashboard](#LocalDashboard)\n",
+        "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
+        "    1. Computing Fairness Metrics\n",
+        "    1. Uploading to Azure\n",
+        "1. [Conclusion](#Conclusion)\n",
+        "    \n",
+        "\n",
+        "<a id=\"Introduction\"></a>\n",
+        "## Introduction\n",
+        "\n",
+        "In this notebook, we walk through a simple example of using the `azureml-contrib-fairness` package to upload a collection of fairness statistics for a fairness dashboard. It is an example of integrating the [open source Fairlearn package](https://www.github.com/fairlearn/fairlearn) with Azure Machine Learning. This is not an example of fairness analysis or mitigation - this notebook simply shows how to get a fairness dashboard into the Azure Machine Learning portal. We will load the data and train a couple of simple models. We will then use Fairlearn to generate data for a Fairness dashboard, which we can upload to Azure Machine Learning portal and view there.\n",
+        "\n",
+        "### Setup\n",
+        "\n",
+        "To use this notebook, an Azure Machine Learning workspace is required.\n",
+        "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
+        "This notebook also requires the following packages:\n",
+        "* `azureml-contrib-fairness`\n",
+        "* `fairlearn==0.4.6`\n",
+        "* `joblib`\n",
+        "* `shap`\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n",
+        "<a id=\"LoadingData\"></a>\n",
+        "## Loading the Data\n",
+        "We use the well-known `adult` census dataset, which we load using `shap` (for convenience). We start with a fairly unremarkable set of imports:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn import svm\n",
+        "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
+        "from sklearn.linear_model import LogisticRegression\n",
+        "import pandas as pd\n",
+        "import shap"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we can load the data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_raw, Y = shap.datasets.adult()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can take a look at some of the data. For example, the next cells shows the counts of the different races identified in the dataset:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(X_raw[\"Race\"].value_counts().to_dict())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"ProcessingData\"></a>\n",
+        "## Processing the Data\n",
+        "\n",
+        "With the data loaded, we process it for our needs. First, we extract the sensitive features of interest into `A` (conventionally used in the literature) and put the rest of the feature data into `X`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "A = X_raw[['Sex','Race']]\n",
+        "X = X_raw.drop(labels=['Sex', 'Race'],axis = 1)\n",
+        "X = pd.get_dummies(X)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Next, we apply a standard set of scalings:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sc = StandardScaler()\n",
+        "X_scaled = sc.fit_transform(X)\n",
+        "X_scaled = pd.DataFrame(X_scaled, columns=X.columns)\n",
+        "\n",
+        "le = LabelEncoder()\n",
+        "Y = le.fit_transform(Y)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, we can then split our data into training and test sets, and also make the labels on our test portion of `A` human-readable:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.model_selection import train_test_split\n",
+        "X_train, X_test, Y_train, Y_test, A_train, A_test = train_test_split(X_scaled, \n",
+        "                                                    Y, \n",
+        "                                                    A,\n",
+        "                                                    test_size = 0.2,\n",
+        "                                                    random_state=0,\n",
+        "                                                    stratify=Y)\n",
+        "\n",
+        "# Work around indexing issue\n",
+        "X_train = X_train.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "X_test = X_test.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)\n",
+        "\n",
+        "# Improve labels\n",
+        "A_test.Sex.loc[(A_test['Sex'] == 0)] = 'female'\n",
+        "A_test.Sex.loc[(A_test['Sex'] == 1)] = 'male'\n",
+        "\n",
+        "\n",
+        "A_test.Race.loc[(A_test['Race'] == 0)] = 'Amer-Indian-Eskimo'\n",
+        "A_test.Race.loc[(A_test['Race'] == 1)] = 'Asian-Pac-Islander'\n",
+        "A_test.Race.loc[(A_test['Race'] == 2)] = 'Black'\n",
+        "A_test.Race.loc[(A_test['Race'] == 3)] = 'Other'\n",
+        "A_test.Race.loc[(A_test['Race'] == 4)] = 'White'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"TrainingModels\"></a>\n",
+        "## Training Models\n",
+        "\n",
+        "We now train a couple of different models on our data. The `adult` census dataset is a classification problem - the goal is to predict whether a particular individual exceeds an income threshold. For the purpose of generating a dashboard to upload, it is sufficient to train two basic classifiers. First, a logistic regression classifier:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "lr_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
+        "\n",
+        "lr_predictor.fit(X_train, Y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "And for comparison, a support vector classifier:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "svm_predictor = svm.SVC()\n",
+        "\n",
+        "svm_predictor.fit(X_train, Y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LoginAzureML\"></a>\n",
+        "## Logging in to AzureML\n",
+        "\n",
+        "With our two classifiers trained, we can log into our AzureML workspace:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace, Experiment, Model\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "ws.get_details()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"RegisterModels\"></a>\n",
+        "## Registering the Models\n",
+        "\n",
+        "Next, we register our models. By default, the subroutine which uploads the models checks that the names provided correspond to registered models in the workspace. We define a utility routine to do the registering:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import joblib\n",
+        "import os\n",
+        "\n",
+        "os.makedirs('models', exist_ok=True)\n",
+        "def register_model(name, model):\n",
+        "    print(\"Registering \", name)\n",
+        "    model_path = \"models/{0}.pkl\".format(name)\n",
+        "    joblib.dump(value=model, filename=model_path)\n",
+        "    registered_model = Model.register(model_path=model_path,\n",
+        "                                      model_name=name,\n",
+        "                                      workspace=ws)\n",
+        "    print(\"Registered \", registered_model.id)\n",
+        "    return registered_model.id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, we register the models. For convenience in subsequent method calls, we store the results in a dictionary, which maps the `id` of the registered model (a string in `name:version` format) to the predictor itself:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "model_dict = {}\n",
+        "\n",
+        "lr_reg_id = register_model(\"fairness_linear_regression\", lr_predictor)\n",
+        "model_dict[lr_reg_id] = lr_predictor\n",
+        "svm_reg_id = register_model(\"fairness_svm\", svm_predictor)\n",
+        "model_dict[svm_reg_id] = svm_predictor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LocalDashboard\"></a>\n",
+        "## Using the Fairlearn Dashboard\n",
+        "\n",
+        "We can now examine the fairness of the two models we have training, both as a function of race and (binary) sex. Before uploading the dashboard to the AzureML portal, we will first instantiate a local instance of the Fairlearn dashboard.\n",
+        "\n",
+        "Regardless of the viewing location, the dashboard is based on three things - the true values, the model predictions and the sensitive feature values. The dashboard can use predictions from multiple models and multiple sensitive features if desired (as we are doing here).\n",
+        "\n",
+        "Our first step is to generate a dictionary mapping the `id` of the registered model to the corresponding array of predictions:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ys_pred = {}\n",
+        "for n, p in model_dict.items():\n",
+        "    ys_pred[n] = p.predict(X_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can examine these predictions in a locally invoked Fairlearn dashboard. This can be compared to the dashboard uploaded to the portal (in the next section):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairlearn.widget import FairlearnDashboard\n",
+        "\n",
+        "FairlearnDashboard(sensitive_features=A_test, \n",
+        "                   sensitive_feature_names=['Sex', 'Race'],\n",
+        "                   y_true=Y_test.tolist(),\n",
+        "                   y_pred=ys_pred)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"AzureUpload\"></a>\n",
+        "## Uploading a Fairness Dashboard to Azure\n",
+        "\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairlearnDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. The required stages are therefore:\n",
+        "1. Precompute all the required metrics\n",
+        "1. Upload to Azure\n",
+        "\n",
+        "\n",
+        "### Computing Fairness Metrics\n",
+        "We use Fairlearn to create a dictionary which contains all the data required to display a dashboard. This includes both the raw data (true values, predicted values and sensitive features), and also the fairness metrics. The API is similar to that used to invoke the Dashboard locally. However, there are a few minor changes to the API, and the type of problem being examined (binary classification, regression etc.) needs to be specified explicitly:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sf = { 'Race': A_test.Race, 'Sex': A_test.Sex }\n",
+        "\n",
+        "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
+        "\n",
+        "dash_dict = _create_group_metric_set(y_true=Y_test,\n",
+        "                                     predictions=ys_pred,\n",
+        "                                     sensitive_features=sf,\n",
+        "                                     prediction_type='binary_classification')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The `_create_group_metric_set()` method is currently underscored since its exact design is not yet final in Fairlearn."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Uploading to Azure\n",
+        "\n",
+        "We can now import the `azureml.contrib.fairness` package itself. We will round-trip the data, so there are two required subroutines:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, we can upload the generated dictionary to AzureML. The upload method requires a run, so we first create an experiment and a run. The uploaded dashboard can be seen on the corresponding Run Details page in AzureML Studio. For completeness, we also download the dashboard dictionary which we uploaded."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "exp = Experiment(ws, \"notebook-01\")\n",
+        "print(exp)\n",
+        "\n",
+        "run = exp.start_logging()\n",
+        "try:\n",
+        "    dashboard_title = \"Sample notebook upload\"\n",
+        "    upload_id = upload_dashboard_dictionary(run,\n",
+        "                                            dash_dict,\n",
+        "                                            dashboard_name=dashboard_title)\n",
+        "    print(\"\\nUploaded to id: {0}\\n\".format(upload_id))\n",
+        "\n",
+        "    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)\n",
+        "finally:\n",
+        "    run.complete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, we can verify that the dashboard dictionary which we downloaded matches our upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(dash_dict == downloaded_dict)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Conclusion\"></a>\n",
+        "## Conclusion\n",
+        "\n",
+        "In this notebook we have demonstrated how to generate and upload a fairness dashboard to AzureML Studio. We have not discussed how to analyse the results and apply mitigations. Those topics will be covered elsewhere."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "riedgar"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.8"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}

contrib/fairness/upload-fairness-dashboard.yml

@@ -0,0 +1,8 @@
+name: upload-fairness-dashboard
+dependencies:
+- pip:
+  - azureml-sdk
+  - azureml-contrib-fairness
+  - fairlearn==0.4.6
+  - joblib
+  - shap

how-to-use-azureml/automated-machine-learning/README.md

@@ -144,7 +144,7 @@ jupyter notebook
     - Dataset: forecasting for a bike-sharing
     - Example of training an automated ML forecasting model on multiple time-series
 
-- [auto-ml-forecasting-function.ipynb](forecasting-high-frequency/auto-ml-forecasting-function.ipynb)
+- [auto-ml-forecasting-function.ipynb](forecasting-forecast-function/auto-ml-forecasting-function.ipynb)
     - Example of training an automated ML forecasting model on multiple time-series
 
 - [auto-ml-forecasting-beer-remote.ipynb](forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb)

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb

@@ -105,7 +105,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb

@@ -93,7 +93,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb

@@ -97,7 +97,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -491,8 +491,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run, test_dataset,\n",
-        "                 target_column_name, model_name)"
+        "test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run,\n",
+        "                         train_dataset, test_dataset, target_column_name, model_name)"
       ]
     },
     {

how-to-use-azureml/automated-machine-learning/classification-text-dnn/helper.py

@@ -6,7 +6,7 @@ from azureml.core.run import Run
 
 
 def run_inference(test_experiment, compute_target, script_folder, train_run,
-                  test_dataset, target_column_name, model_name):
+                  train_dataset, test_dataset, target_column_name, model_name):
 
     train_run.download_file('outputs/conda_env_v_1_0_0.yml',
                             'inference/condafile.yml')
@@ -22,7 +22,10 @@ def run_inference(test_experiment, compute_target, script_folder, train_run,
                         '--target_column_name': target_column_name,
                         '--model_name': model_name
                     },
-                    inputs=[test_dataset.as_named_input('test_data')],
+                    inputs=[
+                        train_dataset.as_named_input('train_data'),
+                        test_dataset.as_named_input('test_data')
+                    ],
                     compute_target=compute_target,
                     environment_definition=inference_env)
 

how-to-use-azureml/automated-machine-learning/classification-text-dnn/infer.py

@@ -1,8 +1,11 @@
-import numpy as np
 import argparse
-from azureml.core import Run
+
+import numpy as np
+
 from sklearn.externals import joblib
-from azureml.automl.core.shared import constants, metrics
+
+from azureml.automl.runtime.shared.score import scoring, constants
+from azureml.core import Run
 from azureml.core.model import Model
 
 
@@ -29,22 +32,26 @@ model = joblib.load(model_path)
 run = Run.get_context()
 # get input dataset by name
 test_dataset = run.input_datasets['test_data']
+train_dataset = run.input_datasets['train_data']
 
 X_test_df = test_dataset.drop_columns(columns=[target_column_name]) \
                         .to_pandas_dataframe()
 y_test_df = test_dataset.with_timestamp_columns(None) \
                         .keep_columns(columns=[target_column_name]) \
                         .to_pandas_dataframe()
+y_train_df = test_dataset.with_timestamp_columns(None) \
+                         .keep_columns(columns=[target_column_name]) \
+                         .to_pandas_dataframe()
 
 predicted = model.predict_proba(X_test_df)
 
-# use automl metrics module
-scores = metrics.compute_metrics_classification(
-    np.array(predicted),
-    np.array(y_test_df),
-    class_labels=model.classes_,
-    metrics=list(constants.Metric.SCALAR_CLASSIFICATION_SET)
-)
+# Use the AutoML scoring module
+class_labels = np.unique(np.concatenate((y_train_df.values, y_test_df.values)))
+train_labels = model.classes_
+classification_metrics = list(constants.CLASSIFICATION_SCALAR_SET)
+scores = scoring.score_classification(y_test_df.values, predicted,
+                                      classification_metrics,
+                                      class_labels, train_labels)
 
 print("scores:")
 print(scores)

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.ipynb

@@ -88,7 +88,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb

@@ -114,7 +114,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/infer.py

@@ -1,11 +1,14 @@
-import pandas as pd
-import numpy as np
 import argparse
-from azureml.core import Run
+
+import numpy as np
+import pandas as pd
+
+from pandas.tseries.frequencies import to_offset
 from sklearn.externals import joblib
 from sklearn.metrics import mean_absolute_error, mean_squared_error
-from azureml.automl.core.shared import constants, metrics
-from pandas.tseries.frequencies import to_offset
+
+from azureml.automl.runtime.shared.score import scoring, constants
+from azureml.core import Run
 
 
 def align_outputs(y_predicted, X_trans, X_test, y_test,
@@ -299,12 +302,11 @@ print(df_all[target_column_name])
 print("predicted values:::")
 print(df_all['predicted'])
 
-# use automl metrics module
-scores = metrics.compute_metrics_regression(
-    df_all['predicted'],
-    df_all[target_column_name],
-    list(constants.Metric.SCALAR_REGRESSION_SET),
-    None, None, None)
+# Use the AutoML scoring module
+regression_metrics = list(constants.REGRESSION_SCALAR_SET)
+y_test = np.array(df_all[target_column_name])
+y_pred = np.array(df_all['predicted'])
+scores = scoring.score_regression(y_test, y_pred, regression_metrics)
 
 print("scores:")
 print(scores)

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb

@@ -87,7 +87,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -510,16 +510,16 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.automl.core.shared import constants, metrics\n",
+        "from azureml.automl.core.shared import constants\n",
+        "from azureml.automl.runtime.shared.score import scoring\n",
         "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
         "from matplotlib import pyplot as plt\n",
         "\n",
         "# use automl metrics module\n",
-        "scores = metrics.compute_metrics_regression(\n",
-        "    df_all['predicted'],\n",
-        "    df_all[target_column_name],\n",
-        "    list(constants.Metric.SCALAR_REGRESSION_SET),\n",
-        "    None, None, None)\n",
+        "scores = scoring.score_regression(\n",
+        "    y_test=df_all[target_column_name],\n",
+        "    y_pred=df_all['predicted'],\n",
+        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
         "for key, value in scores.items():    \n",

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb

@@ -97,7 +97,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -465,7 +465,7 @@
       "metadata": {},
       "source": [
         "### Forecast Function\n",
-        "For forecasting, we will use the forecast function instead of the predict function. Using the predict method would result in getting predictions for EVERY horizon the forecaster can predict at. This is useful when training and evaluating the performance of the forecaster at various horizons, but the level of detail is excessive for normal use. Forecast function also can handle more complicated scenarios, see notebook on [high frequency forecasting](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/forecasting-high-frequency/auto-ml-forecasting-function.ipynb)."
+        "For forecasting, we will use the forecast function instead of the predict function. Using the predict method would result in getting predictions for EVERY horizon the forecaster can predict at. This is useful when training and evaluating the performance of the forecaster at various horizons, but the level of detail is excessive for normal use. Forecast function also can handle more complicated scenarios, see the [forecast function notebook](../forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
       ]
     },
     {
@@ -507,15 +507,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.automl.core.shared import constants, metrics\n",
+        "from azureml.automl.core.shared import constants\n",
+        "from azureml.automl.runtime.shared.score import scoring\n",
         "from matplotlib import pyplot as plt\n",
         "\n",
         "# use automl metrics module\n",
-        "scores = metrics.compute_metrics_regression(\n",
-        "    df_all['predicted'],\n",
-        "    df_all[target_column_name],\n",
-        "    list(constants.Metric.SCALAR_REGRESSION_SET),\n",
-        "    None, None, None)\n",
+        "scores = scoring.score_regression(\n",
+        "    y_test=df_all[target_column_name],\n",
+        "    y_pred=df_all['predicted'],\n",
+        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
         "for key, value in scores.items():    \n",
@@ -667,15 +667,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.automl.core.shared import constants, metrics\n",
+        "from azureml.automl.core.shared import constants\n",
+        "from azureml.automl.runtime.shared.score import scoring\n",
         "from matplotlib import pyplot as plt\n",
         "\n",
         "# use automl metrics module\n",
-        "scores = metrics.compute_metrics_regression(\n",
-        "    df_all['predicted'],\n",
-        "    df_all[target_column_name],\n",
-        "    list(constants.Metric.SCALAR_REGRESSION_SET),\n",
-        "    None, None, None)\n",
+        "scores = scoring.score_regression(\n",
+        "    y_test=df_all[target_column_name],\n",
+        "    y_pred=df_all['predicted'],\n",
+        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
         "for key, value in scores.items():    \n",

how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.ipynb

@@ -35,7 +35,6 @@
         "Terminology:\n",
         "* forecast origin: the last period when the target value is known\n",
         "* forecast periods(s): the period(s) for which the value of the target is desired.\n",
-        "* forecast horizon: the number of forecast periods\n",
         "* lookback: how many past periods (before forecast origin) the model function depends on. The larger of number of lags and length of rolling window.\n",
         "* prediction context: `lookback` periods immediately preceding the forecast origin\n",
         "\n",
@@ -95,7 +94,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -720,6 +719,90 @@
         "X_show[['date', 'grain', 'ext_predictor', '_automl_target_col']]\n",
         "# prediction is in _automl_target_col"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Forecasting farther than the maximum horizon <a id=\"recursive forecasting\"></a>\n",
+        "When the forecast destination, or the latest date in the prediction data frame, is farther into the future than the specified maximum horizon, the `forecast()` function will still make point predictions out to the later date using a recursive operation mode. Internally, the method recursively applies the regular forecaster to generate context so that we can forecast further into the future. \n",
+        "\n",
+        "To illustrate the use-case and operation of recursive forecasting, we'll consider an example with a single time-series where the forecasting period directly follows the training period and is twice as long as the maximum horizon given at training time.\n",
+        "\n",
+        "![Recursive_forecast_overview](recursive_forecast_overview_small.png)\n",
+        "\n",
+        "Internally, we apply the forecaster in an iterative manner and finish the forecast task in two interations. In the first iteration, we apply the forecaster and get the prediction for the first max-horizon periods (y_pred1). In the second iteraction, y_pred1 is used as the context to produce the prediction for the next max-horizon periods (y_pred2). The combination of (y_pred1 and y_pred2) gives the results for the total forecast periods. \n",
+        "\n",
+        "A caveat: forecast accuracy will likely be worse the farther we predict into the future since errors are compounded with recursive application of the forecaster.\n",
+        "\n",
+        "![Recursive_forecast_iter1](recursive_forecast_iter1.png)\n",
+        "![Recursive_forecast_iter2](recursive_forecast_iter2.png)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# generate the same kind of test data we trained on, but with a single grain/time-series and test period twice as long as the max_horizon\n",
+        "_, _, X_test_long, y_test_long = get_timeseries(train_len=n_train_periods,\n",
+        "                                                  test_len=max_horizon*2,\n",
+        "                                                  time_column_name=TIME_COLUMN_NAME,\n",
+        "                                                  target_column_name=TARGET_COLUMN_NAME,\n",
+        "                                                  grain_column_name=GRAIN_COLUMN_NAME,\n",
+        "                                                  grains=1)\n",
+        "\n",
+        "print(X_test_long.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].min())\n",
+        "print(X_test_long.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].max())"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# forecast() function will invoke the recursive forecast method internally.\n",
+        "y_pred_long, X_trans_long = fitted_model.forecast(X_test_long)\n",
+        "y_pred_long"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# What forecast() function does in this case is equivalent to iterating it twice over the test set as the following. \n",
+        "y_pred1, _ = fitted_model.forecast(X_test_long[:max_horizon])\n",
+        "y_pred_all, _ = fitted_model.forecast(X_test_long, np.concatenate((y_pred1, np.full(max_horizon, np.nan))))\n",
+        "np.array_equal(y_pred_all, y_pred_long)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Confidence interval and distributional forecasts\n",
+        "AutoML cannot currently estimate forecast errors beyond the maximum horizon set during training, so the `forecast_quantiles()` function will return missing values for quantiles not equal to 0.5 beyond the maximum horizon. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "fitted_model.forecast_quantiles(X_test_long)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Similarly with the simple senarios illustrated above, forecasting farther than the max horizon in other senarios like 'multiple grain', 'Destination-date forecast', and 'forecast away from the training data' are also automatically handled by the `forecast()` function. "
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb

@@ -82,7 +82,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -336,7 +336,11 @@
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
+      "metadata": {
+        "tags": [
+          "sample-featurizationconfig-remarks"
+        ]
+      },
       "outputs": [],
       "source": [
         "featurization_config = FeaturizationConfig()\n",
@@ -545,7 +549,7 @@
       "source": [
         "If you are used to scikit pipelines, perhaps you expected `predict(X_test)`. However, forecasting requires a more general interface that also supplies the past target `y` values. Please use `forecast(X,y)` as `predict(X)` is reserved for internal purposes on forecasting models.\n",
         "\n",
-        "The [forecast function notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/forecasting-high-frequency/auto-ml-forecasting-function.ipynb) demonstrates the use of the forecast function for a variety of use cases. Also, please see the [API documentation for the forecast function](https://docs.microsoft.com/en-us/python/api/azureml-automl-runtime/azureml.automl.runtime.shared.model_wrappers.forecastingpipelinewrapper?view=azure-ml-py#forecast-x-pred--typing-union-pandas-core-frame-dataframe--nonetype----none--y-pred--typing-union-pandas-core-frame-dataframe--numpy-ndarray--nonetype----none--forecast-destination--typing-union-pandas--libs-tslibs-timestamps-timestamp--nonetype----none--ignore-data-errors--bool---false-----typing-tuple-numpy-ndarray--pandas-core-frame-dataframe-)."
+        "The [forecast function notebook](../forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
       ]
     },
     {
@@ -576,15 +580,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.automl.core.shared import constants, metrics\n",
+        "from azureml.automl.core.shared import constants\n",
+        "from azureml.automl.runtime.shared.score import scoring\n",
         "from matplotlib import pyplot as plt\n",
         "\n",
-        "# use automl metrics module\n",
-        "scores = metrics.compute_metrics_regression(\n",
-        "    df_all['predicted'],\n",
-        "    df_all[target_column_name],\n",
-        "    list(constants.Metric.SCALAR_REGRESSION_SET),\n",
-        "    None, None, None)\n",
+        "# use automl scoring module\n",
+        "scores = scoring.score_regression(\n",
+        "    y_test=df_all[target_column_name],\n",
+        "    y_pred=df_all['predicted'],\n",
+        "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
         "for key, value in scores.items():    \n",

how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb

@@ -28,7 +28,8 @@
         "1. [Setup](#Setup)\n",
         "1. [Train](#Train)\n",
         "1. [Results](#Results)\n",
-        "1. [Test](#Test)\n",
+        "1. [Test](#Tests)\n",
+        "1. [Explanation](#Explanation)\n",
         "1. [Acknowledgements](#Acknowledgements)"
       ]
     },
@@ -49,9 +50,9 @@
         "2. Configure AutoML using `AutoMLConfig`.\n",
         "3. Train the model.\n",
         "4. Explore the results.\n",
-        "5. Visualization model's feature importance in azure portal\n",
-        "6. Explore any model's explanation and explore feature importance in azure portal\n",
-        "7. Test the fitted model."
+        "5. Test the fitted model.\n",
+        "6. Explore any model's explanation and explore feature importance in azure portal.\n",
+        "7. Create an AKS cluster, deploy the webservice of AutoML scoring model and the explainer model to the AKS and consume the web service."
       ]
     },
     {
@@ -95,7 +96,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -255,9 +256,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Analyze results\n",
+        "### Analyze results\n",
         "\n",
-        "### Retrieve the Best Model\n",
+        "#### Retrieve the Best Model\n",
         "\n",
         "Below we select the best pipeline from our iterations. The `get_output` method on `automl_classifier` returns the best run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
       ]
@@ -284,9 +285,80 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Best Model 's explanation\n",
-        "Retrieve the explanation from the best_run which includes explanations for engineered features and raw features.\n",
+        "## Tests\n",
         "\n",
+        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# convert the test data to dataframe\n",
+        "X_test_df = validation_data.drop_columns(columns=[label_column_name]).to_pandas_dataframe()\n",
+        "y_test_df = validation_data.keep_columns(columns=[label_column_name], validate=True).to_pandas_dataframe()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# call the predict functions on the model\n",
+        "y_pred = fitted_model.predict(X_test_df)\n",
+        "y_pred"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Calculate metrics for the prediction\n",
+        "\n",
+        "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
+        "from the trained model that was returned."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn.metrics import confusion_matrix\n",
+        "import numpy as np\n",
+        "import itertools\n",
+        "\n",
+        "cf =confusion_matrix(y_test_df.values,y_pred)\n",
+        "plt.imshow(cf,cmap=plt.cm.Blues,interpolation='nearest')\n",
+        "plt.colorbar()\n",
+        "plt.title('Confusion Matrix')\n",
+        "plt.xlabel('Predicted')\n",
+        "plt.ylabel('Actual')\n",
+        "class_labels = ['False','True']\n",
+        "tick_marks = np.arange(len(class_labels))\n",
+        "plt.xticks(tick_marks,class_labels)\n",
+        "plt.yticks([-0.5,0,1,1.5],['','False','True',''])\n",
+        "# plotting text value inside cells\n",
+        "thresh = cf.max() / 2.\n",
+        "for i,j in itertools.product(range(cf.shape[0]),range(cf.shape[1])):\n",
+        "    plt.text(j,i,format(cf[i,j],'d'),horizontalalignment='center',color='white' if cf[i,j] >thresh else 'black')\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Explanation\n",
+        "In this section, we will show how to compute model explanations and visualize the explanations using azureml-explain-model package. We will also show how to run the automl model and the explainer model through deploying an AKS web service.\n",
+        "\n",
+        "Besides retrieving an existing model explanation for an AutoML model, you can also explain your AutoML model with different test data. The following steps will allow you to compute and visualize engineered feature importance based on your test data.\n",
+        "\n",
+        "### Run the explanation\n",
         "#### Download engineered feature importance from artifact store\n",
         "You can use ExplanationClient to download the engineered feature explanations from the artifact store of the best_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features."
       ]
@@ -303,14 +375,6 @@
         "print(\"You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + best_run.get_portal_url())"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Explanations\n",
-        "In this section, we will show how to compute model explanations and visualize the explanations using azureml-explain-model package. Besides retrieving an existing model explanation for an AutoML model, you can also explain your AutoML model with different test data. The following steps will allow you to compute and visualize engineered feature importance based on your test data."
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -403,6 +467,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "# Compute the engineered explanations\n",
         "engineered_explanations = explainer.explain(['local', 'global'], eval_dataset=automl_explainer_setup_obj.X_test_transform)\n",
         "print(engineered_explanations.get_feature_importance_dict())\n",
         "print(\"You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + automl_run.get_portal_url())"
@@ -412,41 +477,37 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Test the fitted model\n",
+        "#### Initialize the scoring Explainer, save and upload it for later use in scoring explanation"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.explain.model.scoring.scoring_explainer import TreeScoringExplainer\n",
+        "import joblib\n",
         "\n",
-        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# convert the test data to dataframe\n",
-        "X_test_df = validation_data.drop_columns(columns=[label_column_name]).to_pandas_dataframe()\n",
-        "y_test_df = validation_data.keep_columns(columns=[label_column_name], validate=True).to_pandas_dataframe()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# call the predict functions on the model\n",
-        "y_pred = fitted_model.predict(X_test_df)\n",
-        "y_pred"
+        "# Initialize the ScoringExplainer\n",
+        "scoring_explainer = TreeScoringExplainer(explainer.explainer, feature_maps=[automl_explainer_setup_obj.feature_map])\n",
+        "\n",
+        "# Pickle scoring explainer locally to './scoring_explainer.pkl'\n",
+        "scoring_explainer_file_name = 'scoring_explainer.pkl'\n",
+        "with open(scoring_explainer_file_name, 'wb') as stream:\n",
+        "    joblib.dump(scoring_explainer, stream)\n",
+        "\n",
+        "# Upload the scoring explainer to the automl run\n",
+        "automl_run.upload_file('outputs/scoring_explainer.pkl', scoring_explainer_file_name)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Calculate metrics for the prediction\n",
+        "### Deploying the scoring and explainer models to a web service to Azure Kubernetes Service (AKS)\n",
         "\n",
-        "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
-        "from the trained model that was returned."
+        "We use the TreeScoringExplainer from azureml.explain.model package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service."
       ]
     },
     {
@@ -455,25 +516,238 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from sklearn.metrics import confusion_matrix\n",
-        "import numpy as np\n",
-        "import itertools\n",
+        "# Register trained automl model present in the 'outputs' folder in the artifacts\n",
+        "original_model = automl_run.register_model(model_name='automl_model', \n",
+        "                                           model_path='outputs/model.pkl')\n",
+        "scoring_explainer_model = automl_run.register_model(model_name='scoring_explainer',\n",
+        "                                                    model_path='outputs/scoring_explainer.pkl')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Create the conda dependencies for setting up the service\n",
         "\n",
-        "cf =confusion_matrix(y_test_df.values,y_pred)\n",
-        "plt.imshow(cf,cmap=plt.cm.Blues,interpolation='nearest')\n",
-        "plt.colorbar()\n",
-        "plt.title('Confusion Matrix')\n",
-        "plt.xlabel('Predicted')\n",
-        "plt.ylabel('Actual')\n",
-        "class_labels = ['False','True']\n",
-        "tick_marks = np.arange(len(class_labels))\n",
-        "plt.xticks(tick_marks,class_labels)\n",
-        "plt.yticks([-0.5,0,1,1.5],['','False','True',''])\n",
-        "# plotting text value inside cells\n",
-        "thresh = cf.max() / 2.\n",
-        "for i,j in itertools.product(range(cf.shape[0]),range(cf.shape[1])):\n",
-        "    plt.text(j,i,format(cf[i,j],'d'),horizontalalignment='center',color='white' if cf[i,j] >thresh else 'black')\n",
-        "plt.show()"
+        "We need to create the conda dependencies comprising of the azureml-explain-model, azureml-train-automl and azureml-defaults packages."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.automl.core.shared import constants\n",
+        "from azureml.core.environment import Environment\n",
+        "\n",
+        "automl_run.download_file(constants.CONDA_ENV_FILE_PATH, 'myenv.yml')\n",
+        "myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"myenv.yml\")\n",
+        "myenv"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Write the Entry Script\n",
+        "Write the script that will be used to predict on your model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile score.py\n",
+        "import numpy as np\n",
+        "import pandas as pd\n",
+        "import os\n",
+        "import pickle\n",
+        "import azureml.train.automl\n",
+        "import azureml.explain.model\n",
+        "from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \\\n",
+        "    automl_setup_model_explanations\n",
+        "import joblib\n",
+        "from azureml.core.model import Model\n",
+        "\n",
+        "\n",
+        "def init():\n",
+        "    global automl_model\n",
+        "    global scoring_explainer\n",
+        "\n",
+        "    # Retrieve the path to the model file using the model name\n",
+        "    # Assume original model is named original_prediction_model\n",
+        "    automl_model_path = Model.get_model_path('automl_model')\n",
+        "    scoring_explainer_path = Model.get_model_path('scoring_explainer')\n",
+        "\n",
+        "    automl_model = joblib.load(automl_model_path)\n",
+        "    scoring_explainer = joblib.load(scoring_explainer_path)\n",
+        "\n",
+        "\n",
+        "def run(raw_data):\n",
+        "    data = pd.read_json(raw_data, orient='records')    \n",
+        "    # Make prediction\n",
+        "    predictions = automl_model.predict(data)\n",
+        "    # Setup for inferencing explanations\n",
+        "    automl_explainer_setup_obj = automl_setup_model_explanations(automl_model,\n",
+        "                                                                 X_test=data, task='classification')\n",
+        "    # Retrieve model explanations for engineered explanations\n",
+        "    engineered_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform)    \n",
+        "    # You can return any data type as long as it is JSON-serializable\n",
+        "    return {'predictions': predictions.tolist(),\n",
+        "            'engineered_local_importance_values': engineered_local_importance_values}\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Create the InferenceConfig \n",
+        "Create the inference config that will be used when deploying the model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.model import InferenceConfig\n",
+        "\n",
+        "inf_config = InferenceConfig(entry_script='score.py', environment=myenv)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Provision the AKS Cluster\n",
+        "This is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, AksCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your cluster.\n",
+        "aks_name = 'scoring-explain'\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    aks_target = ComputeTarget(workspace=ws, name=aks_name)\n",
+        "    print('Found existing cluster, use it.')\n",
+        "except ComputeTargetException:\n",
+        "    prov_config = AksCompute.provisioning_configuration(vm_size='STANDARD_D3_V2')\n",
+        "    aks_target = ComputeTarget.create(workspace=ws, \n",
+        "                                      name=aks_name,\n",
+        "                                      provisioning_configuration=prov_config)\n",
+        "aks_target.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Deploy web service to AKS"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Set the web service configuration (using default here)\n",
+        "from azureml.core.webservice import AksWebservice\n",
+        "from azureml.core.model import Model\n",
+        "\n",
+        "aks_config = AksWebservice.deploy_configuration()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "aks_service_name ='model-scoring-local-aks'\n",
+        "\n",
+        "aks_service = Model.deploy(workspace=ws,\n",
+        "                           name=aks_service_name,\n",
+        "                           models=[scoring_explainer_model, original_model],\n",
+        "                           inference_config=inf_config,\n",
+        "                           deployment_config=aks_config,\n",
+        "                           deployment_target=aks_target)\n",
+        "\n",
+        "aks_service.wait_for_deployment(show_output = True)\n",
+        "print(aks_service.state)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### View the service logs"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "aks_service.get_logs()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Consume the web service using run method to do the scoring and explanation of scoring.\n",
+        "We test the web sevice by passing data. Run() method retrieves API keys behind the scenes to make sure that call is authenticated."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Serialize the first row of the test data into json\n",
+        "X_test_json = X_test_df[:1].to_json(orient='records')\n",
+        "print(X_test_json)\n",
+        "\n",
+        "# Call the service to get the predictions and the engineered and raw explanations\n",
+        "output = aks_service.run(X_test_json)\n",
+        "\n",
+        "# Print the predicted value\n",
+        "print('predictions:\\n{}\\n'.format(output['predictions']))\n",
+        "# Print the engineered feature importances for the predicted value\n",
+        "print('engineered_local_importance_values:\\n{}\\n'.format(output['engineered_local_importance_values']))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Clean up\n",
+        "Delete the service."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "aks_service.delete()"
       ]
     },
     {

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/auto-ml-regression-explanation-featurization.ipynb

@@ -98,7 +98,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -242,7 +242,11 @@
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
+      "metadata": {
+        "tags": [
+          "sample-featurizationconfig-remarks2"
+        ]
+      },
       "outputs": [],
       "source": [
         "featurization_config = FeaturizationConfig()\n",
@@ -260,7 +264,11 @@
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
+      "metadata": {
+        "tags": [
+          "sample-featurizationconfig-remarks3"
+        ]
+      },
       "outputs": [],
       "source": [
         "automl_settings = {\n",

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/score_explain.py

@@ -7,7 +7,7 @@ import azureml.train.automl
 import azureml.explain.model
 from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \
     automl_setup_model_explanations
-from sklearn.externals import joblib
+import joblib
 from azureml.core.model import Model
 
 

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/train_explainer.py

@@ -4,15 +4,14 @@ import os
 
 from azureml.core.run import Run
 from azureml.core.experiment import Experiment
-from sklearn.externals import joblib
 from azureml.core.dataset import Dataset
 from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \
     automl_setup_model_explanations, automl_check_model_if_explainable
 from azureml.explain.model.mimic.models.lightgbm_model import LGBMExplainableModel
 from azureml.explain.model.mimic_wrapper import MimicWrapper
 from azureml.automl.core.shared.constants import MODEL_PATH
-from azureml.explain.model.scoring.scoring_explainer import TreeScoringExplainer, save
-
+from azureml.explain.model.scoring.scoring_explainer import TreeScoringExplainer
+import joblib
 
 OUTPUT_DIR = './outputs/'
 os.makedirs(OUTPUT_DIR, exist_ok=True)
@@ -74,7 +73,8 @@ print("Engineered and raw explanations computed successfully")
 scoring_explainer = TreeScoringExplainer(explainer.explainer, feature_maps=[automl_explainer_setup_obj.feature_map])
 
 # Pickle scoring explainer locally
-save(scoring_explainer, exist_ok=True)
+with open('scoring_explainer.pkl', 'wb') as stream:
+    joblib.dump(scoring_explainer, stream)
 
 # Upload the scoring explainer to the automl run
 automl_run.upload_file('outputs/scoring_explainer.pkl', 'scoring_explainer.pkl')

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.ipynb

@@ -92,7 +92,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/deployment/deploy-to-local/myenv.yml

@@ -3,6 +3,6 @@ dependencies:
   - python=3.6.2
   - pip:
     - azureml-defaults
-    - scikit-learn
+    - scikit-learn==0.19.1
     - numpy
     - inference-schema[numpy-support]

how-to-use-azureml/deployment/deploy-to-local/register-model-deploy-local.ipynb

@@ -233,7 +233,8 @@
         "    'inference-schema[numpy-support]',\n",
         "    'joblib',\n",
         "    'numpy',\n",
-        "    'scikit-learn'\n",
+        "    'scikit-learn==0.19.1',\n",
+        "    'scipy'\n",
         "])\n",
         "inference_config = InferenceConfig(entry_script='score.py', environment=environment)\n",
         "# if cpu and memory_in_gb parameters are not provided\n",

how-to-use-azureml/deployment/deploy-with-controlled-rollout/deploy-aks-with-controlled-rollout.ipynb

@@ -108,9 +108,9 @@
         "environment.python.conda_dependencies = CondaDependencies.create(pip_packages=[\n",
         "    'azureml-defaults',\n",
         "    'inference-schema[numpy-support]',\n",
-        "    'joblib',\n",
         "    'numpy',\n",
-        "    'scikit-learn'\n",
+        "    'scikit-learn==0.19.1',\n",
+        "    'scipy'\n",
         "])"
       ]
     },

how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb

@@ -5,7 +5,7 @@
       "metadata": {},
       "source": [
         "# Enabling App Insights for Services in Production\n",
-        "With this notebook, you can learn how to enable App Insights for standard service monitoring, plus, we provide examples for doing custom logging within a scoring files in a model. \n",
+        "With this notebook, you can learn how to enable App Insights for standard service monitoring, plus, we provide examples for doing custom logging within a scoring files in a model.\n",
         "\n",
         "\n",
         "## What does Application Insights monitor?\n",
@@ -45,11 +45,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "import azureml.core\n",
+        "import json\n",
+        "\n",
         "from azureml.core import Workspace\n",
         "from azureml.core.compute import AksCompute, ComputeTarget\n",
         "from azureml.core.webservice import AksWebservice\n",
-        "import azureml.core\n",
-        "import json\n",
+        "\n",
         "print(azureml.core.VERSION)"
       ]
     },
@@ -67,7 +69,7 @@
       "outputs": [],
       "source": [
         "ws = Workspace.from_config()\n",
-        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
       ]
     },
     {
@@ -84,13 +86,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "#Register the model\n",
-        "from azureml.core.model import Model\n",
-        "model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
-        "                       model_name = \"sklearn_regression_model.pkl\", # this is the name the model is registered as\n",
-        "                       tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
-        "                       description = \"Ridge regression model to predict diabetes\",\n",
-        "                       workspace = ws)\n",
+        "from azureml.core import Model\n",
+        "\n",
+        "model = Model.register(model_path=\"sklearn_regression_model.pkl\", # This points to a local file.\n",
+        "                       model_name=\"sklearn_regression_model.pkl\", # This is the name the model is registered as.\n",
+        "                       tags={'area': \"diabetes\", 'type': \"regression\"},\n",
+        "                       description=\"Ridge regression model to predict diabetes\",\n",
+        "                       workspace=ws)\n",
         "\n",
         "print(model.name, model.description, model.version)"
       ]
@@ -120,7 +122,7 @@
         "import os\n",
         "import pickle\n",
         "import json\n",
-        "import numpy \n",
+        "import numpy\n",
         "from sklearn.externals import joblib\n",
         "from sklearn.linear_model import Ridge\n",
         "import time\n",
@@ -129,15 +131,15 @@
         "    global model\n",
         "    #Print statement for appinsights custom traces:\n",
         "    print (\"model initialized\" + time.strftime(\"%H:%M:%S\"))\n",
-        "    \n",
+        "\n",
         "    # AZUREML_MODEL_DIR is an environment variable created during deployment.\n",
         "    # It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)\n",
         "    # For multiple models, it points to the folder containing all deployed models (./azureml-models)\n",
         "    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'sklearn_regression_model.pkl')\n",
-        "    \n",
+        "\n",
         "    # deserialize the model file back into a sklearn model\n",
         "    model = joblib.load(model_path)\n",
-        "  \n",
+        "\n",
         "\n",
         "# note you can pass in multiple rows for scoring\n",
         "def run(raw_data):\n",
@@ -168,7 +170,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.conda_dependencies import CondaDependencies \n",
+        "from azureml.core.conda_dependencies import CondaDependencies\n",
         "\n",
         "myenv = CondaDependencies.create(conda_packages=['numpy','scikit-learn'],\n",
         "                                 pip_packages=['azureml-defaults'])\n",
@@ -190,9 +192,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.model import InferenceConfig\n",
         "from azureml.core.environment import Environment\n",
-        "\n",
+        "from azureml.core.model import InferenceConfig\n",
         "\n",
         "myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"myenv.yml\")\n",
         "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)"
@@ -213,11 +214,11 @@
       "source": [
         "from azureml.core.webservice import AciWebservice\n",
         "\n",
-        "aci_deployment_config = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
-        "                                               memory_gb = 1, \n",
-        "                                               tags = {'area': \"diabetes\", 'type': \"regression\"}, \n",
-        "                                               description = 'Predict diabetes using regression model',\n",
-        "                                               enable_app_insights = True)"
+        "aci_deployment_config = AciWebservice.deploy_configuration(cpu_cores=1,\n",
+        "                                                           memory_gb=1,\n",
+        "                                                           tags={'area': \"diabetes\", 'type': \"regression\"},\n",
+        "                                                           description=\"Predict diabetes using regression model\",\n",
+        "                                                           enable_app_insights=True)"
       ]
     },
     {
@@ -226,29 +227,14 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.webservice import Webservice\n",
+        "aci_service_name = \"aci-service-appinsights\"\n",
+        "\n",
+        "aci_service = Model.deploy(ws, aci_service_name, [model], inference_config, aci_deployment_config, overwrite=True)\n",
+        "aci_service.wait_for_deployment(show_output=True)\n",
         "\n",
-        "aci_service_name = 'my-aci-service-4'\n",
-        "aci_service = Model.deploy(ws, aci_service_name, [model], inference_config, aci_deployment_config)\n",
-        "aci_service.wait_for_deployment(True)\n",
         "print(aci_service.state)"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "%%time\n",
-        "\n",
-        "test_sample = json.dumps({'data': [\n",
-        "    [1,28,13,45,54,6,57,8,8,10], \n",
-        "    [101,9,8,37,6,45,4,3,2,41]\n",
-        "]})\n",
-        "test_sample = bytes(test_sample,encoding='utf8')"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -256,7 +242,15 @@
       "outputs": [],
       "source": [
         "if aci_service.state == \"Healthy\":\n",
-        "    prediction = aci_service.run(input_data=test_sample)\n",
+        "    test_sample = json.dumps({\n",
+        "        \"data\": [\n",
+        "            [1,28,13,45,54,6,57,8,8,10],\n",
+        "            [101,9,8,37,6,45,4,3,2,41]\n",
+        "        ]\n",
+        "    })\n",
+        "\n",
+        "    prediction = aci_service.run(test_sample)\n",
+        "\n",
         "    print(prediction)\n",
         "else:\n",
         "    raise ValueError(\"Service deployment isn't healthy, can't call the service. Error: \", aci_service.error)"
@@ -282,14 +276,21 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Use the default configuration (can also provide parameters to customize)\n",
-        "prov_config = AksCompute.provisioning_configuration()\n",
+        "from azureml.exceptions import ComputeTargetException\n",
         "\n",
-        "aks_name = 'my-aks-test3' \n",
-        "# Create the cluster\n",
-        "aks_target = ComputeTarget.create(workspace = ws, \n",
-        "                                  name = aks_name, \n",
-        "                                  provisioning_configuration = prov_config)"
+        "aks_name = \"my-aks\"\n",
+        "\n",
+        "try:\n",
+        "    aks_target = ComputeTarget(ws, aks_name)\n",
+        "    print(\"Using existing AKS cluster {}.\".format(aks_name))\n",
+        "except ComputeTargetException:\n",
+        "    print(\"Creating a new AKS cluster {}.\".format(aks_name))\n",
+        "\n",
+        "    # Use the default configuration (can also provide parameters to customize).\n",
+        "    prov_config = AksCompute.provisioning_configuration()\n",
+        "    aks_target = ComputeTarget.create(workspace=ws,\n",
+        "                                      name=aks_name,\n",
+        "                                      provisioning_configuration=prov_config)"
       ]
     },
     {
@@ -299,7 +300,8 @@
       "outputs": [],
       "source": [
         "%%time\n",
-        "aks_target.wait_for_completion(show_output = True)"
+        "if aks_target.provisioning_state != \"Succeeded\":\n",
+        "    aks_target.wait_for_completion(show_output=True)"
       ]
     },
     {
@@ -323,13 +325,13 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "```python \n",
+        "```python\n",
         "%%time\n",
         "resource_id = '/subscriptions/<subscriptionid>/resourcegroups/<resourcegroupname>/providers/Microsoft.ContainerService/managedClusters/<aksservername>'\n",
         "create_name= 'myaks4'\n",
         "attach_config = AksCompute.attach_configuration(resource_id=resource_id)\n",
-        "aks_target = ComputeTarget.attach(workspace = ws, \n",
-        "                                  name = create_name, \n",
+        "aks_target = ComputeTarget.attach(workspace=ws,\n",
+        "                                  name=create_name,\n",
         "                                  attach_configuration=attach_config)\n",
         "## Wait for the operation to complete\n",
         "aks_target.wait_for_provisioning(True)```"
@@ -349,7 +351,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "#Set the web service configuration\n",
+        "# Set the web service configuration.\n",
         "aks_deployment_config = AksWebservice.deploy_configuration(enable_app_insights=True)"
       ]
     },
@@ -366,15 +368,16 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "if aks_target.provisioning_state== \"Succeeded\": \n",
-        "    aks_service_name ='aks-w-dc5'\n",
+        "if aks_target.provisioning_state == \"Succeeded\":\n",
+        "    aks_service_name = \"aks-service-appinsights\"\n",
         "    aks_service = Model.deploy(ws,\n",
-        "                               aks_service_name, \n",
-        "                               [model], \n",
-        "                               inference_config, \n",
-        "                               aks_deployment_config, \n",
-        "                               deployment_target = aks_target)    \n",
-        "    aks_service.wait_for_deployment(show_output = True)\n",
+        "                               aks_service_name,\n",
+        "                               [model],\n",
+        "                               inference_config,\n",
+        "                               aks_deployment_config,\n",
+        "                               deployment_target=aks_target,\n",
+        "                               overwrite=True)\n",
+        "    aks_service.wait_for_deployment(show_output=True)\n",
         "    print(aks_service.state)\n",
         "else:\n",
         "    raise ValueError(\"AKS provisioning failed. Error: \", aks_service.error)"
@@ -395,13 +398,14 @@
       "source": [
         "%%time\n",
         "\n",
-        "test_sample = json.dumps({'data': [\n",
-        "    [1,28,13,45,54,6,57,8,8,10], \n",
-        "    [101,9,8,37,6,45,4,3,2,41]\n",
-        "]})\n",
-        "test_sample = bytes(test_sample,encoding='utf8')\n",
-        "\n",
         "if aks_service.state == \"Healthy\":\n",
+        "    test_sample = json.dumps({\n",
+        "        \"data\": [\n",
+        "            [1,28,13,45,54,6,57,8,8,10],\n",
+        "            [101,9,8,37,6,45,4,3,2,41]\n",
+        "        ]\n",
+        "    })\n",
+        "\n",
         "    prediction = aks_service.run(input_data=test_sample)\n",
         "    print(prediction)\n",
         "else:\n",
@@ -435,7 +439,7 @@
       "outputs": [],
       "source": [
         "aks_service.update(enable_app_insights=False)\n",
-        "aks_service.wait_for_deployment(show_output = True)"
+        "aks_service.wait_for_deployment(show_output=True)"
       ]
     },
     {

how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb

@@ -115,6 +115,11 @@
         "# Convert from CoreML into ONNX\n",
         "onnx_model = onnxmltools.convert_coreml(coreml_model, 'TinyYOLOv2')\n",
         "\n",
+        "# Fix the preprocessor bias in the ImageScaler\n",
+        "for init in onnx_model.graph.initializer:\n",
+        "    if init.name == 'scalerPreprocessor_bias':\n",
+        "        init.dims[1] = 1\n",
+        "\n",
         "# Save ONNX model\n",
         "onnxmltools.utils.save_model(onnx_model, 'tinyyolov2.onnx')\n",
         "\n",
@@ -255,7 +260,7 @@
       "source": [
         "from azureml.core.conda_dependencies import CondaDependencies \n",
         "\n",
-        "myenv = CondaDependencies.create(pip_packages=[\"numpy\", \"onnxruntime==0.4.0\", \"azureml-core\", \"azureml-defaults\"])\n",
+        "myenv = CondaDependencies.create(pip_packages=[\"numpy\", \"onnxruntime\", \"azureml-core\", \"azureml-defaults\"])\n",
         "\n",
         "with open(\"myenv.yml\",\"w\") as f:\n",
         "    f.write(myenv.serialize_to_string())"
@@ -316,7 +321,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "aci_service_name = 'my-aci-service-15ad'\n",
+        "aci_service_name = 'my-aci-service-tiny-yolo'\n",
         "print(\"Service\", aci_service_name)\n",
         "aci_service = Model.deploy(ws, aci_service_name, [model], inference_config, aciconfig)\n",
         "aci_service.wait_for_deployment(True)\n",

how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.yml

@@ -4,4 +4,5 @@ dependencies:
   - azureml-sdk
   - numpy
   - git+https://github.com/apple/coremltools@v2.1
+  - onnx<1.7.0
   - onnxmltools

how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.yml

@@ -5,5 +5,5 @@ dependencies:
   - azureml-widgets
   - matplotlib
   - numpy
-  - onnx
+  - onnx<1.7.0
   - opencv-python-headless

how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.yml

@@ -5,5 +5,5 @@ dependencies:
   - azureml-widgets
   - matplotlib
   - numpy
-  - onnx
+  - onnx<1.7.0
   - opencv-python-headless

how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks-ssl.ipynb

@@ -0,0 +1,354 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Deploying a web service to Azure Kubernetes Service (AKS)\n",
+        "This notebook shows the steps for deploying a service: registering a model, provisioning a cluster with ssl (one time action), and deploying a service to it. \n",
+        "We then test and delete the service, image and model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace\n",
+        "from azureml.core.compute import AksCompute, ComputeTarget\n",
+        "from azureml.core.webservice import Webservice, AksWebservice\n",
+        "from azureml.core.model import Model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import azureml.core\n",
+        "print(azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Get workspace\n",
+        "Load existing workspace from the config file info."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.workspace import Workspace\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Register the model\n",
+        "Register an existing trained model, add descirption and tags."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "#Register the model\n",
+        "from azureml.core.model import Model\n",
+        "model = Model.register(model_path = \"sklearn_regression_model.pkl\", # this points to a local file\n",
+        "                       model_name = \"sklearn_model\", # this is the name the model is registered as\n",
+        "                       tags = {'area': \"diabetes\", 'type': \"regression\"},\n",
+        "                       description = \"Ridge regression model to predict diabetes\",\n",
+        "                       workspace = ws)\n",
+        "\n",
+        "print(model.name, model.description, model.version)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Create the Environment\n",
+        "Create an environment that the model will be deployed with"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Environment\n",
+        "from azureml.core.conda_dependencies import CondaDependencies \n",
+        "\n",
+        "conda_deps = CondaDependencies.create(conda_packages=['numpy', 'scikit-learn==0.19.1', 'scipy'], pip_packages=['azureml-defaults', 'inference-schema'])\n",
+        "myenv = Environment(name='myenv')\n",
+        "myenv.python.conda_dependencies = conda_deps"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Use a custom Docker image\n",
+        "\n",
+        "You can also specify a custom Docker image to be used as base image if you don't want to use the default base image provided by Azure ML. Please make sure the custom Docker image has Ubuntu >= 16.04, Conda >= 4.5.\\* and Python(3.5.\\* or 3.6.\\*).\n",
+        "\n",
+        "Only supported with `python` runtime.\n",
+        "```python\n",
+        "# use an image available in public Container Registry without authentication\n",
+        "myenv.docker.base_image = \"mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda\"\n",
+        "\n",
+        "# or, use an image available in a private Container Registry\n",
+        "myenv.docker.base_image = \"myregistry.azurecr.io/mycustomimage:1.0\"\n",
+        "myenv.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
+        "myenv.docker.base_image_registry.username = \"username\"\n",
+        "myenv.docker.base_image_registry.password = \"password\"\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Write the Entry Script\n",
+        "Write the script that will be used to predict on your model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile score.py\n",
+        "import os\n",
+        "import pickle\n",
+        "import json\n",
+        "import numpy\n",
+        "from sklearn.externals import joblib\n",
+        "from sklearn.linear_model import Ridge\n",
+        "from inference_schema.schema_decorators import input_schema, output_schema\n",
+        "from inference_schema.parameter_types.standard_py_parameter_type import StandardPythonParameterType\n",
+        "\n",
+        "def init():\n",
+        "    global model\n",
+        "    # AZUREML_MODEL_DIR is an environment variable created during deployment.\n",
+        "    # It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)\n",
+        "    # For multiple models, it points to the folder containing all deployed models (./azureml-models)\n",
+        "    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'sklearn_regression_model.pkl')\n",
+        "    # deserialize the model file back into a sklearn model\n",
+        "    model = joblib.load(model_path)\n",
+        "\n",
+        "\n",
+        "standard_sample_input = {'a': 10, 'b': 9, 'c': 8, 'd': 7, 'e': 6, 'f': 5, 'g': 4, 'h': 3, 'i': 2, 'j': 1 }\n",
+        "standard_sample_output = {'outcome': 1}\n",
+        "\n",
+        "@input_schema('param', StandardPythonParameterType(standard_sample_input))\n",
+        "@output_schema(StandardPythonParameterType(standard_sample_output))\n",
+        "def run(param):\n",
+        "    try:\n",
+        "        raw_data = [param['a'], param['b'], param['c'], param['d'], param['e'], param['f'], param['g'], param['h'], param['i'], param['j']]\n",
+        "        data = numpy.array([raw_data])\n",
+        "        result = model.predict(data)\n",
+        "        return { 'outcome' : result[0] }\n",
+        "    except Exception as e:\n",
+        "        error = str(e)\n",
+        "        return error"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Create the InferenceConfig\n",
+        "Create the inference config that will be used when deploying the model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.model import InferenceConfig\n",
+        "\n",
+        "inf_config = InferenceConfig(entry_script='score.py', environment=myenv)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Provision the AKS Cluster with SSL\n",
+        "This is a one time setup. You can reuse this cluster for multiple deployments after it has been created. If you delete the cluster or the resource group that contains it, then you would have to recreate it.\n",
+        "\n",
+        "See code snippet below. Check the documentation [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-secure-web-service) for more details"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Use the default configuration (can also provide parameters to customize)\n",
+        "\n",
+        "provisioning_config = AksCompute.provisioning_configuration()\n",
+        "# Leaf domain label generates a name using the formula\n",
+        "#  \"<leaf-domain-label>######.<azure-region>.cloudapp.azure.net\"\n",
+        "#  where \"######\" is a random series of characters\n",
+        "provisioning_config.enable_ssl(leaf_domain_label = \"contoso\")\n",
+        "\n",
+        "aks_name = 'my-aks-ssl-1' \n",
+        "# Create the cluster\n",
+        "aks_target = ComputeTarget.create(workspace = ws, \n",
+        "                                  name = aks_name, \n",
+        "                                  provisioning_configuration = provisioning_config)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%time\n",
+        "aks_target.wait_for_completion(show_output = True)\n",
+        "print(aks_target.provisioning_state)\n",
+        "print(aks_target.provisioning_errors)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Deploy web service to AKS"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "tags": [
+          "sample-deploy-to-aks"
+        ]
+      },
+      "outputs": [],
+      "source": [
+        "%%time\n",
+        "\n",
+        "aks_config = AksWebservice.deploy_configuration()\n",
+        "\n",
+        "aks_service_name ='aks-service-ssl-1'\n",
+        "\n",
+        "aks_service = Model.deploy(workspace=ws,\n",
+        "                           name=aks_service_name,\n",
+        "                           models=[model],\n",
+        "                           inference_config=inf_config,\n",
+        "                           deployment_config=aks_config,\n",
+        "                           deployment_target=aks_target,\n",
+        "                          overwrite=True)\n",
+        "\n",
+        "aks_service.wait_for_deployment(show_output = True)\n",
+        "print(aks_service.state)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Test the web service using run method\n",
+        "We test the web sevice by passing data.\n",
+        "Run() method retrieves API keys behind the scenes to make sure that call is authenticated."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%time\n",
+        "import json\n",
+        "\n",
+        "standard_sample_input = json.dumps({'param': {'a': 10, 'b': 9, 'c': 8, 'd': 7, 'e': 6, 'f': 5, 'g': 4, 'h': 3, 'i': 2, 'j': 1 }})\n",
+        "\n",
+        "aks_service.run(input_data=standard_sample_input)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Clean up\n",
+        "Delete the service, image and model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%time\n",
+        "aks_service.delete()\n",
+        "model.delete()"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "vaidyas"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.6"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks-ssl.yml

@@ -0,0 +1,8 @@
+name: production-deploy-to-aks-ssl
+dependencies:
+- pip:
+  - azureml-sdk
+  - matplotlib
+  - tqdm
+  - scipy
+  - sklearn

how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb

@@ -109,7 +109,7 @@
         "from azureml.core import Environment\n",
         "from azureml.core.conda_dependencies import CondaDependencies \n",
         "\n",
-        "conda_deps = CondaDependencies.create(conda_packages=['numpy','scikit-learn'], pip_packages=['azureml-defaults'])\n",
+        "conda_deps = CondaDependencies.create(conda_packages=['numpy','scikit-learn==0.19.1','scipy'], pip_packages=['azureml-defaults'])\n",
         "myenv = Environment(name='myenv')\n",
         "myenv.python.conda_dependencies = conda_deps"
       ]
@@ -300,7 +300,8 @@
         "    'inference-schema[numpy-support]',\n",
         "    'joblib',\n",
         "    'numpy',\n",
-        "    'scikit-learn'\n",
+        "    'scikit-learn==0.19.1',\n",
+        "    'scipy'\n",
         "])\n",
         "inference_config = InferenceConfig(entry_script='score.py', environment=environment)\n",
         "# if cpu and memory_in_gb parameters are not provided\n",

how-to-use-azureml/deployment/tensorflow/tensorflow-model-register-and-deploy.ipynb

@@ -1,260 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/tensorflow/tensorflow-model-register-and-deploy.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Register TensorFlow SavedModel and deploy as webservice\n",
-        "\n",
-        "Following this notebook, you will:\n",
-        "\n",
-        " - Learn how to register a TF SavedModel in your Azure Machine Learning Workspace.\n",
-        " - Deploy your model as a web service in an Azure Container Instance."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Prerequisites\n",
-        "\n",
-        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration notebook](../../../configuration.ipynb) to install the Azure Machine Learning Python SDK and create a workspace."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import azureml.core\n",
-        "\n",
-        "# Check core SDK version number.\n",
-        "print('SDK version:', azureml.core.VERSION)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Initialize workspace\n",
-        "\n",
-        "Create a [Workspace](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.workspace%28class%29?view=azure-ml-py) object from your persisted configuration."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "create workspace"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.from_config()\n",
-        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Download the Model\n",
-        "\n",
-        "Download and extract the model from https://amlsamplenotebooksdata.blob.core.windows.net/data/flowers_model.tar.gz to \"models\" directory"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "import tarfile\n",
-        "import urllib.request\n",
-        "\n",
-        "# create directory for model\n",
-        "model_dir = 'models'\n",
-        "if not os.path.isdir(model_dir):\n",
-        "    os.mkdir(model_dir)\n",
-        "\n",
-        "url=\"https://amlsamplenotebooksdata.blob.core.windows.net/data/flowers_model.tar.gz\"\n",
-        "response = urllib.request.urlretrieve(url, model_dir + \"/flowers_model.tar.gz\")\n",
-        "tar = tarfile.open(model_dir + \"/flowers_model.tar.gz\", \"r:gz\")\n",
-        "tar.extractall(model_dir)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Register model\n",
-        "\n",
-        "Register a file or folder as a model by calling [Model.register()](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.model.model?view=azure-ml-py#register-workspace--model-path--model-name--tags-none--properties-none--description-none--datasets-none--model-framework-none--model-framework-version-none--child-paths-none-). For this example, we have provided a TensorFlow SavedModel (`flowers_model` in the notebook's directory).\n",
-        "\n",
-        "In addition to the content of the model file itself, your registered model will also store model metadata -- model description, tags, and framework information -- that will be useful when managing and deploying models in your workspace. Using tags, for instance, you can categorize your models and apply filters when listing models in your workspace. Also, marking this model with the scikit-learn framework will simplify deploying it as a web service, as we'll see later."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "register model from file"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "from azureml.core import Model\n",
-        "\n",
-        "model = Model.register(workspace=ws,\n",
-        "                       model_name='flowers',                        # Name of the registered model in your workspace.\n",
-        "                       model_path= model_dir + '/flowers_model',    # Local Tensorflow SavedModel folder to upload and register as a model.\n",
-        "                       model_framework=Model.Framework.TENSORFLOW,  # Framework used to create the model.\n",
-        "                       model_framework_version='1.14.0',            # Version of Tensorflow used to create the model.\n",
-        "                       description='Flowers model')\n",
-        "\n",
-        "print('Name:', model.name)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Deploy model\n",
-        "\n",
-        "Deploy your model as a web service using [Model.deploy()](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.model.model?view=azure-ml-py#deploy-workspace--name--models--inference-config--deployment-config-none--deployment-target-none-). Web services take one or more models, load them in an environment, and run them on one of several supported deployment targets.\n",
-        "\n",
-        "For this example, we will deploy your TensorFlow SavedModel to an Azure Container Instance (ACI)."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Use a default environment (for supported models)\n",
-        "\n",
-        "The Azure Machine Learning service provides a default environment for supported model frameworks, including TensorFlow, based on the metadata you provided when registering your model. This is the easiest way to deploy your model.\n",
-        "\n",
-        "**Note**: This step can take several minutes."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Webservice\n",
-        "from azureml.exceptions import WebserviceException\n",
-        "\n",
-        "service_name = 'tensorflow-flower-service'\n",
-        "\n",
-        "# Remove any existing service under the same name.\n",
-        "try:\n",
-        "    Webservice(ws, service_name).delete()\n",
-        "except WebserviceException:\n",
-        "    pass\n",
-        "\n",
-        "service = Model.deploy(ws, service_name, [model])\n",
-        "service.wait_for_deployment(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "After your model is deployed, perform a call to the web service."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import requests\n",
-        "\n",
-        "headers = {'Content-Type': 'application/json'}\n",
-        "\n",
-        "if service.auth_enabled:\n",
-        "    headers['Authorization'] = 'Bearer '+ service.get_keys()[0]\n",
-        "elif service.token_auth_enabled:\n",
-        "    headers['Authorization'] = 'Bearer '+ service.get_token()[0]\n",
-        "\n",
-        "scoring_uri = service.scoring_uri # If you have a SavedModel with classify and regress, \n",
-        "                                  #  you can change the scoring_uri from 'uri:predict' to 'uri:classify' or 'uri:regress'.\n",
-        "print(scoring_uri)\n",
-        "\n",
-        "with open('tensorflow-flower-predict-input.json', 'rb') as data_file:\n",
-        "    response = requests.post(\n",
-        "        scoring_uri, data=data_file, headers=headers)\n",
-        "print(response.status_code)\n",
-        "print(response.elapsed)\n",
-        "print(response.json())"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "When you are finished testing your service, clean up the deployment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "service.delete()"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "vaidyas"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.7.0"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}

how-to-use-azureml/deployment/tensorflow/tensorflow-model-register-and-deploy.yml

@@ -1,4 +0,0 @@
-name: tensorflow-model-register-and-deploy
-dependencies:
-- pip:
-  - azureml-sdk

how-to-use-azureml/explain-model/azure-integration/remote-explanation/explain-model-on-amlcompute.ipynb

@@ -58,7 +58,7 @@
         "\n",
         "Problem: Boston Housing Price Prediction with scikit-learn (train a model and run an explainer remotely via AMLCompute, and download and visualize the remotely-calculated explanations.)\n",
         "\n",
-        "| ![explanations-run-history](./img/explanations-run-history.PNG) |\n",
+        "| ![explanations-run-history](./img/explanations-run-history.png) |\n",
         "|:--:|\n"
       ]
     },
@@ -261,6 +261,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                                                            pip_packages=azureml_pip_packages)\n",
         "\n",
@@ -379,6 +383,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                                                            pip_packages=azureml_pip_packages)\n",
         "\n",
@@ -509,6 +517,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                                                            pip_packages=azureml_pip_packages)\n",
         "\n",
@@ -672,7 +684,7 @@
       "source": [
         "# retrieve model for visualization and deployment\n",
         "from azureml.core.model import Model\n",
-        "from sklearn.externals import joblib\n",
+        "import joblib\n",
         "original_model = Model(ws, 'model_explain_model_on_amlcomp')\n",
         "model_path = original_model.download(exist_ok=True)\n",
         "original_model = joblib.load(model_path)"
@@ -692,7 +704,7 @@
       "outputs": [],
       "source": [
         "# retrieve x_test for visualization\n",
-        "from sklearn.externals import joblib\n",
+        "import joblib\n",
         "x_test_path = './x_test_boston_housing.pkl'\n",
         "run.download_file('x_test_boston_housing.pkl', output_file_path=x_test_path)"
       ]

how-to-use-azureml/explain-model/azure-integration/remote-explanation/train_explain.py

@@ -7,7 +7,7 @@ from interpret.ext.blackbox import TabularExplainer
 from azureml.contrib.interpret.explanation.explanation_client import ExplanationClient
 from sklearn.model_selection import train_test_split
 from azureml.core.run import Run
-from sklearn.externals import joblib
+import joblib
 import os
 import numpy as np
 

how-to-use-azureml/explain-model/azure-integration/scoring-time/score_local_explain.py

@@ -3,7 +3,7 @@ import numpy as np
 import pandas as pd
 import os
 import pickle
-from sklearn.externals import joblib
+import joblib
 from sklearn.linear_model import LogisticRegression
 from azureml.core.model import Model
 

how-to-use-azureml/explain-model/azure-integration/scoring-time/score_remote_explain.py

@@ -3,7 +3,7 @@ import numpy as np
 import pandas as pd
 import os
 import pickle
-from sklearn.externals import joblib
+import joblib
 from sklearn.linear_model import LogisticRegression
 from azureml.core.model import Model
 

how-to-use-azureml/explain-model/azure-integration/scoring-time/train-explain-model-locally-and-deploy.ipynb

@@ -165,7 +165,7 @@
       "outputs": [],
       "source": [
         "from sklearn.model_selection import train_test_split\n",
-        "from sklearn.externals import joblib\n",
+        "import joblib\n",
         "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
         "from sklearn.impute import SimpleImputer\n",
         "from sklearn.pipeline import Pipeline\n",
@@ -346,6 +346,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "myenv = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                 pip_packages=['sklearn-pandas', 'pyyaml'] + azureml_pip_packages,\n",
         "                                 pin_sdk_version=False)\n",
@@ -413,9 +417,9 @@
         "headers = {'Content-Type':'application/json'}\n",
         "\n",
         "# send request to service\n",
+        "print(\"POST to url\", service.scoring_uri)\n",
         "resp = requests.post(service.scoring_uri, sample_data, headers=headers)\n",
         "\n",
-        "print(\"POST to url\", service.scoring_uri)\n",
         "# can covert back to Python objects from json string if desired\n",
         "print(\"prediction:\", resp.text)\n",
         "result = json.loads(resp.text)"

how-to-use-azureml/explain-model/azure-integration/scoring-time/train-explain-model-on-amlcompute-and-deploy.ipynb

@@ -63,7 +63,7 @@
         "7.\tCreate an image and register it in the image registry.\n",
         "8.\tDeploy the image as a web service in Azure.\n",
         "\n",
-        "| ![azure-machine-learning-cycle](./img/azure-machine-learning-cycle.PNG) |\n",
+        "| ![azure-machine-learning-cycle](./img/azure-machine-learning-cycle.png) |\n",
         "|:--:|"
       ]
     },
@@ -264,6 +264,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                pip_packages=['sklearn_pandas', 'pyyaml'] + azureml_pip_packages,\n",
         "                                pin_sdk_version=False)\n",
@@ -325,7 +329,7 @@
       "source": [
         "# retrieve model for visualization and deployment\n",
         "from azureml.core.model import Model\n",
-        "from sklearn.externals import joblib\n",
+        "import joblib\n",
         "original_model = Model(ws, 'amlcompute_deploy_model')\n",
         "model_path = original_model.download(exist_ok=True)\n",
         "original_svm_model = joblib.load(model_path)"
@@ -352,7 +356,7 @@
       "outputs": [],
       "source": [
         "# retrieve x_test for visualization\n",
-        "from sklearn.externals import joblib\n",
+        "import joblib\n",
         "x_test_path = './x_test.pkl'\n",
         "run.download_file('x_test_ibm.pkl', output_file_path=x_test_path)\n",
         "x_test = joblib.load(x_test_path)"
@@ -432,6 +436,10 @@
         "if pandas_ver:\n",
         "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
         "# specify CondaDependencies obj\n",
+        "# The CondaDependencies specifies the conda and pip packages that are installed in the environment\n",
+        "# the submitted job is run in.  Note the remote environment(s) needs to be similar to the local\n",
+        "# environment, otherwise if a model is trained or deployed in a different environment this can\n",
+        "# cause errors.  Please take extra care when specifying your dependencies in a production environment.\n",
         "myenv = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
         "                                 pip_packages=['sklearn-pandas', 'pyyaml'] + azureml_pip_packages,\n",
         "                                 pin_sdk_version=False)\n",
@@ -495,9 +503,9 @@
         "headers = {'Content-Type':'application/json'}\n",
         "\n",
         "# send request to service\n",
+        "print(\"POST to url\", service.scoring_uri)\n",
         "resp = requests.post(service.scoring_uri, input_data, headers=headers)\n",
         "\n",
-        "print(\"POST to url\", service.scoring_uri)\n",
         "# can covert back to Python objects from json string if desired\n",
         "print(\"prediction:\", resp.text)"
       ]

how-to-use-azureml/explain-model/azure-integration/scoring-time/train_explain.py

@@ -6,7 +6,7 @@ import os
 import pandas as pd
 import zipfile
 from sklearn.model_selection import train_test_split
-from sklearn.externals import joblib
+import joblib
 from sklearn.preprocessing import StandardScaler, OneHotEncoder
 from sklearn.impute import SimpleImputer
 from sklearn.pipeline import Pipeline

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-azurebatch-to-run-a-windows-executable.ipynb

@@ -252,7 +252,7 @@
       "source": [
         "binaries_folder = \"azurebatch/job_binaries\"\n",
         "if not os.path.isdir(binaries_folder):\n",
-        "    os.mkdir(binaries_folder)\n",
+        "    os.makedirs(binaries_folder)\n",
         "\n",
         "file_name=\"azurebatch.cmd\"\n",
         "with open(path.join(binaries_folder, file_name), 'w') as f:\n",

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-showcasing-dataset-and-pipelineparameter.ipynb

@@ -0,0 +1,510 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-data-dependency-steps.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Showcasing Dataset and PipelineParameter\n",
+        "\n",
+        "This notebook demonstrates how a [**FileDataset**](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.filedataset?view=azure-ml-py) or [**TabularDataset**](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) can be parametrized with [**PipelineParameters**](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelineparameter?view=azure-ml-py) in an AML [Pipeline](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline(class)?view=azure-ml-py). By parametrizing datasets, you can dynamically run pipeline experiments with different datasets without any code change.\n",
+        "\n",
+        "A common use case is building a training pipeline with a sample of your training data for quick iterative development. When you're ready to test and deploy your pipeline at scale, you can pass in your full training dataset to the pipeline experiment without making any changes to your training script. \n",
+        " \n",
+        "To see more about how parameters work between steps, please refer [aml-pipelines-with-data-dependency-steps](https://aka.ms/pl-data-dep).\n",
+        "\n",
+        "* [How to create a Pipeline with a Dataset PipelineParameter](#index1)\n",
+        "* [How to submit a Pipeline with a Dataset PipelineParameter](#index2)\n",
+        "* [How to submit a Pipeline and change the Dataset PipelineParameter value from the sdk](#index3)\n",
+        "* [How to submit a Pipeline and change the Dataset PipelineParameter value using a REST call](#index4)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Azure Machine Learning and Pipeline SDK-specific imports"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import azureml.core\n",
+        "from azureml.core import Workspace, Experiment, Dataset\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.data.dataset_consumption_config import DatasetConsumptionConfig\n",
+        "from azureml.widgets import RunDetails\n",
+        "\n",
+        "from azureml.pipeline.core import PipelineParameter\n",
+        "from azureml.pipeline.core import Pipeline, PipelineRun\n",
+        "from azureml.pipeline.steps import PythonScriptStep\n",
+        "\n",
+        "# Check core SDK version number\n",
+        "print(\"SDK version:\", azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Initialize Workspace\n",
+        "\n",
+        "Initialize a workspace object from persisted configuration. If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure the config file is present at .\\config.json\n",
+        "\n",
+        "If you don't have a config.json file, go through the [configuration Notebook](https://aka.ms/pl-config) first.\n",
+        "\n",
+        "This sets you up with a working config file that has information on your workspace, subscription id, etc."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Create an Azure ML experiment\n",
+        "\n",
+        "Let's create an experiment named \"showcasing-dataset\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Choose a name for the run history container in the workspace.\n",
+        "experiment_name = 'showcasing-dataset'\n",
+        "source_directory  = '.'\n",
+        "\n",
+        "experiment = Experiment(ws, experiment_name)\n",
+        "experiment"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Create or Attach an AmlCompute cluster\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Choose a name for your cluster.\n",
+        "amlcompute_cluster_name = \"cpu-cluster\"\n",
+        "\n",
+        "found = False\n",
+        "# Check if this compute target already exists in the workspace.\n",
+        "cts = ws.compute_targets\n",
+        "if amlcompute_cluster_name in cts and cts[amlcompute_cluster_name].type == 'AmlCompute':\n",
+        "    found = True\n",
+        "    print('Found existing compute target.')\n",
+        "    compute_target = cts[amlcompute_cluster_name]\n",
+        "    \n",
+        "if not found:\n",
+        "    print('Creating a new compute target...')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\", # for GPU, use \"STANDARD_NC6\"\n",
+        "                                                                #vm_priority = 'lowpriority', # optional\n",
+        "                                                                max_nodes = 4)\n",
+        "\n",
+        "    # Create the cluster.\n",
+        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
+        "    \n",
+        "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
+        "    # If no min_node_count is provided, it will use the scale settings for the cluster.\n",
+        "    compute_target.wait_for_completion(show_output = True, timeout_in_minutes = 10)\n",
+        "    \n",
+        "     # For a more detailed view of current AmlCompute status, use get_status()."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Dataset Configuration\n",
+        "\n",
+        "The following steps detail how to create a [FileDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.filedataset?view=azure-ml-py) and [TabularDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) from an external CSV file, and configure them to be used by a [Pipeline](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline(class)?view=azure-ml-py):\n",
+        "\n",
+        "1. Create a dataset from a csv file\n",
+        "2. Create a [PipelineParameter](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelineparameter?view=azure-ml-py) object and set the `default_value` to the dataset. [PipelineParameter](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelineparameter?view=azure-ml-py) objects enabled arguments to be passed into Pipelines when they are resubmitted after creation. The `name` is referenced later on when we submit additional pipeline runs with different input datasets. \n",
+        "3. Create a [DatasetConsumptionConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_consumption_config.datasetconsumptionconfig?view=azure-ml-py) object from the [PiepelineParameter](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelineparameter?view=azure-ml-py). The [DatasetConsumptionConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_consumption_config.datasetconsumptionconfig?view=azure-ml-py) object specifies how the dataset should be used by the remote compute where the pipeline is run. **NOTE** only [DatasetConsumptionConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_consumption_config.datasetconsumptionconfig?view=azure-ml-py) objects built on [FileDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.filedataset?view=azure-ml-py) can be set `as_mount()` or `as_download()` on the remote compute."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "tags": [
+          "datapath-remarks-sample"
+        ]
+      },
+      "outputs": [],
+      "source": [
+        "file_dataset = Dataset.File.from_files('https://dprepdata.blob.core.windows.net/demo/Titanic.csv')\n",
+        "file_pipeline_param = PipelineParameter(name=\"file_ds_param\", default_value=file_dataset)\n",
+        "file_ds_consumption = DatasetConsumptionConfig(\"file_dataset\", file_pipeline_param).as_mount()\n",
+        "\n",
+        "tabular_dataset = Dataset.Tabular.from_delimited_files('https://dprepdata.blob.core.windows.net/demo/Titanic.csv')\n",
+        "tabular_pipeline_param = PipelineParameter(name=\"tabular_ds_param\", default_value=tabular_dataset)\n",
+        "tabular_ds_consumption = DatasetConsumptionConfig(\"tabular_dataset\", tabular_pipeline_param)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We will setup a training script to ingest our passed-in datasets and print their contents. **NOTE** the names of the datasets referenced inside the training script correspond to the `name` of their respective [DatasetConsumptionConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_consumption_config.datasetconsumptionconfig?view=azure-ml-py) objects we defined above."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile train_with_dataset.py\n",
+        "from azureml.core import Run\n",
+        "\n",
+        "input_file_ds_path = Run.get_context().input_datasets['file_dataset']\n",
+        "with open(input_file_ds_path, 'r') as f:\n",
+        "    content = f.read()\n",
+        "    print(content)\n",
+        "\n",
+        "input_tabular_ds = Run.get_context().input_datasets['tabular_dataset']\n",
+        "tabular_df = input_tabular_ds.to_pandas_dataframe()\n",
+        "print(tabular_df)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id='index1'></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Create a Pipeline with a Dataset PipelineParameter\n",
+        "\n",
+        "Note that the ```file_ds_consumption``` and ```tabular_ds_consumption``` are specified as both arguments and inputs to create a step."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "train_step = PythonScriptStep(\n",
+        "    name=\"train_step\",\n",
+        "    script_name=\"train_with_dataset.py\",\n",
+        "    arguments=[\"--param1\", file_ds_consumption, \"--param2\", tabular_ds_consumption],\n",
+        "    inputs=[file_ds_consumption, tabular_ds_consumption],\n",
+        "    compute_target=compute_target,\n",
+        "    source_directory=source_directory)\n",
+        "\n",
+        "print(\"train_step created\")\n",
+        "\n",
+        "pipeline = Pipeline(workspace=ws, steps=[train_step])\n",
+        "print(\"pipeline with the train_step created\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id='index2'></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Submit a Pipeline with a Dataset PipelineParameter\n",
+        "\n",
+        "Pipelines can be submitted with default values of PipelineParameters by not specifying any parameters."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Pipeline will run with default file_ds and tabular_ds\n",
+        "pipeline_run = experiment.submit(pipeline)\n",
+        "print(\"Pipeline is submitted for execution\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "RunDetails(pipeline_run).show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run.wait_for_completion()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id='index3'></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Submit a Pipeline with a different Dataset PipelineParameter value from the SDK\n",
+        "\n",
+        "The training pipeline can be reused with different input datasets by passing them in as PipelineParameters"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "iris_file_ds = Dataset.File.from_files('https://raw.githubusercontent.com/Azure/MachineLearningNotebooks/'\n",
+        "                                        '4e7b3784d50e81c313c62bcdf9a330194153d9cd/how-to-use-azureml/work-with-data/'\n",
+        "                                        'datasets-tutorial/train-with-datasets/train-dataset/iris.csv')\n",
+        "\n",
+        "iris_tabular_ds = Dataset.Tabular.from_delimited_files('https://raw.githubusercontent.com/Azure/MachineLearningNotebooks/'\n",
+        "                                                       '4e7b3784d50e81c313c62bcdf9a330194153d9cd/how-to-use-azureml/work-with-data/'\n",
+        "                                                       'datasets-tutorial/train-with-datasets/train-dataset/iris.csv')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run_with_params = experiment.submit(pipeline, pipeline_parameters={'file_ds_param': iris_file_ds, 'tabular_ds_param': iris_tabular_ds}) "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "RunDetails(pipeline_run_with_params).show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run_with_params.wait_for_completion()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id='index4'></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Dynamically Set the Dataset PipelineParameter Values using a REST Call\n",
+        "\n",
+        "Let's publish the pipeline we created previously, so we can generate a pipeline endpoint. We can then submit the iris datasets to the pipeline REST endpoint by passing in their IDs. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "published_pipeline = pipeline.publish(name=\"Dataset_Pipeline\", description=\"Pipeline to test Dataset PipelineParameter\", continue_on_step_failure=True)\n",
+        "published_pipeline"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "published_pipeline.submit(ws, experiment_name=\"publishedexperiment\", pipeline_parameters={'file_ds_param': iris_file_ds, 'tabular_ds_param': iris_tabular_ds})"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
+        "import requests\n",
+        "\n",
+        "auth = InteractiveLoginAuthentication()\n",
+        "aad_token = auth.get_authentication_header()\n",
+        "\n",
+        "rest_endpoint = published_pipeline.endpoint\n",
+        "\n",
+        "print(\"You can perform HTTP POST on URL {} to trigger this pipeline\".format(rest_endpoint))"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# specify the param when running the pipeline\n",
+        "response = requests.post(rest_endpoint, \n",
+        "                         headers=aad_token, \n",
+        "                         json={\"ExperimentName\": \"MyRestPipeline\",\n",
+        "                               \"RunSource\": \"SDK\",\n",
+        "                               \"DataSetDefinitionValueAssignments\": {\"file_ds_param\": {\"SavedDataSetReference\": {\"Id\": iris_file_ds.id}},\n",
+        "                                                                     \"tabular_ds_param\": {\"SavedDataSetReference\": {\"Id\": iris_tabular_ds.id}}}\n",
+        "                              }\n",
+        "                        )"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "try:\n",
+        "    response.raise_for_status()\n",
+        "except Exception:    \n",
+        "    raise Exception('Received bad response from the endpoint: {}\\n'\n",
+        "                    'Response Code: {}\\n'\n",
+        "                    'Headers: {}\\n'\n",
+        "                    'Content: {}'.format(rest_endpoint, response.status_code, response.headers, response.content))\n",
+        "\n",
+        "run_id = response.json().get('Id')\n",
+        "print('Submitted pipeline run: ', run_id)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "published_pipeline_run_via_rest = PipelineRun(ws.experiments[\"MyRestPipeline\"], run_id)\n",
+        "RunDetails(published_pipeline_run_via_rest).show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "published_pipeline_run_via_rest.wait_for_completion()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id='index5'></a>"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "rafarmah"
+      }
+    ],
+    "category": "tutorial",
+    "compute": [
+      "AML Compute"
+    ],
+    "datasets": [
+      "Custom"
+    ],
+    "deployment": [
+      "None"
+    ],
+    "exclude_from_index": false,
+    "framework": [
+      "Azure ML"
+    ],
+    "friendly_name": "How to use Dataset as a PipelineParameter",
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.7"
+    },
+    "order_index": 13,
+    "star_tag": [
+      "featured"
+    ],
+    "tags": [
+      "None"
+    ],
+    "task": "Demonstrates the use of Dataset as a PipelineParameter"
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-showcasing-dataset-and-pipelineparameter.yml

@@ -0,0 +1,5 @@
+name: aml-pipelines-showcasing-dataset-and-pipelineparameter
+dependencies:
+- pip:
+  - azureml-sdk
+  - azureml-widgets

how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-databricks-as-compute-target.ipynb

@@ -510,7 +510,7 @@
         "    inputs=[step_1_input],\n",
         "    num_workers=1,\n",
         "    python_script_path=python_script_path,\n",
-        "    python_script_params={'arg1', pipeline_param, 'arg2},\n",
+        "    python_script_params={'arg1', pipeline_param, 'arg2'},\n",
         "    run_name='DB_Python_demo',\n",
         "    compute_target=databricks_compute,\n",
         "    allow_reuse=True\n",

how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/nyc-taxi-data-regression-model-building.ipynb

@@ -279,8 +279,7 @@
         "# Specify CondaDependencies obj, add necessary packages\n",
         "aml_run_config.environment.python.conda_dependencies = CondaDependencies.create(\n",
         "    conda_packages=['pandas','scikit-learn'], \n",
-        "    pip_packages=['azureml-sdk[automl,explain]', 'pyarrow'], \n",
-        "    pin_sdk_version=False)\n",
+        "    pip_packages=['azureml-sdk[automl,explain]', 'pyarrow'])\n",
         "\n",
         "print (\"Run configuration created.\")"
       ]
@@ -692,7 +691,6 @@
         "                             debug_log = 'automated_ml_errors.log',\n",
         "                             path = train_model_folder,\n",
         "                             compute_target = aml_compute,\n",
-        "                             run_configuration = aml_run_config,\n",
         "                             featurization = 'auto',\n",
         "                             training_data = training_dataset,\n",
         "                             label_column_name = 'cost',\n",
@@ -718,7 +716,6 @@
         "\n",
         "trainWithAutomlStep = AutoMLStep(name='AutoML_Regression',\n",
         "                                 automl_config=automl_config,\n",
-        "                                 passthru_automl_config=False,\n",
         "                                 allow_reuse=True)\n",
         "print(\"trainWithAutomlStep created.\")"
       ]

how-to-use-azureml/machine-learning-pipelines/parallel-run/Code/digit_identification.py

@@ -13,7 +13,7 @@ def init():
     global g_tf_sess
 
     # pull down model from workspace
-    model_path = Model.get_model_path("mnist")
+    model_path = Model.get_model_path("mnist-prs")
 
     # contruct graph to execute
     tf.reset_default_graph()

how-to-use-azureml/machine-learning-pipelines/parallel-run/README.md

@@ -2,18 +2,16 @@
 
 Azure Machine Learning Batch Inference targets large inference jobs that are not time-sensitive. Batch Inference provides cost-effective inference compute scaling, with unparalleled throughput for asynchronous applications. It is optimized for high-throughput, fire-and-forget inference over large collections of data.
 
-# Getting Started with Batch Inference Public Preview
+# Getting Started with Batch Inference
 
-Batch inference public preview offers a platform in which to do large inference or generic parallel map-style operations. Below introduces the major steps to use this new functionality. For a quick try, please follow the prerequisites and simply run the sample notebooks provided in this directory.
+Batch inference offers a platform in which to do large inference or generic parallel map-style operations. Below introduces the major steps to use this new functionality. For a quick try, please follow the prerequisites and simply run the sample notebooks provided in this directory.
 
 ## Prerequisites
 
 ### Python package installation
-Following the convention of most AzureML Public Preview features, Batch Inference SDK is currently available as a contrib package.
-
 If you're unfamiliar with creating a new Python environment, you may follow this example for [creating a conda environment](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-environment#local). Batch Inference package can be installed through the following pip command.
 ```
-pip install azureml-contrib-pipeline-steps
+pip install azureml-pipeline-steps
 ```
 
 ### Creation of Azure Machine Learning Workspace
@@ -66,9 +64,8 @@ base_image_registry.password = "password"
 
 ## Create a batch inference job
 
-**ParallelRunStep** is a newly added step in the azureml.contrib.pipeline.steps package. You will use it to add a step to create a batch inference job with your Azure machine learning pipeline. (Use batch inference without an Azure machine learning pipeline is not supported yet). ParallelRunStep has all the following parameters:
+**ParallelRunStep** is a newly added step in the azureml.pipeline.steps package. You will use it to add a step to create a batch inference job with your Azure machine learning pipeline. (Use batch inference without an Azure machine learning pipeline is not supported yet). ParallelRunStep has all the following parameters:
   - **name**: this name will be used to register batch inference service, has the following naming restrictions: (unique, 3-32 chars and regex ^\[a-z\]([-a-z0-9]*[a-z0-9])?$)
-  - **models**: zero or more model names already registered in Azure Machine Learning model registry.
   - **parallel_run_config**: ParallelRunConfig as defined above.
   - **inputs**: one or more Dataset objects.
   - **output**: this should be a PipelineData object encapsulating an Azure BLOB container path.

how-to-use-azureml/machine-learning-pipelines/parallel-run/file-dataset-image-inference-mnist.ipynb

@@ -23,11 +23,6 @@
         "\n",
         "In this notebook, we will demonstrate how to make predictions on large quantities of data asynchronously using the ML pipelines with Azure Machine Learning. Batch inference (or batch scoring) provides cost-effective inference, with unparalleled throughput for asynchronous applications. Batch prediction pipelines can scale to perform inference on terabytes of production data. Batch prediction is optimized for high throughput, fire-and-forget predictions for a large collection of data.\n",
         "\n",
-        "> **Note**\n",
-        "This notebook uses public preview functionality (ParallelRunStep). Please install azureml-contrib-pipeline-steps package before running this notebook. Pandas is used to display job results.\n",
-        "```\n",
-        "pip install azureml-contrib-pipeline-steps pandas\n",
-        "```\n",
         "> **Tip**\n",
         "If your system requires low-latency processing (to process a single document or small set of documents quickly), use [real-time scoring](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-consume-web-service) instead of batch prediction.\n",
         "\n",
@@ -86,7 +81,6 @@
       "source": [
         "import os\n",
         "from azureml.core.compute import AmlCompute, ComputeTarget\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
         "# choose a name for your cluster\n",
         "compute_name = os.environ.get(\"AML_COMPUTE_CLUSTER_NAME\", \"cpu-cluster\")\n",
@@ -184,9 +178,20 @@
         "mnist_ds_name = 'mnist_sample_data'\n",
         "\n",
         "path_on_datastore = mnist_data.path('mnist')\n",
-        "input_mnist_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)\n",
-        "registered_mnist_ds = input_mnist_ds.register(ws, mnist_ds_name, create_new_version=True)\n",
-        "named_mnist_ds = registered_mnist_ds.as_named_input(mnist_ds_name)"
+        "input_mnist_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.data.dataset_consumption_config import DatasetConsumptionConfig\n",
+        "from azureml.pipeline.core import PipelineParameter\n",
+        "\n",
+        "pipeline_param = PipelineParameter(name=\"mnist_param\", default_value=input_mnist_ds)\n",
+        "input_mnist_ds_consumption = DatasetConsumptionConfig(\"minist_param_config\", pipeline_param).as_mount()"
       ]
     },
     {
@@ -269,7 +274,7 @@
         "\n",
         "# register downloaded model \n",
         "model = Model.register(model_path = \"models/\",\n",
-        "                       model_name = \"mnist\", # this is the name the model is registered as\n",
+        "                       model_name = \"mnist-prs\", # this is the name the model is registered as\n",
         "                       tags = {'pretrained': \"mnist\"},\n",
         "                       description = \"Mnist trained tensorflow model\",\n",
         "                       workspace = ws)"
@@ -306,8 +311,6 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "import os\n",
-        "\n",
         "scripts_folder = \"Code\"\n",
         "script_file = \"digit_identification.py\"\n",
         "\n",
@@ -341,8 +344,8 @@
         "from azureml.core import Environment\n",
         "from azureml.core.runconfig import CondaDependencies, DEFAULT_CPU_IMAGE\n",
         "\n",
-        "batch_conda_deps = CondaDependencies.create(pip_packages=[\"tensorflow==1.15.2\", \"pillow\"])\n",
-        "\n",
+        "batch_conda_deps = CondaDependencies.create(pip_packages=[\"tensorflow==1.15.2\", \"pillow\", \n",
+        "                                                          \"azureml-core\", \"azureml-dataprep[fuse]\"])\n",
         "batch_env = Environment(name=\"batch_environment\")\n",
         "batch_env.python.conda_dependencies = batch_conda_deps\n",
         "batch_env.docker.enabled = True\n",
@@ -362,17 +365,21 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunStep, ParallelRunConfig\n",
+        "from azureml.pipeline.core import PipelineParameter\n",
+        "from azureml.pipeline.steps import ParallelRunStep, ParallelRunConfig\n",
         "\n",
         "parallel_run_config = ParallelRunConfig(\n",
         "    source_directory=scripts_folder,\n",
         "    entry_script=script_file,\n",
-        "    mini_batch_size=\"5\",\n",
+        "    mini_batch_size=PipelineParameter(name=\"batch_size_param\", default_value=\"5\"),\n",
         "    error_threshold=10,\n",
         "    output_action=\"append_row\",\n",
+        "    append_row_file_name=\"mnist_outputs.txt\",\n",
         "    environment=batch_env,\n",
         "    compute_target=compute_target,\n",
-        "    node_count=2)"
+        "    process_count_per_node=PipelineParameter(name=\"process_count_param\", default_value=2),\n",
+        "    node_count=2\n",
+        ")"
       ]
     },
     {
@@ -392,10 +399,8 @@
         "parallelrun_step = ParallelRunStep(\n",
         "    name=\"predict-digits-mnist\",\n",
         "    parallel_run_config=parallel_run_config,\n",
-        "    inputs=[ named_mnist_ds ],\n",
+        "    inputs=[ input_mnist_ds_consumption ],\n",
         "    output=output_dir,\n",
-        "    models=[ model ],\n",
-        "    arguments=[ ],\n",
         "    allow_reuse=True\n",
         ")"
       ]
@@ -454,6 +459,46 @@
         "pipeline_run.wait_for_completion(show_output=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Resubmit a with different dataset\n",
+        "Since we made the input a `PipelineParameter`, we can resubmit with a different dataset without having to create an entirely new experiment. We'll use the same datastore but use only a single image."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "path_on_datastore = mnist_data.path('mnist/0.png')\n",
+        "single_image_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run_2 = experiment.submit(pipeline, \n",
+        "                                   pipeline_parameters={\"mnist_param\": single_image_ds, \n",
+        "                                                        \"batch_size_param\": \"1\",\n",
+        "                                                        \"process_count_param\": 1}\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline_run_2.wait_for_completion(show_output=True)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -480,7 +525,7 @@
         "\n",
         "for root, dirs, files in os.walk(\"mnist_results\"):\n",
         "    for file in files:\n",
-        "        if file.endswith('parallel_run_step.txt'):\n",
+        "        if file.endswith('mnist_outputs.txt'):\n",
         "            result_file = os.path.join(root,file)\n",
         "\n",
         "df = pd.read_csv(result_file, delimiter=\":\", header=None)\n",

how-to-use-azureml/machine-learning-pipelines/parallel-run/file-dataset-image-inference-mnist.yml

@@ -2,6 +2,6 @@ name: file-dataset-image-inference-mnist
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-contrib-pipeline-steps
+  - azureml-pipeline-steps
   - azureml-widgets
   - pandas

how-to-use-azureml/machine-learning-pipelines/parallel-run/tabular-dataset-inference-iris.ipynb

@@ -23,11 +23,6 @@
         "\n",
         "In this notebook, we will demonstrate how to make predictions on large quantities of data asynchronously using the ML pipelines with Azure Machine Learning. Batch inference (or batch scoring) provides cost-effective inference, with unparalleled throughput for asynchronous applications. Batch prediction pipelines can scale to perform inference on terabytes of production data. Batch prediction is optimized for high throughput, fire-and-forget predictions for a large collection of data.\n",
         "\n",
-        "> **Note**\n",
-        "This notebook uses public preview functionality (ParallelRunStep). Please install azureml-contrib-pipeline-steps package before running this notebook. Pandas is used to display job results.\n",
-        "```\n",
-        "pip install azureml-contrib-pipeline-steps pandas\n",
-        "```\n",
         "> **Tip**\n",
         "If your system requires low-latency processing (to process a single document or small set of documents quickly), use [real-time scoring](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-consume-web-service) instead of batch prediction.\n",
         "\n",
@@ -84,7 +79,6 @@
       "source": [
         "import os\n",
         "from azureml.core.compute import AmlCompute, ComputeTarget\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
         "# choose a name for your cluster\n",
         "compute_name = os.environ.get(\"AML_COMPUTE_CLUSTER_NAME\", \"cpu-cluster\")\n",
@@ -233,7 +227,7 @@
         "\n",
         "# register downloaded model\n",
         "model = Model.register(model_path = \"iris_model.pkl/iris_model.pkl\",\n",
-        "                       model_name = \"iris\", # this is the name the model is registered as\n",
+        "                       model_name = \"iris-prs\", # this is the name the model is registered as\n",
         "                       tags = {'pretrained': \"iris\"},\n",
         "                       workspace = ws)"
       ]
@@ -304,7 +298,8 @@
         "from azureml.core import Environment\n",
         "from azureml.core.runconfig import CondaDependencies\n",
         "\n",
-        "predict_conda_deps = CondaDependencies.create(pip_packages=[ \"scikit-learn==0.20.3\" ])\n",
+        "predict_conda_deps = CondaDependencies.create(pip_packages=[\"scikit-learn==0.20.3\",\n",
+        "                                                            \"azureml-core\", \"azureml-dataprep[pandas,fuse]\"])\n",
         "\n",
         "predict_env = Environment(name=\"predict_environment\")\n",
         "predict_env.python.conda_dependencies = predict_conda_deps\n",
@@ -325,19 +320,21 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunStep, ParallelRunConfig\n",
+        "from azureml.pipeline.steps import ParallelRunStep, ParallelRunConfig\n",
         "\n",
         "# In a real-world scenario, you'll want to shape your process per node and nodes to fit your problem domain.\n",
         "parallel_run_config = ParallelRunConfig(\n",
-        "                    source_directory=scripts_folder,\n",
-        "                    entry_script=script_file,  # the user script to run against each input\n",
-        "                    mini_batch_size='5MB',\n",
-        "                    error_threshold=5,\n",
-        "                    output_action='append_row',\n",
-        "                    environment=predict_env,\n",
-        "                    compute_target=compute_target, \n",
-        "                    node_count=3,\n",
-        "                    run_invocation_timeout=600)"
+        "    source_directory=scripts_folder,\n",
+        "    entry_script=script_file,  # the user script to run against each input\n",
+        "    mini_batch_size='5MB',\n",
+        "    error_threshold=5,\n",
+        "    output_action='append_row',\n",
+        "    append_row_file_name=\"iris_outputs.txt\",\n",
+        "    environment=predict_env,\n",
+        "    compute_target=compute_target, \n",
+        "    node_count=3,\n",
+        "    run_invocation_timeout=600\n",
+        ")"
       ]
     },
     {
@@ -359,8 +356,7 @@
         "    inputs=[named_iris_ds],\n",
         "    output=output_folder,\n",
         "    parallel_run_config=parallel_run_config,\n",
-        "    models=[model],\n",
-        "    arguments=['--model_name', 'iris'],\n",
+        "    arguments=['--model_name', 'iris-prs'],\n",
         "    allow_reuse=True\n",
         ")"
       ]
@@ -384,7 +380,7 @@
         "\n",
         "pipeline = Pipeline(workspace=ws, steps=[distributed_csv_iris_step])\n",
         "\n",
-        "pipeline_run = Experiment(ws, 'iris').submit(pipeline)"
+        "pipeline_run = Experiment(ws, 'iris-prs').submit(pipeline)"
       ]
     },
     {
@@ -453,7 +449,7 @@
         "\n",
         "for root, dirs, files in os.walk(\"iris_results\"):\n",
         "    for file in files:\n",
-        "        if file.endswith('parallel_run_step.txt'):\n",
+        "        if file.endswith('iris_outputs.txt'):\n",
         "            result_file = os.path.join(root,file)\n",
         "\n",
         "# cleanup output format\n",

how-to-use-azureml/machine-learning-pipelines/parallel-run/tabular-dataset-inference-iris.yml

@@ -2,6 +2,6 @@ name: tabular-dataset-inference-iris
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-contrib-pipeline-steps
+  - azureml-pipeline-steps
   - azureml-widgets
   - pandas

how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.ipynb

@@ -26,11 +26,8 @@
         "2. Run neural style on each image using one of the provided models (from `pytorch` pretrained models for this example).\n",
         "3. Stitch the image back into a video.\n",
         "\n",
-        "> **Note**\n",
-        "This notebook uses public preview functionality (ParallelRunStep). Please install azureml-contrib-pipeline-steps package before running this notebook.\n",
-        "```\n",
-        "pip install azureml-contrib-pipeline-steps\n",
-        "```"
+        "> **Tip**\n",
+        "If your system requires low-latency processing (to process a single document or small set of documents quickly), use [real-time scoring](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-consume-web-service) instead of batch prediction."
       ]
     },
     {
@@ -356,7 +353,9 @@
       "source": [
         "from azureml.pipeline.core.graph import PipelineParameter\n",
         "# create a parameter for style (one of \"candy\", \"mosaic\") to transfer the images to\n",
-        "style_param = PipelineParameter(name=\"style\", default_value=\"mosaic\")"
+        "style_param = PipelineParameter(name=\"style\", default_value=\"mosaic\")\n",
+        "# create a parameter for the number of nodes to use in step no. 2 (style transfer)\n",
+        "nodecount_param = PipelineParameter(name=\"nodecount\", default_value=2)"
       ]
     },
     {
@@ -415,6 +414,8 @@
         "parallel_cd.add_conda_package(\"pytorch\")\n",
         "parallel_cd.add_conda_package(\"torchvision\")\n",
         "parallel_cd.add_conda_package(\"pillow<7\") # needed for torchvision==0.4.0\n",
+        "parallel_cd.add_pip_package(\"azureml-core\")\n",
+        "parallel_cd.add_pip_package(\"azureml-dataprep[fuse]\")\n",
         "\n",
         "styleenvironment = Environment(name=\"styleenvironment\")\n",
         "styleenvironment.python.conda_dependencies=parallel_cd\n",
@@ -427,17 +428,20 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunConfig\n",
+        "from azureml.pipeline.core import PipelineParameter\n",
+        "from azureml.pipeline.steps import ParallelRunConfig\n",
         "\n",
         "parallel_run_config = ParallelRunConfig(\n",
-        "                    environment=styleenvironment,\n",
-        "                    entry_script='transform.py',\n",
-        "                    output_action='summary_only',\n",
-        "                    mini_batch_size=\"1\",\n",
-        "                    error_threshold=1,\n",
-        "                    source_directory=scripts_folder,\n",
-        "                    compute_target=gpu_cluster, \n",
-        "                    node_count=3)"
+        "    environment=styleenvironment,\n",
+        "    entry_script='transform.py',\n",
+        "    output_action='summary_only',\n",
+        "    mini_batch_size=\"1\",\n",
+        "    error_threshold=1,\n",
+        "    source_directory=scripts_folder,\n",
+        "    compute_target=gpu_cluster, \n",
+        "    node_count=nodecount_param,\n",
+        "    process_count_per_node=2\n",
+        ")"
       ]
     },
     {
@@ -446,7 +450,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunStep\n",
+        "from azureml.pipeline.steps import ParallelRunStep\n",
         "from datetime import datetime\n",
         "\n",
         "parallel_step_name = 'styletransfer-' + datetime.now().strftime('%Y%m%d%H%M')\n",
@@ -455,9 +459,6 @@
         "    name=parallel_step_name,\n",
         "    inputs=[ffmpeg_images_file_dataset], # Input file share/blob container/file dataset\n",
         "    output=processed_images,  # Output file share/blob container\n",
-        "    models=[mosaic_model, candy_model],\n",
-        "    tags = {'scenario': \"batch inference\", 'type': \"demo\"},\n",
-        "    properties = {'area': \"style transfer\"},\n",
         "    arguments=[\"--style\", style_param],\n",
         "    parallel_run_config=parallel_run_config,\n",
         "    allow_reuse=True #[optional - default value True]\n",
@@ -666,7 +667,8 @@
         "response = requests.post(rest_endpoint, \n",
         "                         headers=aad_token,\n",
         "                         json={\"ExperimentName\": experiment_name,\n",
-        "                               \"ParameterAssignments\": {\"style\": \"candy\", \"aml_node_count\": 2}})\n",
+        "                               \"ParameterAssignments\": {\"style\": \"candy\", \"NodeCount\": 3}})\n",
+        "\n",
         "run_id = response.json()[\"Id\"]\n",
         "\n",
         "from azureml.pipeline.core.run import PipelineRun\n",

how-to-use-azureml/machine-learning-pipelines/pipeline-style-transfer/pipeline-style-transfer.yml

@@ -2,7 +2,6 @@ name: pipeline-style-transfer
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-contrib-pipeline-steps
   - azureml-pipeline-steps
   - azureml-widgets
   - requests

how-to-use-azureml/ml-frameworks/scikit-learn/training/train-hyperparameter-tune-deploy-with-sklearn/train-hyperparameter-tune-deploy-with-sklearn.ipynb

@@ -381,7 +381,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "run.cancel()"
+        "run.wait_for_completion(show_output=True)"
       ]
     },
     {

how-to-use-azureml/monitor-models/data-drift/drift-on-aks.ipynb

@@ -456,6 +456,24 @@
         "monitor.enable_schedule()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Delete the DataDriftDetector\n",
+        "\n",
+        "Invoking the `delete()` method on the object deletes the the drift monitor permanently and cannot be undone. You will no longer be able to find it in the UI and the `list()` or `get()` methods. The object on which delete() was called will have its state set to deleted and name suffixed with deleted. The baseline and target datasets and model data that was collected, if any, are not deleted. The compute is not deleted. The DataDrift schedule pipeline is disabled and archived."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "monitor.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/monitor-models/data-drift/myenv.yml

@@ -7,4 +7,5 @@ dependencies:
     - azureml-monitoring
     - scikit-learn
     - numpy
+    - packaging
     - inference-schema[numpy-support]

how-to-use-azureml/monitor-models/data-drift/score.py

@@ -4,7 +4,13 @@ import numpy as np
 from azureml.monitoring import ModelDataCollector
 from inference_schema.parameter_types.numpy_parameter_type import NumpyParameterType
 from inference_schema.schema_decorators import input_schema, output_schema
-from sklearn.externals import joblib
+# sklearn.externals.joblib is removed in 0.23
+from sklearn import __version__ as sklearnver
+from packaging.version import Version
+if Version(sklearnver) < Version("0.23.0"):
+    from sklearn.externals import joblib
+else:
+    import joblib
 
 
 def init():

how-to-use-azureml/reinforcement-learning/README.md

@@ -11,19 +11,29 @@ Taxonomies for products and languages: https://review.docs.microsoft.com/new-hop
 
 This is an introduction to the [Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/service/) Reinforcement Learning (Public Preview) using the [Ray](https://github.com/ray-project/ray/) framework.
 
-Using these samples, you will be able to do the following.
+## What is reinforcement learning?
 
-1. Use an Azure Machine Learning workspace, set up virtual network and create compute clusters for running Ray.
-2. Run some experiments to train a reinforcement learning agent using Ray and RLlib.
+Reinforcement learning is an approach to machine learning to train agents to make a sequence of decisions.  This technique has gained popularity over the last few years as breakthroughs have been made to teach reinforcement learning agents to excel at complex tasks like playing video games.  There are many practical real-world use cases as well, including robotics, chemistry, online recommendations, advertising and more.
+
+In reinforcement learning, the goal is to train an agent *policy* that outputs actions based on the agent’s observations of its environment.  Actions result in further observations and *rewards* for taking the actions.  In reinforcement learning, the full reward for policy actions may take many steps to obtain.  Learning a policy involves many trial-and-error runs of the agent interacting with the environment and improving its policy. 
+
+## Reinforcement learning on Azure Machine Learning
+
+Reinforcement learning support in Azure Machine Learning service enables data scientists to scale training to many powerful CPU or GPU enabled VMs using [Azure Machine Learning compute clusters](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-set-up-training-targets#amlcompute) which automatically provision, manage, and scale down these VMs to help manage your costs.
+
+Using these samples, you will learn how to do the following.
+
+1. Use an Azure Machine Learning workspace, set up virtual network and create compute clusters for distributed training.
+2. Train reinforcement learning agents using Ray RLlib.
 
 ## Contents
 
 | File/folder       | Description                                |
 |-------------------|--------------------------------------------|
-| [devenv_setup.ipynb](setup/devenv_setup.ipynb) | Notebook to setup development environment for Azure ML RL |
-| [cartpole_ci.ipynb](cartpole-on-compute-instance/cartpole_ci.ipynb)  | Notebook to train a Cartpole playing agent on an Azure ML Compute Instance |
-| [cartpole_cc.ipynb](cartpole-on-single-compute/cartpole_cc.ipynb)  | Notebook to train a Cartpole playing agent on an Azure ML Compute Cluster (single node) |
-| [pong_rllib.ipynb](atari-on-distributed-compute/pong_rllib.ipynb)   | Notebook to train Pong agent using RLlib on multiple compute targets |
+| [devenv_setup.ipynb](setup/devenv_setup.ipynb) | Notebook to setup virtual network for using Azure Machine Learning.  Needed for the Pong and Minecraft examples. |
+| [cartpole_ci.ipynb](cartpole-on-compute-instance/cartpole_ci.ipynb)  | Notebook to train a Cartpole playing agent on an Azure Machine Learning Compute Instance |
+| [cartpole_sc.ipynb](cartpole-on-single-compute/cartpole_sc.ipynb)  | Notebook to train a Cartpole playing agent on an Azure Machine Learning Compute Cluster (single node) |
+| [pong_rllib.ipynb](atari-on-distributed-compute/pong_rllib.ipynb)   | Notebook for distributed training of Pong agent using RLlib on multiple compute targets |
 | [minecraft.ipynb](minecraft-on-distributed-compute/minecraft.ipynb)   | Notebook to train an agent to navigate through a lava maze in the Minecraft game |
 
 ## Prerequisites
@@ -32,9 +42,10 @@ To make use of these samples, you need the following.
 
 * A Microsoft Azure subscription.
 * A Microsoft Azure resource group.
-* An Azure Machine Learning Workspace in the resource group. Please make sure that the VM sizes `STANDARD_NC6` and `STANDARD_D2_V2` are supported in the workspace's region.
-* A virtual network set up in the resource group.
-  * A virtual network is needed for the examples training on multiple compute targets.
+* An Azure Machine Learning Workspace in the resource group.
+* Azure Machine Learning training compute. These samples use the VM sizes `STANDARD_NC6` and `STANDARD_D2_V2`.  If these are not available in your region,
+you can replace them with other sizes.
+* A virtual network set up in the resource group for samples that use multiple compute targets.  The Cartpole examples do not need a virtual network.
   * The [devenv_setup.ipynb](setup/devenv_setup.ipynb) notebook shows you how to create a virtual network. You can alternatively use an existing virtual network, make sure it's in the same region as workspace is.
   * Any network security group defined on the virtual network must allow network traffic on ports used by Azure infrastructure services. This is described in more detail in the [devenv_setup.ipynb](setup/devenv_setup.ipynb) notebook.
 
@@ -43,10 +54,10 @@ To make use of these samples, you need the following.
 
 You can run these samples in the following ways.
 
-* On an Azure ML Compute Instance or Notebook VM.
+* On an Azure Machine Learning Compute Instance or Azure Data Science Virtual Machine (DSVM).
 * On a workstation with Python and the Azure ML Python SDK installed.
 
-### Azure ML Compute Instance or Notebook VM
+### Compute Instance or DSVM
 #### Update packages
 
 

how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/files/networkutils.py

@@ -0,0 +1,237 @@
+import sys
+import csv
+from azure.mgmt.network import NetworkManagementClient
+
+
+def check_port_in_port_range(expected_port: str,
+                             dest_port_range: str):
+    """
+    Check if a port is within a port range
+    Port range maybe like *, 8080 or 8888-8889
+    """
+
+    if dest_port_range == '*':
+        return True
+
+    dest_ports = dest_port_range.split('-')
+
+    if len(dest_ports) == 1 and \
+       int(dest_ports[0]) == int(expected_port):
+        return True
+
+    if len(dest_ports) == 2 and \
+       int(dest_ports[0]) <= int(expected_port) and \
+       int(dest_ports[1]) >= int(expected_port):
+        return True
+
+    return False
+
+
+def check_port_in_destination_port_ranges(expected_port: str,
+                                          dest_port_ranges: list):
+    """
+    Check if a port is within a given list of port ranges
+    i.e. check if port 8080 is in port ranges of 22,80,8080-8090,443
+    """
+
+    for dest_port_range in dest_port_ranges:
+        if check_port_in_port_range(expected_port, dest_port_range) is True:
+            return True
+
+    return False
+
+
+def check_ports_in_destination_port_ranges(expected_ports: list,
+                                           dest_port_ranges: list):
+    """
+    Check if all ports in a given port list are within a given list
+    of port ranges
+    i.e. check if port 8080,8081 are in port ranges of 22,80,8080-8090,443
+    """
+
+    for expected_port in expected_ports:
+        if check_port_in_destination_port_ranges(
+           expected_port, dest_port_ranges) is False:
+            return False
+
+    return True
+
+
+def check_source_address_prefix(source_address_prefix: str):
+    """Check if source address prefix is BatchNodeManagement or default"""
+
+    required_prefix = 'BatchNodeManagement'
+    default_prefix = 'default'
+
+    if source_address_prefix.lower() == required_prefix.lower() or \
+       source_address_prefix.lower() == default_prefix.lower():
+        return True
+
+    return False
+
+
+def check_protocol(protocol: str):
+    """Check if protocol is supported - Tcp/Any"""
+
+    required_protocol = 'Tcp'
+    any_protocol = 'Any'
+
+    if required_protocol.lower() == protocol.lower() or \
+       any_protocol.lower() == protocol.lower():
+        return True
+
+    return False
+
+
+def check_direction(direction: str):
+    """Check if port direction is inbound"""
+
+    required_direction = 'Inbound'
+
+    if required_direction.lower() == direction.lower():
+        return True
+
+    return False
+
+
+def check_provisioning_state(provisioning_state: str):
+    """Check if the provisioning state is succeeded"""
+
+    required_provisioning_state = 'Succeeded'
+
+    if required_provisioning_state.lower() == provisioning_state.lower():
+        return True
+
+    return False
+
+
+def check_rule_for_Azure_ML(rule):
+    """Check if the ports required for Azure Machine Learning are open"""
+
+    required_ports = ['29876', '29877']
+
+    if check_source_address_prefix(rule.source_address_prefix) is False:
+        return False
+
+    if check_protocol(rule.protocol) is False:
+        return False
+
+    if check_direction(rule.direction) is False:
+        return False
+
+    if check_provisioning_state(rule.provisioning_state) is False:
+        return False
+
+    if rule.destination_port_range is not None:
+        if check_ports_in_destination_port_ranges(
+           required_ports,
+           [rule.destination_port_range]) is False:
+            return False
+    else:
+        if check_ports_in_destination_port_ranges(
+           required_ports,
+           rule.destination_port_ranges) is False:
+            return False
+
+    return True
+
+
+def check_vnet_security_rules(auth_object,
+                              vnet_subscription_id,
+                              vnet_resource_group,
+                              vnet_name,
+                              save_to_file=False):
+    """
+    Check all the rules of virtual network if required ports for Azure Machine
+    Learning are open
+    """
+
+    network_client = NetworkManagementClient(
+        auth_object,
+        vnet_subscription_id)
+
+    # get the vnet
+    vnet = network_client.virtual_networks.get(
+        resource_group_name=vnet_resource_group,
+        virtual_network_name=vnet_name)
+
+    vnet_location = vnet.location
+    vnet_info = []
+
+    if vnet.subnets is None or len(vnet.subnets) == 0:
+        print('WARNING: No subnet found for VNet:', vnet_name)
+
+    # for each subnet of the vnet
+    for subnet in vnet.subnets:
+        if subnet.network_security_group is None:
+            print('WARNING: No network security group found for subnet.',
+                  'Subnet',
+                  subnet.id.split("/")[-1])
+        else:
+            # get all the rules
+            network_security_group_name = \
+                subnet.network_security_group.id.split("/")[-1]
+            network_security_group_resource_group_name = \
+                subnet.network_security_group.id.split("/")[4]
+            network_security_group_subscription_id = \
+                subnet.network_security_group.id.split("/")[2]
+
+            security_rules = list(network_client.security_rules.list(
+                network_security_group_resource_group_name,
+                network_security_group_name))
+
+            rule_matched = None
+            for rule in security_rules:
+                rule_info = []
+                # add vnet details
+                rule_info.append(vnet_name)
+                rule_info.append(vnet_subscription_id)
+                rule_info.append(vnet_resource_group)
+                rule_info.append(vnet_location)
+                # add subnet details
+                rule_info.append(subnet.id.split("/")[-1])
+                rule_info.append(network_security_group_name)
+                rule_info.append(network_security_group_subscription_id)
+                rule_info.append(network_security_group_resource_group_name)
+                # add rule details
+                rule_info.append(rule.priority)
+                rule_info.append(rule.name)
+                rule_info.append(rule.source_address_prefix)
+                if rule.destination_port_range is not None:
+                    rule_info.append(rule.destination_port_range)
+                else:
+                    rule_info.append(rule.destination_port_ranges)
+                rule_info.append(rule.direction)
+                rule_info.append(rule.provisioning_state)
+                vnet_info.append(rule_info)
+
+                if check_rule_for_Azure_ML(rule) is True:
+                    rule_matched = rule
+
+            if rule_matched is not None:
+                print("INFORMATION: Rule matched with required ports. Subnet:",
+                      subnet.id.split("/")[-1], "Rule:", rule.name)
+            else:
+                print("WARNING: No rule matched with required ports. Subnet:",
+                      subnet.id.split("/")[-1])
+
+    if save_to_file is True:
+        file_name = vnet_name + ".csv"
+        with open(file_name, mode='w') as vnet_rule_file:
+            vnet_rule_file_writer = csv.writer(
+                vnet_rule_file,
+                delimiter=',',
+                quotechar='"',
+                quoting=csv.QUOTE_MINIMAL)
+            header = ['VNet_Name', 'VNet_Subscription_ID',
+                      'VNet_Resource_Group', 'VNet_Location',
+                      'Subnet_Name', 'NSG_Name',
+                      'NSG_Subscription_ID', 'NSG_Resource_Group',
+                      'Rule_Priority', 'Rule_Name', 'Rule_Source',
+                      'Rule_Destination_Ports', 'Rule_Direction',
+                      'Rule_Provisioning_State']
+            vnet_rule_file_writer.writerow(header)
+            vnet_rule_file_writer.writerows(vnet_info)
+
+        print("INFORMATION: Network security group rules for your virtual \
+network are saved in file", file_name)

how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/files/pong_rllib.py

@@ -23,17 +23,18 @@ if __name__ == "__main__":
 
     ray.init(address=args.ray_address)
 
-    tune.run(run_or_experiment=args.run,
-             config={
-                 "env": args.env,
-                 "num_gpus": args.config["num_gpus"],
-                 "num_workers": args.config["num_workers"],
-                 "callbacks": {"on_train_result": callbacks.on_train_result},
-                 "sample_batch_size": 50,
-                 "train_batch_size": 1000,
-                 "num_sgd_iter": 2,
-                 "num_data_loader_buffers": 2,
-                 "model": {"dim": 42},
-             },
-             stop=args.stop,
-             local_dir='./logs')
+    tune.run(
+        run_or_experiment=args.run,
+        config={
+            "env": args.env,
+            "num_gpus": args.config["num_gpus"],
+            "num_workers": args.config["num_workers"],
+            "callbacks": {"on_train_result": callbacks.on_train_result},
+            "sample_batch_size": 50,
+            "train_batch_size": 1000,
+            "num_sgd_iter": 2,
+            "num_data_loader_buffers": 2,
+            "model": {"dim": 42},
+        },
+        stop=args.stop,
+        local_dir='./logs')

how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.ipynb

@@ -13,15 +13,15 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.png)"
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.png)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Azure ML Reinforcement Learning Sample - Pong problem\n",
-        "Azure ML Reinforcement Learning (Azure ML RL) is a managed service for running distributed RL (reinforcement learning) simulation and training using the Ray framework.\n",
+        "# Reinforcement Learning in Azure Machine Learning - Pong problem\n",
+        "Reinforcement Learning in Azure Machine Learning is a managed service for running distributed reinforcement learning training and simulation using the open source Ray framework.\n",
         "This example uses Ray RLlib to train a Pong playing agent on a multi-node cluster.\n",
         "\n",
         "## Pong problem\n",
@@ -48,7 +48,7 @@
       "source": [
         "The goal here is to train an agent to win an episode of Pong game against opponent with the score of at least 18 points. An episode in Pong runs until one of the players reaches a score of 21. Episodes are a terminology that is used across all the [OpenAI gym](https://gym.openai.com/envs/Pong-v0/) environments that contains a strictly defined task.\n",
         "\n",
-        "Training a Pong agent is a CPU intensive task and this example demonstrates the use of Azure ML RL service to train an agent faster in a distributed, parallel environment. You'll learn more about using the head and the worker compute targets to train an agent in this notebook below."
+        "Training a Pong agent is a compute-intensive task and this example demonstrates the use of Reinforcement Learning in Azure Machine Learning service to train an agent faster in a distributed, parallel environment. You'll learn more about using the head and the worker compute targets to train an agent in this notebook below."
       ]
     },
     {
@@ -57,7 +57,7 @@
       "source": [
         "## Prerequisite\n",
         "\n",
-        "The user should have completed the [Azure ML Reinforcement Learning Sample - Setting Up Development Environment](../setup/devenv_setup.ipynb) to setup a virtual network. This virtual network will be used here for head and worker compute targets. It is highly recommended that the user should go through the [Azure ML Reinforcement Learning Sample - Cartpole Problem](../cartpole-on-single-compute/cartpole_cc.ipynb) to understand the basics of Azure ML RL and Ray RLlib used in this notebook."
+        "The user should have completed the [Reinforcement Learning in Azure Machine Learning - Setting Up Development Environment](../setup/devenv_setup.ipynb) to setup a virtual network. This virtual network will be used here for head and worker compute targets. It is highly recommended that the user should go through the [Reinforcement Learning in Azure Machine Learning - Cartpole Problem on Single Compute](../cartpole-on-single-compute/cartpole_sc.ipynb) to understand the basics of Reinforcement Learning in Azure Machine Learning and Ray RLlib used in this notebook."
       ]
     },
     {
@@ -69,7 +69,7 @@
         "\n",
         "* Connecting to a workspace to enable communication between your local machine and remote resources\n",
         "* Creating an experiment to track all your runs\n",
-        "* Creating a remote head and worker compute target on a vnet to use for training"
+        "* Creating remote head and worker compute target on a virtual network to use for training"
       ]
     },
     {
@@ -88,19 +88,19 @@
       "source": [
         "%matplotlib inline\n",
         "\n",
-        "# Azure ML core imports\n",
+        "# Azure Machine Learning core imports\n",
         "import azureml.core\n",
         "\n",
         "# Check core SDK version number\n",
-        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
+        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Get Azure ML workspace\n",
-        "Get a reference to an existing Azure ML workspace."
+        "### Get Azure Machine Learning workspace\n",
+        "Get a reference to an existing Azure Machine Learning workspace."
       ]
     },
     {
@@ -119,7 +119,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Create Azure ML experiment\n",
+        "### Create Azure Machine Learning experiment\n",
         "Create an experiment to track the runs in your workspace."
       ]
     },
@@ -140,9 +140,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Specify the name of your vnet\n",
+        "### Specify the name of your virtual network\n",
         "\n",
-        "The resource group you use must contain a vnet.  Specify the name of the vnet here created in the [Azure ML Reinforcement Learning Sample - Setting Up Development Environment](../setup/devenv_setup.ipynb)."
+        "The resource group you use must contain a virtual network. Specify the name of the virtual network here created in the [Azure Machine Learning Reinforcement Learning Sample - Setting Up Development Environment](../setup/devenv_setup.ipynb)."
       ]
     },
     {
@@ -159,9 +159,27 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Create head computing cluster\n",
+        "Ensure that the virtual network is configured correctly with required ports open. It is possible that you have configured rules with broader range of ports that allows ports 29876-29877 to be opened. Kindly review your network security group rules. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from files.networkutils import *\n",
         "\n",
-        "In this example, we show how to set up separate compute clusters for the Ray head and Ray worker nodes. First we define the head cluster with GPU for the Ray head node. One CPU of the head node will be used for the Ray head process and the rest of the CPUs will be used by the Ray worker processes."
+        "check_vnet_security_rules(ws._auth_object, ws.subscription_id, ws.resource_group, vnet_name, True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Create head compute target\n",
+        "\n",
+        "In this example, we show how to set up separate compute targets for the Ray head and Ray worker nodes. First we define the head cluster with GPU for the Ray head node. One CPU of the head node will be used for the Ray head process and the rest of the CPUs will be used by the Ray worker processes."
       ]
     },
     {
@@ -186,15 +204,17 @@
         "        if head_compute_target.provisioning_state == 'Succeeded':\n",
         "            print('found head compute target. just use it', head_compute_name)\n",
         "        else: \n",
-        "            raise Exception('found head compute target but it is in state', head_compute_target.provisioning_state)\n",
+        "            raise Exception(\n",
+        "                'found head compute target but it is in state', head_compute_target.provisioning_state)\n",
         "else:\n",
         "    print('creating a new head compute target...')\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size=head_vm_size,\n",
-        "                                                                min_nodes=head_compute_min_nodes, \n",
-        "                                                                max_nodes=head_compute_max_nodes,\n",
-        "                                                                vnet_resourcegroup_name=ws.resource_group,\n",
-        "                                                                vnet_name=vnet_name,\n",
-        "                                                                subnet_name='default')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(\n",
+        "        vm_size=head_vm_size,\n",
+        "        min_nodes=head_compute_min_nodes, \n",
+        "        max_nodes=head_compute_max_nodes,\n",
+        "        vnet_resourcegroup_name=ws.resource_group,\n",
+        "        vnet_name=vnet_name,\n",
+        "        subnet_name='default')\n",
         "\n",
         "    # Create the cluster\n",
         "    head_compute_target = ComputeTarget.create(ws, head_compute_name, provisioning_config)\n",
@@ -203,7 +223,7 @@
         "    # If no min node count is provided it will use the scale settings for the cluster\n",
         "    head_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "    \n",
-        "     # For a more detailed view of current AmlCompute status, use get_status()\n",
+        "    # For a more detailed view of current AmlCompute status, use get_status()\n",
         "    print(head_compute_target.get_status().serialize())"
       ]
     },
@@ -211,9 +231,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Create worker computing cluster\n",
+        "### Create worker compute target\n",
         "\n",
-        "Now we create a compute cluster with CPUs for the additional Ray worker nodes. CPUs in these worker nodes are used by Ray worker processes. Each Ray worker node may have multiple Ray worker processes depending on CPUs on the worker node. Ray can distribute multiple worker tasks on each worker node."
+        "Now we create a compute target with CPUs for the additional Ray worker nodes. CPUs in these worker nodes are used by Ray worker processes. Each Ray worker node, depending on the CPUs on the node, may have multiple Ray worker processes. There can be multiple worker tasks on each worker process (core)."
       ]
     },
     {
@@ -222,7 +242,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Choose a name for your Ray worker cluster\n",
+        "# Choose a name for your Ray worker compute target\n",
         "worker_compute_name = 'worker-cpu'\n",
         "worker_compute_min_nodes = 0 \n",
         "worker_compute_max_nodes = 4\n",
@@ -237,24 +257,26 @@
         "        if worker_compute_target.provisioning_state == 'Succeeded':\n",
         "            print('found worker compute target. just use it', worker_compute_name)\n",
         "        else: \n",
-        "            raise Exception('found worker compute target but it is in state', head_compute_target.provisioning_state)\n",
+        "            raise Exception(\n",
+        "                'found worker compute target but it is in state', head_compute_target.provisioning_state)\n",
         "else:\n",
         "    print('creating a new worker compute target...')\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size=worker_vm_size,\n",
-        "                                                                min_nodes=worker_compute_min_nodes, \n",
-        "                                                                max_nodes=worker_compute_max_nodes,\n",
-        "                                                                vnet_resourcegroup_name=ws.resource_group,\n",
-        "                                                                vnet_name=vnet_name,\n",
-        "                                                                subnet_name='default')\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(\n",
+        "        vm_size=worker_vm_size,\n",
+        "        min_nodes=worker_compute_min_nodes,\n",
+        "        max_nodes=worker_compute_max_nodes,\n",
+        "        vnet_resourcegroup_name=ws.resource_group,\n",
+        "        vnet_name=vnet_name,\n",
+        "        subnet_name='default')\n",
         "\n",
-        "    # Create the cluster\n",
+        "    # Create the compute target\n",
         "    worker_compute_target = ComputeTarget.create(ws, worker_compute_name, provisioning_config)\n",
         "    \n",
         "    # Can poll for a minimum number of nodes and for a specific timeout. \n",
         "    # If no min node count is provided it will use the scale settings for the cluster\n",
         "    worker_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
         "    \n",
-        "     # For a more detailed view of current AmlCompute status, use get_status()\n",
+        "    # For a more detailed view of current AmlCompute status, use get_status()\n",
         "    print(worker_compute_target.get_status().serialize())"
       ]
     },
@@ -262,12 +284,12 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Train Pong Agent Using Azure ML RL\n",
-        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct RL run configurations for the underlying RL framework. Azure ML RL initially supports the [Ray framework](https://ray.io/) and its highly customizable [RLLib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLLib framework to train a Pong playing agent.\n",
+        "## Train Pong Agent\n",
+        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct  reinforcement learning run configurations for the underlying reinforcement learning framework.  Reinforcement Learning in Azure Machine Learning supports the open source [Ray framework](https://ray.io/) and its highly customizable [RLLib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLLib framework to train a Pong playing agent.\n",
         "\n",
         "\n",
         "### Define worker configuration\n",
-        "Define a `WorkerConfiguration` using your worker compute target. We also specify the number of nodes in the worker compute target to be used for training and additional PIP packages to install on those nodes as a part of setup.\n",
+        "Define a `WorkerConfiguration` using your worker compute target. We specify the number of nodes in the worker compute target to be used for training and additional PIP packages to install on those nodes as a part of setup.\n",
         "In this case, we define the PIP packages as dependencies for both head and worker nodes. With this setup, the game simulations will run directly on the worker compute nodes."
       ]
     },
@@ -285,7 +307,7 @@
         "# Specify the Ray worker configuration\n",
         "worker_conf = WorkerConfiguration(\n",
         "    \n",
-        "    # Azure ML compute cluster to run Ray workers\n",
+        "    # Azure Machine Learning compute target to run Ray workers\n",
         "    compute_target=worker_compute_target, \n",
         "    \n",
         "    # Number of worker nodes\n",
@@ -305,7 +327,7 @@
       "source": [
         "### Create reinforcement learning estimator\n",
         "\n",
-        "The `ReinforcementLearningEstimator` is used to submit a job to Azure Machine Learning to start the Ray experiment run. We define the training script parameters here that will be passed to estimator. \n",
+        "The `ReinforcementLearningEstimator` is used to submit a job to Azure Machine Learning to start the Ray experiment run. We define the training script parameters here that will be passed to the estimator. \n",
         "\n",
         "We specify `episode_reward_mean` to 18 as we want to stop the training as soon as the trained agent reaches an average win margin of at least 18 point over opponent over all episodes in the training epoch.\n",
         "Number of Ray worker processes are defined by parameter `num_workers`. We set it to 13 as we have 13 CPUs available in our compute targets. Multiple Ray worker processes parallelizes agent training and helps in achieving our goal faster. \n",
@@ -348,7 +370,7 @@
         "    \"--stop\": '\\'{\"episode_reward_mean\": 18, \"time_total_s\": 3600}\\'',\n",
         "}\n",
         "\n",
-        "# RL estimator\n",
+        "#  Reinforcement learning estimator\n",
         "rl_estimator = ReinforcementLearningEstimator(\n",
         "    \n",
         "    # Location of source files\n",
@@ -361,7 +383,7 @@
         "    # Defined above.\n",
         "    script_params=script_params,\n",
         "    \n",
-        "    # The Azure ML compute target set up for Ray head nodes\n",
+        "    # The Azure Machine Learning compute target set up for Ray head nodes\n",
         "    compute_target=head_compute_target,\n",
         "    \n",
         "    # Pip packages\n",
@@ -370,7 +392,7 @@
         "    # GPU usage\n",
         "    use_gpu=True,\n",
         "    \n",
-        "    # RL framework.  Currently must be Ray.\n",
+        "    # Reinforcement learning framework. Currently must be Ray.\n",
         "    rl_framework=Ray(),\n",
         "    \n",
         "    # Ray worker configuration defined above.\n",
@@ -394,23 +416,24 @@
       "metadata": {},
       "source": [
         "### Training script\n",
-        "As recommended in [RLLib](https://ray.readthedocs.io/en/latest/rllib.html) documentations, we use Ray [Tune](https://ray.readthedocs.io/en/latest/tune.html) API to run training algorithm. All the RLLib built-in trainers are compatible with the Tune API. Here we use tune.run() to execute a built-in training algorithm. For convenience, down below you can see part of the entry script where we make this call.\n",
+        "As recommended in [RLlib](https://ray.readthedocs.io/en/latest/rllib.html) documentations, we use Ray [Tune](https://ray.readthedocs.io/en/latest/tune.html) API to run the training algorithm. All the RLlib built-in trainers are compatible with the Tune API. Here we use tune.run() to execute a built-in training algorithm. For convenience, down below you can see part of the entry script where we make this call.\n",
         "\n",
         "```python\n",
-        "    tune.run(run_or_experiment=args.run,\n",
-        "             config={\n",
-        "                 \"env\": args.env,\n",
-        "                 \"num_gpus\": args.config[\"num_gpus\"],\n",
-        "                 \"num_workers\": args.config[\"num_workers\"],\n",
-        "                 \"callbacks\": {\"on_train_result\": callbacks.on_train_result},\n",
-        "                 \"sample_batch_size\": 50,\n",
-        "                 \"train_batch_size\": 1000,\n",
-        "                 \"num_sgd_iter\": 2,\n",
-        "                 \"num_data_loader_buffers\": 2,\n",
-        "                 \"model\": {\"dim\": 42},\n",
-        "             },\n",
-        "             stop=args.stop,\n",
-        "             local_dir='./logs')\n",
+        "    tune.run(\n",
+        "        run_or_experiment=args.run,\n",
+        "        config={\n",
+        "            \"env\": args.env,\n",
+        "            \"num_gpus\": args.config[\"num_gpus\"],\n",
+        "            \"num_workers\": args.config[\"num_workers\"],\n",
+        "            \"callbacks\": {\"on_train_result\": callbacks.on_train_result},\n",
+        "            \"sample_batch_size\": 50,\n",
+        "            \"train_batch_size\": 1000,\n",
+        "            \"num_sgd_iter\": 2,\n",
+        "            \"num_data_loader_buffers\": 2,\n",
+        "            \"model\": {\"dim\": 42},\n",
+        "        },\n",
+        "        stop=args.stop,\n",
+        "        local_dir='./logs')\n",
         "```"
       ]
     },
@@ -437,7 +460,7 @@
       "source": [
         "### Monitor the run\n",
         "\n",
-        "Azure ML provides a Jupyter widget to show the real-time status of an experiment run. You could use this widget to monitor the status of runs. The widget shows the list of two child runs, one for head compute target run and one for worker compute target run, as well. You can click on the link under Status to see the details of the child run."
+        "Azure Machine Learning provides a Jupyter widget to show the status of an experiment run. You could use this widget to monitor the status of the runs. The widget shows the list of two child runs, one for head compute target run and one for worker compute target run. You can click on the link under **Status** to see the details of the child run. It will also show the metrics being logged."
       ]
     },
     {
@@ -455,9 +478,29 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
+        "### Stop the run\n",
+        "\n",
+        "To stop the run, call `run.cancel()`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Uncomment line below to cancel the run\n",
+        "# run.cancel()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Wait for completion\n",
         "Wait for the run to complete before proceeding. If you want to stop the run, you may skip this and move to next section below. \n",
         "\n",
-        "**Note: the run may take anywhere from 30 minutes to 45 minutes to complete.**"
+        "**Note: The run may take anywhere from 30 minutes to 45 minutes to complete.**"
       ]
     },
     {
@@ -469,24 +512,6 @@
         "run.wait_for_completion()"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Stop the run\n",
-        "\n",
-        "To cancel the run, call run.cancel()."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# run.cancel()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -539,8 +564,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "We observe that during the training over multiple episodes, the agent learn to win the Pong game against opponent with our target of 18 points in each episode of 21 points.\n",
-        "**Congratulations!! You have trained your Pong agent to win a game marvelously.**"
+        "We observe that during the training over multiple episodes, the agent learns to win the Pong game against opponent with our target of 18 points in each episode of 21 points.\n",
+        "**Congratulations!! You have trained your Pong agent to win a game.**"
       ]
     },
     {
@@ -570,7 +595,7 @@
       "metadata": {},
       "source": [
         "## Next\n",
-        "In this example, you learnt how to solve distributed RL training problems using head and worker compute targets. This is currently the last introductory tutorial for Azure Machine Learning service's Reinforcement Learning offering. We would love to hear your feedback to build the features you need!"
+        "In this example, you learned how to solve distributed reinforcement learning training problems using head and worker compute targets. This was an introductory tutorial on Reinforement Learning in Azure Machine Learning service offering. We would love to hear your feedback to build the features you need!"
       ]
     }
   ],
@@ -595,7 +620,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.7.4"
+      "version": "3.6.9"
     },
     "notice": "Copyright (c) Microsoft Corporation. All rights reserved.\u00e2\u20ac\u00afLicensed under the MIT License.\u00e2\u20ac\u00af "
   },

how-to-use-azureml/reinforcement-learning/cartpole-on-compute-instance/cartpole_ci.ipynb

@@ -13,18 +13,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/how-to-use-azureml/reinforcement-learning/cartpole-on-compute-instance/cartpole_ci.png)"
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/reinforcement-learning/cartpole-on-compute-instance/cartpole_ci.png)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Azure ML Reinforcement Learning Sample - Cartpole Problem on Compute Instance\n",
+        "# Reinforcement Learning in Azure Machine Learning - Cartpole Problem on Compute Instance\n",
         "\n",
-        "Azure ML Reinforcement Learning (Azure ML RL) is a managed service for running reinforcement learning training and simulation. With Azure MLRL, data scientists can start developing RL systems on one machine, and scale to compute clusters with 100\u00e2\u20ac\u2122s of nodes if needed.\n",
+        "Reinforcement Learning in Azure Machine Learning is a managed service for running reinforcement learning training and simulation. With Reinforcement Learning in Azure Machine Learning, data scientists can start developing reinforcement learning systems on one machine, and scale to compute targets with 100\u00e2\u20ac\u2122s of nodes if needed.\n",
         "\n",
-        "This example shows how to use Azure ML RL to train a Cartpole playing agent on a compute instance."
+        "This example shows how to use Reinforcement Learning in Azure Machine Learning to train a Cartpole playing agent on a compute instance."
       ]
     },
     {
@@ -56,7 +56,7 @@
       "metadata": {},
       "source": [
         "### Prerequisite\n",
-        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription id, a resource group and a workspace. All datastores and datasets you use should be associated with your workspace."
+        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription ID, a resource group, and an Azure Machine Learning workspace. All datastores and datasets you use should be associated with your workspace."
       ]
     },
     {
@@ -75,8 +75,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Azure ML SDK \n",
-        "Display the Azure ML SDK version."
+        "### Azure Machine Learning SDK \n",
+        "Display the Azure Machine Learning SDK version."
       ]
     },
     {
@@ -86,15 +86,15 @@
       "outputs": [],
       "source": [
         "import azureml.core\n",
-        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
+        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Get Azure ML workspace\n",
-        "Get a reference to an existing Azure ML workspace."
+        "### Get Azure Machine Learning workspace\n",
+        "Get a reference to an existing Azure Machine Learning workspace."
       ]
     },
     {
@@ -161,21 +161,34 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "from azureml.core.compute import ComputeTarget, ComputeInstance\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
         "# Load current compute instance info\n",
         "current_compute_instance = load_nbvm()\n",
-        "print(\"Current compute instance:\", current_compute_instance)\n",
         "\n",
         "# For this demo, let's use the current compute instance as the compute target, if available\n",
         "if current_compute_instance:\n",
+        "    print(\"Current compute instance:\", current_compute_instance)\n",
         "    instance_name = current_compute_instance['instance']\n",
         "else:\n",
-        "    instance_name = next(iter(ws.compute_targets))\n",
+        "    instance_name = \"cartpole-ci-stdd2v2\"\n",
+        "    try:\n",
+        "        instance = ComputeInstance(workspace=ws, name=instance_name)\n",
+        "        print('Found existing instance, use it.')\n",
+        "    except ComputeTargetException:\n",
+        "        print(\"Creating new compute instance...\")\n",
+        "        compute_config = ComputeInstance.provisioning_configuration(\n",
+        "            vm_size='STANDARD_D2_V2'\n",
+        "        )\n",
+        "        instance = ComputeInstance.create(ws, instance_name, compute_config)\n",
+        "        instance.wait_for_completion(show_output=True)\n",
+        "    print(\"Instance name:\", instance_name)\n",
         "\n",
         "compute_target = ws.compute_targets[instance_name]\n",
         "\n",
         "print(\"Compute target status:\")\n",
         "print(compute_target.get_status().serialize())\n",
-        "\n",
         "print(\"Compute target size:\")\n",
         "print(compute_target.size(ws))"
       ]
@@ -184,7 +197,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Create Azure ML experiment\n",
+        "### Create Azure Machine Learning experiment\n",
         "Create an experiment to track the runs in your workspace. "
       ]
     },
@@ -204,8 +217,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Train Cartpole Agent Using Azure ML RL\n",
-        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct RL run configurations for the underlying RL framework. Azure ML RL initially supports the [Ray framework](https://ray.io/) and its highly customizable [RLlib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLlib framework to train a cartpole playing agent. "
+        "## Train Cartpole Agent\n",
+        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct reinforcement learning run configurations for the underlying reinforcement learning framework. Reinforcement Learning in Azure Machine Learning supports the open source [Ray framework](https://ray.io/) and its highly customizable [RLlib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLlib framework to train a cartpole playing agent. "
       ]
     },
     {
@@ -222,7 +235,7 @@
         "- `entry_script`, path to your entry script relative to the source directory,\n",
         "- `script_params`, constant parameters to be passed to each run of training script,\n",
         "- `compute_target`, reference to the compute target in which the trainer and worker(s) jobs will be executed,\n",
-        "- `rl_framework`, the RL framework to be used (currently must be Ray).\n",
+        "- `rl_framework`, the reinforcement learning framework to be used (currently must be Ray).\n",
         "\n",
         "We use the `script_params` parameter to pass in general and algorithm-specific parameters to the training script.\n"
       ]
@@ -273,10 +286,10 @@
         "    # A dictionary of arguments to pass to the training script specified in ``entry_script``\n",
         "    script_params=script_params,\n",
         "    \n",
-        "    # The Azure ML compute target set up for Ray head nodes\n",
+        "    # The Azure Machine Learning compute target set up for Ray head nodes\n",
         "    compute_target=compute_target,\n",
         "    \n",
-        "    # RL framework.  Currently must be Ray.\n",
+        "    # Reinforcement learning framework. Currently must be Ray.\n",
         "    rl_framework=Ray()\n",
         ")"
       ]
@@ -345,11 +358,11 @@
       "metadata": {},
       "source": [
         "### Monitor experiment\n",
-        "Azure ML provides a Jupyter widget to show the real-time status of an experiment run. You could use this widget to monitor status of the runs.\n",
+        "Azure Machine Learning provides a Jupyter widget to show the status of an experiment run. You could use this widget to monitor the status of the runs.\n",
         "\n",
         "Note that _ReinforcementLearningEstimator_ creates at least two runs: (a) A parent run, i.e. the run returned above, and (b) a collection of child runs. The number of the child runs depends on the configuration of the reinforcement learning estimator. In our simple scenario, configured above, only one child run will be created.\n",
         "\n",
-        "The widget will show a list of the child runs as well. You can click on the link under **Status** to see the details of a child run."
+        "The widget will show a list of the child runs as well. You can click on the link under **Status** to see the details of a child run. It will also show the metrics being logged."
       ]
     },
     {
@@ -369,7 +382,7 @@
       "source": [
         "### Stop the run\n",
         "\n",
-        "To cancel the run, call `training_run.cancel()`."
+        "To stop the run, call `training_run.cancel()`."
       ]
     },
     {
@@ -577,10 +590,10 @@
         "        training_artifacts_ds.as_named_input('artifacts_dataset'),\n",
         "        training_artifacts_ds.as_named_input('artifacts_path').as_mount()],\n",
         "    \n",
-        "    # The Azure ML compute target\n",
+        "    # The Azure Machine Learning compute target\n",
         "    compute_target=compute_target,\n",
         "    \n",
-        "    # RL framework. Currently must be Ray.\n",
+        "    # Reinforcement learning framework. Currently must be Ray.\n",
         "    rl_framework=Ray(),\n",
         "    \n",
         "    # Additional pip packages to install\n",
@@ -654,7 +667,11 @@
       "outputs": [],
       "source": [
         "# To archive the created experiment:\n",
-        "#exp.archive()"
+        "#exp.archive()\n",
+        "\n",
+        "# To delete created compute instance\n",
+        "if not current_compute_instance:\n",
+        "    compute_target.delete()"
       ]
     },
     {
@@ -662,7 +679,7 @@
       "metadata": {},
       "source": [
         "## Next\n",
-        "This example was about running Azure ML RL (Ray/RLlib Framework) on compute instance. Please see [Cartpole problem](../cartpole-on-single-compute/cartpole_cc.ipynb)\n",
+        "This example was about running Reinforcement Learning in Azure Machine Learning (Ray/RLlib Framework) on a compute instance. Please see [Cartpole Problem on Single Compute](../cartpole-on-single-compute/cartpole_sc.ipynb)\n",
         "example which uses Ray RLlib to train a Cartpole playing agent on a single node remote compute.\n"
       ]
     }

how-to-use-azureml/reinforcement-learning/cartpole-on-compute-instance/cartpole_ci.yml

@@ -4,3 +4,4 @@ dependencies:
   - azureml-sdk
   - azureml-contrib-reinforcementlearning
   - azureml-widgets
+  - azureml-dataprep

how-to-use-azureml/reinforcement-learning/cartpole-on-single-compute/cartpole_sc.ipynb

@@ -13,18 +13,18 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/how-to-use-azureml/reinforcement-learning/cartpole_on_single_compute/cartpole_cc.png)"
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/reinforcement-learning/cartpole_on_single_compute/cartpole_sc.png)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Azure ML Reinforcement Learning Sample - Cartpole Problem\n",
+        "# Reinforcement Learning in Azure Machine Learning - Cartpole Problem on Single Compute\n",
         "\n",
-        "Azure ML Reinforcement Learning (Azure ML RL) is a managed service for running reinforcement learning training and simulation. With Azure MLRL, data scientists can start developing RL systems on one machine, and scale to compute clusters with 100\u00e2\u20ac\u2122s of nodes if needed.\n",
+        "Reinforcement Learning in Azure Machine Learning is a managed service for running reinforcement learning training and simulation. With Reinforcement Learning in Azure Machine Learning, data scientists can start developing reinforcement learning systems on one machine, and scale to compute targets with 100\u00e2\u20ac\u2122s of nodes if needed.\n",
         "\n",
-        "This example shows how to use Azure ML RL to train a Cartpole playing agent on a single machine. "
+        "This example shows how to use Reinforcement Learning in Azure Machine Learning to train a Cartpole playing agent on a single compute. "
       ]
     },
     {
@@ -56,7 +56,7 @@
       "metadata": {},
       "source": [
         "### Prerequisite\n",
-        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription id, a resource group and a workspace. All datastores and datasets you use should be associated with your workspace."
+        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription ID, a resource group, and an Azure Machine Learning workspace. All datastores and datasets you use should be associated with your workspace."
       ]
     },
     {
@@ -75,8 +75,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Azure ML SDK \n",
-        "Display the Azure ML SDK version."
+        "### Azure Machine Learning SDK \n",
+        "Display the Azure Machine Learning SDK version."
       ]
     },
     {
@@ -87,15 +87,15 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
+        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Get Azure ML workspace\n",
-        "Get a reference to an existing Azure ML workspace."
+        "### Get Azure Machine Learning workspace\n",
+        "Get a reference to an existing Azure Machine Learning workspace."
       ]
     },
     {
@@ -118,7 +118,7 @@
         "\n",
         "A compute target is a designated compute resource where you run your training and simulation scripts. This location may be your local machine or a cloud-based compute resource. The code below shows how to create a cloud-based compute target. For more information see [What are compute targets in Azure Machine Learning?](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target)\n",
         "\n",
-        "**Note: Creation of a compute resource can take several minutes**"
+        "**Note: Creation of a compute resource can take several minutes**. Please make sure to change `STANDARD_D2_V2` to a [size available in your region](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=virtual-machines)."
       ]
     },
     {
@@ -158,7 +158,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Create Azure ML experiment\n",
+        "### Create Azure Machine Learning experiment\n",
         "Create an experiment to track the runs in your workspace. "
       ]
     },
@@ -170,7 +170,7 @@
       "source": [
         "from azureml.core.experiment import Experiment\n",
         "\n",
-        "experiment_name = 'CartPole-v0-CC'\n",
+        "experiment_name = 'CartPole-v0-SC'\n",
         "exp = Experiment(workspace=ws, name=experiment_name)"
       ]
     },
@@ -178,8 +178,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Train Cartpole Agent Using Azure ML RL\n",
-        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct RL run configurations for the underlying RL framework. Azure ML RL initially supports the [Ray framework](https://ray.io/) and its highly customizable [RLlib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLlib framework to train a cartpole playing agent. "
+        "## Train Cartpole Agent\n",
+        "To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct reinforcement learning run configurations for the underlying reinforcement learning framework. Reinforcement Learning in Azure Machine Learning supports the open source [Ray framework](https://ray.io/) and its highly customizable [RLlib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLlib framework to train a cartpole playing agent. "
       ]
     },
     {
@@ -196,7 +196,7 @@
         "- `entry_script`, path to your entry script relative to the source directory,\n",
         "- `script_params`, constant parameters to be passed to each run of training script,\n",
         "- `compute_target`, reference to the compute target in which the trainer and worker(s) jobs will be executed,\n",
-        "- `rl_framework`, the RL framework to be used (currently must be Ray).\n",
+        "- `rl_framework`, the reinforcement learning framework to be used (currently must be Ray).\n",
         "\n",
         "We use the `script_params` parameter to pass in general and algorithm-specific parameters to the training script.\n"
       ]
@@ -249,7 +249,7 @@
         "    # There are two parts to this:\n",
         "    # 1. Use a custom docker file with proper instructions to install xvfb, ffmpeg, python-opengl\n",
         "    # and other dependencies.        \n",
-        "    # TODO: Add these instructions to default rl base image and drop this docker file.\n",
+        "    # TODO: Add these instructions to default reinforcement learning base image and drop this docker file.\n",
         "    \n",
         "    with open(\"files/docker/Dockerfile\", \"r\") as f:\n",
         "        dockerfile=f.read()\n",
@@ -274,10 +274,10 @@
         "    # A dictionary of arguments to pass to the training script specified in ``entry_script``\n",
         "    script_params=script_params,\n",
         "    \n",
-        "    # The Azure ML compute target set up for Ray head nodes\n",
+        "    # The Azure Machine Learning compute target set up for Ray head nodes\n",
         "    compute_target=compute_target,\n",
         "    \n",
-        "    # RL framework.  Currently must be Ray.\n",
+        "    # Reinforcement learning framework. Currently must be Ray.\n",
         "    rl_framework=Ray(),\n",
         "    \n",
         "    # Custom environmnet for Xvfb\n",
@@ -350,11 +350,11 @@
       "source": [
         "### Monitor experiment\n",
         "\n",
-        "Azure ML provides a Jupyter widget to show the real-time status of an experiment run. You could use this widget to monitor status of the runs.\n",
+        "Azure Machine Learning provides a Jupyter widget to show the status of an experiment run. You could use this widget to monitor the status of the runs.\n",
         "\n",
         "Note that _ReinforcementLearningEstimator_ creates at least two runs: (a) A parent run, i.e. the run returned above, and (b) a collection of child runs. The number of the child runs depends on the configuration of the reinforcement learning estimator. In our simple scenario, configured above, only one child run will be created.\n",
         "\n",
-        "The widget will show a list of the child runs as well. You can click on the link under **Status** to see the details of a child run."
+        "The widget will show a list of the child runs as well. You can click on the link under **Status** to see the details of a child run. It will also show the metrics being logged."
       ]
     },
     {
@@ -373,7 +373,7 @@
       "metadata": {},
       "source": [
         "### Stop the run\n",
-        "To cancel the run, call `training_run.cancel()`."
+        "To stop the run, call `training_run.cancel()`."
       ]
     },
     {
@@ -393,7 +393,7 @@
         "### Wait for completion\n",
         "Wait for the run to complete before proceeding.\n",
         "\n",
-        "**Note: The length of the run depends on the provisioning time of the compute target and may take several minutes to complete.**"
+        "**Note: The length of the run depends on the provisioning time of the compute target and it may take several minutes to complete.**"
       ]
     },
     {
@@ -560,18 +560,20 @@
         "    dir_util.mkpath(destination)\n",
         "    \n",
         "    try:\n",
-        "        # Mount dataset and copy movies\n",
+        "        pirnt(\"Trying mounting dataset and copying movies.\")\n",
         "        # Note: We assume movie paths start with '\\'\n",
         "        mount_context = artifacts_ds.mount()\n",
         "        mount_context.start()\n",
-        "        print('Download started.')\n",
         "        for movie in movies:\n",
         "            print('Copying {} ...'.format(movie))\n",
         "            shutil.copy2(path.join(mount_context.mount_point, movie[1:]), destination)\n",
         "        mount_context.stop()\n",
         "    except:\n",
-        "        print(\"Mounting error! Downloading all artifacts ...\")\n",
-        "        artifacts_ds.download(target_path=destination, overwrite=True)\n",
+        "        print(\"Mounting failed! Going with dataset download.\")\n",
+        "        for i, file in enumerate(artifacts_ds.to_path()):\n",
+        "            if file in movies:\n",
+        "                print('Downloading {} ...'.format(file))\n",
+        "                artifacts_ds.skip(i).take(1).download(target_path=destination, overwrite=True)\n",
         "        \n",
         "    print('Downloading movies completed!')\n",
         "\n",
@@ -625,7 +627,7 @@
         "print(\"Last movie:\", last_movie)\n",
         "\n",
         "# Download movies\n",
-        "training_movies_path = \"training\"\n",
+        "training_movies_path = path.join(\"training\", \"videos\")\n",
         "download_movies(training_artifacts_ds, [first_movie, last_movie], training_movies_path)"
       ]
     },
@@ -781,7 +783,7 @@
         "# 1. Use a custom docker file with proper instructions to install xvfb, ffmpeg, python-opengl\n",
         "# and other dependencies.\n",
         "# Note: Even when the rendering is off pyhton-opengl is needed.\n",
-        "# TODO: Add these instructions to default rl base image and drop this docker file.\n",
+        "# TODO: Add these instructions to default reinforcement learning base image and drop this docker file.\n",
         "\n",
         "with open(\"files/docker/Dockerfile\", \"r\") as f:\n",
         "    dockerfile=f.read()\n",
@@ -811,10 +813,10 @@
         "        training_artifacts_ds.as_named_input('artifacts_dataset'),\n",
         "        training_artifacts_ds.as_named_input('artifacts_path').as_mount()],\n",
         "    \n",
-        "    # The Azure ML compute target set up for Ray head nodes\n",
+        "    # The Azure Machine Learning compute target set up for Ray head nodes\n",
         "    compute_target=compute_target,\n",
         "    \n",
-        "    # RL framework.  Currently must be Ray.\n",
+        "    # Reinforcement learning framework. Currently must be Ray.\n",
         "    rl_framework=Ray(),\n",
         "    \n",
         "    # Custom environmnet for Xvfb\n",
@@ -928,7 +930,7 @@
         "print(\"Last movie:\", last_movie)\n",
         "\n",
         "# Download last movie\n",
-        "rollout_movies_path = \"rollout\"\n",
+        "rollout_movies_path = path.join(\"rollout\", \"videos\")\n",
         "download_movies(rollout_artifacts_ds, [last_movie], rollout_movies_path)\n",
         "\n",
         "# Look for the downloaded movie in local directory\n",
@@ -996,7 +998,7 @@
       "metadata": {},
       "source": [
         "## Next\n",
-        "This example was about running Azure ML RL (Ray/RLlib Framework) on a single node. Please see [Pong problem](../atari-on-distributed-compute/pong_rllib.ipynb)\n",
+        "This example was about running Reinforcement Learning in Azure Machine Learning (Ray/RLlib Framework) on a single compute. Please see [Pong Problem](../atari-on-distributed-compute/pong_rllib.ipynb)\n",
         "example which uses Ray RLlib to train a Pong playing agent on a multi-node cluster."
       ]
     }

how-to-use-azureml/reinforcement-learning/cartpole-on-single-compute/cartpole_sc.yml

@@ -1,6 +1,7 @@
-name: cartpole_cc
+name: cartpole_sc
 dependencies:
 - pip:
   - azureml-sdk
   - azureml-contrib-reinforcementlearning
   - azureml-widgets
+  - azureml-dataprep

how-to-use-azureml/reinforcement-learning/minecraft-on-distributed-compute/files/minecraft_train.py

@@ -1,3 +1,5 @@
+import os
+
 import ray
 import ray.tune as tune
 
@@ -6,8 +8,10 @@ from minecraft_environment import create_env
 
 
 def stop(trial_id, result):
+    max_train_time = int(os.environ.get("AML_MAX_TRAIN_TIME_SECONDS", 5 * 60 * 60))
+
     return result["episode_reward_mean"] >= 1 \
-        or result["time_total_s"] > 5 * 60 * 60
+        or result["time_total_s"] >= max_train_time
 
 
 if __name__ == '__main__':

how-to-use-azureml/reinforcement-learning/minecraft-on-distributed-compute/minecraft.ipynb

@@ -110,7 +110,7 @@
       "outputs": [],
       "source": [
         "import azureml.core\n",
-        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
+        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION) "
       ]
     },
     {
@@ -209,7 +209,7 @@
         "# name of the Virtual Network subnet ('default' the default name)\n",
         "subnet_name = 'default'\n",
         "\n",
-        "gpu_cluster_name = 'gpu-cluster-nc6'\n",
+        "gpu_cluster_name = 'gpu-cl-nc6-vnet'\n",
         "\n",
         "try:\n",
         "    gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
@@ -250,7 +250,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "cpu_cluster_name = 'cpu-cluster-d2'\n",
+        "cpu_cluster_name = 'cpu-cl-d2-vnet'\n",
         "\n",
         "try:\n",
         "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
@@ -297,8 +297,11 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "import os\n",
         "from azureml.core import Environment\n",
         "\n",
+        "max_train_time = os.environ.get(\"AML_MAX_TRAIN_TIME_SECONDS\", 5 * 60 * 60)\n",
+        "\n",
         "def create_env(env_type):\n",
         "    env = Environment(name='minecraft-{env_type}'.format(env_type=env_type))\n",
         "\n",
@@ -306,6 +309,7 @@
         "    env.docker.base_image = 'akdmsft/minecraft-{env_type}'.format(env_type=env_type)\n",
         "\n",
         "    env.python.interpreter_path = \"xvfb-run -s '-screen 0 640x480x16 -ac +extension GLX +render' python\"\n",
+        "    env.environment_variables[\"AML_MAX_TRAIN_TIME_SECONDS\"] = str(max_train_time)\n",
         "    env.python.user_managed_dependencies = True\n",
         "    \n",
         "    return env\n",
@@ -590,7 +594,6 @@
       "outputs": [],
       "source": [
         "import re\n",
-        "import os\n",
         "import tempfile\n",
         "\n",
         "from azureml.core import Dataset\n",

how-to-use-azureml/reinforcement-learning/setup/devenv_setup.ipynb

@@ -13,14 +13,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/how-to-use-azureml/reinforcement-learning/setup/devenv_setup.png)"
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/reinforcement-learning/setup/devenv_setup.png)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Azure ML Reinforcement Learning Sample - Setting Up Development Environment\n",
+        "# Reinforcement Learning in Azure Machine Learning - Setting Up Development Environment\n",
         "\n",
         "Ray multi-node cluster setup requires all worker nodes to be able to communicate with the head node. This notebook explains you how to setup a virtual network, to be used by the Ray head and worker compute targets, created and used in other notebook examples."
       ]
@@ -31,7 +31,7 @@
       "source": [
         "### Prerequisite\n",
         "\n",
-        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription id, a resource group and a workspace."
+        "The user should have completed the Azure Machine Learning Tutorial: [Get started creating your first ML experiment with the Python SDK](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-setup). You will need to make sure that you have a valid subscription ID, a resource group, and an Azure Machine Learning workspace."
       ]
     },
     {
@@ -48,19 +48,17 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Azure ML Core imports\n",
         "import azureml.core\n",
         "\n",
-        "# Check core SDK version number\n",
-        "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
+        "print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Get Azure ML workspace\n",
-        "Get a reference to an existing Azure ML workspace. Please make sure that the VM sizes `STANDARD_NC6` and `STANDARD_D2_V2` are supported in the workspace's region.\n"
+        "### Get Azure Machine Learning workspace\n",
+        "Get a reference to an existing Azure Machine Learning workspace. Please make sure to change `STANDARD_NC6` and `STANDARD_D2_V2` to [the ones available in your region](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=virtual-machines).\n"
       ]
     },
     {
@@ -72,7 +70,7 @@
         "from azureml.core import Workspace\n",
         "\n",
         "ws = Workspace.from_config()\n",
-        "print(ws.name, ws.location, ws.resource_group, sep = ' | ') "
+        "print(ws.name, ws.location, ws.resource_group, sep = ' | ')"
       ]
     },
     {
@@ -115,7 +113,7 @@
         "# The Azure subscription you are using\n",
         "subscription_id=ws.subscription_id\n",
         "\n",
-        "# The resource group for the RL cluster\n",
+        "# The resource group for the reinforcement learning cluster\n",
         "resource_group=ws.resource_group\n",
         "\n",
         "# Azure region of the resource group\n",
@@ -135,7 +133,7 @@
         ")\n",
         "\n",
         "async_vnet_creation.wait()\n",
-        "print(\"VNet created successfully: \", async_vnet_creation.result())"
+        "print(\"Virtual network created successfully: \", async_vnet_creation.result())"
       ]
     },
     {
@@ -169,7 +167,7 @@
         "        azure.mgmt.network.models.SecurityRule(\n",
         "            name=security_rule_name,\n",
         "            access=azure.mgmt.network.models.SecurityRuleAccess.allow,\n",
-        "            description='Azure ML RL rule',\n",
+        "            description='Reinforcement Learning in Azure Machine Learning rule',\n",
         "            destination_address_prefix='*',\n",
         "            destination_port_range='29876-29877',\n",
         "            direction=azure.mgmt.network.models.SecurityRuleDirection.inbound,\n",
@@ -188,7 +186,7 @@
         ")\n",
         "\n",
         "async_nsg_creation.wait() \n",
-        "print(\"Network security group created successfully: \", async_nsg_creation.result())\n",
+        "print(\"Network security group created successfully:\", async_nsg_creation.result())\n",
         "\n",
         "network_security_group = network_client.network_security_groups.get(\n",
         "    resource_group,\n",
@@ -202,7 +200,7 @@
         "            network_security_group=network_security_group\n",
         "            )\n",
         "    \n",
-        "# Create subnet on vnet\n",
+        "# Create subnet on virtual network\n",
         "async_subnet_creation = network_client.subnets.create_or_update(\n",
         "    resource_group_name=resource_group,\n",
         "    virtual_network_name=vnet_name,\n",
@@ -213,6 +211,25 @@
         "async_subnet_creation.wait()\n",
         "print(\"Subnet created successfully:\", async_subnet_creation.result())"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Review the virtual network security rules\n",
+        "Ensure that the virtual network is configured correctly with required ports open. It is possible that you have configured rules with broader range of ports that allows ports 29876-29877 to be opened. Kindly review your network security group rules. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from files.networkutils import *\n",
+        "\n",
+        "check_vnet_security_rules(ws._auth_object, ws.subscription_id, ws.resource_group, vnet_name, True)"
+      ]
     }
   ],
   "metadata": {

how-to-use-azureml/reinforcement-learning/setup/files/networkutils.py

@@ -0,0 +1,237 @@
+import sys
+import csv
+from azure.mgmt.network import NetworkManagementClient
+
+
+def check_port_in_port_range(expected_port: str,
+                             dest_port_range: str):
+    """
+    Check if a port is within a port range
+    Port range maybe like *, 8080 or 8888-8889
+    """
+
+    if dest_port_range == '*':
+        return True
+
+    dest_ports = dest_port_range.split('-')
+
+    if len(dest_ports) == 1 and \
+       int(dest_ports[0]) == int(expected_port):
+        return True
+
+    if len(dest_ports) == 2 and \
+       int(dest_ports[0]) <= int(expected_port) and \
+       int(dest_ports[1]) >= int(expected_port):
+        return True
+
+    return False
+
+
+def check_port_in_destination_port_ranges(expected_port: str,
+                                          dest_port_ranges: list):
+    """
+    Check if a port is within a given list of port ranges
+    i.e. check if port 8080 is in port ranges of 22,80,8080-8090,443
+    """
+
+    for dest_port_range in dest_port_ranges:
+        if check_port_in_port_range(expected_port, dest_port_range) is True:
+            return True
+
+    return False
+
+
+def check_ports_in_destination_port_ranges(expected_ports: list,
+                                           dest_port_ranges: list):
+    """
+    Check if all ports in a given port list are within a given list
+    of port ranges
+    i.e. check if port 8080,8081 are in port ranges of 22,80,8080-8090,443
+    """
+
+    for expected_port in expected_ports:
+        if check_port_in_destination_port_ranges(
+           expected_port, dest_port_ranges) is False:
+            return False
+
+    return True
+
+
+def check_source_address_prefix(source_address_prefix: str):
+    """Check if source address prefix is BatchNodeManagement or default"""
+
+    required_prefix = 'BatchNodeManagement'
+    default_prefix = 'default'
+
+    if source_address_prefix.lower() == required_prefix.lower() or \
+       source_address_prefix.lower() == default_prefix.lower():
+        return True
+
+    return False
+
+
+def check_protocol(protocol: str):
+    """Check if protocol is supported - Tcp/Any"""
+
+    required_protocol = 'Tcp'
+    any_protocol = 'Any'
+
+    if required_protocol.lower() == protocol.lower() or \
+       any_protocol.lower() == protocol.lower():
+        return True
+
+    return False
+
+
+def check_direction(direction: str):
+    """Check if port direction is inbound"""
+
+    required_direction = 'Inbound'
+
+    if required_direction.lower() == direction.lower():
+        return True
+
+    return False
+
+
+def check_provisioning_state(provisioning_state: str):
+    """Check if the provisioning state is succeeded"""
+
+    required_provisioning_state = 'Succeeded'
+
+    if required_provisioning_state.lower() == provisioning_state.lower():
+        return True
+
+    return False
+
+
+def check_rule_for_Azure_ML(rule):
+    """Check if the ports required for Azure Machine Learning are open"""
+
+    required_ports = ['29876', '29877']
+
+    if check_source_address_prefix(rule.source_address_prefix) is False:
+        return False
+
+    if check_protocol(rule.protocol) is False:
+        return False
+
+    if check_direction(rule.direction) is False:
+        return False
+
+    if check_provisioning_state(rule.provisioning_state) is False:
+        return False
+
+    if rule.destination_port_range is not None:
+        if check_ports_in_destination_port_ranges(
+           required_ports,
+           [rule.destination_port_range]) is False:
+            return False
+    else:
+        if check_ports_in_destination_port_ranges(
+           required_ports,
+           rule.destination_port_ranges) is False:
+            return False
+
+    return True
+
+
+def check_vnet_security_rules(auth_object,
+                              vnet_subscription_id,
+                              vnet_resource_group,
+                              vnet_name,
+                              save_to_file=False):
+    """
+    Check all the rules of virtual network if required ports for Azure Machine
+    Learning are open
+    """
+
+    network_client = NetworkManagementClient(
+        auth_object,
+        vnet_subscription_id)
+
+    # get the vnet
+    vnet = network_client.virtual_networks.get(
+        resource_group_name=vnet_resource_group,
+        virtual_network_name=vnet_name)
+
+    vnet_location = vnet.location
+    vnet_info = []
+
+    if vnet.subnets is None or len(vnet.subnets) == 0:
+        print('WARNING: No subnet found for VNet:', vnet_name)
+
+    # for each subnet of the vnet
+    for subnet in vnet.subnets:
+        if subnet.network_security_group is None:
+            print('WARNING: No network security group found for subnet.',
+                  'Subnet',
+                  subnet.id.split("/")[-1])
+        else:
+            # get all the rules
+            network_security_group_name = \
+                subnet.network_security_group.id.split("/")[-1]
+            network_security_group_resource_group_name = \
+                subnet.network_security_group.id.split("/")[4]
+            network_security_group_subscription_id = \
+                subnet.network_security_group.id.split("/")[2]
+
+            security_rules = list(network_client.security_rules.list(
+                network_security_group_resource_group_name,
+                network_security_group_name))
+
+            rule_matched = None
+            for rule in security_rules:
+                rule_info = []
+                # add vnet details
+                rule_info.append(vnet_name)
+                rule_info.append(vnet_subscription_id)
+                rule_info.append(vnet_resource_group)
+                rule_info.append(vnet_location)
+                # add subnet details
+                rule_info.append(subnet.id.split("/")[-1])
+                rule_info.append(network_security_group_name)
+                rule_info.append(network_security_group_subscription_id)
+                rule_info.append(network_security_group_resource_group_name)
+                # add rule details
+                rule_info.append(rule.priority)
+                rule_info.append(rule.name)
+                rule_info.append(rule.source_address_prefix)
+                if rule.destination_port_range is not None:
+                    rule_info.append(rule.destination_port_range)
+                else:
+                    rule_info.append(rule.destination_port_ranges)
+                rule_info.append(rule.direction)
+                rule_info.append(rule.provisioning_state)
+                vnet_info.append(rule_info)
+
+                if check_rule_for_Azure_ML(rule) is True:
+                    rule_matched = rule
+
+            if rule_matched is not None:
+                print("INFORMATION: Rule matched with required ports. Subnet:",
+                      subnet.id.split("/")[-1], "Rule:", rule.name)
+            else:
+                print("WARNING: No rule matched with required ports. Subnet:",
+                      subnet.id.split("/")[-1])
+
+    if save_to_file is True:
+        file_name = vnet_name + ".csv"
+        with open(file_name, mode='w') as vnet_rule_file:
+            vnet_rule_file_writer = csv.writer(
+                vnet_rule_file,
+                delimiter=',',
+                quotechar='"',
+                quoting=csv.QUOTE_MINIMAL)
+            header = ['VNet_Name', 'VNet_Subscription_ID',
+                      'VNet_Resource_Group', 'VNet_Location',
+                      'Subnet_Name', 'NSG_Name',
+                      'NSG_Subscription_ID', 'NSG_Resource_Group',
+                      'Rule_Priority', 'Rule_Name', 'Rule_Source',
+                      'Rule_Destination_Ports', 'Rule_Direction',
+                      'Rule_Provisioning_State']
+            vnet_rule_file_writer.writerow(header)
+            vnet_rule_file_writer.writerows(vnet_info)
+
+        print("INFORMATION: Network security group rules for your virtual \
+network are saved in file", file_name)

how-to-use-azureml/track-and-monitor-experiments/logging-api/logging-api.ipynb

@@ -100,7 +100,7 @@
         "\n",
         "# Check core SDK version number\n",
         "\n",
-        "print(\"This notebook was created using SDK version 1.5.0, you are currently running version\", azureml.core.VERSION)"
+        "print(\"This notebook was created using SDK version 1.8.0, you are currently running version\", azureml.core.VERSION)"
       ]
     },
     {

how-to-use-azureml/track-and-monitor-experiments/tensorboard/tensorboard.ipynb

@@ -439,6 +439,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "from azureml.train.dnn import TensorFlow\n",
+        "\n",
         "script_params = {\"--log_dir\": \"./logs\"}\n",
         "\n",
         "# If you want the run to go longer, set --max-steps to a higher number.\n",

how-to-use-azureml/training/train-in-spark/train-in-spark.ipynb

@@ -144,25 +144,18 @@
         "import os\n",
         "\n",
         "try:\n",
-        "    # if you want to connect using SSH key instead of username/password you can provide parameters private_key_file and private_key_passphrase\n",
-        "    attach_config = HDInsightCompute.attach_configuration(address=os.environ.get('hdiservername', '<my_hdi_cluster_name>-ssh.azurehdinsight.net'), \n",
-        "                                                          ssh_port=22, \n",
-        "                                                          username=os.environ.get('hdiusername', '<ssh_username>'), \n",
+        "# If you want to connect using SSH key instead of username/password you can provide parameters private_key_file and private_key_passphrase\n",
+        "\n",
+        "# Attaching a HDInsight cluster using the public address of the HDInsight cluster is no longer supported.\n",
+        "# Instead, use resourceId of the HDInsight cluster.\n",
+        "# The resourceId of the HDInsight Cluster can be constructed using the following string format:\n",
+        "# /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.HDInsight/clusters/<cluster_name>.\n",
+        "# You can also use subscription_id, resource_group and cluster_name without constructing resourceId.\n",
+        "    attach_config = HDInsightCompute.attach_configuration(resource_id='<resource_id>',\n",
+        "                                                          ssh_port=22,\n",
+        "                                                          username=os.environ.get('hdiusername', '<ssh_username>'),\n",
         "                                                          password=os.environ.get('hdipassword', '<my_password>'))\n",
         "\n",
-        "# The following Azure regions do not support attaching a HDI Cluster using the public IP address of the HDI Cluster.\n",
-        "# Instead, use the Azure Resource Manager ID of the HDI Cluster with the resource_id parameter:\n",
-        "# US East\n",
-        "# US West 2\n",
-        "# US South Central\n",
-        "# The resource ID of the HDI Cluster can be constructed using the\n",
-        "# subscription ID, resource group name, and cluster name using the following string format:\n",
-        "# /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.HDInsight/clusters/<cluster_name>.    \n",
-        "# If in US East, US West 2, or US South Central, use the following instead:\n",
-        "# attach_config = HDInsightCompute.attach_configuration(resource_id='<resource_id>',\n",
-        "#                                                       ssh_port=22,\n",
-        "#                                                       username=os.environ.get('hdiusername', '<ssh_username>'),\n",
-        "#                                                       password=os.environ.get('hdipassword', '<my_password>'))\n",
         "    hdi_compute = ComputeTarget.attach(workspace=ws, \n",
         "                                       name='myhdi', \n",
         "                                       attach_configuration=attach_config)\n",

how-to-use-azureml/training/train-on-computeinstance/train-on-computeinstance.ipynb

@@ -0,0 +1,374 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/train-on-amlcompute/train-on-computeinstance.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#  Train using Azure Machine Learning Compute Instance\n",
+        "\n",
+        "* Initialize Workspace\n",
+        "* Introduction to ComputeInstance\n",
+        "* Create an Experiment\n",
+        "* Submit ComputeInstance run\n",
+        "* Additional operations to perform on ComputeInstance"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Prerequisites\n",
+        "If you are using an Azure Machine Learning ComputeInstance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Check core SDK version number\n",
+        "import azureml.core\n",
+        "\n",
+        "print(\"SDK version:\", azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Initialize Workspace\n",
+        "\n",
+        "Initialize a workspace object"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "tags": [
+          "create workspace"
+        ]
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Introduction to ComputeInstance\n",
+        "\n",
+        "\n",
+        "Azure Machine Learning compute instance is a fully-managed cloud-based workstation optimized for your machine learning development environment. It is created **within your workspace region**.\n",
+        "\n",
+        "For more information on ComputeInstance, please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-instance)\n",
+        "\n",
+        "**Note**: As with other Azure services, there are limits on certain resources (for eg. AmlCompute quota) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Create ComputeInstance\n",
+        "First lets check which VM families are available in your region. Azure is a regional service and some specialized SKUs (especially GPUs) are only available in certain regions. Since ComputeInstance is created in the region of your workspace, we will use the supported_vms () function to see if the VM family we want to use ('STANDARD_D3_V2') is supported.\n",
+        "\n",
+        "You can also pass a different region to check availability and then re-create your workspace in that region through the [configuration notebook](../../../configuration.ipynb)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "msdoc": "how-to-auto-train-remote.md",
+        "name": "check_region"
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, ComputeInstance\n",
+        "\n",
+        "ComputeInstance.supported_vmsizes(workspace = ws)\n",
+        "# ComputeInstance.supported_vmsizes(workspace = ws, location='eastus')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "msdoc": "how-to-auto-train-remote.md",
+        "name": "create_instance"
+      },
+      "outputs": [],
+      "source": [
+        "import datetime\n",
+        "import time\n",
+        "\n",
+        "from azureml.core.compute import ComputeTarget, ComputeInstance\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your instance\n",
+        "compute_name = \"compute-instance\"\n",
+        "\n",
+        "# Verify that instance does not exist already\n",
+        "try:\n",
+        "    instance = ComputeInstance(workspace=ws, name=compute_name)\n",
+        "    print('Found existing instance, use it.')\n",
+        "except ComputeTargetException:\n",
+        "    compute_config = ComputeInstance.provisioning_configuration(\n",
+        "        vm_size='STANDARD_D3_V2',\n",
+        "        ssh_public_access=False,\n",
+        "        # vnet_resourcegroup_name='<my-resource-group>',\n",
+        "        # vnet_name='<my-vnet-name>',\n",
+        "        # subnet_name='default',\n",
+        "        # admin_user_ssh_public_key='<my-sshkey>'\n",
+        "    )\n",
+        "    instance = ComputeInstance.create(ws, compute_name, compute_config)\n",
+        "    instance.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Create An Experiment\n",
+        "\n",
+        "**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Experiment\n",
+        "experiment_name = 'train-on-computeinstance'\n",
+        "experiment = Experiment(workspace = ws, name = experiment_name)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Submit ComputeInstance run\n",
+        "The training script `train.py` is already created for you"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Create environment\n",
+        "\n",
+        "Create Docker based environment with scikit-learn installed."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Environment\n",
+        "from azureml.core.conda_dependencies import CondaDependencies\n",
+        "\n",
+        "myenv = Environment(\"myenv\")\n",
+        "\n",
+        "myenv.docker.enabled = True\n",
+        "myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Configure & Run"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import ScriptRunConfig\n",
+        "from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
+        "\n",
+        "src = ScriptRunConfig(source_directory='', script='train.py')\n",
+        "\n",
+        "# Set compute target to the one created in previous step\n",
+        "src.run_config.target = instance\n",
+        "\n",
+        "# Set environment\n",
+        "src.run_config.environment = myenv\n",
+        " \n",
+        "run = experiment.submit(config=src)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%time\n",
+        "# Shows output of the run on stdout.\n",
+        "run.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(run.get_metrics())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Additional operations to perform on ComputeInstance\n",
+        "\n",
+        "You can perform more operations on ComputeInstance such as get status, change the state or deleting the compute."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "msdoc": "how-to-auto-train-remote.md",
+        "name": "get_status"
+      },
+      "outputs": [],
+      "source": [
+        "# get_status() gets the latest status of the ComputeInstance target\n",
+        "instance.get_status()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "msdoc": "how-to-auto-train-remote.md",
+        "name": "stop"
+      },
+      "outputs": [],
+      "source": [
+        "# stop() is used to stop the ComputeInstance\n",
+        "# Stopping ComputeInstance will stop the billing meter and persist the state on the disk.\n",
+        "# Available Quota will not be changed with this operation.\n",
+        "instance.stop(wait_for_completion=True, show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "msdoc": "how-to-auto-train-remote.md",
+        "name": "start"
+      },
+      "outputs": [],
+      "source": [
+        "# start() is used to start the ComputeInstance if it is in stopped state\n",
+        "instance.start(wait_for_completion=True, show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# restart() is used to restart the ComputeInstance\n",
+        "instance.restart(wait_for_completion=True, show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# delete() is used to delete the ComputeInstance target. Useful if you want to re-use the compute name \n",
+        "# instance.delete()"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "ramagott"
+      }
+    ],
+    "category": "training",
+    "compute": [
+      "Compute Instance"
+    ],
+    "datasets": [
+      "Diabetes"
+    ],
+    "deployment": [
+      "None"
+    ],
+    "exclude_from_index": false,
+    "framework": [
+      "None"
+    ],
+    "friendly_name": "Train on Azure Machine Learning Compute Instance",
+    "index_order": 1,
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.7.7"
+    },
+    "tags": [
+      "None"
+    ],
+    "task": "Submit a run on Azure Machine Learning Compute Instance."
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/training/train-on-computeinstance/train-on-computeinstance.yml

@@ -0,0 +1,6 @@
+name: train-on-computeinstance
+dependencies:
+- scikit-learn
+- pip:
+  - azureml-sdk
+  - azureml-widgets

how-to-use-azureml/training/train-on-computeinstance/train.py

@@ -0,0 +1,44 @@
+# Copyright (c) Microsoft. All rights reserved.
+# Licensed under the MIT license.
+
+from sklearn.datasets import load_diabetes
+from sklearn.linear_model import Ridge
+from sklearn.metrics import mean_squared_error
+from sklearn.model_selection import train_test_split
+from azureml.core.run import Run
+from sklearn.externals import joblib
+import os
+import numpy as np
+
+os.makedirs('./outputs', exist_ok=True)
+
+X, y = load_diabetes(return_X_y=True)
+
+run = Run.get_context()
+
+X_train, X_test, y_train, y_test = train_test_split(X, y,
+                                                    test_size=0.2,
+                                                    random_state=0)
+data = {"train": {"X": X_train, "y": y_train},
+        "test": {"X": X_test, "y": y_test}}
+
+# list of numbers from 0.0 to 1.0 with a 0.05 interval
+alphas = np.arange(0.0, 1.0, 0.05)
+
+for alpha in alphas:
+    # Use Ridge algorithm to create a regression model
+    reg = Ridge(alpha=alpha)
+    reg.fit(data["train"]["X"], data["train"]["y"])
+
+    preds = reg.predict(data["test"]["X"])
+    mse = mean_squared_error(preds, data["test"]["y"])
+    run.log('alpha', alpha)
+    run.log('mse', mse)
+
+    model_file_name = 'ridge_{0:.2f}.pkl'.format(alpha)
+    # save model in the outputs folder so it automatically get uploaded
+    with open(model_file_name, "wb") as file:
+        joblib.dump(value=reg, filename=os.path.join('./outputs/',
+                                                     model_file_name))
+
+    print('alpha is {0:.2f}, and mse is {1:0.2f}'.format(alpha, mse))

how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb

@@ -268,23 +268,15 @@
         "                                                       private_key_file='./.ssh/id_rsa')\n",
         "\n",
         "\n",
-        "# The following Azure regions do not support attaching a virtual machine using the public IP address of the VM.\n",
-        "# Instead, use the Azure Resource Manager ID of the VM with the resource_id parameter:\n",
-        "# US East\n",
-        "# US West 2\n",
-        "# US South Central\n",
-        "# The resource ID of the VM can be constructed using the\n",
-        "# subscription ID, resource group name, and VM name using the following string format:\n",
-        "# /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Compute/virtualMachines/<vm_name>.    \n",
-        "# If in US East, US West 2, or US South Central, use the following instead:\n",
-        "# attach_config = RemoteCompute.attach_configuration(resource_id='<resource_id>',\n",
-        "#                                                    ssh_port=22,\n",
-        "#                                                    username='username',\n",
-        "#                                                    private_key_file='./.ssh/id_rsa')\n",
-        "\n",
-        "    attached_dsvm_compute = ComputeTarget.attach(workspace=ws,\n",
-        "                                                 name=compute_target_name,\n",
-        "                                                 attach_configuration=attach_config)\n",
+        "# Attaching a virtual machine using the public IP address of the VM is no longer supported.\n",
+        "# Instead, use resourceId of the VM.\n",
+        "# The resourceId of the VM can be constructed using the following string format:\n",
+        "# /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Compute/virtualMachines/<vm_name>.\n",
+        "# You can also use subscription_id, resource_group and vm_name without constructing resourceId.\n",
+        "    attach_config = RemoteCompute.attach_configuration(resource_id='<resource_id>',\n",
+        "                                                       ssh_port=22,\n",
+        "                                                       username='username',\n",
+        "                                                       private_key_file='./.ssh/id_rsa')\n",
         "    attached_dsvm_compute.wait_for_completion(show_output=True)"
       ]
     },

how-to-use-azureml/work-with-data/datadrift-tutorial/datadrift-tutorial.ipynb

@@ -380,6 +380,24 @@
         "#compute_target.delete()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Delete the DataDriftDetector\n",
+        "\n",
+        "Invoking the `delete()` method on the object deletes the the drift monitor permanently and cannot be undone. You will no longer be able to find it in the UI and the `list()` or `get()` methods. The object on which delete() was called will have its state set to deleted and name suffixed with deleted. The baseline and target datasets and model data that was collected, if any, are not deleted. The compute is not deleted. The DataDrift schedule pipeline is disabled and archived."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "monitor.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},

how-to-use-azureml/work-with-data/datasets-tutorial/pipeline-with-datasets/pipeline-for-image-classification.ipynb

@@ -279,7 +279,8 @@
         "                             outputs=[prepared_fashion_ds],\n",
         "                             source_directory=script_folder,\n",
         "                             compute_target=compute_target,\n",
-        "                             runconfig=run_config)"
+        "                             runconfig=run_config,\n",
+        "                             allow_reuse=False)"
       ]
     },
     {

how-to-use-azureml/work-with-data/datasets-tutorial/pipeline-with-datasets/pipeline-for-image-classification.yml

@@ -2,6 +2,5 @@ name: pipeline-for-image-classification
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-dataprep
   - pandas<=0.23.4
   - fuse

how-to-use-azureml/work-with-data/datasets-tutorial/timeseries-datasets/tabular-timeseries-dataset-filtering.yml

@@ -2,5 +2,4 @@ name: tabular-timeseries-dataset-filtering
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-dataprep
   - pandas<=0.23.4

how-to-use-azureml/work-with-data/datasets-tutorial/train-with-datasets/train-with-datasets.ipynb

@@ -227,7 +227,13 @@
         "from azureml.core import Dataset, Run\n",
         "from sklearn.model_selection import train_test_split\n",
         "from sklearn.tree import DecisionTreeClassifier\n",
-        "from sklearn.externals import joblib\n",
+        "# sklearn.externals.joblib is removed in 0.23\n",
+        "from sklearn import __version__ as sklearnver\n",
+        "from packaging.version import Version\n",
+        "if Version(sklearnver) < Version(\"0.23.0\"):\n",
+        "    from sklearn.externals import joblib\n",
+        "else:\n",
+        "    import joblib\n",
         "\n",
         "run = Run.get_context()\n",
         "# get input dataset by name\n",
@@ -291,7 +297,7 @@
         "              entry_script='train_iris.py', \n",
         "              # pass dataset object as an input with name 'titanic'\n",
         "              inputs=[dataset.as_named_input('iris')],\n",
-        "              pip_packages=['azureml-dataprep[fuse]'],\n",
+        "              pip_packages=['azureml-dataprep[fuse]', 'packaging'],\n",
         "              compute_target=compute_target) "
       ]
     },
@@ -415,11 +421,17 @@
         "import os\n",
         "import glob\n",
         "\n",
+        "from azureml.core.run import Run\n",
         "from sklearn.linear_model import Ridge\n",
         "from sklearn.metrics import mean_squared_error\n",
         "from sklearn.model_selection import train_test_split\n",
-        "from azureml.core.run import Run\n",
-        "from sklearn.externals import joblib\n",
+        "# sklearn.externals.joblib is removed in 0.23\n",
+        "from sklearn import __version__ as sklearnver\n",
+        "from packaging.version import Version\n",
+        "if Version(sklearnver) < Version(\"0.23.0\"):\n",
+        "    from sklearn.externals import joblib\n",
+        "else:\n",
+        "    import joblib\n",
         "\n",
         "import numpy as np\n",
         "\n",
@@ -480,9 +492,10 @@
         "from azureml.core.conda_dependencies import CondaDependencies\n",
         "\n",
         "conda_env = Environment('conda-env')\n",
-        "conda_env.python.conda_dependencies = CondaDependencies.create(pip_packages=['azureml-sdk',\n",
+        "conda_env.python.conda_dependencies = CondaDependencies.create(pip_packages=['azureml-core',\n",
         "                                                                             'azureml-dataprep[pandas,fuse]',\n",
-        "                                                                             'scikit-learn'])"
+        "                                                                             'scikit-learn',\n",
+        "                                                                             'packaging'])"
       ]
     },
     {

how-to-use-azureml/work-with-data/datasets-tutorial/train-with-datasets/train-with-datasets.yml

@@ -3,7 +3,6 @@ dependencies:
 - pip:
   - azureml-sdk
   - azureml-widgets
-  - azureml-dataprep
   - pandas<=0.23.4
   - fuse
   - scikit-learn

index.md

@@ -26,7 +26,7 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 | :star:[Datasets with ML Pipeline](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/work-with-data/datasets-tutorial/pipeline-with-datasets/pipeline-for-image-classification.ipynb) | Train | Fashion MNIST | Remote | None | Azure ML | Dataset, Pipeline, Estimator, ScriptRun |
 | :star:[Filtering data using Tabular Timeseiries Dataset related API](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/work-with-data/datasets-tutorial/timeseries-datasets/tabular-timeseries-dataset-filtering.ipynb) | Filtering | NOAA | Local | None | Azure ML | Dataset, Tabular Timeseries |
 | :star:[Train with Datasets (Tabular and File)](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/work-with-data/datasets-tutorial/train-with-datasets/train-with-datasets.ipynb) | Train | Iris, Diabetes | Remote | None | Azure ML | Dataset, Estimator, ScriptRun |
-| [Forecasting away from training data](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/forecasting-high-frequency/auto-ml-forecasting-function.ipynb) | Forecasting | None | Remote | None | Azure ML AutoML | Forecasting, Confidence Intervals |
+| [Forecasting away from training data](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.ipynb) | Forecasting | None | Remote | None | Azure ML AutoML | Forecasting, Confidence Intervals |
 | [Automated ML run with basic edition features.](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb) | Classification | Bankmarketing | AML | ACI | None | featurization, explainability, remote_run, AutomatedML |
 | [Classification of credit card fraudulent transactions using Automated ML](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb) | Classification | Creditcard | AML Compute | None | None | remote_run, AutomatedML |
 | [Automated ML run with featurization and model explainability.](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/auto-ml-regression-explanation-featurization.ipynb) | Regression | MachineData | AML | ACI | None | featurization, explainability, remote_run, AutomatedML |
@@ -41,6 +41,7 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 | :star:[How to Setup a Schedule for a Published Pipeline](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb) | Demonstrates the use of Schedules for Published Pipelines | Custom | AML Compute | None | Azure ML | None |
 | [How to setup a versioned Pipeline Endpoint](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-versioned-pipeline-endpoints.ipynb) | Demonstrates the use of PipelineEndpoint to run a specific version of the Published Pipeline | Custom | AML Compute | None | Azure ML | None |
 | :star:[How to use DataPath as a PipelineParameter](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-showcasing-datapath-and-pipelineparameter.ipynb) | Demonstrates the use of DataPath as a PipelineParameter | Custom | AML Compute | None | Azure ML | None |
+| :star:[How to use Dataset as a PipelineParameter](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-showcasing-dataset-and-pipelineparameter.ipynb) | Demonstrates the use of Dataset as a PipelineParameter | Custom | AML Compute | None | Azure ML | None |
 | [How to use AdlaStep with AML Pipelines](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-adla-as-compute-target.ipynb) | Demonstrates the use of AdlaStep | Custom | Azure Data Lake Analytics | None | Azure ML | None |
 | :star:[How to use DatabricksStep with AML Pipelines](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-use-databricks-as-compute-target.ipynb) | Demonstrates the use of DatabricksStep | Custom | Azure Databricks | None | Azure ML, Azure Databricks | None |
 | :star:[How to use AutoMLStep with AML Pipelines](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.ipynb) | Demonstrates the use of AutoMLStep | Custom | AML Compute | None | Automated Machine Learning | None |
@@ -64,6 +65,7 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 | [Resuming a model](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/ml-frameworks/tensorflow/training/train-tensorflow-resume-training/train-tensorflow-resume-training.ipynb) | Resume a model in TensorFlow from a previously submitted run | MNIST | AML Compute | None | TensorFlow | None |
 | [Training in Spark](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training/train-in-spark/train-in-spark.ipynb) | Submiting a run on a spark cluster | None | HDI cluster | None | PySpark | None |
 | [Train on Azure Machine Learning Compute](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb) | Submit a run on Azure Machine Learning Compute. | Diabetes | AML Compute | None | None | None |
+| [Train on Azure Machine Learning Compute Instance](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training/train-on-computeinstance/train-on-computeinstance.ipynb) | Submit a run on Azure Machine Learning Compute Instance. | Diabetes | Compute Instance | None | None | None |
 | [Train on local compute](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training/train-on-local/train-on-local.ipynb) | Train a model locally | Diabetes | Local | None | None | None |
 | [Train in a remote Linux virtual machine](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb) | Configure and execute a run | Diabetes | Data Science Virtual Machine | None | None | None |
 | [Using Tensorboard](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/training-with-deep-learning/export-run-history-to-tensorboard/export-run-history-to-tensorboard.ipynb) | Export the run history as Tensorboard logs | None | None | None | TensorFlow | None |
@@ -94,6 +96,8 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 |:----|:-----|:-------:|:----------------:|:-----------------:|:------------:|:------------:|
 | [DNN Text Featurization](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb) | Text featurization using DNNs for classification | None | AML Compute | None | None | None |
 | [configuration](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) |  |  |  |  |  |  |
+| [fairlearn-azureml-mitigation](https://github.com/Azure/MachineLearningNotebooks/blob/master//contrib/fairness/fairlearn-azureml-mitigation.ipynb) |  |  |  |  |  |  |
+| [upload-fairness-dashboard](https://github.com/Azure/MachineLearningNotebooks/blob/master//contrib/fairness/upload-fairness-dashboard.ipynb) |  |  |  |  |  |  |
 | [lightgbm-example](https://github.com/Azure/MachineLearningNotebooks/blob/master//contrib/gbdt/lightgbm/lightgbm-example.ipynb) |  |  |  |  |  |  |
 | [azure-ml-with-nvidia-rapids](https://github.com/Azure/MachineLearningNotebooks/blob/master//contrib/RAPIDS/azure-ml-with-nvidia-rapids.ipynb) |  |  |  |  |  |  |
 | [auto-ml-continuous-retraining](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.ipynb) |  |  |  |  |  |  |
@@ -111,9 +115,9 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 | [register-model-deploy-local-advanced](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/deploy-to-local/register-model-deploy-local-advanced.ipynb) |  |  |  |  |  |  |
 | [enable-app-insights-in-production-service](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb) |  |  |  |  |  |  |
 | [onnx-model-register-and-deploy](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/onnx/onnx-model-register-and-deploy.ipynb) |  |  |  |  |  |  |
+| [production-deploy-to-aks-ssl](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks-ssl.ipynb) |  |  |  |  |  |  |
 | [production-deploy-to-aks](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb) |  |  |  |  |  |  |
 | [production-deploy-to-aks-gpu](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/production-deploy-to-aks-gpu/production-deploy-to-aks-gpu.ipynb) |  |  |  |  |  |  |
-| [tensorflow-model-register-and-deploy](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/deployment/tensorflow/tensorflow-model-register-and-deploy.ipynb) |  |  |  |  |  |  |
 | [explain-model-on-amlcompute](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/explain-model/azure-integration/remote-explanation/explain-model-on-amlcompute.ipynb) |  |  |  |  |  |  |
 | [save-retrieve-explanations-run-history](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/explain-model/azure-integration/run-history/save-retrieve-explanations-run-history.ipynb) |  |  |  |  |  |  |
 | [train-explain-model-locally-and-deploy](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/explain-model/azure-integration/scoring-time/train-explain-model-locally-and-deploy.ipynb) |  |  |  |  |  |  |
@@ -123,7 +127,7 @@ Machine Learning notebook samples and encourage efficient retrieval of topics an
 | [authentication-in-azureml](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb) |  |  |  |  |  |  |
 | [pong_rllib](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.ipynb) |  |  |  |  |  |  |
 | [cartpole_ci](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/cartpole-on-compute-instance/cartpole_ci.ipynb) |  |  |  |  |  |  |
-| [cartpole_cc](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/cartpole-on-single-compute/cartpole_cc.ipynb) |  |  |  |  |  |  |
+| [cartpole_sc](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/cartpole-on-single-compute/cartpole_sc.ipynb) |  |  |  |  |  |  |
 | [minecraft](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/minecraft-on-distributed-compute/minecraft.ipynb) |  |  |  |  |  |  |
 | [devenv_setup](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/reinforcement-learning/setup/devenv_setup.ipynb) |  |  |  |  |  |  |
 | [Logging APIs](https://github.com/Azure/MachineLearningNotebooks/blob/master//how-to-use-azureml/track-and-monitor-experiments/logging-api/logging-api.ipynb) | Logging APIs and analyzing results | None | None | None | None | None |

setup-environment/configuration.ipynb

@@ -102,7 +102,7 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"This notebook was created using version 1.5.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

tutorials/create-first-ml-experiment/tutorial-1st-experiment-sdk-train.ipynb

@@ -212,7 +212,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Main Experiment page in the studio](../imgs/experiment_main.png)"
+        "![Main Experiment page in the studio](./imgs/experiment_main.png)"
       ]
     },
     {
@@ -226,7 +226,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "![Run details page in the studio](../imgs/model_download.png)"
+        "![Run details page in the studio](./imgs/model_download.png)"
       ]
     },
     {

tutorials/image-classification-mnist-data/img-classification-part2-deploy.ipynb

@@ -217,6 +217,7 @@
       "outputs": [],
       "source": [
         "%%time\n",
+        "import uuid\n",
         "from azureml.core.webservice import Webservice\n",
         "from azureml.core.model import InferenceConfig\n",
         "from azureml.core.environment import Environment\n",
@@ -230,8 +231,9 @@
         "myenv = Environment.get(workspace=ws, name=\"tutorial-env\", version=\"1\")\n",
         "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)\n",
         "\n",
+        "service_name = 'sklearn-mnist-svc-' + str(uuid.uuid4())[:4]\n",
         "service = Model.deploy(workspace=ws, \n",
-        "                       name='sklearn-mnist-svc', \n",
+        "                       name=service_name, \n",
         "                       models=[model], \n",
         "                       inference_config=inference_config, \n",
         "                       deployment_config=aciconfig)\n",

tutorials/image-classification-mnist-data/img-classification-part3-deploy-encrypted.ipynb

@@ -272,6 +272,7 @@
       "outputs": [],
       "source": [
         "%%time\n",
+        "import uuid\n",
         "from azureml.core.webservice import Webservice\n",
         "from azureml.core.model import InferenceConfig\n",
         "from azureml.core.environment import Environment\n",
@@ -284,8 +285,9 @@
         "myenv = Environment.get(workspace=ws, name=\"tutorial-env\")\n",
         "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)\n",
         "\n",
+        "service_name = 'sklearn-mnist-svc-' + str(uuid.uuid4())[:4]\n",
         "service = Model.deploy(workspace=ws, \n",
-        "                       name='sklearn-mnist-svc', \n",
+        "                       name=service_name, \n",
         "                       models=[model], \n",
         "                       inference_config=inference_config, \n",
         "                       deployment_config=aciconfig)\n",
@@ -489,7 +491,7 @@
         "import json\n",
         "from azureml.core import Webservice\n",
         "\n",
-        "service = Webservice(ws, 'sklearn-mnist-svc')\n",
+        "service = Webservice(ws, service_name)\n",
         "\n",
         "#pass the connection string for blob storage to give the server access to the uploaded public keys \n",
         "conn_str_template = 'DefaultEndpointsProtocol={};AccountName={};AccountKey={};EndpointSuffix=core.windows.net'\n",

tutorials/machine-learning-pipelines-advanced/scripts/batch_scoring.py

@@ -21,9 +21,10 @@ image_size = 299
 num_channel = 3
 
 
-def get_class_label_dict():
+def get_class_label_dict(labels_dir):
     label = []
-    proto_as_ascii_lines = tf.gfile.GFile("labels.txt").readlines()
+    labels_path = os.path.join(labels_dir, 'labels.txt')
+    proto_as_ascii_lines = tf.gfile.GFile(labels_path).readlines()
     for l in proto_as_ascii_lines:
         label.append(l.rstrip())
     return label
@@ -34,14 +35,10 @@ def init():
 
     parser = argparse.ArgumentParser(description="Start a tensorflow model serving")
     parser.add_argument('--model_name', dest="model_name", required=True)
-    parser.add_argument('--labels_name', dest="labels_name", required=True)
+    parser.add_argument('--labels_dir', dest="labels_dir", required=True)
     args, _ = parser.parse_known_args()
 
-    workspace = Run.get_context(allow_offline=False).experiment.workspace
-    label_ds = Dataset.get_by_name(workspace=workspace, name=args.labels_name)
-    label_ds.download(target_path='.', overwrite=True)
-
-    label_dict = get_class_label_dict()
+    label_dict = get_class_label_dict(args.labels_dir)
     classes_num = len(label_dict)
 
     with slim.arg_scope(inception_v3.inception_v3_arg_scope()):

tutorials/machine-learning-pipelines-advanced/tutorial-pipeline-batch-scoring-classification.ipynb

@@ -20,14 +20,8 @@
       "metadata": {},
       "source": [
         "# Use Azure Machine Learning Pipelines for batch prediction\n",
-        "\n",
-        "## Note\n",
-        "This notebook uses public preview functionality (ParallelRunStep). Please install azureml-contrib-pipeline-steps package before running this notebook.\n",
-        "\n",
-        "\n",
         "In this tutorial, you use Azure Machine Learning service pipelines to run a batch scoring image classification job. The example job uses the pre-trained [Inception-V3](https://arxiv.org/abs/1512.00567) CNN (convolutional neural network) Tensorflow model to classify unlabeled images. Machine learning pipelines optimize your workflow with speed, portability, and reuse so you can focus on your expertise, machine learning, rather than on infrastructure and automation. After building and publishing a pipeline, you can configure a REST endpoint to enable triggering the pipeline from any HTTP library on any platform.\n",
         "\n",
-        "\n",
         "In this tutorial, you learn the following tasks:\n",
         "\n",
         "> * Configure workspace and download sample data\n",
@@ -38,7 +32,7 @@
         "> * Build, run, and publish a pipeline\n",
         "> * Enable a REST endpoint for the pipeline\n",
         "\n",
-        "If you don\u00e2\u20ac\u2122t have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning service](https://aka.ms/AMLFree) today."
+        "If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning service](https://aka.ms/AMLFree) today."
       ]
     },
     {
@@ -129,7 +123,7 @@
         "from azureml.pipeline.core import PipelineData\n",
         "\n",
         "input_images = Dataset.File.from_files((batchscore_blob, \"batchscoring/images/\"))\n",
-        "label_ds = Dataset.File.from_files((batchscore_blob, \"batchscoring/labels/*.txt\"))\n",
+        "label_ds = Dataset.File.from_files((batchscore_blob, \"batchscoring/labels/\"))\n",
         "output_dir = PipelineData(name=\"scores\", \n",
         "                          datastore=def_data_store, \n",
         "                          output_path_on_compute=\"batchscoring/results\")"
@@ -149,7 +143,7 @@
       "outputs": [],
       "source": [
         "input_images = input_images.register(workspace = ws, name = \"input_images\")\n",
-        "label_ds = label_ds.register(workspace = ws, name = \"label_ds\")"
+        "label_ds = label_ds.register(workspace = ws, name = \"label_ds\", create_new_version=True)"
       ]
     },
     {
@@ -260,7 +254,7 @@
         "The script `batch_scoring.py` takes the following parameters, which get passed from the `ParallelRunStep` that you create later:\n",
         "\n",
         "- `--model_name`: the name of the model being used\n",
-        "- `--labels_name` : the name of the `Dataset` holding the `labels.txt` file \n",
+        "- `--labels_dir` : the directory path having the `labels.txt` file \n",
         "\n",
         "The pipelines infrastructure uses the `ArgumentParser` class to pass parameters into pipeline steps. For example, in the code below the first argument `--model_name` is given the property identifier `model_name`. In the `main()` function, this property is accessed using `Model.get_model_path(args.model_name)`."
       ]
@@ -296,7 +290,8 @@
         "from azureml.core.conda_dependencies import CondaDependencies\n",
         "from azureml.core.runconfig import DEFAULT_GPU_IMAGE\n",
         "\n",
-        "cd = CondaDependencies.create(pip_packages=[\"tensorflow-gpu==1.15.2\", \"azureml-defaults\"])\n",
+        "cd = CondaDependencies.create(pip_packages=[\"tensorflow-gpu==1.15.2\",\n",
+        "                                            \"azureml-core\", \"azureml-dataprep[fuse]\"])\n",
         "\n",
         "env = Environment(name=\"parallelenv\")\n",
         "env.python.conda_dependencies=cd\n",
@@ -317,7 +312,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunConfig\n",
+        "from azureml.pipeline.steps import ParallelRunConfig\n",
         "\n",
         "parallel_run_config = ParallelRunConfig(\n",
         "    environment=env,\n",
@@ -356,18 +351,20 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.contrib.pipeline.steps import ParallelRunStep\n",
+        "from azureml.pipeline.steps import ParallelRunStep\n",
         "from datetime import datetime\n",
         "\n",
         "parallel_step_name = \"batchscoring-\" + datetime.now().strftime(\"%Y%m%d%H%M\")\n",
         "\n",
+        "label_config = label_ds.as_named_input(\"labels_input\")\n",
+        "\n",
         "batch_score_step = ParallelRunStep(\n",
         "    name=parallel_step_name,\n",
         "    inputs=[input_images.as_named_input(\"input_images\")],\n",
         "    output=output_dir,\n",
-        "    models=[model],\n",
         "    arguments=[\"--model_name\", \"inception\",\n",
-        "               \"--labels_name\", \"label_ds\"],\n",
+        "               \"--labels_dir\", label_config],\n",
+        "    side_inputs=[label_config],\n",
         "    parallel_run_config=parallel_run_config,\n",
         "    allow_reuse=False\n",
         ")"