updating spark notebook

update samples from Release-106 as a part of  SDK release

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns

README.md

@@ -1,77 +1,43 @@
-# Azure Machine Learning service example notebooks
+# Azure Machine Learning Python SDK notebooks
 
 > a community-driven repository of examples using mlflow for tracking can be found at https://github.com/Azure/azureml-examples
 
-This repository contains example notebooks demonstrating the [Azure Machine Learning](https://azure.microsoft.com/services/machine-learning-service/) Python SDK which allows you to build, train, deploy and manage machine learning solutions using Azure.  The AML SDK allows you the choice of using local or cloud compute resources, while managing and maintaining the complete data science workflow from the cloud.
+Welcome to the Azure Machine Learning Python SDK notebooks repository!
 
-![Azure ML Workflow](https://raw.githubusercontent.com/MicrosoftDocs/azure-docs/master/articles/machine-learning/media/concept-azure-machine-learning-architecture/workflow.png)
+## Getting started
 
+These notebooks are recommended for use in an Azure Machine Learning [Compute Instance](https://docs.microsoft.com/azure/machine-learning/concept-compute-instance), where you can run them without any additional set up.
 
-## Quick installation
+However, the notebooks can be run in any development environment with the correct `azureml` packages installed.
-```sh
-pip install azureml-sdk
-```
-Read more detailed instructions on [how to set up your environment](./NBSETUP.md) using Azure Notebook service, your own Jupyter notebook server, or Docker.
 
-## How to navigate and use the example notebooks?
+Install the `azureml.core` Python package:
-If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, you should always run the [Configuration](./configuration.ipynb) notebook first when setting up a notebook library on a new machine or in a new environment. It configures your notebook library to connect to an Azure Machine Learning workspace, and sets up your workspace and compute to be used by many of the other examples. 
-This [index](./index.md) should assist in navigating the Azure Machine Learning notebook samples and encourage efficient retrieval of topics and content. 
-
-If you want to...
-
- * ...try out and explore Azure ML, start with image classification tutorials: [Part 1 (Training)](./tutorials/image-classification-mnist-data/img-classification-part1-training.ipynb) and [Part 2 (Deployment)](./tutorials/image-classification-mnist-data/img-classification-part2-deploy.ipynb).
- * ...learn about experimentation and tracking run history: [track and monitor experiments](./how-to-use-azureml/track-and-monitor-experiments).
- * ...train deep learning models at scale, first learn about [Machine Learning Compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and then try [distributed hyperparameter tuning](./how-to-use-azureml/ml-frameworks/pytorch/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) and [distributed training](./how-to-use-azureml/ml-frameworks/pytorch/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb).
- * ...deploy models as a realtime scoring service, first learn the basics by [deploying to Azure Container Instance](./how-to-use-azureml/deployment/deploy-to-cloud/model-register-and-deploy.ipynb), then learn how to [production deploy models on Azure Kubernetes Cluster](./how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb).
- * ...deploy models as a batch scoring service: [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb) and [use Machine Learning Pipelines to deploy your model](https://aka.ms/pl-batch-scoring).
- * ...monitor your deployed models, learn about using [App Insights](./how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb).
-
-## Tutorials
-
-The [Tutorials](./tutorials) folder contains notebooks for the tutorials described in the [Azure Machine Learning documentation](https://aka.ms/aml-docs).
-  
-## How to use Azure ML
-
-The [How to use Azure ML](./how-to-use-azureml) folder contains specific examples demonstrating the features of the Azure Machine Learning SDK
-
-- [Training](./how-to-use-azureml/training) - Examples of how to build models using Azure ML's logging and execution capabilities on local and remote compute targets
-- [Training with ML and DL frameworks](./how-to-use-azureml/ml-frameworks) - Examples demonstrating how to build and train machine learning models at scale on Azure ML and perform hyperparameter tuning.
-- [Manage Azure ML Service](./how-to-use-azureml/manage-azureml-service) - Examples how to perform tasks, such as authenticate against Azure ML service in different ways.
-- [Automated Machine Learning](./how-to-use-azureml/automated-machine-learning) - Examples using Automated Machine Learning to automatically generate optimal machine learning pipelines and models
-- [Machine Learning Pipelines](./how-to-use-azureml/machine-learning-pipelines) - Examples showing how to create and use reusable pipelines for training and batch scoring
-- [Deployment](./how-to-use-azureml/deployment) - Examples showing how to deploy and manage machine learning models and solutions
-- [Azure Databricks](./how-to-use-azureml/azure-databricks) - Examples showing how to use Azure ML with Azure Databricks
-- [Reinforcement Learning](./how-to-use-azureml/reinforcement-learning) - Examples showing how to train reinforcement learning agents
-
----
-## Documentation
-
- * Quickstarts, end-to-end tutorials, and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).
- * [Python SDK reference](https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py)
- * Azure ML Data Prep SDK [overview](https://aka.ms/data-prep-sdk), [Python SDK reference](https://aka.ms/aml-data-prep-apiref), and [tutorials and how-tos](https://aka.ms/aml-data-prep-notebooks).
-
----
-
-
-## Community Repository 
-Visit this [community repository](https://github.com/microsoft/MLOps/tree/master/examples) to find useful end-to-end sample notebooks. Also, please follow these [contribution guidelines](https://github.com/microsoft/MLOps/blob/master/contributing.md) when contributing to this repository.   
-
-## Projects using Azure Machine Learning
-
-Visit following repos to see projects contributed by Azure ML users:
- - [Learn about Natural Language Processing best practices using Azure Machine Learning service](https://github.com/microsoft/nlp)
- - [Pre-Train BERT models using Azure Machine Learning service](https://github.com/Microsoft/AzureML-BERT)
- - [Fashion MNIST with Azure ML SDK](https://github.com/amynic/azureml-sdk-fashion)
- - [UMass Amherst Student Samples](https://github.com/katiehouse3/microsoft-azure-ml-notebooks) - A number of end-to-end machine learning notebooks, including machine translation, image classification, and customer churn, created by students in the 696DS course at UMass Amherst.
- 
-## Data/Telemetry 
-This repository collects usage data and sends it to Microsoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
-
-To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
 
 ```sh
-    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)"
+pip install azureml-core
 ```
-This URL will be slightly different depending on the file. 
 
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/README.png)
+Install additional packages as needed:
+
+```sh
+pip install azureml-mlflow
+pip install azureml-dataset-runtime
+pip install azureml-automl-runtime
+pip install azureml-pipeline
+pip install azureml-pipeline-steps
+...
+```
+
+We recommend starting with one of the [quickstarts](tutorials/compute-instance-quickstarts).
+
+## Contributing
+
+This repository is a push-only mirror. Pull requests are ignored.
+
+## Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Please see the [code of conduct](CODE_OF_CONDUCT.md) for details.
+
+## Reference
+
+- [Documentation](https://docs.microsoft.com/azure/machine-learning)
+

how-to-use-azureml/deployment/spark/model-register-and-deploy-spark.ipynb

@@ -2,23 +2,22 @@
   "cells": [
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "Copyright (c) Microsoft Corporation. All rights reserved.\n",
         "\n",
         "Licensed under the MIT License."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/deployment/deploy-to-cloud/model-register-and-deploy.png)"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "# Register Spark Model and deploy as Webservice\n",
         "\n",
@@ -26,120 +25,128 @@
         "\n",
         " 1. Register Spark Model\n",
         " 2. Deploy Spark Model as Webservice"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "## Prerequisites\n",
         "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the [configuration](../../../configuration.ipynb) Notebook first if you haven't."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
-      "outputs": [],
       "source": [
-        "# Check core SDK version number\n",
+        "# Check core SDK version number\r\n",
-        "import azureml.core\n",
+        "import azureml.core\r\n",
-        "\n",
+        "\r\n",
         "print(\"SDK version:\", azureml.core.VERSION)"
-      ]
+      ],
+      "outputs": [],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "## Initialize Workspace\n",
         "\n",
         "Initialize a workspace object from persisted configuration."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
+      "source": [
+        "from azureml.core import Workspace\r\n",
+        "\r\n",
+        "ws = Workspace.from_config()\r\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
+      ],
+      "outputs": [],
       "metadata": {
         "tags": [
           "create workspace"
         ]
-      },
+      }
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.from_config()\n",
-        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
-      ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "### Register Model"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "You can add tags and descriptions to your Models. Note you need to have a `iris.model` file in the current directory. This model file is generated using [train in spark](../training/train-in-spark/train-in-spark.ipynb) notebook. The below call registers that file as a Model with the same name `iris.model` in the workspace.\n",
         "\n",
         "Using tags, you can track useful information such as the name and version of the machine learning library used to train the model. Note that tags must be alphanumeric."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
+      "source": [
+        "from azureml.core.model import Model\r\n",
+        "\r\n",
+        "model = Model.register(model_path=\"iris.model\",\r\n",
+        "                       model_name=\"iris.model\",\r\n",
+        "                       tags={'type': \"regression\"},\r\n",
+        "                       description=\"Logistic regression model to predict iris species\",\r\n",
+        "                       workspace=ws)"
+      ],
+      "outputs": [],
       "metadata": {
         "tags": [
           "register model from file"
         ]
-      },
+      }
-      "outputs": [],
-      "source": [
-        "from azureml.core.model import Model\n",
-        "\n",
-        "model = Model.register(model_path=\"iris.model\",\n",
-        "                       model_name=\"iris.model\",\n",
-        "                       tags={'type': \"regression\"},\n",
-        "                       description=\"Logistic regression model to predict iris species\",\n",
-        "                       workspace=ws)"
-      ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "### Fetch Environment"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "You can now create and/or use an Environment object when deploying a Webservice. The Environment can have been previously registered with your Workspace, or it will be registered with it as a part of the Webservice deployment.\n",
         "\n",
         "In this notebook, we will be using 'AzureML-PySpark-MmlSpark-0.15', a curated environment.\n",
         "\n",
         "More information can be found in our [using environments notebook](../training/using-environments/using-environments.ipynb)."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
-      "outputs": [],
       "source": [
-        "from azureml.core import Environment\n",
+        "from azureml.core import Environment\r\n",
-        "\n",
+        "from azureml.core.environment import SparkPackage\r\n",
-        "env = Environment.get(ws, name='AzureML-PySpark-MmlSpark-0.15')\n"
+        "from azureml.core.conda_dependencies import CondaDependencies\r\n",
-      ]
+        "\r\n",
+        "myenv = Environment('my-pyspark-environment')\r\n",
+        "myenv.docker.base_image = \"mcr.microsoft.com/mmlspark/release:0.15\"\r\n",
+        "myenv.inferencing_stack_version = \"latest\"\r\n",
+        "myenv.python.conda_dependencies = CondaDependencies.create(pip_packages=[\"azureml-core\",\"azureml-defaults\",\"azureml-telemetry\",\"azureml-train-restclients-hyperdrive\",\"azureml-train-core\"], python_version=\"3.6.2\")\r\n",
+        "myenv.python.conda_dependencies.add_channel(\"conda-forge\")\r\n",
+        "myenv.spark.packages = [SparkPackage(\"com.microsoft.ml.spark\", \"mmlspark_2.11\", \"0.15\"), SparkPackage(\"com.microsoft.azure\", \"azure-storage\", \"2.0.0\"), SparkPackage(\"org.apache.hadoop\", \"hadoop-azure\", \"2.7.0\")]\r\n",
+        "myenv.spark.repositories = [\"https://mmlspark.azureedge.net/maven\"]\r\n"
+      ],
+      "outputs": [],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "## Create Inference Configuration\n",
         "\n",
@@ -157,109 +164,109 @@
         " - source_directory = holds source path as string, this entire folder gets added in image so its really easy to access any files within this folder or subfolder\n",
         " - entry_script = contains logic specific to initializing your model and running predictions\n",
         " - environment = An environment object to use for the deployment. Doesn't have to be registered"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
+      "source": [
+        "from azureml.core.model import InferenceConfig\r\n",
+        "\r\n",
+        "inference_config = InferenceConfig(entry_script=\"score.py\", environment=myenv)"
+      ],
+      "outputs": [],
       "metadata": {
         "tags": [
           "create image"
         ]
-      },
+      }
-      "outputs": [],
-      "source": [
-        "from azureml.core.model import InferenceConfig\n",
-        "\n",
-        "inference_config = InferenceConfig(entry_script=\"score.py\", environment=env)"
-      ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "### Deploy Model as Webservice on Azure Container Instance\n",
         "\n",
         "Note that the service creation can take few minutes."
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
+      "source": [
+        "from azureml.core.webservice import AciWebservice, Webservice\r\n",
+        "from azureml.exceptions import WebserviceException\r\n",
+        "\r\n",
+        "deployment_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1)\r\n",
+        "aci_service_name = 'aciservice1'\r\n",
+        "\r\n",
+        "try:\r\n",
+        "    # if you want to get existing service below is the command\r\n",
+        "    # since aci name needs to be unique in subscription deleting existing aci if any\r\n",
+        "    # we use aci_service_name to create azure aci\r\n",
+        "    service = Webservice(ws, name=aci_service_name)\r\n",
+        "    if service:\r\n",
+        "        service.delete()\r\n",
+        "except WebserviceException as e:\r\n",
+        "    print()\r\n",
+        "\r\n",
+        "service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config)\r\n",
+        "\r\n",
+        "service.wait_for_deployment(True)\r\n",
+        "print(service.state)"
+      ],
+      "outputs": [],
       "metadata": {
         "tags": [
           "azuremlexception-remarks-sample"
         ]
-      },
+      }
-      "outputs": [],
-      "source": [
-        "from azureml.core.webservice import AciWebservice, Webservice\n",
-        "from azureml.exceptions import WebserviceException\n",
-        "\n",
-        "deployment_config = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1)\n",
-        "aci_service_name = 'aciservice1'\n",
-        "\n",
-        "try:\n",
-        "    # if you want to get existing service below is the command\n",
-        "    # since aci name needs to be unique in subscription deleting existing aci if any\n",
-        "    # we use aci_service_name to create azure aci\n",
-        "    service = Webservice(ws, name=aci_service_name)\n",
-        "    if service:\n",
-        "        service.delete()\n",
-        "except WebserviceException as e:\n",
-        "    print()\n",
-        "\n",
-        "service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config)\n",
-        "\n",
-        "service.wait_for_deployment(True)\n",
-        "print(service.state)"
-      ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "#### Test web service"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "metadata": {},
-      "outputs": [],
       "source": [
-        "import json\n",
+        "import json\r\n",
-        "test_sample = json.dumps({'features':{'type':1,'values':[4.3,3.0,1.1,0.1]},'label':2.0})\n",
+        "test_sample = json.dumps({'features':{'type':1,'values':[4.3,3.0,1.1,0.1]},'label':2.0})\r\n",
-        "\n",
+        "\r\n",
-        "test_sample_encoded = bytes(test_sample, encoding='utf8')\n",
+        "test_sample_encoded = bytes(test_sample, encoding='utf8')\r\n",
-        "prediction = service.run(input_data=test_sample_encoded)\n",
+        "prediction = service.run(input_data=test_sample_encoded)\r\n",
         "print(prediction)"
-      ]
+      ],
+      "outputs": [],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "#### Delete ACI to clean up"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "code",
       "execution_count": null,
+      "source": [
+        "service.delete()"
+      ],
+      "outputs": [],
       "metadata": {
         "tags": [
           "deploy service",
           "aci"
         ]
-      },
+      }
-      "outputs": [],
-      "source": [
-        "service.delete()"
-      ]
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "### Model Profiling\n",
         "\n",
@@ -271,11 +278,11 @@
         "profiling_results = profile.get_results()\n",
         "print(profiling_results)\n",
         "```"
-      ]
+      ],
+      "metadata": {}
     },
     {
       "cell_type": "markdown",
-      "metadata": {},
       "source": [
         "### Model Packaging\n",
         "\n",
@@ -296,7 +303,8 @@
         "package.wait_for_creation(show_output=True)\n",
         "package.save(\"./local_context_dir\")\n",
         "```"
-      ]
+      ],
+      "metadata": {}
     }
   ],
   "metadata": {