jprdonnelly/MachineLearningNotebooks
1
0
Fork 0
mirror of https://github.com/Azure/MachineLearningNotebooks.git synced 2025-12-19 17:17:04 -05:00
Code Issues Projects Releases Wiki Activity

version 1.0.41

Browse Source
This commit is contained in:
Roope Astala
2019-05-29 10:59:59 -04:00
parent 2a479bb01e
commit db6ae67940
53 changed files with 2088 additions and 803 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
README.md
View File

@@ -55,62 +55,8 @@ Visit following repos to see projects contributed by Azure ML users:
- [Fine tune natural language processing models using Azure Machine Learning service](https://github.com/Microsoft/AzureML-BERT)
- [Fashion MNIST with Azure ML SDK](https://github.com/amynic/azureml-sdk-fashion)
## Azure Machine Learning Resources & Links
## Product Documentation
- [Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/)
- [Azure Machine Learning Studio](https://docs.microsoft.com/en-us/azure/machine-learning/studio/)
## Product Team Blogs
- [What’s new in Azure Machine Learning service](https://aka.ms/aml-blog-whats-new)
- [Announcing automated ML capability in Azure Machine Learning](https://aka.ms/aml-blog-automl)
- [Experimentation using Azure Machine Learning](https://aka.ms/aml-blog-experimentation)
- [Azure AI – Making AI real for business](https://aka.ms/aml-blog-overview)
## Community Blogs
- [Power Bat – How Spektacom is Powering the Game of Cricket with Microsoft AI](https://blogs.technet.microsoft.com/machinelearning/2018/10/11/power-bat-how-spektacom-is-powering-the-game-of-cricket-with-microsoft-ai/)
## Ignite 2018 Public Preview Launch Sessions
- [AI with Azure Machine Learning services: Simplifying the data science process](https://myignite.techcommunity.microsoft.com/sessions/66248)
- [AI TechTalk: Azure Machine Learning SDK - a walkthrough](https://myignite.techcommunity.microsoft.com/sessions/66265)
- [AI for an intelligent cloud and intelligent edge: Discover, deploy, and manage with Azure ML services](https://myignite.techcommunity.microsoft.com/sessions/65389)
- [Generating high quality models efficiently using Automated ML and Hyperparameter Tuning](https://myignite.techcommunity.microsoft.com/sessions/66245)
- [AI for pros: Deep learning with PyTorch using the Azure Data Science Virtual Machine and scaling training with Azure ML](https://myignite.techcommunity.microsoft.com/sessions/66244)
## Get-started Videos on YouTube
- [Get started with Python SDK](https://youtu.be/VIsXeTuW3FU)
- [Get started from Azure Portal](https://youtu.be/lCkYUHV86Mk)
## Third Party Articles
- [Azure’s new machine learning features embrace Python](https://www.infoworld.com/article/3306840/azure/azures-new-machine-learning-features-embrace-python.html) (InfoWorld)
- [How to use Azure ML in Windows 10](https://www.infoworld.com/article/3308381/azure/how-to-use-azure-ml-in-windows-10.html) (InfoWorld)
- [How Azure ML Streamlines Cloud-based Machine Learning](https://thenewstack.io/how-the-azure-ml-streamlines-cloud-based-machine-learning/) (The New Stack)
- [Facebook launches PyTorch 1.0 with integrations for Google Cloud, AWS, and Azure Machine Learning](https://venturebeat.com/2018/10/02/facebook-launches-pytorch-1-0-integrations-for-google-cloud-aws-and-azure-machine-learning/) (VentureBeat)
- [How Microsoft Uses Machine Learning to Help You Build Machine Learning Pipelines](https://towardsdatascience.com/how-microsoft-uses-machine-learning-to-help-you-build-machine-learning-pipelines-be75f710613b) (Towards Data Science)
- [Microsoft's Machine Learning Tools for Developers Get Smarter](https://techcrunch.com/2018/09/24/microsofts-machine-learning-tools-for-developers-get-smarter/) (TechCrunch)
- [Microsoft introduces Azure service to automatically build AI models](https://venturebeat.com/2018/09/24/microsoft-introduces-azure-service-to-automatically-build-ai-models/) (VentureBeat)
## Community Projects
- [Use Papermill with Azure ML](https://github.com/jreynolds01/papermill_execution_azureml/)
- [Fashion MNIST](https://github.com/amynic/azureml-sdk-fashion)
- Keras on Databricks
- [Samples from CSS](https://github.com/Azure/AMLSamples)
## Azure Machine Learning Studio Resources
- [A-Z Machine Learning using Azure Machine Learning (AzureML)](https://www.udemy.com/machine-learning-using-azureml/)
- [Machine Learning In The Cloud With Azure Machine Learning](https://www.udemy.com/machine-learning-in-the-cloud-with-azure-machine-learning/)
- [How to Become A Data Scientist Using Azure Machine Learning](https://www.udemy.com/azure-machine-learning-introduction/)
- [Learn Azure Machine Learning from scratch](https://www.udemy.com/learn-azure-machine-learning-from-scratch/)
- [Azure Machine Learning Studio PowerShell Module](https://aka.ms/amlps)
## Forum Help
- [Azure Machine Learning service](https://social.msdn.microsoft.com/Forums/en-US/home?forum=AzureMachineLearningService)
- [Azure Machine Learning Studio](https://social.msdn.microsoft.com/forums/azure/en-US/home?forum=MachineLearning)
## Data/Telemetry
This repository collects usage data and sends it to Microsoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
This repository collects usage data and sends it to Mircosoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
@@ -119,5 +65,14 @@ To opt out of tracking, please go to the raw markdown or .ipynb files and remove
```
This URL will be slightly different depending on the file.
## Data/Telemetry
This repository collects usage data and sends it to Mircosoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
```sh
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)"
```
This URL will be slightly different depending on the file.
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/README.png)

5
how-to-use-azureml/README.md
View File

@@ -2,7 +2,7 @@
Learn how to use Azure Machine Learning services for experimentation and model management.
If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, go through the [configuration Notebook](../configuration.ipynb) first if you haven't already to establish your connection to the AzureML Workspace. Then, run the notebooks in following recommended order.
As a pre-requisite, run the [configuration Notebook](../configuration.ipynb) notebook first to set up your Azure ML Workspace. Then, run the notebooks in following recommended order.
* [train-within-notebook](./training/train-within-notebook): Train a model hile tracking run history, and learn how to deploy the model as web service to Azure Container Instance.
* [train-on-local](./training/train-on-local): Learn how to submit a run to local computer and use Azure ML managed run configuration.
@@ -15,6 +15,3 @@ If you are using an Azure Machine Learning Notebook VM, you are all set. Otherw
* [enable-app-insights-in-production-service](./deployment/enable-app-insights-in-production-service) Learn how to use App Insights with production web service.
Find quickstarts, end-to-end tutorials, and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)

2
how-to-use-azureml/automated-machine-learning/automl_env.yml
View File

@@ -10,7 +10,7 @@ dependencies:
- urllib3<1.24
- scipy>=1.0.0,<=1.1.0
- scikit-learn>=0.19.0,<=0.20.3
- pandas>=0.22.0,<0.23.0
- pandas>=0.22.0,<=0.23.4
- py-xgboost<=0.80
- pip:

9
how-to-use-azureml/automated-machine-learning/classification-with-deployment/auto-ml-classification-with-deployment.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-deployment/auto-ml-classification-with-deployment.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -18,7 +25,7 @@
"\n",
"## Contents\n",
"1. [Introduction](#Introduction)\n",
"1. [Setup](#setup)\n",
"1. [Setup](#Setup)\n",
"1. [Train](#Train)\n",
"1. [Deploy](#Deploy)\n",
"1. [Test](#Test)"

86
how-to-use-azureml/automated-machine-learning/classification-with-onnx/auto-ml-classification-with-onnx.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-onnx/auto-ml-classification-with-onnx.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -66,11 +73,12 @@
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn import datasets\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"import azureml.core\n",
"from azureml.core.experiment import Experiment\n",
"from azureml.core.workspace import Workspace\n",
"from azureml.train.automl import AutoMLConfig"
"from azureml.train.automl import AutoMLConfig, constants"
]
},
{
@@ -106,7 +114,7 @@
"source": [
"## Data\n",
"\n",
"This uses scikit-learn's [load_digits](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) method."
"This uses scikit-learn's [load_iris](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html) method."
]
},
{
@@ -115,11 +123,17 @@
"metadata": {},
"outputs": [],
"source": [
"digits = datasets.load_digits()\n",
"iris = datasets.load_iris()\n",
"X_train, X_test, y_train, y_test = train_test_split(iris.data, \n",
" iris.target, \n",
" test_size=0.2, \n",
" random_state=0)\n",
"\n",
"# Exclude the first 100 rows from training so that they can be used for test.\n",
"X_train = digits.data[100:,:]\n",
"y_train = digits.target[100:]"
"# Convert the X_train and X_test to pandas DataFrame and set column names,\n",
"# This is needed for initializing the input variable names of ONNX model, \n",
"# and the prediction with the ONNX model using the inference helper.\n",
"X_train = pd.DataFrame(X_train, columns=['c1', 'c2', 'c3', 'c4'])\n",
"X_test = pd.DataFrame(X_test, columns=['c1', 'c2', 'c3', 'c4'])"
]
},
{
@@ -158,6 +172,7 @@
" verbosity = logging.INFO, \n",
" X = X_train, \n",
" y = y_train,\n",
" preprocess=True,\n",
" enable_onnx_compatible_models=True,\n",
" path = project_folder)"
]
@@ -253,6 +268,65 @@
"onnx_fl_path = \"./best_model.onnx\"\n",
"OnnxConverter.save_onnx_model(onnx_mdl, onnx_fl_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Predict with the ONNX model, using onnxruntime package"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import json\n",
"from azureml.automl.core.onnx_convert import OnnxConvertConstants\n",
"\n",
"if sys.version_info < OnnxConvertConstants.OnnxIncompatiblePythonVersion:\n",
" python_version_compatible = True\n",
"else:\n",
" python_version_compatible = False\n",
"\n",
"try:\n",
" import onnxruntime\n",
" from azureml.automl.core.onnx_convert import OnnxInferenceHelper \n",
" onnxrt_present = True\n",
"except ImportError:\n",
" onnxrt_present = False\n",
"\n",
"def get_onnx_res(run):\n",
" res_path = '_debug_y_trans_converter.json'\n",
" run.download_file(name=constants.MODEL_RESOURCE_PATH_ONNX, output_file_path=res_path)\n",
" with open(res_path) as f:\n",
" onnx_res = json.load(f)\n",
" return onnx_res\n",
"\n",
"if onnxrt_present and python_version_compatible: \n",
" mdl_bytes = onnx_mdl.SerializeToString()\n",
" onnx_res = get_onnx_res(best_run)\n",
"\n",
" onnxrt_helper = OnnxInferenceHelper(mdl_bytes, onnx_res)\n",
" pred_onnx, pred_prob_onnx = onnxrt_helper.predict(X_test)\n",
"\n",
" print(pred_onnx)\n",
" print(pred_prob_onnx)\n",
"else:\n",
" if not python_version_compatible:\n",
" print('Please use Python version 3.6 to run the inference helper.') \n",
" if not onnxrt_present:\n",
" print('Please install the onnxruntime package to do the prediction with ONNX model.')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {

7
how-to-use-azureml/automated-machine-learning/classification-with-whitelisting/auto-ml-classification-with-whitelisting.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-with-whitelisting/auto-ml-classification-with-whitelisting.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/classification/auto-ml-classification.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification/auto-ml-classification.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/dataprep-remote-execution/auto-ml-dataprep-remote-execution.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/dataprep-remote-execution/auto-ml-dataprep-remote-execution.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/dataprep/auto-ml-dataprep.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/dataprep/auto-ml-dataprep.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/exploring-previous-runs/auto-ml-exploring-previous-runs.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/exploring-previous-runs/auto-ml-exploring-previous-runs.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/missing-data-blacklist-early-termination/auto-ml-missing-data-blacklist-early-termination.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/missing-data-blacklist-early-termination/auto-ml-missing-data-blacklist-early-termination.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/model-explanation/auto-ml-model-explanation.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/model-explanation/auto-ml-model-explanation.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/remote-amlcompute/auto-ml-remote-amlcompute.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-amlcompute/auto-ml-remote-amlcompute.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/remote-attach/auto-ml-remote-attach.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-attach/auto-ml-remote-attach.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-execution-with-datastore/auto-ml-remote-execution-with-datastore.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/remote-execution/auto-ml-remote-execution.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/remote-execution/auto-ml-remote-execution.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/sample-weight/auto-ml-sample-weight.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/sample-weight/auto-ml-sample-weight.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/sparse-data-train-test-split/auto-ml-sparse-data-train-test-split.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/sparse-data-train-test-split/auto-ml-sparse-data-train-test-split.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

7
how-to-use-azureml/automated-machine-learning/subsampling/auto-ml-subsampling-local.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/subsampling/auto-ml-subsampling-local.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

4
how-to-use-azureml/azure-databricks/README.md
View File

@@ -27,3 +27,7 @@ You can use Azure Databricks as a compute target from [Azure Machine Learning Pi
For more on SDK concepts, please refer to [notebooks](https://github.com/Azure/MachineLearningNotebooks).
**Please let us know your feedback.**
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/README.png)

14
how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.ipynb
View File

@@ -11,6 +11,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/build-model-run-history-03.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -333,6 +340,13 @@
"source": [
"dbutils.notebook.exit(\"success\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.png)"
]
}
],
"metadata": {

14
how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.ipynb
View File

@@ -11,6 +11,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/deploy-to-aci-04.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -277,6 +284,13 @@
"#comment to not delete the web service\n",
"myservice.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.png)"
]
}
],
"metadata": {

14
how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.ipynb
View File

@@ -11,6 +11,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -203,6 +210,13 @@
"#model.delete()\n",
"aks_target.delete() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aks-existingimage-05.png)"
]
}
],
"metadata": {

14
how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.ipynb
View File

@@ -11,6 +11,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/ingest-data-02.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -139,6 +146,13 @@
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.png)"
]
}
],
"metadata": {

14
how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.ipynb
View File

@@ -11,6 +11,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/azure-databricks/amlsdk/installation-and-configuration-01.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -143,6 +150,13 @@
" 'Subscription id: ' + ws.subscription_id, \n",
" 'Resource group: ' + ws.resource_group, sep = '\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.png)"
]
}
],
"metadata": {

7
how-to-use-azureml/azure-databricks/automl/automl-databricks-local-01.ipynb
View File

@@ -660,6 +660,13 @@
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/automl/automl-databricks-local-01.png)"
]
}
],
"metadata": {

7
how-to-use-azureml/azure-databricks/automl/automl-databricks-local-with-deployment.ipynb
View File

@@ -796,6 +796,13 @@
"source": [
"myservice.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/automl/automl-databricks-local-with-deployment.png)"
]
}
],
"metadata": {

7
how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.ipynb
View File

@@ -677,6 +677,13 @@
"# Next: ADLA as a Compute Target\n",
"To use ADLA as a compute target from Azure Machine Learning Pipeline, a AdlaStep is used. This [notebook](./aml-pipelines-use-adla-as-compute-target.ipynb) demonstrates the use of AdlaStep in Azure Machine Learning Pipeline."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.png)"
]
}
],
"metadata": {

7
how-to-use-azureml/deploy-to-cloud/model-register-and-deploy.ipynb
View File

@@ -9,6 +9,13 @@
"Licensed under the MIT License."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/deploy-to-cloud/model-register-and-deploy.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},

2
how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb
View File

@@ -255,7 +255,7 @@
"source": [
"from azureml.core.conda_dependencies import CondaDependencies \n",
"\n",
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime\",\"azureml-core\"])\n",
"myenv = CondaDependencies.create(pip_packages=[\"numpy\",\"onnxruntime==0.4.0\",\"azureml-core\"])\n",
"\n",
"with open(\"myenv.yml\",\"w\") as f:\n",
" f.write(myenv.serialize_to_string())"

7
how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb
View File

@@ -63,7 +63,7 @@
"metadata": {},
"outputs": [],
"source": [
"aml_compute = ws.get_default_compute_target(\"CPU\")"
"aml_compute_target = ws.get_default_compute_target(\"CPU\")"
]
},
{
@@ -284,7 +284,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Disable the schedule"
"### Disable the schedule\n",
"It is important to note the best practice of disabling schedules when not in use.\n",
"The number of schedule triggers allowed per month per region per subscription is 100,000.\n",
"This is calculated using the project trigger counts for all active schedules."
]
},
{

1
how-to-use-azureml/training/README.md
View File

@@ -8,5 +8,6 @@ Follow these sample notebooks to learn:
4. [Train on AmlCompute](train-on-amlcompute): train a model using an AmlCompute cluster as compute target.
5. [Train in an HDI Spark cluster](train-in-spark): train a Spark ML model using an HDInsight Spark cluster as compute target.
6. [Logging API](logging-api): experiment with various logging functions to create runs and automatically generate graphs.
7. [Train and hyperparameter tune on Iris Dataset with Scikit-learn](train-hyperparameter-tune-deploy-with-sklearn): train a model using the Scikit-learn estimator and tune hyperparameters with Hyperdrive.
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/training/README.png)

2
how-to-use-azureml/training/logging-api/logging-api.ipynb
View File

@@ -100,7 +100,7 @@
"\n",
"# Check core SDK version number\n",
"\n",
"print(\"This notebook was created using SDK version 1.0.39, you are currently running version\", azureml.core.VERSION)"
"print(\"This notebook was created using SDK version 1.0.41, you are currently running version\", azureml.core.VERSION)"
]
},
{

118
how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb
View File

@@ -164,6 +164,30 @@
"shutil.copy('train.py', project_folder)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create environment\n",
"\n",
"Create Docker based environment with scikit-learn installed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from azureml.core import Environment\n",
"from azureml.core.conda_dependencies import CondaDependencies\n",
"\n",
"myenv = Environment(\"myenv\")\n",
"\n",
"myenv.docker.enabled = True\n",
"myenv.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -179,38 +203,32 @@
"metadata": {},
"outputs": [],
"source": [
"from azureml.core.runconfig import RunConfiguration\n",
"from azureml.core.conda_dependencies import CondaDependencies\n",
"from azureml.core import ScriptRunConfig\n",
"from azureml.core.runconfig import DEFAULT_CPU_IMAGE\n",
"\n",
"# create a new runconfig object\n",
"run_config = RunConfiguration()\n",
"src = ScriptRunConfig(source_directory=project_folder, script='train.py')\n",
"\n",
"default_compute_target = ws.get_default_compute_target(type=\"CPU\")\n",
"# Use default compute target\n",
"src.run_config.target = ws.get_default_compute_target(type=\"CPU\").name\n",
"\n",
"# signal that you want to use AmlCompute to execute script.\n",
"run_config.target = default_compute_target.name\n",
"\n",
"# enable Docker \n",
"run_config.environment.docker.enabled = True\n",
"\n",
"# set Docker base image to the default CPU-based image\n",
"run_config.environment.docker.base_image = DEFAULT_CPU_IMAGE\n",
"\n",
"# use conda_dependencies.yml to create a conda environment in the Docker image for execution\n",
"run_config.environment.python.user_managed_dependencies = False\n",
"\n",
"# specify CondaDependencies obj\n",
"run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
"\n",
"# Now submit a run on AmlCompute\n",
"from azureml.core.script_run_config import ScriptRunConfig\n",
"\n",
"script_run_config = ScriptRunConfig(source_directory=project_folder,\n",
" script='train.py',\n",
" run_config=run_config)\n",
"\n",
"run = experiment.submit(script_run_config)\n",
"# Set environment\n",
"src.run_config.environment = myenv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Submit run"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"run = experiment.submit(src)\n",
"\n",
"# Show run details\n",
"run"
@@ -292,27 +310,9 @@
"metadata": {},
"outputs": [],
"source": [
"from azureml.core.runconfig import RunConfiguration\n",
"from azureml.core.conda_dependencies import CondaDependencies\n",
"# Set compute target to the one created in previous step\n",
"src.run_config.target = cpu_cluster.name\n",
" \n",
"# create a new RunConfig object\n",
"run_config = RunConfiguration(framework=\"python\")\n",
"\n",
"# Set compute target to AmlCompute target created in previous step\n",
"run_config.target = cpu_cluster.name\n",
"\n",
"# enable Docker \n",
"run_config.environment.docker.enabled = True\n",
"\n",
"# specify CondaDependencies obj\n",
"run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
"\n",
"from azureml.core import Run\n",
"from azureml.core import ScriptRunConfig\n",
"\n",
"src = ScriptRunConfig(source_directory=project_folder, \n",
" script='train.py', \n",
" run_config=run_config) \n",
"run = experiment.submit(config=src)\n",
"run"
]
@@ -397,27 +397,9 @@
"metadata": {},
"outputs": [],
"source": [
"from azureml.core.runconfig import RunConfiguration\n",
"from azureml.core.conda_dependencies import CondaDependencies\n",
"# Set compute target to the one created in previous step\n",
"src.run_config.target = cpu_cluster.name\n",
" \n",
"# create a new RunConfig object\n",
"run_config = RunConfiguration(framework=\"python\")\n",
"\n",
"# Set compute target to AmlCompute target created in previous step\n",
"run_config.target = cpu_cluster.name\n",
"\n",
"# enable Docker \n",
"run_config.environment.docker.enabled = True\n",
"\n",
"# specify CondaDependencies obj\n",
"run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
"\n",
"from azureml.core import Run\n",
"from azureml.core import ScriptRunConfig\n",
"\n",
"src = ScriptRunConfig(source_directory=project_folder, \n",
" script='train.py', \n",
" run_config=run_config) \n",
"run = experiment.submit(config=src)\n",
"run"
]

45
how-to-use-azureml/training/train-on-local/train-on-local.ipynb
View File

@@ -170,15 +170,15 @@
"metadata": {},
"outputs": [],
"source": [
"from azureml.core.runconfig import RunConfiguration\n",
"from azureml.core import Environment\n",
"\n",
"# Editing a run configuration property on-fly.\n",
"run_config_user_managed = RunConfiguration()\n",
"user_managed_env = Environment(\"user-managed-env\")\n",
"\n",
"run_config_user_managed.environment.python.user_managed_dependencies = True\n",
"user_managed_env.python.user_managed_dependencies = True\n",
"\n",
"# You can choose a specific Python environment by pointing to a Python path \n",
"#run_config.environment.python.interpreter_path = '/home/johndoe/miniconda3/envs/myenv/bin/python'"
"#user_managed_env.python.interpreter_path = '/home/johndoe/miniconda3/envs/myenv/bin/python'"
]
},
{
@@ -197,7 +197,16 @@
"source": [
"from azureml.core import ScriptRunConfig\n",
"\n",
"src = ScriptRunConfig(source_directory='./', script='train.py', run_config=run_config_user_managed)\n",
"src = ScriptRunConfig(source_directory='./', script='train.py')\n",
"src.run_config.environment = user_managed_env"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"run = exp.submit(src)"
]
},
@@ -277,13 +286,13 @@
"source": [
"from azureml.core.conda_dependencies import CondaDependencies\n",
"\n",
"run_config_system_managed = RunConfiguration()\n",
"system_managed_env = Environment(\"system-managed-env\")\n",
"\n",
"run_config_system_managed.environment.python.user_managed_dependencies = False\n",
"system_managed_env.python.user_managed_dependencies = False\n",
"\n",
"# Specify conda dependencies with scikit-learn\n",
"cd = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
"run_config_system_managed.environment.python.conda_dependencies = cd"
"system_managed_env.python.conda_dependencies = cd"
]
},
{
@@ -302,7 +311,7 @@
"metadata": {},
"outputs": [],
"source": [
"src = ScriptRunConfig(source_directory=\"./\", script='train.py', run_config=run_config_system_managed)\n",
"src.run_config.environment = system_managed_env\n",
"run = exp.submit(src)"
]
},
@@ -371,18 +380,16 @@
"metadata": {},
"outputs": [],
"source": [
"run_config_docker = RunConfiguration()\n",
"run_config_docker.environment.python.user_managed_dependencies = False\n",
"run_config_docker.environment.docker.enabled = True\n",
"docker_env = Environment(\"docker-env\")\n",
"\n",
"docker_env.python.user_managed_dependencies = False\n",
"docker_env.docker.enabled = True\n",
"\n",
"# use the default CPU-based Docker image from Azure ML\n",
"run_config_docker.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE # Reference Docker image\n",
"print(docker_env.docker.base_image)\n",
"\n",
"# Specify conda dependencies with scikit-learn\n",
"cd = CondaDependencies.create(conda_packages=['scikit-learn'])\n",
"run_config_docker.environment.python.conda_dependencies = cd\n",
"\n",
"src = ScriptRunConfig(source_directory=\"./\", script='train.py', run_config=run_config_docker)"
"docker_env.python.conda_dependencies = cd"
]
},
{
@@ -402,6 +409,8 @@
"source": [
"import subprocess\n",
"\n",
"src.run_config.environment = docker_env\n",
"\n",
"# Check if Docker is installed and Linux containers are enabled\n",
"if subprocess.run(\"docker -v\", shell=True).returncode == 0:\n",
" out = subprocess.check_output(\"docker system info\", shell=True).decode('ascii')\n",
@@ -657,7 +666,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
"version": "3.6.5"
}
},
"nbformat": 4,

121
how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb
View File

@@ -280,20 +280,11 @@
"metadata": {},
"outputs": [],
"source": [
"from azureml.core.runconfig import RunConfiguration\n",
"from azureml.core import Environment\n",
"from azureml.core.conda_dependencies import CondaDependencies\n",
"\n",
"# create a new RunConfig object\n",
"conda_run_config = RunConfiguration(framework=\"python\")\n",
"\n",
"# Set compute target to the Linux DSVM\n",
"conda_run_config.target = attached_dsvm_compute.name\n",
"\n",
"# set the data reference of the run configuration\n",
"conda_run_config.data_references = {ds.name: dr}\n",
"\n",
"# specify CondaDependencies obj\n",
"conda_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
"conda_env = Environment(\"conda-env\")\n",
"conda_env.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
]
},
{
@@ -306,18 +297,14 @@
"\n",
"src = ScriptRunConfig(source_directory=script_folder, \n",
" script='train.py', \n",
" run_config=conda_run_config, \n",
" # pass the datastore reference as a parameter to the training script\n",
" arguments=['--data-folder', str(ds.as_download())] \n",
" ) \n",
"run = exp.submit(config=src)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
"\n",
"src.run_config.framework = \"python\"\n",
"src.run_config.environment = conda_env\n",
"src.run_config.target = attached_dsvm_compute.name\n",
"src.run_config.data_references = {ds.name: dr}"
]
},
{
@@ -326,9 +313,18 @@
"metadata": {},
"outputs": [],
"source": [
"run = exp.submit(config=src)\n",
"\n",
"run.wait_for_completion(show_output=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -359,17 +355,7 @@
"metadata": {},
"outputs": [],
"source": [
"# create a new RunConfig object\n",
"vm_run_config = RunConfiguration(framework=\"python\")\n",
"\n",
"# Set compute target to the Linux DSVM\n",
"vm_run_config.target = attached_dsvm_compute.name\n",
"\n",
"# set the data reference of the run coonfiguration\n",
"conda_run_config.data_references = {ds.name: dr}\n",
"\n",
"# Let system know that you will configure the Python environment yourself.\n",
"vm_run_config.environment.python.user_managed_dependencies = True"
"conda_env.python.user_managed_dependencies = True"
]
},
{
@@ -385,11 +371,6 @@
"metadata": {},
"outputs": [],
"source": [
"src = ScriptRunConfig(source_directory=script_folder, \n",
" script='train.py', \n",
" run_config=vm_run_config,\n",
" # pass the datastore reference as a parameter to the training script\n",
" arguments=['--data-folder', str(ds.as_download())])\n",
"run = exp.submit(config=src)\n",
"run.wait_for_completion(show_output=True)"
]
@@ -398,7 +379,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script `train2.py` that doesn't have azureml dependencies, and submit it instead."
"You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script `train2.py` that doesn't have azureml or data store dependencies, and submit it instead."
]
},
{
@@ -411,7 +392,10 @@
"shutil.copy('./train2.py', os.path.join(script_folder, 'train2.py'))\n",
"\n",
"with open(os.path.join(script_folder, './train2.py'), 'r') as training_script:\n",
" print(training_script.read())"
" print(training_script.read())\n",
" \n",
"src.run_config.data_references = {}\n",
"src.script = \"train2.py\""
]
},
{
@@ -427,10 +411,8 @@
"metadata": {},
"outputs": [],
"source": [
"src = ScriptRunConfig(source_directory=script_folder, \n",
" script='train2.py', \n",
" run_config=vm_run_config)\n",
"run = exp.submit(config=src)\n",
"\n",
"run.wait_for_completion(show_output=True)"
]
},
@@ -464,24 +446,10 @@
"metadata": {},
"outputs": [],
"source": [
"# Load the \"cpu-dsvm.runconfig\" file (created by the above attach operation) in memory\n",
"docker_run_config = RunConfiguration(framework=\"python\")\n",
"conda_env.docker.enabled = True\n",
"conda_env.python.user_managed_dependencies = False\n",
"\n",
"# Set compute target to the Linux DSVM\n",
"docker_run_config.target = attached_dsvm_compute.name\n",
"\n",
"# Use Docker in the remote VM\n",
"docker_run_config.environment.docker.enabled = True\n",
"\n",
"# Use CPU base image from DockerHub\n",
"docker_run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE\n",
"print('Base Docker image is:', docker_run_config.environment.docker.base_image)\n",
"\n",
"# set the data reference of the run coonfiguration\n",
"docker_run_config.data_references = {ds.name: dr}\n",
"\n",
"# specify CondaDependencies obj\n",
"docker_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
"print('Base Docker image is:', conda_env.docker.base_image)"
]
},
{
@@ -498,20 +466,11 @@
"metadata": {},
"outputs": [],
"source": [
"src = ScriptRunConfig(source_directory=script_folder, \n",
" script='train.py', \n",
" run_config=docker_run_config,\n",
" # pass the datastore reference as a parameter to the training script\n",
" arguments=['--data-folder', str(ds.as_download())])\n",
"run = exp.submit(config=src)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"src.script = \"train.py\"\n",
"src.run_config.data_references = {ds.name: dr}\n",
"\n",
"run = exp.submit(config=src)\n",
"\n",
"run.wait_for_completion(show_output=True)"
]
},
@@ -525,20 +484,20 @@
"\n",
"```python\n",
"# use an image available in Docker Hub without authentication\n",
"run_config_docker.environment.docker.base_image = \"continuumio/miniconda3\"\n",
"conda_env.docker.base_image = \"continuumio/miniconda3\"\n",
"\n",
"# or, use an image available in a private Azure Container Registry\n",
"run_config_docker.environment.docker.base_image = \"mycustomimage:1.0\"\n",
"run_config_docker.environment.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
"run_config_docker.environment.docker.base_image_registry.username = \"username\"\n",
"run_config_docker.environment.docker.base_image_registry.password = \"password\"\n",
"conda_env.docker.base_image = \"mycustomimage:1.0\"\n",
"conda_env.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
"conda_env.docker.base_image_registry.username = \"username\"\n",
"conda_env.docker.base_image_registry.password = \"password\"\n",
"```\n",
"\n",
"When you are using a custom Docker image, you might already have your environment setup properly in a Python environment in the Docker image. In that case, you can skip specifying conda dependencies, and just use `user_managed_dependencies` option instead:\n",
"```python\n",
"run_config_docker.environment.python.user_managed_dependencies = True\n",
"conda_env.python.user_managed_dependencies = True\n",
"# path to the Python environment in the custom Docker image\n",
"run_config.environment.python.interpreter_path = '/opt/conda/bin/python'\n",
"conda_env.python.interpreter_path = '/opt/conda/bin/python'\n",
"```"
]
},
@@ -640,7 +599,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
"version": "3.6.5"
}
},
"nbformat": 4,

9
how-to-use-azureml/work-with-data/README.md Normal file
View File

@@ -0,0 +1,9 @@
# Work With Data Using Azure Machine Learning Service
Azure Machine Learning Datasets (preview) make it easier to access and work with your data. Datasets manage data in various scenarios such as model training and pipeline creation. Using the Azure Machine Learning SDK, you can access underlying storage, explore and prepare data, manage the life cycle of different Dataset definitions, and compare between Datasets used in training and in production.
- For an example of using Datasets, see the [sample](datasets).
- For advanced data preparation examples, see [dataprep](dataprep).
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/README..png)

2
how-to-use-azureml/work-with-data/dataprep/README.md
View File

@@ -222,3 +222,5 @@ Bug fixes
IMPORTANT: Please read the notice and find out more about this NYC Taxi and Limousine Commission dataset here: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml
IMPORTANT: Please read the notice and find out more about this Chicago Police Department dataset here: https://catalog.data.gov/dataset/crimes-2001-to-present-398a4
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/dataprep/README.png)

7
how-to-use-azureml/work-with-data/dataprep/case-studies/new-york-taxi/new-york-taxi.ipynb
View File

@@ -477,6 +477,13 @@
"dflow_path = path.join(mkdtemp(), \"new_york_taxi.dprep\")\n",
"combined_df.save(file_path=dflow_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/dataprep/case-studies/new-york-taxi/new-york-taxi.png)"
]
}
],
"metadata": {

7
how-to-use-azureml/work-with-data/dataprep/case-studies/new-york-taxi/new-york-taxi_scale-out.ipynb
View File

@@ -97,6 +97,13 @@
"spark_df = df.take(5).to_pandas_dataframe()\n",
"spark_df.head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/dataprep/case-studies/new-york-taxi/new-york-taxi_scale-out.png)"
]
}
],
"metadata": {

1001
how-to-use-azureml/work-with-data/dataprep/data/crime-full.csv Normal file
View File

File diff suppressed because it is too large Load Diff

7
how-to-use-azureml/work-with-data/dataprep/tutorials/getting-started/getting-started.ipynb
View File

@@ -404,6 +404,13 @@
"* [Sample your data](../../how-to-guides/subsetting-sampling.ipynb)\n",
"* [Reference and link between Dataflows](../../how-to-guides/join.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/dataprep/tutorials/getting-started/getting-started.png)"
]
}
],
"metadata": {

4
how-to-use-azureml/work-with-data/datasets/README.md
View File

@@ -153,3 +153,7 @@ For an end-to-end tutorial, you may refer to [Dataset tutorial](datasets-tutoria
- Take snapshots of data to ensure models can be trained with the same data every time.
- Use registered Dataset in your training script.
- Create and use multiple Dataset definitions to ensure that updates to the definition don't break existing pipelines/scripts.
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/work-with-data/datasets/README.png)

25
how-to-use-azureml/work-with-data/datasets/train-dataset/train.py
View File

@@ -15,22 +15,25 @@ from sklearn.tree import DecisionTreeClassifier
run = Run.get_context()
workspace = run.experiment.workspace
dataset_name = 'training_data'
dataset_name = 'clean_Titanic_tutorial'
snapshot_name = 'train_snapshot'
dataset = Dataset.get(workspace=workspace, name=dataset_name)
dflow = dataset.get_definition()
dflow_val, dflow_train = dflow.random_split(percentage=0.3)
df = dataset.get_snapshot(snapshot_name=snapshot_name).to_pandas_dataframe()
y_df = dflow_train.keep_columns(['HasDetections']).to_pandas_dataframe()
x_df = dflow_train.drop_columns(['HasDetections']).to_pandas_dataframe()
y_val = dflow_val.keep_columns(['HasDetections']).to_pandas_dataframe()
x_val = dflow_val.drop_columns(['HasDetections']).to_pandas_dataframe()
x_col = ['Pclass', 'Sex', 'SibSp', 'Parch']
y_col = ['Survived']
x_df = df.loc[:, x_col]
y_df = df.loc[:, y_col]
data = {"train": {"X": x_df, "y": y_df},
x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=223)
"validation": {"X": x_val, "y": y_val}}
data = {"train": {"X": x_train, "y": y_train},
"test": {"X": x_test, "y": y_test}}
clf = DecisionTreeClassifier().fit(data["train"]["X"], data["train"]["y"])
print('Accuracy of Decision Tree classifier on training set: {:.2f}'.format(clf.score(x_df, y_df)))
print('Accuracy of Decision Tree classifier on validation set: {:.2f}'.format(clf.score(x_val, y_val)))
print('Accuracy of Decision Tree classifier on training set: {:.2f}'.format(clf.score(x_train, y_train)))
print('Accuracy of Decision Tree classifier on test set: {:.2f}'.format(clf.score(x_test, y_test)))

2
tutorials/README.md
View File

@@ -18,3 +18,5 @@ If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwi
* [Part 2](regression-part2-automated-ml.ipynb): Train a model using Automated Machine Learning.
Also find quickstarts and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).
![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/README.png)

7
tutorials/img-classification-part1-training.ipynb
View File

@@ -626,6 +626,13 @@
"\n",
"> [Tutorial 2 - Deploy models](img-classification-part2-deploy.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/img-classification-part1-training.png)"
]
}
],
"metadata": {

7
tutorials/img-classification-part2-deploy.ipynb
View File

@@ -587,6 +587,13 @@
" \n",
"You can also try out the [regression tutorial](regression-part1-data-prep.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/img-classification-part2-deploy.png)"
]
}
],
"metadata": {

7
tutorials/regression-part1-data-prep.ipynb
View File

@@ -599,6 +599,13 @@
"\n",
"> [Tutorial #2: Train regression model](regression-part2-automated-ml.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/regression-part1-data-prep.png)"
]
}
],
"metadata": {

11
tutorials/regression-part2-automated-ml.ipynb
View File

@@ -25,7 +25,7 @@
"> * Run the model locally with custom parameters\n",
"> * Explore the results\n",
"\n",
"If you don\u00e2\u20ac\u2122t have an Azure subscription, create a [free account](https://aka.ms/AMLfree) before you begin. \n",
"If you do not have an Azure subscription, create a [free account](https://aka.ms/AMLfree) before you begin. \n",
"\n",
"> Code in this article was tested with Azure Machine Learning SDK version 1.0.0\n",
"\n",
@@ -485,7 +485,7 @@
">The resources you created can be used as prerequisites to other Azure Machine Learning service tutorials and how-to articles. \n",
"\n",
"\n",
"If you don't plan to use the resources you created, delete them, so you don't incur any charges:\n",
"If you do not plan to use the resources you created, delete them, so you do not incur any charges:\n",
"\n",
"1. In the Azure portal, select **Resource groups** on the far left.\n",
"\n",
@@ -510,6 +510,13 @@
"\n",
"[Deploy your model](https://docs.microsoft.com/azure/machine-learning/service/tutorial-deploy-models-with-aml) with Azure Machine Learning."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/regression-part2-automated-ml.png)"
]
}
],
"metadata": {