updating spark notebook

update samples from Release-106 as a part of  SDK release

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns

NBSETUP.md

@@ -28,7 +28,7 @@ git clone https://github.com/Azure/MachineLearningNotebooks.git
 pip install azureml-sdk[notebooks,tensorboard]
 
 # install model explainability component
-pip install azureml-sdk[explain]
+pip install azureml-sdk[interpret]
 
 # install automated ml components
 pip install azureml-sdk[automl]
@@ -86,7 +86,7 @@ If you need additional Azure ML SDK components, you can either modify the Docker
 pip install azureml-sdk[automl]
 
 # install the core SDK and model explainability component
-pip install azureml-sdk[explain]
+pip install azureml-sdk[interpret]
 
 # install the core SDK and experimental components
 pip install azureml-sdk[contrib]

README.md

@@ -1,77 +1,43 @@
-# Azure Machine Learning service example notebooks
+# Azure Machine Learning Python SDK notebooks
 
-This repository contains example notebooks demonstrating the [Azure Machine Learning](https://azure.microsoft.com/en-us/services/machine-learning-service/) Python SDK which allows you to build, train, deploy and manage machine learning solutions using Azure.  The AML SDK allows you the choice of using local or cloud compute resources, while managing and maintaining the complete data science workflow from the cloud.
+> a community-driven repository of examples using mlflow for tracking can be found at https://github.com/Azure/azureml-examples
 
-![Azure ML Workflow](https://raw.githubusercontent.com/MicrosoftDocs/azure-docs/master/articles/machine-learning/media/concept-azure-machine-learning-architecture/workflow.png)
+Welcome to the Azure Machine Learning Python SDK notebooks repository!
 
+## Getting started
 
-## Quick installation
-```sh
-pip install azureml-sdk
-```
-Read more detailed instructions on [how to set up your environment](./NBSETUP.md) using Azure Notebook service, your own Jupyter notebook server, or Docker.
+These notebooks are recommended for use in an Azure Machine Learning [Compute Instance](https://docs.microsoft.com/azure/machine-learning/concept-compute-instance), where you can run them without any additional set up.
 
-## How to navigate and use the example notebooks?
-If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, you should always run the [Configuration](./configuration.ipynb) notebook first when setting up a notebook library on a new machine or in a new environment. It configures your notebook library to connect to an Azure Machine Learning workspace, and sets up your workspace and compute to be used by many of the other examples. 
-This [index](./index.md) should assist in navigating the Azure Machine Learning notebook samples and encourage efficient retrieval of topics and content. 
+However, the notebooks can be run in any development environment with the correct `azureml` packages installed.
 
-If you want to...
-
- * ...try out and explore Azure ML, start with image classification tutorials: [Part 1 (Training)](./tutorials/image-classification-mnist-data/img-classification-part1-training.ipynb) and [Part 2 (Deployment)](./tutorials/image-classification-mnist-data/img-classification-part2-deploy.ipynb).
- * ...learn about experimentation and tracking run history, first [train within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then try [training on remote VM](./how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb) and [using logging APIs](./how-to-use-azureml/training/logging-api/logging-api.ipynb).
- * ...train deep learning models at scale, first learn about [Machine Learning Compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and then try [distributed hyperparameter tuning](./how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) and [distributed training](./how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb).
- * ...deploy models as a realtime scoring service, first learn the basics by [training within Notebook and deploying to Azure Container Instance](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then learn how to [production deploy models on Azure Kubernetes Cluster](./how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb).
- * ...deploy models as a batch scoring service, first [train a model within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and [use Machine Learning Pipelines to deploy your model](https://aka.ms/pl-batch-scoring).
- * ...monitor your deployed models, learn about using [App Insights](./how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb).
-
-## Tutorials
-
-The [Tutorials](./tutorials) folder contains notebooks for the tutorials described in the [Azure Machine Learning documentation](https://aka.ms/aml-docs).
-  
-## How to use Azure ML
-
-The [How to use Azure ML](./how-to-use-azureml) folder contains specific examples demonstrating the features of the Azure Machine Learning SDK
-
-- [Training](./how-to-use-azureml/training) - Examples of how to build models using Azure ML's logging and execution capabilities on local and remote compute targets
-- [Training with Deep Learning](./how-to-use-azureml/training-with-deep-learning) - Examples demonstrating how to build deep learning models using estimators and parameter sweeps
-- [Manage Azure ML Service](./how-to-use-azureml/manage-azureml-service) - Examples how to perform tasks, such as authenticate against Azure ML service in different ways.
-- [Automated Machine Learning](./how-to-use-azureml/automated-machine-learning) - Examples using Automated Machine Learning to automatically generate optimal machine learning pipelines and models
-- [Machine Learning Pipelines](./how-to-use-azureml/machine-learning-pipelines) - Examples showing how to create and use reusable pipelines for training and batch scoring
-- [Deployment](./how-to-use-azureml/deployment) - Examples showing how to deploy and manage machine learning models and solutions
-- [Azure Databricks](./how-to-use-azureml/azure-databricks) - Examples showing how to use Azure ML with Azure Databricks
-- [Monitor Models](./how-to-use-azureml/monitor-models) - Examples showing how to enable model monitoring services such as DataDrift
-- [Reinforcement Learning](./how-to-use-azureml/reinforcement-learning) - Examples showing how to train reinforcement learning agents
-
----
-## Documentation
-
- * Quickstarts, end-to-end tutorials, and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).
- * [Python SDK reference](https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py)
- * Azure ML Data Prep SDK [overview](https://aka.ms/data-prep-sdk), [Python SDK reference](https://aka.ms/aml-data-prep-apiref), and [tutorials and how-tos](https://aka.ms/aml-data-prep-notebooks).
-
----
-
-
-## Community Repository 
-Visit this [community repository](https://github.com/microsoft/MLOps/tree/master/examples) to find useful end-to-end sample notebooks. Also, please follow these [contribution guidelines](https://github.com/microsoft/MLOps/blob/master/contributing.md) when contributing to this repository.   
-
-## Projects using Azure Machine Learning
-
-Visit following repos to see projects contributed by Azure ML users:
- - [AMLSamples](https://github.com/Azure/AMLSamples) Number of end-to-end examples, including face recognition, predictive maintenance, customer churn and sentiment analysis.
- - [Learn about Natural Language Processing best practices using Azure Machine Learning service](https://github.com/microsoft/nlp)
- - [Pre-Train BERT models using Azure Machine Learning service](https://github.com/Microsoft/AzureML-BERT)
- - [Fashion MNIST with Azure ML SDK](https://github.com/amynic/azureml-sdk-fashion)
- - [UMass Amherst Student Samples](https://github.com/katiehouse3/microsoft-azure-ml-notebooks) - A number of end-to-end machine learning notebooks, including machine translation, image classification, and customer churn, created by students in the 696DS course at UMass Amherst.
- 
-## Data/Telemetry 
-This repository collects usage data and sends it to Mircosoft to help improve our products and services. Read Microsoft's [privacy statement to learn more](https://privacy.microsoft.com/en-US/privacystatement)
-
-To opt out of tracking, please go to the raw markdown or .ipynb files and remove the following line of code:
+Install the `azureml.core` Python package:
 
 ```sh
-    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/README.png)"
+pip install azureml-core
 ```
-This URL will be slightly different depending on the file. 
 
- ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/README.png)
+Install additional packages as needed:
+
+```sh
+pip install azureml-mlflow
+pip install azureml-dataset-runtime
+pip install azureml-automl-runtime
+pip install azureml-pipeline
+pip install azureml-pipeline-steps
+...
+```
+
+We recommend starting with one of the [quickstarts](tutorials/compute-instance-quickstarts).
+
+## Contributing
+
+This repository is a push-only mirror. Pull requests are ignored.
+
+## Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Please see the [code of conduct](CODE_OF_CONDUCT.md) for details.
+
+## Reference
+
+- [Documentation](https://docs.microsoft.com/azure/machine-learning)
+

configuration.ipynb

@@ -103,7 +103,7 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -254,6 +254,8 @@
         "\n",
         "Many of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n",
         "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "To create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors.  Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n",
         "\n",
         "The cluster parameters are:\n",

contrib/fairness/fairlearn-azureml-mitigation.ipynb

@@ -36,9 +36,9 @@
         "\n",
         "<a id=\"Introduction\"></a>\n",
         "## Introduction\n",
-        "This notebook shows how to use [Fairlearn (an open source fairness assessment and unfairness mitigation package)](http://fairlearn.github.io) and Azure Machine Learning Studio for a binary classification problem. This example uses the well-known adult census dataset. For the purposes of this notebook, we shall treat this as a loan decision problem. We will pretend that the label indicates whether or not each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan. Its purpose is purely illustrative of a workflow including a fairness dashboard - in particular, we do **not** include a full discussion of the detailed issues which arise when considering fairness in machine learning. For such discussions, please [refer to the Fairlearn website](http://fairlearn.github.io/).\n",
+        "This notebook shows how to use [Fairlearn (an open source fairness assessment and unfairness mitigation package)](http://fairlearn.org) and Azure Machine Learning Studio for a binary classification problem. This example uses the well-known adult census dataset. For the purposes of this notebook, we shall treat this as a loan decision problem. We will pretend that the label indicates whether or not each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan. Its purpose is purely illustrative of a workflow including a fairness dashboard - in particular, we do **not** include a full discussion of the detailed issues which arise when considering fairness in machine learning. For such discussions, please [refer to the Fairlearn website](http://fairlearn.org/).\n",
         "\n",
-        "We will apply the [grid search algorithm](https://fairlearn.github.io/api_reference/fairlearn.reductions.html#fairlearn.reductions.GridSearch) from the Fairlearn package using a specific notion of fairness called Demographic Parity. This produces a set of models, and we will view these in a dashboard both locally and in the Azure Machine Learning Studio.\n",
+        "We will apply the [grid search algorithm](https://fairlearn.org/v0.4.6/api_reference/fairlearn.reductions.html#fairlearn.reductions.GridSearch) from the Fairlearn package using a specific notion of fairness called Demographic Parity. This produces a set of models, and we will view these in a dashboard both locally and in the Azure Machine Learning Studio.\n",
         "\n",
         "### Setup\n",
         "\n",
@@ -46,14 +46,37 @@
         "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
         "This notebook also requires the following packages:\n",
         "* `azureml-contrib-fairness`\n",
-        "* `fairlearn==0.4.6`\n",
+        "* `fairlearn>=0.6.2` (pre-v0.5.0 will work with minor modifications)\n",
         "* `joblib`\n",
-        "* `shap`\n",
-        "\n",
+        "* `liac-arff`\n",
+        "* `raiwidgets~=0.7.0`\n",
         "\n",
+        "Fairlearn relies on features introduced in v0.22.1 of `scikit-learn`. If you have an older version already installed, please uncomment and run the following cell:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# !pip install --upgrade scikit-learn>=0.22.1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, please ensure that when you downloaded this notebook, you also downloaded the `fairness_nb_utils.py` file from the same location, and placed it in the same directory as this notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
         "<a id=\"LoadingData\"></a>\n",
         "## Loading the Data\n",
-        "We use the well-known `adult` census dataset, which we load using `shap` (for convenience). We start with a fairly unremarkable set of imports:"
+        "We use the well-known `adult` census dataset, which we will fetch from the OpenML website. We start with a fairly unremarkable set of imports:"
       ]
     },
     {
@@ -63,90 +86,141 @@
       "outputs": [],
       "source": [
         "from fairlearn.reductions import GridSearch, DemographicParity, ErrorRate\n",
-        "from fairlearn.widget import FairlearnDashboard\n",
-        "from sklearn import svm\n",
-        "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
+        "from raiwidgets import FairnessDashboard\n",
+        "\n",
+        "from sklearn.compose import ColumnTransformer\n",
+        "from sklearn.impute import SimpleImputer\n",
         "from sklearn.linear_model import LogisticRegression\n",
-        "import pandas as pd\n",
-        "import shap"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "We can now load and inspect the data from the `shap` package:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "X_raw, Y = shap.datasets.adult()\n",
-        "X_raw[\"Race\"].value_counts().to_dict()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "We are going to treat the sex of each individual as a protected attribute (where 0 indicates female and 1 indicates male), and in this particular case we are going separate this attribute out and drop it from the main data (this is not always the best option - see the [Fairlearn website](http://fairlearn.github.io/) for further discussion). We also separate out the Race column, but we will not perform any mitigation based on it. Finally, we perform some standard data preprocessing steps to convert the data into a format suitable for the ML algorithms"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "A = X_raw[['Sex','Race']]\n",
-        "X = X_raw.drop(labels=['Sex', 'Race'],axis = 1)\n",
-        "X = pd.get_dummies(X)\n",
-        "\n",
-        "\n",
-        "le = LabelEncoder()\n",
-        "Y = le.fit_transform(Y)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "With our data prepared, we can make the conventional split in to 'test' and 'train' subsets:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
         "from sklearn.model_selection import train_test_split\n",
-        "X_train, X_test, Y_train, Y_test, A_train, A_test = train_test_split(X_raw, \n",
-        "                                                    Y, \n",
-        "                                                    A,\n",
-        "                                                    test_size = 0.2,\n",
-        "                                                    random_state=0,\n",
-        "                                                    stratify=Y)\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn.compose import make_column_selector as selector\n",
+        "from sklearn.pipeline import Pipeline\n",
+        "\n",
+        "import pandas as pd"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can now load and inspect the data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairness_nb_utils import fetch_census_dataset\n",
+        "\n",
+        "data = fetch_census_dataset()\n",
+        "    \n",
+        "# Extract the items we want\n",
+        "X_raw = data.data\n",
+        "y = (data.target == '>50K') * 1\n",
+        "\n",
+        "X_raw[\"race\"].value_counts().to_dict()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We are going to treat the sex and race of each individual as protected attributes, and in this particular case we are going to remove these attributes from the main data (this is not always the best option - see the [Fairlearn website](http://fairlearn.github.io/) for further discussion). Protected attributes are often denoted by 'A' in the literature, and we follow that convention here:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "A = X_raw[['sex','race']]\n",
+        "X_raw = X_raw.drop(labels=['sex', 'race'], axis = 1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We now preprocess our data. To avoid the problem of data leakage, we split our data into training and test sets before performing any other transformations. Subsequent transformations (such as scalings) will be fit to the training data set, and then applied to the test dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "(X_train, X_test, y_train, y_test, A_train, A_test) = train_test_split(\n",
+        "    X_raw, y, A, test_size=0.3, random_state=12345, stratify=y\n",
+        ")\n",
+        "\n",
+        "# Ensure indices are aligned between X, y and A,\n",
+        "# after all the slicing and splitting of DataFrames\n",
+        "# and Series\n",
         "\n",
-        "# Work around indexing issue\n",
         "X_train = X_train.reset_index(drop=True)\n",
-        "A_train = A_train.reset_index(drop=True)\n",
         "X_test = X_test.reset_index(drop=True)\n",
-        "A_test = A_test.reset_index(drop=True)\n",
+        "y_train = y_train.reset_index(drop=True)\n",
+        "y_test = y_test.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We have two types of column in the dataset - categorical columns which will need to be one-hot encoded, and numeric ones which will need to be rescaled. We also need to take care of missing values. We use a simple approach here, but please bear in mind that this is another way that bias could be introduced (especially if one subgroup tends to have more missing values).\n",
         "\n",
-        "# Improve labels\n",
-        "A_test.Sex.loc[(A_test['Sex'] == 0)] = 'female'\n",
-        "A_test.Sex.loc[(A_test['Sex'] == 1)] = 'male'\n",
+        "For this preprocessing, we make use of `Pipeline` objects from `sklearn`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "numeric_transformer = Pipeline(\n",
+        "    steps=[\n",
+        "        (\"impute\", SimpleImputer()),\n",
+        "        (\"scaler\", StandardScaler()),\n",
+        "    ]\n",
+        ")\n",
         "\n",
+        "categorical_transformer = Pipeline(\n",
+        "    [\n",
+        "        (\"impute\", SimpleImputer(strategy=\"most_frequent\")),\n",
+        "        (\"ohe\", OneHotEncoder(handle_unknown=\"ignore\", sparse=False)),\n",
+        "    ]\n",
+        ")\n",
         "\n",
-        "A_test.Race.loc[(A_test['Race'] == 0)] = 'Amer-Indian-Eskimo'\n",
-        "A_test.Race.loc[(A_test['Race'] == 1)] = 'Asian-Pac-Islander'\n",
-        "A_test.Race.loc[(A_test['Race'] == 2)] = 'Black'\n",
-        "A_test.Race.loc[(A_test['Race'] == 3)] = 'Other'\n",
-        "A_test.Race.loc[(A_test['Race'] == 4)] = 'White'"
+        "preprocessor = ColumnTransformer(\n",
+        "    transformers=[\n",
+        "        (\"num\", numeric_transformer, selector(dtype_exclude=\"category\")),\n",
+        "        (\"cat\", categorical_transformer, selector(dtype_include=\"category\")),\n",
+        "    ]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, the preprocessing pipeline is defined, we can run it on our training data, and apply the generated transform to our test data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_train = preprocessor.fit_transform(X_train)\n",
+        "X_test = preprocessor.transform(X_test)"
       ]
     },
     {
@@ -167,7 +241,7 @@
       "source": [
         "unmitigated_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
         "\n",
-        "unmitigated_predictor.fit(X_train, Y_train)"
+        "unmitigated_predictor.fit(X_train, y_train)"
       ]
     },
     {
@@ -183,8 +257,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "FairlearnDashboard(sensitive_features=A_test, sensitive_feature_names=['Sex', 'Race'],\n",
-        "                   y_true=Y_test,\n",
+        "FairnessDashboard(sensitive_features=A_test,\n",
+        "                  y_true=y_test,\n",
         "                  y_pred={\"unmitigated\": unmitigated_predictor.predict(X_test)})"
       ]
     },
@@ -235,10 +309,11 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "sweep.fit(X_train, Y_train,\n",
-        "          sensitive_features=A_train.Sex)\n",
+        "sweep.fit(X_train, y_train,\n",
+        "          sensitive_features=A_train.sex)\n",
         "\n",
-        "predictors = sweep._predictors"
+        "# For Fairlearn pre-v0.5.0, need sweep._predictors\n",
+        "predictors = sweep.predictors_"
       ]
     },
     {
@@ -255,16 +330,14 @@
       "outputs": [],
       "source": [
         "errors, disparities = [], []\n",
-        "for m in predictors:\n",
-        "    classifier = lambda X: m.predict(X)\n",
-        "    \n",
+        "for predictor in predictors:\n",
         "    error = ErrorRate()\n",
-        "    error.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
+        "    error.load_data(X_train, pd.Series(y_train), sensitive_features=A_train.sex)\n",
         "    disparity = DemographicParity()\n",
-        "    disparity.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
+        "    disparity.load_data(X_train, pd.Series(y_train), sensitive_features=A_train.sex)\n",
         "    \n",
-        "    errors.append(error.gamma(classifier)[0])\n",
-        "    disparities.append(disparity.gamma(classifier).max())\n",
+        "    errors.append(error.gamma(predictor.predict)[0])\n",
+        "    disparities.append(disparity.gamma(predictor.predict).max())\n",
         "    \n",
         "all_results = pd.DataFrame( {\"predictor\": predictors, \"error\": errors, \"disparity\": disparities})\n",
         "\n",
@@ -313,9 +386,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "FairlearnDashboard(sensitive_features=A_test, \n",
-        "                   sensitive_feature_names=['Sex', 'Race'],\n",
-        "                   y_true=Y_test.tolist(),\n",
+        "FairnessDashboard(sensitive_features=A_test, \n",
+        "                  y_true=y_test.tolist(),\n",
         "                  y_pred=predictions_dominant)"
       ]
     },
@@ -323,7 +395,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "When using sex as the sensitive feature, we see a Pareto front forming - the set of predictors which represent optimal tradeoffs between accuracy and disparity in predictions. In the ideal case, we would have a predictor at (1,0) - perfectly accurate and without any unfairness under demographic parity (with respect to the protected attribute \"sex\"). The Pareto front represents the closest we can come to this ideal based on our data and choice of estimator. Note the range of the axes - the disparity axis covers more values than the accuracy, so we can reduce disparity substantially for a small loss in accuracy. Finally, we also see that the unmitigated model is towards the top right of the plot, with high accuracy, but worst disparity.\n",
+        "When using sex as the sensitive feature and accuracy as the metric, we see a Pareto front forming - the set of predictors which represent optimal tradeoffs between accuracy and disparity in predictions. In the ideal case, we would have a predictor at (1,0) - perfectly accurate and without any unfairness under demographic parity (with respect to the protected attribute \"sex\"). The Pareto front represents the closest we can come to this ideal based on our data and choice of estimator. Note the range of the axes - the disparity axis covers more values than the accuracy, so we can reduce disparity substantially for a small loss in accuracy. Finally, we also see that the unmitigated model is towards the top right of the plot, with high accuracy, but worst disparity.\n",
         "\n",
         "By clicking on individual models on the plot, we can inspect their metrics for disparity and accuracy in greater detail. In a real example, we would then pick the model which represented the best trade-off between accuracy and disparity given the relevant business constraints."
       ]
@@ -335,7 +407,7 @@
         "<a id=\"AzureUpload\"></a>\n",
         "## Uploading a Fairness Dashboard to Azure\n",
         "\n",
-        "Uploading a fairness dashboard to Azure is a two stage process. The `FairlearnDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. By default, the dashboard in Azure Machine Learning Studio also requires the models to be registered. The required stages are therefore:\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairnessDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. By default, the dashboard in Azure Machine Learning Studio also requires the models to be registered. The required stages are therefore:\n",
         "1. Register the dominant models\n",
         "1. Precompute all the required metrics\n",
         "1. Upload to Azure\n",
@@ -425,12 +497,12 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "sf = { 'sex': A_test.Sex, 'race': A_test.Race }\n",
+        "sf = { 'sex': A_test.sex, 'race': A_test.race }\n",
         "\n",
         "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
         "\n",
         "\n",
-        "dash_dict = _create_group_metric_set(y_true=Y_test,\n",
+        "dash_dict = _create_group_metric_set(y_true=y_test,\n",
         "                                     predictions=predictions_dominant_ids,\n",
         "                                     sensitive_features=sf,\n",
         "                                     prediction_type='binary_classification')"
@@ -509,7 +581,7 @@
         "<a id=\"Conclusion\"></a>\n",
         "## Conclusion\n",
         "\n",
-        "In this notebook we have demonstrated how to use the `GridSearch` algorithm from Fairlearn to generate a collection of models, and then present them in the fairness dashboard in Azure Machine Learning Studio. Please remember that this notebook has not attempted to discuss the many considerations which should be part of any approach to unfairness mitigation. The [Fairlearn website](http://fairlearn.github.io/) provides that discussion"
+        "In this notebook we have demonstrated how to use the `GridSearch` algorithm from Fairlearn to generate a collection of models, and then present them in the fairness dashboard in Azure Machine Learning Studio. Please remember that this notebook has not attempted to discuss the many considerations which should be part of any approach to unfairness mitigation. The [Fairlearn website](http://fairlearn.org/) provides that discussion"
       ]
     },
     {

contrib/fairness/fairlearn-azureml-mitigation.yml

@@ -3,6 +3,7 @@ dependencies:
 - pip:
   - azureml-sdk
   - azureml-contrib-fairness
-  - fairlearn==0.4.6
+  - fairlearn>=0.6.2
   - joblib
-  - shap
+  - liac-arff
+  - raiwidgets~=0.7.0

contrib/fairness/fairness_nb_utils.py

@@ -0,0 +1,111 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+"""Utilities for azureml-contrib-fairness notebooks."""
+
+import arff
+from collections import OrderedDict
+from contextlib import closing
+import gzip
+import pandas as pd
+from sklearn.datasets import fetch_openml
+from sklearn.utils import Bunch
+import time
+
+
+def fetch_openml_with_retries(data_id, max_retries=4, retry_delay=60):
+    """Fetch a given dataset from OpenML with retries as specified."""
+    for i in range(max_retries):
+        try:
+            print("Download attempt {0} of {1}".format(i + 1, max_retries))
+            data = fetch_openml(data_id=data_id, as_frame=True)
+            break
+        except Exception as e:  # noqa: B902
+            print("Download attempt failed with exception:")
+            print(e)
+            if i + 1 != max_retries:
+                print("Will retry after {0} seconds".format(retry_delay))
+                time.sleep(retry_delay)
+                retry_delay = retry_delay * 2
+    else:
+        raise RuntimeError("Unable to download dataset from OpenML")
+
+    return data
+
+
+_categorical_columns = [
+    'workclass',
+    'education',
+    'marital-status',
+    'occupation',
+    'relationship',
+    'race',
+    'sex',
+    'native-country'
+]
+
+
+def fetch_census_dataset():
+    """Fetch the Adult Census Dataset.
+
+    This uses a particular URL for the Adult Census dataset. The code
+    is a simplified version of fetch_openml() in sklearn.
+
+    The data are copied from:
+    https://openml.org/data/v1/download/1595261.gz
+    (as of 2021-03-31)
+    """
+    try:
+        from urllib import urlretrieve
+    except ImportError:
+        from urllib.request import urlretrieve
+
+    filename = "1595261.gz"
+    data_url = "https://rainotebookscdn.blob.core.windows.net/datasets/"
+
+    remaining_attempts = 5
+    sleep_duration = 10
+    while remaining_attempts > 0:
+        try:
+            urlretrieve(data_url + filename, filename)
+
+            http_stream = gzip.GzipFile(filename=filename, mode='rb')
+
+            with closing(http_stream):
+                def _stream_generator(response):
+                    for line in response:
+                        yield line.decode('utf-8')
+
+                stream = _stream_generator(http_stream)
+                data = arff.load(stream)
+        except Exception as exc:  # noqa: B902
+            remaining_attempts -= 1
+            print("Error downloading dataset from {} ({} attempt(s) remaining)"
+                  .format(data_url, remaining_attempts))
+            print(exc)
+            time.sleep(sleep_duration)
+            sleep_duration *= 2
+            continue
+        else:
+            # dataset successfully downloaded
+            break
+    else:
+        raise Exception("Could not retrieve dataset from {}.".format(data_url))
+
+    attributes = OrderedDict(data['attributes'])
+    arff_columns = list(attributes)
+
+    raw_df = pd.DataFrame(data=data['data'], columns=arff_columns)
+
+    target_column_name = 'class'
+    target = raw_df.pop(target_column_name)
+    for col_name in _categorical_columns:
+        dtype = pd.api.types.CategoricalDtype(attributes[col_name])
+        raw_df[col_name] = raw_df[col_name].astype(dtype, copy=False)
+
+    result = Bunch()
+    result.data = raw_df
+    result.target = target
+
+    return result

contrib/fairness/upload-fairness-dashboard.ipynb

@@ -30,7 +30,7 @@
         "1. [Training Models](#TrainingModels)\n",
         "1. [Logging in to AzureML](#LoginAzureML)\n",
         "1. [Registering the Models](#RegisterModels)\n",
-        "1. [Using the Fairlearn Dashboard](#LocalDashboard)\n",
+        "1. [Using the Fairness Dashboard](#LocalDashboard)\n",
         "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
         "    1. Computing Fairness Metrics\n",
         "    1. Uploading to Azure\n",
@@ -48,16 +48,37 @@
         "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
         "This notebook also requires the following packages:\n",
         "* `azureml-contrib-fairness`\n",
-        "* `fairlearn==0.4.6`\n",
+        "* `fairlearn>=0.6.2` (also works for pre-v0.5.0 with slight modifications)\n",
         "* `joblib`\n",
-        "* `shap`\n",
-        "\n",
-        "\n",
-        "\n",
+        "* `liac-arff`\n",
+        "* `raiwidgets~=0.7.0`\n",
         "\n",
+        "Fairlearn relies on features introduced in v0.22.1 of `scikit-learn`. If you have an older version already installed, please uncomment and run the following cell:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# !pip install --upgrade scikit-learn>=0.22.1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, please ensure that when you downloaded this notebook, you also downloaded the `fairness_nb_utils.py` file from the same location, and placed it in the same directory as this notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
         "<a id=\"LoadingData\"></a>\n",
         "## Loading the Data\n",
-        "We use the well-known `adult` census dataset, which we load using `shap` (for convenience). We start with a fairly unremarkable set of imports:"
+        "We use the well-known `adult` census dataset, which we fetch from the OpenML website. We start with a fairly unremarkable set of imports:"
       ]
     },
     {
@@ -67,10 +88,13 @@
       "outputs": [],
       "source": [
         "from sklearn import svm\n",
-        "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
+        "from sklearn.compose import ColumnTransformer\n",
+        "from sklearn.impute import SimpleImputer\n",
         "from sklearn.linear_model import LogisticRegression\n",
-        "import pandas as pd\n",
-        "import shap"
+        "from sklearn.model_selection import train_test_split\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn.compose import make_column_selector as selector\n",
+        "from sklearn.pipeline import Pipeline"
       ]
     },
     {
@@ -86,7 +110,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X_raw, Y = shap.datasets.adult()"
+        "from fairness_nb_utils import fetch_census_dataset\n",
+        "\n",
+        "data = fetch_census_dataset()\n",
+        "    \n",
+        "# Extract the items we want\n",
+        "X_raw = data.data\n",
+        "y = (data.target == '>50K') * 1"
       ]
     },
     {
@@ -102,7 +132,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(X_raw[\"Race\"].value_counts().to_dict())"
+        "print(X_raw[\"race\"].value_counts().to_dict())"
       ]
     },
     {
@@ -112,7 +142,7 @@
         "<a id=\"ProcessingData\"></a>\n",
         "## Processing the Data\n",
         "\n",
-        "With the data loaded, we process it for our needs. First, we extract the sensitive features of interest into `A` (conventionally used in the literature) and put the rest of the feature data into `X`:"
+        "With the data loaded, we process it for our needs. First, we extract the sensitive features of interest into `A` (conventionally used in the literature) and leave the rest of the feature data in `X_raw`:"
       ]
     },
     {
@@ -121,16 +151,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "A = X_raw[['Sex','Race']]\n",
-        "X = X_raw.drop(labels=['Sex', 'Race'],axis = 1)\n",
-        "X = pd.get_dummies(X)"
+        "A = X_raw[['sex','race']]\n",
+        "X_raw = X_raw.drop(labels=['sex', 'race'],axis = 1)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Next, we apply a standard set of scalings:"
+        "We now preprocess our data. To avoid the problem of data leakage, we split our data into training and test sets before performing any other transformations. Subsequent transformations (such as scalings) will be fit to the training data set, and then applied to the test dataset."
       ]
     },
     {
@@ -139,51 +168,74 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "sc = StandardScaler()\n",
-        "X_scaled = sc.fit_transform(X)\n",
-        "X_scaled = pd.DataFrame(X_scaled, columns=X.columns)\n",
+        "(X_train, X_test, y_train, y_test, A_train, A_test) = train_test_split(\n",
+        "    X_raw, y, A, test_size=0.3, random_state=12345, stratify=y\n",
+        ")\n",
         "\n",
-        "le = LabelEncoder()\n",
-        "Y = le.fit_transform(Y)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Finally, we can then split our data into training and test sets, and also make the labels on our test portion of `A` human-readable:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from sklearn.model_selection import train_test_split\n",
-        "X_train, X_test, Y_train, Y_test, A_train, A_test = train_test_split(X_scaled, \n",
-        "                                                    Y, \n",
-        "                                                    A,\n",
-        "                                                    test_size = 0.2,\n",
-        "                                                    random_state=0,\n",
-        "                                                    stratify=Y)\n",
+        "# Ensure indices are aligned between X, y and A,\n",
+        "# after all the slicing and splitting of DataFrames\n",
+        "# and Series\n",
         "\n",
-        "# Work around indexing issue\n",
         "X_train = X_train.reset_index(drop=True)\n",
-        "A_train = A_train.reset_index(drop=True)\n",
         "X_test = X_test.reset_index(drop=True)\n",
-        "A_test = A_test.reset_index(drop=True)\n",
+        "y_train = y_train.reset_index(drop=True)\n",
+        "y_test = y_test.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We have two types of column in the dataset - categorical columns which will need to be one-hot encoded, and numeric ones which will need to be rescaled. We also need to take care of missing values. We use a simple approach here, but please bear in mind that this is another way that bias could be introduced (especially if one subgroup tends to have more missing values).\n",
         "\n",
-        "# Improve labels\n",
-        "A_test.Sex.loc[(A_test['Sex'] == 0)] = 'female'\n",
-        "A_test.Sex.loc[(A_test['Sex'] == 1)] = 'male'\n",
+        "For this preprocessing, we make use of `Pipeline` objects from `sklearn`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "numeric_transformer = Pipeline(\n",
+        "    steps=[\n",
+        "        (\"impute\", SimpleImputer()),\n",
+        "        (\"scaler\", StandardScaler()),\n",
+        "    ]\n",
+        ")\n",
         "\n",
+        "categorical_transformer = Pipeline(\n",
+        "    [\n",
+        "        (\"impute\", SimpleImputer(strategy=\"most_frequent\")),\n",
+        "        (\"ohe\", OneHotEncoder(handle_unknown=\"ignore\", sparse=False)),\n",
+        "    ]\n",
+        ")\n",
         "\n",
-        "A_test.Race.loc[(A_test['Race'] == 0)] = 'Amer-Indian-Eskimo'\n",
-        "A_test.Race.loc[(A_test['Race'] == 1)] = 'Asian-Pac-Islander'\n",
-        "A_test.Race.loc[(A_test['Race'] == 2)] = 'Black'\n",
-        "A_test.Race.loc[(A_test['Race'] == 3)] = 'Other'\n",
-        "A_test.Race.loc[(A_test['Race'] == 4)] = 'White'"
+        "preprocessor = ColumnTransformer(\n",
+        "    transformers=[\n",
+        "        (\"num\", numeric_transformer, selector(dtype_exclude=\"category\")),\n",
+        "        (\"cat\", categorical_transformer, selector(dtype_include=\"category\")),\n",
+        "    ]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, the preprocessing pipeline is defined, we can run it on our training data, and apply the generated transform to our test data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_train = preprocessor.fit_transform(X_train)\n",
+        "X_test = preprocessor.transform(X_test)"
       ]
     },
     {
@@ -204,7 +256,7 @@
       "source": [
         "lr_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
         "\n",
-        "lr_predictor.fit(X_train, Y_train)"
+        "lr_predictor.fit(X_train, y_train)"
       ]
     },
     {
@@ -222,7 +274,7 @@
       "source": [
         "svm_predictor = svm.SVC()\n",
         "\n",
-        "svm_predictor.fit(X_train, Y_train)"
+        "svm_predictor.fit(X_train, y_train)"
       ]
     },
     {
@@ -337,11 +389,10 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from fairlearn.widget import FairlearnDashboard\n",
+        "from raiwidgets import FairnessDashboard\n",
         "\n",
-        "FairlearnDashboard(sensitive_features=A_test, \n",
-        "                   sensitive_feature_names=['Sex', 'Race'],\n",
-        "                   y_true=Y_test.tolist(),\n",
+        "FairnessDashboard(sensitive_features=A_test, \n",
+        "                  y_true=y_test.tolist(),\n",
         "                  y_pred=ys_pred)"
       ]
     },
@@ -352,7 +403,7 @@
         "<a id=\"AzureUpload\"></a>\n",
         "## Uploading a Fairness Dashboard to Azure\n",
         "\n",
-        "Uploading a fairness dashboard to Azure is a two stage process. The `FairlearnDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. The required stages are therefore:\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairnessDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. The required stages are therefore:\n",
         "1. Precompute all the required metrics\n",
         "1. Upload to Azure\n",
         "\n",
@@ -367,11 +418,11 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "sf = { 'Race': A_test.Race, 'Sex': A_test.Sex }\n",
+        "sf = { 'Race': A_test.race, 'Sex': A_test.sex }\n",
         "\n",
         "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
         "\n",
-        "dash_dict = _create_group_metric_set(y_true=Y_test,\n",
+        "dash_dict = _create_group_metric_set(y_true=y_test,\n",
         "                                     predictions=ys_pred,\n",
         "                                     sensitive_features=sf,\n",
         "                                     prediction_type='binary_classification')"
@@ -486,7 +537,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.8"
+      "version": "3.6.10"
     }
   },
   "nbformat": 4,

contrib/fairness/upload-fairness-dashboard.yml

@@ -3,6 +3,7 @@ dependencies:
 - pip:
   - azureml-sdk
   - azureml-contrib-fairness
-  - fairlearn==0.4.6
+  - fairlearn>=0.6.2
   - joblib
-  - shap
+  - liac-arff
+  - raiwidgets~=0.7.0

contrib/gbdt/lightgbm/binary0.test

@@ -1,500 +0,0 @@
-1	0.644	0.247	-0.447	0.862	0.374	0.854	-1.126	-0.790	2.173	1.015	-0.201	1.400	0.000	1.575	1.807	1.607	0.000	1.585	-0.190	-0.744	3.102	0.958	1.061	0.980	0.875	0.581	0.905	0.796
-0	0.385	1.800	1.037	1.044	0.349	1.502	-0.966	1.734	0.000	0.966	-1.960	-0.249	0.000	1.501	0.465	-0.354	2.548	0.834	-0.440	0.638	3.102	0.695	0.909	0.981	0.803	0.813	1.149	1.116
-0	1.214	-0.166	0.004	0.505	1.434	0.628	-1.174	-1.230	1.087	0.579	-1.047	-0.118	0.000	0.835	0.340	1.234	2.548	0.711	-1.383	1.355	0.000	0.848	0.911	1.043	0.931	1.058	0.744	0.696
-1	0.420	1.111	0.137	1.516	-1.657	0.854	0.623	1.605	1.087	1.511	-1.297	0.251	0.000	0.872	-0.368	-0.721	0.000	0.543	0.731	1.424	3.102	1.597	1.282	1.105	0.730	0.148	1.231	1.234
-0	0.897	-1.703	-1.306	1.022	-0.729	0.836	0.859	-0.333	2.173	1.336	-0.965	0.972	2.215	0.671	1.021	-1.439	0.000	0.493	-2.019	-0.289	0.000	0.805	0.930	0.984	1.430	2.198	1.934	1.684
-0	0.756	1.126	-0.945	2.355	-0.555	0.889	0.800	1.440	0.000	0.585	0.271	0.631	2.215	0.722	1.744	1.051	0.000	0.618	0.924	0.698	1.551	0.976	0.864	0.988	0.803	0.234	0.822	0.911
-0	1.141	-0.741	0.953	1.478	-0.524	1.197	-0.871	1.689	2.173	0.875	1.321	-0.518	1.107	0.540	0.037	-0.987	0.000	0.879	1.187	0.245	0.000	0.888	0.701	1.747	1.358	2.479	1.491	1.223
-1	0.606	-0.936	-0.384	1.257	-1.162	2.719	-0.600	0.100	2.173	3.303	-0.284	1.561	1.107	0.689	1.786	-0.326	0.000	0.780	-0.532	1.216	0.000	0.936	2.022	0.985	1.574	4.323	2.263	1.742
-1	0.603	0.429	-0.279	1.448	1.301	1.008	2.423	-1.295	0.000	0.452	1.305	0.533	0.000	1.076	1.011	1.256	2.548	2.021	1.260	-0.343	0.000	0.890	0.969	1.281	0.763	0.652	0.827	0.785
-0	1.171	-0.962	0.521	0.841	-0.315	1.196	-0.744	-0.882	2.173	0.726	-1.305	1.377	1.107	0.643	-1.790	-1.264	0.000	1.257	0.222	0.817	0.000	0.862	0.911	0.987	0.846	1.293	0.899	0.756
-1	1.392	-0.358	0.235	1.494	-0.461	0.895	-0.848	1.549	2.173	0.841	-0.384	0.666	1.107	1.199	2.509	-0.891	0.000	1.109	-0.364	-0.945	0.000	0.693	2.135	1.170	1.362	0.959	2.056	1.842
-1	1.024	1.076	-0.886	0.851	1.530	0.673	-0.449	0.187	1.087	0.628	-0.895	1.176	2.215	0.696	-0.232	-0.875	0.000	0.411	1.501	0.048	0.000	0.842	0.919	1.063	1.193	0.777	0.964	0.807
-1	0.890	-0.760	1.182	1.369	0.751	0.696	-0.959	-0.710	1.087	0.775	-0.130	-1.409	2.215	0.701	-0.110	-0.739	0.000	0.508	-0.451	0.390	0.000	0.762	0.738	0.998	1.126	0.788	0.940	0.790
-1	0.460	0.537	0.636	1.442	-0.269	0.585	0.323	-1.731	2.173	0.503	1.034	-0.927	0.000	0.928	-1.024	1.006	2.548	0.513	-0.618	-1.336	0.000	0.802	0.831	0.992	1.019	0.925	1.056	0.833
-1	0.364	1.648	0.560	1.720	0.829	1.110	0.811	-0.588	0.000	0.408	1.045	1.054	2.215	0.319	-1.138	1.545	0.000	0.423	1.025	-1.265	3.102	1.656	0.928	1.003	0.544	0.327	0.670	0.746
-1	0.525	-0.096	1.206	0.948	-1.103	1.519	-0.582	0.606	2.173	1.274	-0.572	-0.934	0.000	0.855	-1.028	-1.222	0.000	0.578	-1.000	-1.725	3.102	0.896	0.878	0.981	0.498	0.909	0.772	0.668
-0	0.536	-0.821	-1.029	0.703	1.113	0.363	-0.711	0.022	1.087	0.325	1.503	1.249	2.215	0.673	1.041	-0.401	0.000	0.480	2.127	1.681	0.000	0.767	1.034	0.990	0.671	0.836	0.669	0.663
-1	1.789	-0.583	1.641	0.897	0.799	0.515	-0.100	-1.483	0.000	1.101	0.031	-0.326	2.215	1.195	0.001	0.126	2.548	0.768	-0.148	0.601	0.000	0.916	0.921	1.207	1.069	0.483	0.934	0.795
-1	1.332	-0.571	0.986	0.580	1.508	0.582	0.634	-0.746	1.087	1.084	-0.964	-0.489	0.000	0.785	0.274	0.343	2.548	0.779	0.721	1.489	0.000	1.733	1.145	0.990	1.270	0.715	0.897	0.915
-0	1.123	0.629	-1.708	0.597	-0.882	0.752	0.195	1.522	2.173	1.671	1.515	-0.003	0.000	0.778	0.514	0.139	1.274	0.801	1.260	1.600	0.000	1.495	0.976	0.988	0.676	0.921	1.010	0.943
-0	1.816	-0.515	0.171	0.980	-0.454	0.870	0.202	-1.399	2.173	1.130	1.066	-1.593	0.000	0.844	0.735	1.275	2.548	1.125	-1.133	0.348	0.000	0.837	0.693	0.988	1.112	0.784	1.009	0.974
-1	0.364	0.694	0.445	1.862	0.159	0.963	-1.356	1.260	1.087	0.887	-0.540	-1.533	2.215	0.658	-2.544	-1.236	0.000	0.516	-0.807	0.039	0.000	0.891	1.004	0.991	1.092	0.976	1.000	0.953
-1	0.790	-1.175	0.475	1.846	0.094	0.999	-1.090	0.257	0.000	1.422	0.854	1.112	2.215	1.302	1.004	-1.702	1.274	2.557	-0.787	-1.048	0.000	0.890	1.429	0.993	2.807	0.840	2.248	1.821
-1	0.765	-0.500	-0.603	1.843	-0.560	1.068	0.007	0.746	2.173	1.154	-0.017	1.329	0.000	1.165	1.791	-1.585	0.000	1.116	0.441	-0.886	0.000	0.774	0.982	0.989	1.102	0.633	1.178	1.021
-1	1.407	1.293	-1.418	0.502	-1.527	2.005	-2.122	0.622	0.000	1.699	1.508	-0.649	2.215	1.665	0.748	-0.755	0.000	2.555	0.811	1.423	1.551	7.531	5.520	0.985	1.115	1.881	4.487	3.379
-1	0.772	-0.186	-1.372	0.823	-0.140	0.781	0.763	0.046	2.173	1.128	0.516	1.380	0.000	0.797	-0.640	-0.134	2.548	2.019	-0.972	-1.670	0.000	2.022	1.466	0.989	0.856	0.808	1.230	0.991
-1	0.546	-0.954	0.715	1.335	-1.689	0.783	-0.443	-1.735	2.173	1.081	0.185	-0.435	0.000	1.433	-0.662	-0.389	0.000	0.969	0.924	1.099	0.000	0.910	0.879	0.988	0.683	0.753	0.878	0.865
-1	0.596	0.276	-1.054	1.358	1.355	1.444	1.813	-0.208	0.000	1.175	-0.949	-1.573	0.000	0.855	-1.228	-0.925	2.548	1.837	-0.400	0.913	0.000	0.637	0.901	1.028	0.553	0.790	0.679	0.677
-0	0.458	2.292	1.530	0.291	1.283	0.749	-0.930	-0.198	0.000	0.300	-1.560	0.990	0.000	0.811	-0.176	0.995	2.548	1.085	-0.178	-1.213	3.102	0.891	0.648	0.999	0.732	0.655	0.619	0.620
-0	0.638	-0.575	-1.048	0.125	0.178	0.846	-0.753	-0.339	1.087	0.799	-0.727	1.182	0.000	0.888	0.283	0.717	0.000	1.051	-1.046	-1.557	3.102	0.889	0.871	0.989	0.884	0.923	0.836	0.779
-1	0.434	-1.119	-0.313	2.427	0.461	0.497	0.261	-1.177	2.173	0.618	-0.737	-0.688	0.000	1.150	-1.237	-1.652	2.548	0.757	-0.054	1.700	0.000	0.809	0.741	0.982	1.450	0.936	1.086	0.910
-1	0.431	-1.144	-1.030	0.778	-0.655	0.490	0.047	-1.546	0.000	1.583	-0.014	0.891	2.215	0.516	0.956	0.567	2.548	0.935	-1.123	-0.082	0.000	0.707	0.995	0.995	0.700	0.602	0.770	0.685
-1	1.894	0.222	1.224	1.578	1.715	0.966	2.890	-0.013	0.000	0.922	-0.703	-0.844	0.000	0.691	2.056	1.039	0.000	0.900	-0.733	-1.240	3.102	1.292	1.992	1.026	0.881	0.684	1.759	1.755
-0	0.985	-0.316	0.141	1.067	-0.946	0.819	-1.177	1.307	2.173	1.080	-0.429	0.557	1.107	1.726	1.435	-1.075	0.000	1.100	1.547	-0.647	0.000	0.873	1.696	1.179	1.146	1.015	1.538	1.270
-0	0.998	-0.187	-0.236	0.882	0.755	0.468	0.950	-0.439	2.173	0.579	-0.550	-0.624	0.000	1.847	1.196	1.384	1.274	0.846	1.273	-1.072	0.000	1.194	0.797	1.013	1.319	1.174	0.963	0.898
-0	0.515	0.246	-0.593	1.082	1.591	0.912	-0.623	-0.957	2.173	0.858	0.418	0.844	0.000	0.948	2.519	1.599	0.000	1.158	1.385	-0.095	3.102	0.973	1.033	0.988	0.998	1.716	1.054	0.901
-0	0.919	-1.001	1.506	1.389	0.653	0.507	-0.616	-0.689	2.173	0.808	0.536	-0.467	2.215	0.496	2.187	-0.859	0.000	0.822	0.807	1.163	0.000	0.876	0.861	1.088	0.947	0.614	0.911	1.087
-0	0.794	0.051	1.477	1.504	-1.695	0.716	0.315	0.264	1.087	0.879	-0.135	-1.094	2.215	1.433	-0.741	0.201	0.000	1.566	0.534	-0.989	0.000	0.627	0.882	0.974	0.807	1.130	0.929	0.925
-1	0.455	-0.946	-1.175	1.453	-0.580	0.763	-0.856	0.840	0.000	0.829	1.223	1.174	2.215	0.714	0.638	-0.466	0.000	1.182	0.223	-1.333	0.000	0.977	0.938	0.986	0.713	0.714	0.796	0.843
-1	0.662	-0.296	-1.287	1.212	-0.707	0.641	1.457	0.222	0.000	0.600	0.525	-1.700	2.215	0.784	-0.835	-0.961	2.548	0.865	1.131	1.162	0.000	0.854	0.877	0.978	0.740	0.734	0.888	0.811
-0	0.390	0.698	-1.629	1.888	0.298	0.990	1.614	-1.572	0.000	1.666	0.170	0.719	2.215	1.590	1.064	-0.886	1.274	0.952	0.305	-1.216	0.000	1.048	0.897	1.173	0.891	1.936	1.273	1.102
-0	1.014	0.117	1.384	0.686	-1.047	0.609	-1.245	-0.850	0.000	1.076	-1.158	0.814	1.107	1.598	-0.389	-0.111	0.000	0.907	1.688	-1.673	0.000	1.333	0.866	0.989	0.975	0.442	0.797	0.788
-0	1.530	-1.408	-0.207	0.440	-1.357	0.902	-0.647	1.325	1.087	1.320	-0.819	0.246	1.107	0.503	1.407	-1.683	0.000	1.189	-0.972	-0.925	0.000	0.386	1.273	0.988	0.829	1.335	1.173	1.149
-1	1.689	-0.590	0.915	2.076	1.202	0.644	-0.478	-0.238	0.000	0.809	-1.660	-1.184	0.000	1.227	-0.224	-0.808	2.548	1.655	1.047	-0.623	0.000	0.621	1.192	0.988	1.309	0.866	0.924	1.012
-0	1.102	0.402	-1.622	1.262	1.022	0.576	0.271	-0.269	0.000	0.591	0.495	-1.278	0.000	1.271	0.209	0.575	2.548	0.941	0.964	-0.685	3.102	0.989	0.963	1.124	0.857	0.858	0.716	0.718
-0	2.491	0.825	0.581	1.593	0.205	0.782	-0.815	1.499	0.000	1.179	-0.999	-1.509	0.000	0.926	0.920	-0.522	2.548	2.068	-1.021	-1.050	3.102	0.874	0.943	0.980	0.945	1.525	1.570	1.652
-0	0.666	0.254	1.601	1.303	-0.250	1.236	-1.929	0.793	0.000	1.074	0.447	-0.871	0.000	0.991	1.059	-0.342	0.000	1.703	-0.393	-1.419	3.102	0.921	0.945	1.285	0.931	0.462	0.770	0.729
-0	0.937	-1.126	1.424	1.395	1.743	0.760	0.428	-0.238	2.173	0.846	0.494	1.320	2.215	0.872	-1.826	-0.507	0.000	0.612	1.860	1.403	0.000	3.402	2.109	0.985	1.298	1.165	1.404	1.240
-1	0.881	-1.086	-0.870	0.513	0.266	2.049	-1.870	1.160	0.000	2.259	-0.428	-0.935	2.215	1.321	-0.655	-0.449	2.548	1.350	-1.766	-0.108	0.000	0.911	1.852	0.987	1.167	0.820	1.903	1.443
-0	0.410	0.835	-0.819	1.257	1.112	0.871	-1.737	-0.401	0.000	0.927	0.158	1.253	0.000	1.183	0.405	-1.570	0.000	0.807	-0.704	-0.438	3.102	0.932	0.962	0.987	0.653	0.315	0.616	0.648
-1	0.634	0.196	-1.679	1.379	-0.967	2.260	-0.273	1.114	0.000	1.458	1.070	-0.278	1.107	1.195	0.110	-0.688	2.548	0.907	0.298	-1.359	0.000	0.949	1.129	0.984	0.675	0.877	0.938	0.824
-1	0.632	-1.254	1.201	0.496	-0.106	0.235	2.731	-0.955	0.000	0.615	-0.805	0.600	0.000	0.633	-0.934	1.641	0.000	1.407	-0.483	-0.962	1.551	0.778	0.797	0.989	0.578	0.722	0.576	0.539
-0	0.714	1.122	1.566	2.399	-1.431	1.665	0.299	0.323	0.000	1.489	1.087	-0.861	2.215	1.174	0.140	1.083	2.548	0.404	-0.968	1.105	0.000	0.867	0.969	0.981	1.039	1.552	1.157	1.173
-1	0.477	-0.321	-0.471	1.966	1.034	2.282	1.359	-0.874	0.000	1.672	-0.258	1.109	0.000	1.537	0.604	0.231	2.548	1.534	-0.640	0.827	0.000	0.746	1.337	1.311	0.653	0.721	0.795	0.742
-1	1.351	0.460	0.031	1.194	-1.185	0.670	-1.157	-1.637	2.173	0.599	-0.823	0.680	0.000	0.478	0.373	1.716	0.000	0.809	-0.919	0.010	1.551	0.859	0.839	1.564	0.994	0.777	0.971	0.826
-1	0.520	-1.442	-0.348	0.840	1.654	1.273	-0.760	1.317	0.000	0.861	2.579	-0.791	0.000	1.779	0.257	-0.703	0.000	2.154	1.928	0.457	0.000	1.629	3.194	0.992	0.730	1.107	2.447	2.747
-0	0.700	-0.308	0.920	0.438	-0.879	0.516	1.409	1.101	0.000	0.960	0.701	-0.049	2.215	1.442	-0.416	-1.439	2.548	0.628	1.009	-0.364	0.000	0.848	0.817	0.987	0.759	1.421	0.937	0.920
-1	0.720	1.061	-0.546	0.798	-1.521	1.066	0.173	0.271	1.087	1.453	0.114	1.336	1.107	0.702	0.616	-0.367	0.000	0.543	-0.386	-1.301	0.000	0.653	0.948	0.989	1.031	1.500	0.965	0.790
-1	0.735	-0.416	0.588	1.308	-0.382	1.042	0.344	1.609	0.000	0.926	0.163	-0.520	1.107	1.050	-0.427	1.159	2.548	0.834	0.613	0.948	0.000	0.848	1.189	1.042	0.844	1.099	0.829	0.843
-1	0.777	-0.396	1.540	1.608	0.638	0.955	0.040	0.918	2.173	1.315	1.116	-0.823	0.000	0.781	-0.762	0.564	2.548	0.945	-0.573	1.379	0.000	0.679	0.706	1.124	0.608	0.593	0.515	0.493
-1	0.934	0.319	-0.257	0.970	-0.980	0.726	0.774	0.731	0.000	0.896	0.038	-1.465	1.107	0.773	-0.055	-0.831	2.548	1.439	-0.229	0.698	0.000	0.964	1.031	0.995	0.845	0.480	0.810	0.762
-0	0.461	0.771	0.019	2.055	-1.288	1.043	0.147	0.261	2.173	0.833	-0.156	1.425	0.000	0.832	0.805	-0.491	2.548	0.589	1.252	1.414	0.000	0.850	0.906	1.245	1.364	0.850	0.908	0.863
-1	0.858	-0.116	-0.937	0.966	1.167	0.825	-0.108	1.111	1.087	0.733	1.163	-0.634	0.000	0.894	0.771	0.020	0.000	0.846	-1.124	-1.195	3.102	0.724	1.194	1.195	0.813	0.969	0.985	0.856
-0	0.720	-0.335	-0.307	1.445	0.540	1.108	-0.034	-1.691	1.087	0.883	-1.356	-0.678	2.215	0.440	1.093	0.253	0.000	0.389	-1.582	-1.097	0.000	1.113	1.034	0.988	1.256	1.572	1.062	0.904
-1	0.750	-0.811	-0.542	0.985	0.408	0.471	0.477	0.355	0.000	1.347	-0.875	-1.556	2.215	0.564	1.082	-0.724	0.000	0.793	-0.958	-0.020	3.102	0.836	0.825	0.986	1.066	0.924	0.927	0.883
-0	0.392	-0.468	-0.216	0.680	1.565	1.086	-0.765	-0.581	1.087	1.264	-1.035	1.189	2.215	0.986	-0.338	0.747	0.000	0.884	-1.328	-0.965	0.000	1.228	0.988	0.982	1.135	1.741	1.108	0.956
-1	0.434	-1.269	0.643	0.713	0.608	0.597	0.832	1.627	0.000	0.708	-0.422	0.079	2.215	1.533	-0.823	-1.127	2.548	0.408	-1.357	-0.828	0.000	1.331	1.087	0.999	1.075	1.015	0.875	0.809
-0	0.828	-1.803	0.342	0.847	-0.162	1.585	-1.128	-0.272	2.173	1.974	0.039	-1.717	0.000	0.900	0.764	-1.741	0.000	1.349	-0.079	1.035	3.102	0.984	0.815	0.985	0.780	1.661	1.403	1.184
-1	1.089	-0.350	-0.747	1.472	0.792	1.087	-0.069	-1.192	0.000	0.512	-0.841	-1.284	0.000	2.162	-0.821	0.545	2.548	1.360	2.243	-0.183	0.000	0.977	0.628	1.725	1.168	0.635	0.823	0.822
-1	0.444	0.451	-1.332	1.176	-0.247	0.898	0.194	0.007	0.000	1.958	0.576	-1.618	2.215	0.584	1.203	0.268	0.000	0.939	1.033	1.264	3.102	0.829	0.886	0.985	1.265	0.751	1.032	0.948
-0	0.629	0.114	1.177	0.917	-1.204	0.845	0.828	-0.088	0.000	0.962	-1.302	0.823	2.215	0.732	0.358	-1.334	2.548	0.538	0.582	1.561	0.000	1.028	0.834	0.988	0.904	1.205	1.039	0.885
-1	1.754	-1.259	-0.573	0.959	-1.483	0.358	0.448	-1.452	0.000	0.711	0.313	0.499	2.215	1.482	-0.390	1.474	2.548	1.879	-1.540	0.668	0.000	0.843	0.825	1.313	1.315	0.939	1.048	0.871
-1	0.549	0.706	-1.437	0.894	0.891	0.680	-0.762	-1.568	0.000	0.981	0.499	-0.425	2.215	1.332	0.678	0.485	1.274	0.803	0.022	-0.893	0.000	0.793	1.043	0.987	0.761	0.899	0.915	0.794
-0	0.475	0.542	-0.987	1.569	0.069	0.551	1.543	-1.488	0.000	0.608	0.301	1.734	2.215	0.277	0.499	-0.522	0.000	1.375	1.212	0.696	3.102	0.652	0.756	0.987	0.828	0.830	0.715	0.679
-1	0.723	0.049	-1.153	1.300	0.083	0.723	-0.749	0.630	0.000	1.126	0.412	-0.384	0.000	1.272	1.256	1.358	2.548	3.108	0.777	-1.486	3.102	0.733	1.096	1.206	1.269	0.899	1.015	0.903
-1	1.062	0.296	0.725	0.285	-0.531	0.819	1.277	-0.667	0.000	0.687	0.829	-0.092	0.000	1.158	0.447	1.047	2.548	1.444	-0.186	-1.491	3.102	0.863	1.171	0.986	0.769	0.828	0.919	0.840
-0	0.572	-0.349	1.396	2.023	0.795	0.577	0.457	-0.533	0.000	1.351	0.701	-1.091	0.000	0.724	-1.012	-0.182	2.548	0.923	-0.012	0.789	3.102	0.936	1.025	0.985	1.002	0.600	0.828	0.909
-1	0.563	0.387	0.412	0.553	1.050	0.723	-0.992	-0.447	0.000	0.748	0.948	0.546	2.215	1.761	-0.559	-1.183	0.000	1.114	-0.251	1.192	3.102	0.936	0.912	0.976	0.578	0.722	0.829	0.892
-1	1.632	1.577	-0.697	0.708	-1.263	0.863	0.012	1.197	2.173	0.498	0.990	-0.806	0.000	0.627	2.387	-1.283	0.000	0.607	1.290	-0.174	3.102	0.916	1.328	0.986	0.557	0.971	0.935	0.836
-1	0.562	-0.360	0.399	0.803	-1.334	1.443	-0.116	1.628	2.173	0.750	0.987	0.135	1.107	0.795	0.298	-0.556	0.000	1.150	-0.113	-0.093	0.000	0.493	1.332	0.985	1.001	1.750	1.013	0.886
-1	0.987	0.706	-0.492	0.861	0.607	0.593	0.088	-0.184	0.000	0.802	0.894	1.608	2.215	0.782	-0.471	1.500	2.548	0.521	0.772	-0.960	0.000	0.658	0.893	1.068	0.877	0.664	0.709	0.661
-1	1.052	0.883	-0.581	1.566	0.860	0.931	1.515	-0.873	0.000	0.493	0.145	-0.672	0.000	1.133	0.935	1.581	2.548	1.630	0.695	0.923	3.102	1.105	1.087	1.713	0.948	0.590	0.872	0.883
-1	2.130	-0.516	-0.291	0.776	-1.230	0.689	-0.257	0.800	2.173	0.730	-0.274	-1.437	0.000	0.615	0.241	1.083	0.000	0.834	0.757	1.613	3.102	0.836	0.806	1.333	1.061	0.730	0.889	0.783
-1	0.742	0.797	1.628	0.311	-0.418	0.620	0.685	-1.457	0.000	0.683	1.774	-1.082	0.000	1.700	1.104	0.225	2.548	0.382	-2.184	-1.307	0.000	0.945	1.228	0.984	0.864	0.931	0.988	0.838
-0	0.311	-1.249	-0.927	1.272	-1.262	0.642	-1.228	-0.136	0.000	1.220	-0.804	-1.558	2.215	0.950	-0.828	0.495	1.274	2.149	-1.672	0.634	0.000	1.346	0.887	0.981	0.856	1.101	1.001	1.106
-0	0.660	-1.834	-0.667	0.601	1.236	0.932	-0.933	-0.135	2.173	1.373	-0.122	1.429	0.000	0.654	-0.034	-0.847	2.548	0.711	0.911	0.703	0.000	1.144	0.942	0.984	0.822	0.739	0.992	0.895
-0	3.609	-0.590	0.851	0.615	0.455	1.280	0.003	-0.866	1.087	1.334	0.708	-1.131	0.000	0.669	0.480	0.092	0.000	0.975	0.983	-1.429	3.102	1.301	1.089	0.987	1.476	0.934	1.469	1.352
-1	0.905	-0.403	1.567	2.651	0.953	1.194	-0.241	-0.567	1.087	0.308	-0.384	-0.007	0.000	0.608	-0.175	-1.163	2.548	0.379	0.941	1.662	0.000	0.580	0.721	1.126	0.895	0.544	1.097	0.836
-1	0.983	0.255	1.093	0.905	-0.874	0.863	0.060	-0.368	0.000	0.824	-0.747	-0.633	0.000	0.614	0.961	1.052	0.000	0.792	-0.260	1.632	3.102	0.874	0.883	1.280	0.663	0.406	0.592	0.645
-1	1.160	-1.027	0.274	0.460	0.322	2.085	-1.623	-0.840	0.000	1.634	-1.046	1.182	2.215	0.492	-0.367	1.174	0.000	0.824	-0.998	1.617	0.000	0.943	0.884	1.001	1.209	1.313	1.034	0.866
-0	0.299	0.028	-1.372	1.930	-0.661	0.840	-0.979	0.664	1.087	0.535	-2.041	1.434	0.000	1.087	-1.797	0.344	0.000	0.485	-0.560	-1.105	3.102	0.951	0.890	0.980	0.483	0.684	0.730	0.706
-0	0.293	1.737	-1.418	2.074	0.794	0.679	1.024	-1.457	0.000	1.034	1.094	-0.168	1.107	0.506	1.680	-0.661	0.000	0.523	-0.042	-1.274	3.102	0.820	0.944	0.987	0.842	0.694	0.761	0.750
-0	0.457	-0.393	1.560	0.738	-0.007	0.475	-0.230	0.246	0.000	0.776	-1.264	-0.606	2.215	0.865	-0.731	-1.576	2.548	1.153	0.343	1.436	0.000	1.060	0.883	0.988	0.972	0.703	0.758	0.720
-0	0.935	-0.582	0.240	2.401	0.818	1.231	-0.618	-1.289	0.000	0.799	0.544	-0.228	2.215	0.525	-1.494	-0.969	0.000	0.609	-1.123	1.168	3.102	0.871	0.767	1.035	1.154	0.919	0.868	1.006
-1	0.902	-0.745	-1.215	1.174	-0.501	1.215	0.167	1.162	0.000	0.896	1.217	-0.976	0.000	0.585	-0.429	1.036	0.000	1.431	-0.416	0.151	3.102	0.524	0.952	0.990	0.707	0.271	0.592	0.826
-1	0.653	0.337	-0.320	1.118	-0.934	1.050	0.745	0.529	1.087	1.075	1.742	-1.538	0.000	0.585	1.090	0.973	0.000	1.091	-0.187	1.160	1.551	1.006	1.108	0.978	1.121	0.838	0.947	0.908
-0	1.157	1.401	0.340	0.395	-1.218	0.945	1.928	-0.876	0.000	1.384	0.320	1.002	1.107	1.900	1.177	-0.462	2.548	1.122	1.316	1.720	0.000	1.167	1.096	0.989	0.937	1.879	1.307	1.041
-0	0.960	0.355	-0.152	0.872	-0.338	0.391	0.348	0.956	1.087	0.469	2.664	1.409	0.000	0.756	-1.561	1.500	0.000	0.525	1.436	1.728	3.102	1.032	0.946	0.996	0.929	0.470	0.698	0.898
-1	1.038	0.274	0.825	1.198	0.963	1.078	-0.496	-1.014	2.173	0.739	-0.727	-0.151	2.215	1.035	-0.799	0.398	0.000	1.333	-0.872	-1.498	0.000	0.849	1.033	0.985	0.886	0.936	0.975	0.823
-0	0.490	0.277	0.318	1.303	0.694	1.333	-1.620	-0.563	0.000	1.459	-1.326	1.140	0.000	0.779	-0.673	-1.324	2.548	0.860	-1.247	0.043	0.000	0.857	0.932	0.992	0.792	0.278	0.841	1.498
-0	1.648	-0.688	-1.386	2.790	0.995	1.087	1.359	-0.687	0.000	1.050	-0.223	-0.261	2.215	0.613	-0.889	1.335	0.000	1.204	0.827	0.309	3.102	0.464	0.973	2.493	1.737	0.827	1.319	1.062
-0	1.510	-0.662	1.668	0.860	0.280	0.705	0.974	-1.647	1.087	0.662	-0.393	-0.225	0.000	0.610	-0.996	0.532	2.548	0.464	1.305	0.102	0.000	0.859	1.057	1.498	0.799	1.260	0.946	0.863
-1	0.850	-1.185	-0.117	0.943	-0.449	1.142	0.875	-0.030	0.000	2.223	-0.461	1.627	2.215	0.767	-1.761	-1.692	0.000	1.012	-0.727	0.639	3.102	3.649	2.062	0.985	1.478	1.087	1.659	1.358
-0	0.933	1.259	0.130	0.326	-0.890	0.306	1.136	1.142	0.000	0.964	0.705	-1.373	2.215	0.546	-0.196	-0.001	0.000	0.578	-1.169	1.004	3.102	0.830	0.836	0.988	0.837	1.031	0.749	0.655
-0	0.471	0.697	1.570	1.109	0.201	1.248	0.348	-1.448	0.000	2.103	0.773	0.686	2.215	1.451	-0.087	-0.453	2.548	1.197	-0.045	-1.026	0.000	0.793	1.094	0.987	0.851	1.804	1.378	1.089
-1	2.446	-0.701	-1.568	0.059	0.822	1.401	-0.600	-0.044	2.173	0.324	-0.001	1.344	2.215	0.913	-0.818	1.049	0.000	0.442	-1.088	-0.005	0.000	0.611	1.062	0.979	0.562	0.988	0.998	0.806
-0	0.619	2.029	0.933	0.528	-0.903	0.974	0.760	-0.311	2.173	0.825	0.658	-1.466	1.107	0.894	1.594	0.370	0.000	0.882	-0.258	1.661	0.000	1.498	1.088	0.987	0.867	1.139	0.900	0.779
-1	0.674	-0.131	-0.362	0.518	-1.574	0.876	0.442	0.145	1.087	0.497	-1.526	-1.704	0.000	0.680	2.514	-1.374	0.000	0.792	-0.479	0.773	1.551	0.573	1.198	0.984	0.800	0.667	0.987	0.832
-1	1.447	1.145	-0.937	0.307	-1.458	0.478	1.264	0.816	1.087	0.558	1.015	-0.101	2.215	0.937	-0.190	1.177	0.000	0.699	0.954	-1.512	0.000	0.877	0.838	0.990	0.873	0.566	0.646	0.713
-1	0.976	0.308	-0.844	0.436	0.610	1.253	0.149	-1.585	2.173	1.415	0.568	0.096	2.215	0.953	-0.855	0.441	0.000	0.867	-0.650	1.643	0.000	0.890	1.234	0.988	0.796	2.002	1.179	0.977
-0	0.697	0.401	-0.718	0.920	0.735	0.958	-0.172	0.168	2.173	0.872	-0.097	-1.335	0.000	0.513	-1.192	-1.710	1.274	0.426	-1.637	1.368	0.000	0.997	1.227	1.072	0.800	1.013	0.786	0.749
-1	1.305	-2.157	1.740	0.661	-0.912	0.705	-0.516	0.759	2.173	0.989	-0.716	-0.300	2.215	0.627	-1.052	-1.736	0.000	0.467	-2.467	0.568	0.000	0.807	0.964	0.988	1.427	1.012	1.165	0.926
-0	1.847	1.663	-0.618	0.280	1.258	1.462	-0.054	1.371	0.000	0.900	0.309	-0.544	0.000	0.331	-2.149	-0.341	0.000	1.091	-0.833	0.710	3.102	1.496	0.931	0.989	1.549	0.115	1.140	1.150
-0	0.410	-0.323	1.069	2.160	0.010	0.892	0.942	-1.640	2.173	0.946	0.938	1.314	0.000	1.213	-1.099	-0.794	2.548	0.650	0.053	0.056	0.000	1.041	0.916	1.063	0.985	1.910	1.246	1.107
-1	0.576	1.092	-0.088	0.777	-1.579	0.757	0.271	0.109	0.000	0.819	0.827	-1.554	2.215	1.313	2.341	-1.568	0.000	2.827	0.239	-0.338	0.000	0.876	0.759	0.986	0.692	0.457	0.796	0.791
-1	0.537	0.925	-1.406	0.306	-0.050	0.906	1.051	0.037	0.000	1.469	-0.177	-1.320	2.215	1.872	0.723	1.158	0.000	1.313	0.227	-0.501	3.102	0.953	0.727	0.978	0.755	0.892	0.932	0.781
-0	0.716	-0.065	-0.484	1.313	-1.563	0.596	-0.242	0.678	2.173	0.426	-1.909	0.616	0.000	0.885	-0.406	-1.343	2.548	0.501	-1.327	-0.340	0.000	0.470	0.728	1.109	0.919	0.881	0.665	0.692
-1	0.624	-0.389	0.128	1.636	-1.110	1.025	0.573	-0.843	2.173	0.646	-0.697	1.064	0.000	0.632	-1.442	0.961	0.000	0.863	-0.106	1.717	0.000	0.825	0.917	1.257	0.983	0.713	0.890	0.824
-0	0.484	2.101	1.714	1.131	-0.823	0.750	0.583	-1.304	1.087	0.894	0.421	0.559	2.215	0.921	-0.063	0.282	0.000	0.463	-0.474	-1.387	0.000	0.742	0.886	0.995	0.993	1.201	0.806	0.754
-0	0.570	0.339	-1.478	0.528	0.439	0.978	1.479	-1.411	2.173	0.763	1.541	-0.734	0.000	1.375	0.840	0.903	0.000	0.965	1.599	0.364	0.000	0.887	1.061	0.992	1.322	1.453	1.013	0.969
-0	0.940	1.303	1.636	0.851	-1.732	0.803	-0.030	-0.177	0.000	0.480	-0.125	-0.954	0.000	0.944	0.709	0.296	2.548	1.342	-0.418	1.197	3.102	0.853	0.989	0.979	0.873	0.858	0.719	0.786
-1	0.599	0.544	-0.238	0.816	1.043	0.857	0.660	1.128	2.173	0.864	-0.624	-0.843	0.000	1.159	0.367	0.174	0.000	1.520	-0.543	-1.508	0.000	0.842	0.828	0.984	0.759	0.895	0.918	0.791
-1	1.651	1.897	-0.914	0.423	0.315	0.453	0.619	-1.607	2.173	0.532	-0.424	0.209	1.107	0.369	2.479	0.034	0.000	0.701	0.217	0.984	0.000	0.976	0.951	1.035	0.879	0.825	0.915	0.798
-1	0.926	-0.574	-0.763	0.285	1.094	0.672	2.314	1.545	0.000	1.124	0.415	0.809	0.000	1.387	0.270	-0.949	2.548	1.547	-0.631	-0.200	3.102	0.719	0.920	0.986	0.889	0.933	0.797	0.777
-0	0.677	1.698	-0.890	0.641	-0.449	0.607	1.754	1.720	0.000	0.776	0.372	0.782	2.215	0.511	1.491	-0.480	0.000	0.547	-0.341	0.853	3.102	0.919	1.026	0.997	0.696	0.242	0.694	0.687
-0	1.266	0.602	0.958	0.487	1.256	0.709	0.843	-1.196	0.000	0.893	1.303	-0.594	1.107	1.090	1.320	0.354	0.000	0.797	1.846	1.139	0.000	0.780	0.896	0.986	0.661	0.709	0.790	0.806
-1	0.628	-0.616	-0.329	0.764	-1.150	0.477	-0.715	1.187	2.173	1.250	0.607	1.026	2.215	0.983	-0.023	-0.583	0.000	0.377	1.344	-1.015	0.000	0.744	0.954	0.987	0.837	0.841	0.795	0.694
-1	1.035	-0.828	-1.358	1.870	-1.060	1.075	0.130	0.448	2.173	0.660	0.697	0.641	0.000	0.425	1.006	-1.035	0.000	0.751	1.055	1.364	3.102	0.826	0.822	0.988	0.967	0.901	1.077	0.906
-1	0.830	0.265	-0.150	0.660	1.105	0.592	-0.557	0.908	2.173	0.670	-1.419	-0.671	0.000	1.323	-0.409	1.644	2.548	0.850	-0.033	-0.615	0.000	0.760	0.967	0.984	0.895	0.681	0.747	0.770
-1	1.395	1.100	1.167	1.088	0.218	0.400	-0.132	0.024	2.173	0.743	0.530	-1.361	2.215	0.341	-0.691	-0.238	0.000	0.396	-1.426	-0.933	0.000	0.363	0.472	1.287	0.922	0.810	0.792	0.656
-1	1.070	1.875	-1.298	1.215	-0.106	0.767	0.795	0.514	1.087	0.401	2.780	1.276	0.000	0.686	1.127	1.721	2.548	0.391	-0.259	-1.167	0.000	1.278	1.113	1.389	0.852	0.824	0.838	0.785
-0	1.114	-0.071	1.719	0.399	-1.383	0.849	0.254	0.481	0.000	0.958	-0.579	0.742	0.000	1.190	-0.140	-0.862	2.548	0.479	1.390	0.856	0.000	0.952	0.988	0.985	0.764	0.419	0.835	0.827
-0	0.714	0.376	-0.568	1.578	-1.165	0.648	0.141	0.639	2.173	0.472	0.569	1.449	1.107	0.783	1.483	0.361	0.000	0.540	-0.790	0.032	0.000	0.883	0.811	0.982	0.775	0.572	0.760	0.745
-0	0.401	-1.731	0.765	0.974	1.648	0.652	-1.024	0.191	0.000	0.544	-0.366	-1.246	2.215	0.627	0.140	1.008	2.548	0.810	0.409	0.429	0.000	0.950	0.934	0.977	0.621	0.580	0.677	0.650
-1	0.391	1.679	-1.298	0.605	-0.832	0.549	1.338	0.522	2.173	1.244	0.884	1.070	0.000	1.002	0.846	-1.345	2.548	0.783	-2.464	-0.237	0.000	4.515	2.854	0.981	0.877	0.939	1.942	1.489
-1	0.513	-0.220	-0.444	1.699	0.479	1.109	0.181	-0.999	2.173	0.883	-0.335	-1.716	2.215	1.075	-0.380	1.352	0.000	0.857	0.048	0.147	0.000	0.937	0.758	0.986	1.206	0.958	0.949	0.876
-0	1.367	-0.388	0.798	1.158	1.078	0.811	-1.024	-1.628	0.000	1.504	0.097	-0.999	2.215	1.652	-0.860	0.054	2.548	0.573	-0.142	-1.401	0.000	0.869	0.833	1.006	1.412	1.641	1.214	1.041
-1	1.545	-0.533	-1.517	1.177	1.289	2.331	-0.370	-0.073	0.000	1.295	-0.358	-0.891	2.215	0.476	0.756	0.985	0.000	1.945	-0.016	-1.651	3.102	1.962	1.692	1.073	0.656	0.941	1.312	1.242
-0	0.858	0.978	-1.258	0.286	0.161	0.729	1.230	1.087	2.173	0.561	2.670	-0.109	0.000	0.407	2.346	0.938	0.000	1.078	0.729	-0.658	3.102	0.597	0.921	0.982	0.579	0.954	0.733	0.769
-1	1.454	-1.384	0.870	0.067	0.394	1.033	-0.673	0.318	0.000	1.166	-0.763	-1.533	2.215	2.848	-0.045	-0.856	2.548	0.697	-0.140	1.134	0.000	0.931	1.293	0.977	1.541	1.326	1.201	1.078
-1	0.559	-0.913	0.486	1.104	-0.321	1.073	-0.348	1.345	0.000	0.901	-0.827	-0.842	0.000	0.739	0.047	-0.415	2.548	0.433	-1.132	1.268	0.000	0.797	0.695	0.985	0.868	0.346	0.674	0.623
-1	1.333	0.780	-0.964	0.916	1.202	1.822	-0.071	0.742	2.173	1.486	-0.399	-0.824	0.000	0.740	0.568	-0.134	0.000	0.971	-0.070	-1.589	3.102	1.278	0.929	1.421	1.608	1.214	1.215	1.137
-1	2.417	0.631	-0.317	0.323	0.581	0.841	1.524	-1.738	0.000	0.543	1.176	-0.325	0.000	0.827	0.700	0.866	0.000	0.834	-0.262	-1.702	3.102	0.932	0.820	0.988	0.646	0.287	0.595	0.589
-0	0.955	-1.242	0.938	1.104	0.474	0.798	-0.743	1.535	0.000	1.356	-1.357	-1.080	2.215	1.320	-1.396	-0.132	2.548	0.728	-0.529	-0.633	0.000	0.832	0.841	0.988	0.923	1.077	0.988	0.816
-1	1.305	-1.918	0.391	1.161	0.063	0.724	2.593	1.481	0.000	0.592	-1.207	-0.329	0.000	0.886	-0.836	-1.168	2.548	1.067	-1.481	-1.440	0.000	0.916	0.688	0.991	0.969	0.550	0.665	0.638
-0	1.201	0.071	-1.123	2.242	-1.533	0.702	-0.256	0.688	0.000	0.967	0.491	1.040	2.215	1.271	-0.558	0.095	0.000	1.504	0.676	-0.383	3.102	0.917	1.006	0.985	1.017	1.057	0.928	1.057
-0	0.994	-1.607	1.596	0.774	-1.391	0.625	-0.134	-0.862	2.173	0.746	-0.765	-0.316	2.215	1.131	-0.320	0.869	0.000	0.607	0.826	0.301	0.000	0.798	0.967	0.999	0.880	0.581	0.712	0.774
-1	0.482	-0.467	0.729	1.419	1.458	0.824	0.376	-0.242	0.000	1.368	0.023	1.459	2.215	0.826	0.669	-1.079	2.548	0.936	2.215	-0.309	0.000	1.883	1.216	0.997	1.065	0.946	1.224	1.526
-1	0.383	1.588	1.611	0.748	1.194	0.866	-0.279	-0.636	0.000	0.707	0.536	0.801	2.215	1.647	-1.155	0.367	0.000	1.292	0.303	-1.681	3.102	2.016	1.581	0.986	0.584	0.684	1.107	0.958
-0	0.629	0.203	0.736	0.671	-0.271	1.350	-0.486	0.761	2.173	0.496	-0.805	-1.718	0.000	2.393	0.044	-1.046	1.274	0.651	-0.116	-0.541	0.000	0.697	1.006	0.987	1.069	2.317	1.152	0.902
-0	0.905	-0.564	-0.570	0.263	1.096	1.219	-1.397	-1.414	1.087	1.164	-0.533	-0.208	0.000	1.459	1.965	0.784	0.000	2.220	-1.421	0.452	0.000	0.918	1.360	0.993	0.904	0.389	2.118	1.707
-1	1.676	1.804	1.171	0.529	1.175	1.664	0.354	-0.530	0.000	1.004	0.691	-1.280	2.215	0.838	0.373	0.626	2.548	1.094	1.774	0.501	0.000	0.806	1.100	0.991	0.769	0.976	0.807	0.740
-1	1.364	-1.936	0.020	1.327	0.428	1.021	-1.665	-0.907	2.173	0.818	-2.701	1.303	0.000	0.716	-0.590	-1.629	2.548	0.895	-2.280	-1.602	0.000	1.211	0.849	0.989	1.320	0.864	1.065	0.949
-0	0.629	-0.626	0.609	1.828	1.280	0.644	-0.856	-0.873	2.173	0.555	1.066	-0.640	0.000	0.477	-1.364	-1.021	2.548	1.017	0.036	0.380	0.000	0.947	0.941	0.994	1.128	0.241	0.793	0.815
-1	1.152	-0.843	0.926	1.802	0.800	2.493	-1.449	-1.127	0.000	1.737	0.833	0.488	0.000	1.026	0.929	-0.990	2.548	1.408	0.689	1.142	3.102	1.171	0.956	0.993	2.009	0.867	1.499	1.474
-0	2.204	0.081	0.008	1.021	-0.679	2.676	0.090	1.163	0.000	2.210	-1.686	-1.195	0.000	1.805	0.891	-0.148	2.548	0.450	-0.502	-1.295	3.102	6.959	3.492	1.205	0.908	0.845	2.690	2.183
-1	0.957	0.954	1.702	0.043	-0.503	1.113	0.033	-0.308	0.000	0.757	-0.363	-1.129	2.215	1.635	0.068	1.048	1.274	0.415	-2.098	0.061	0.000	1.010	0.979	0.992	0.704	1.125	0.761	0.715
-0	1.222	0.418	1.059	1.303	1.442	0.282	-1.499	-1.286	0.000	1.567	0.016	-0.164	2.215	0.451	2.229	-1.229	0.000	0.660	-0.513	-0.296	3.102	2.284	1.340	0.985	1.531	0.314	1.032	1.094
-1	0.603	1.675	-0.973	0.703	-1.709	1.023	0.652	1.296	2.173	1.078	0.363	-0.263	0.000	0.734	-0.457	-0.745	1.274	0.561	1.434	-0.042	0.000	0.888	0.771	0.984	0.847	1.234	0.874	0.777
-0	0.897	0.949	-0.848	1.115	-0.085	0.522	-1.267	-1.418	0.000	0.684	-0.599	1.474	0.000	1.176	0.922	0.641	2.548	0.470	0.103	0.148	3.102	0.775	0.697	0.984	0.839	0.358	0.847	1.008
-1	0.987	1.013	-1.504	0.468	-0.259	1.160	0.476	-0.971	2.173	1.266	0.919	0.780	0.000	0.634	1.695	0.233	0.000	0.487	-0.082	0.719	3.102	0.921	0.641	0.991	0.730	0.828	0.952	0.807
-1	0.847	1.581	-1.397	1.629	1.529	1.053	0.816	-0.344	2.173	0.895	0.779	0.332	0.000	0.750	1.311	0.419	2.548	1.604	0.844	1.367	0.000	1.265	0.798	0.989	1.328	0.783	0.930	0.879
-1	0.805	1.416	-1.327	0.397	0.589	0.488	0.982	0.843	0.000	0.664	-0.999	0.129	0.000	0.624	0.613	-0.558	0.000	1.431	-0.667	-1.561	3.102	0.959	1.103	0.989	0.590	0.632	0.926	0.798
-0	1.220	-0.313	-0.489	1.759	0.201	1.698	-0.220	0.241	2.173	1.294	1.390	-1.682	0.000	1.447	-1.623	-1.296	0.000	1.710	0.872	-1.356	3.102	1.198	0.981	1.184	0.859	2.165	1.807	1.661
-0	0.772	-0.611	-0.549	0.465	-1.528	1.103	-0.140	0.001	2.173	0.854	-0.406	1.655	0.000	0.733	-1.250	1.072	0.000	0.883	0.627	-1.132	3.102	0.856	0.927	0.987	1.094	1.013	0.938	0.870
-1	1.910	0.771	0.828	0.231	1.267	1.398	1.455	-0.295	2.173	0.837	-2.564	0.770	0.000	0.540	2.189	1.287	0.000	1.345	1.311	-1.151	0.000	0.861	0.869	0.984	1.359	1.562	1.105	0.963
-1	0.295	0.832	1.399	1.222	-0.517	2.480	0.013	1.591	0.000	2.289	0.436	0.287	2.215	1.995	-0.367	-0.409	1.274	0.375	1.367	-1.716	0.000	1.356	2.171	0.990	1.467	1.664	1.855	1.705
-1	1.228	0.339	-0.575	0.417	1.474	0.480	-1.416	-1.498	2.173	0.614	-0.933	-0.961	0.000	1.189	1.690	1.003	0.000	1.690	-1.065	0.106	3.102	0.963	1.147	0.987	1.086	0.948	0.930	0.866
-0	2.877	-1.014	1.440	0.782	0.483	1.134	-0.735	-0.196	2.173	1.123	0.084	-0.596	0.000	1.796	-0.356	1.044	2.548	1.406	1.582	-0.991	0.000	0.939	1.178	1.576	0.996	1.629	1.216	1.280
-1	2.178	0.259	1.107	0.256	1.222	0.979	-0.440	-0.538	1.087	0.496	-0.760	-0.049	0.000	1.471	1.683	-1.486	0.000	0.646	0.695	-1.577	3.102	1.093	1.070	0.984	0.608	0.889	0.962	0.866
-1	0.604	0.592	1.295	0.964	0.348	1.178	-0.016	0.832	2.173	1.626	-0.420	-0.760	0.000	0.748	0.461	-0.906	0.000	0.728	0.309	-1.269	1.551	0.852	0.604	0.989	0.678	0.949	1.021	0.878
-0	0.428	-1.352	-0.912	1.713	0.797	1.894	-1.452	0.191	2.173	2.378	2.113	-1.190	0.000	0.860	2.174	0.949	0.000	1.693	0.759	1.426	3.102	0.885	1.527	1.186	1.090	3.294	4.492	3.676
-0	0.473	0.485	0.154	1.433	-1.504	0.766	1.257	-1.302	2.173	0.414	0.119	0.238	0.000	0.805	0.242	-0.691	2.548	0.734	0.749	0.753	0.000	0.430	0.893	1.137	0.686	0.724	0.618	0.608
-1	0.763	-0.601	0.876	0.182	-1.678	0.818	0.599	0.481	2.173	0.658	-0.737	-0.553	0.000	0.857	-1.138	-1.435	0.000	1.540	-1.466	-0.447	0.000	0.870	0.566	0.989	0.728	0.658	0.821	0.726
-0	0.619	-0.273	-0.143	0.992	-1.267	0.566	0.876	-1.396	2.173	0.515	0.892	0.618	0.000	0.434	-0.902	0.862	2.548	0.490	-0.539	0.549	0.000	0.568	0.794	0.984	0.667	0.867	0.597	0.578
-0	0.793	0.970	0.324	0.570	0.816	0.761	-0.550	1.519	2.173	1.150	0.496	-0.447	0.000	0.925	0.724	1.008	1.274	1.135	-0.275	-0.843	0.000	0.829	1.068	0.978	1.603	0.892	1.041	1.059
-1	0.480	0.364	-0.067	1.906	-1.582	1.397	1.159	0.140	0.000	0.639	0.398	-1.102	0.000	1.597	-0.668	1.607	2.548	1.306	-0.797	0.288	3.102	0.856	1.259	1.297	1.022	1.032	1.049	0.939
-0	0.514	1.304	1.490	1.741	-0.220	0.648	0.155	0.535	0.000	0.562	-1.016	0.837	0.000	0.863	-0.780	-0.815	2.548	1.688	-0.130	-1.545	3.102	0.887	0.980	1.309	1.269	0.654	1.044	1.035
-0	1.225	0.333	0.656	0.893	0.859	1.037	-0.876	1.603	1.087	1.769	0.272	-0.227	2.215	1.000	0.579	-1.690	0.000	1.385	0.471	-0.860	0.000	0.884	1.207	0.995	1.097	2.336	1.282	1.145
-0	2.044	-1.472	-0.294	0.392	0.369	0.927	0.718	1.492	1.087	1.619	-0.736	0.047	2.215	1.884	-0.101	-1.540	0.000	0.548	-0.441	1.117	0.000	0.798	0.877	0.981	0.750	2.272	1.469	1.276
-0	1.037	-0.276	0.735	3.526	1.156	2.498	0.401	-0.590	1.087	0.714	-1.203	1.393	2.215	0.681	0.629	1.534	0.000	0.719	-0.355	-0.706	0.000	0.831	0.857	0.988	2.864	2.633	1.988	1.466
-1	0.651	-1.218	-0.791	0.770	-1.449	0.610	-0.535	0.960	2.173	0.380	-1.072	-0.031	2.215	0.415	2.123	-1.100	0.000	0.776	0.217	0.420	0.000	0.986	1.008	1.001	0.853	0.588	0.799	0.776
-0	1.586	-0.409	0.085	3.258	0.405	1.647	-0.674	-1.519	0.000	0.640	-1.027	-1.681	0.000	1.452	-0.444	-0.957	2.548	0.927	-0.017	1.215	3.102	0.519	0.866	0.992	0.881	0.847	1.018	1.278
-0	0.712	0.092	-0.466	0.688	1.236	0.921	-1.217	-1.022	2.173	2.236	-1.167	0.868	2.215	0.851	-1.892	-0.753	0.000	0.475	-1.216	-0.383	0.000	0.668	0.758	0.988	1.180	2.093	1.157	0.934
-0	0.419	0.471	0.974	2.805	0.235	1.473	-0.198	1.255	1.087	0.931	1.083	-0.712	0.000	1.569	1.358	-1.179	2.548	2.506	0.199	-0.842	0.000	0.929	0.991	0.992	1.732	2.367	1.549	1.430
-1	0.667	1.003	1.504	0.368	1.061	0.885	-0.318	-0.353	0.000	1.438	-1.939	0.710	0.000	1.851	0.277	-1.460	2.548	1.403	0.517	-0.157	0.000	0.883	1.019	1.000	0.790	0.859	0.938	0.841
-1	1.877	-0.492	0.372	0.441	0.955	1.034	-1.220	-0.846	1.087	0.952	-0.320	1.125	0.000	0.542	0.308	-1.261	2.548	1.018	-1.415	-1.547	0.000	1.280	0.932	0.991	1.273	0.878	0.921	0.906
-0	1.052	0.901	1.176	1.280	1.517	0.562	-1.150	-0.079	2.173	1.228	-0.308	-0.354	0.000	0.790	-1.492	-0.963	0.000	0.942	-0.672	-1.588	3.102	1.116	0.902	0.988	1.993	0.765	1.375	1.325
-1	0.518	-0.254	1.642	0.865	0.725	0.980	0.734	0.023	0.000	1.448	0.780	-1.736	2.215	0.955	0.513	-0.519	0.000	0.365	-0.444	-0.243	3.102	0.833	0.555	0.984	0.827	0.795	0.890	0.786
-0	0.870	0.815	-0.506	0.663	-0.518	0.935	0.289	-1.675	2.173	1.188	0.005	0.635	0.000	0.580	0.066	-1.455	2.548	0.580	-0.634	-0.199	0.000	0.852	0.788	0.979	1.283	0.208	0.856	0.950
-0	0.628	1.382	0.135	0.683	0.571	1.097	0.564	-0.950	2.173	0.617	-0.326	0.371	0.000	1.093	0.918	1.667	2.548	0.460	1.221	0.708	0.000	0.743	0.861	0.975	1.067	1.007	0.843	0.762
-0	4.357	0.816	-1.609	1.845	-1.288	3.292	0.726	0.324	2.173	1.528	0.583	-0.801	2.215	0.605	0.572	1.406	0.000	0.794	-0.791	0.122	0.000	0.967	1.132	1.124	3.602	2.811	2.460	1.861
-0	0.677	-1.265	1.559	0.866	-0.618	0.823	0.260	0.185	0.000	1.133	0.337	1.589	2.215	0.563	-0.830	0.510	0.000	0.777	0.117	-0.941	3.102	0.839	0.763	0.986	1.182	0.649	0.796	0.851
-0	2.466	-1.838	-1.648	1.717	1.533	1.676	-1.553	-0.109	2.173	0.670	-0.666	0.284	0.000	0.334	-2.480	0.316	0.000	0.366	-0.804	-1.298	3.102	0.875	0.894	0.997	0.548	0.770	1.302	1.079
-1	1.403	0.129	-1.307	0.688	0.306	0.579	0.753	0.814	1.087	0.474	0.694	-1.400	0.000	0.520	1.995	0.185	0.000	0.929	-0.504	1.270	3.102	0.972	0.998	1.353	0.948	0.650	0.688	0.724
-1	0.351	1.188	-0.360	0.254	-0.346	1.129	0.545	1.691	0.000	0.652	-0.039	-0.258	2.215	1.089	0.655	0.472	2.548	0.554	-0.493	1.366	0.000	0.808	1.045	0.992	0.570	0.649	0.809	0.744
-0	1.875	-0.013	-0.128	0.236	1.163	0.902	0.426	0.590	2.173	1.251	-1.210	-0.616	0.000	1.035	1.534	0.912	0.000	1.944	1.789	-1.691	0.000	0.974	1.113	0.990	0.925	1.120	0.956	0.912
-0	0.298	0.750	-0.507	1.555	1.463	0.804	1.200	-0.665	0.000	0.439	-0.829	-0.252	1.107	0.770	-1.090	0.947	2.548	1.165	-0.166	-0.763	0.000	1.140	0.997	0.988	1.330	0.555	1.005	1.012
-0	0.647	0.342	0.245	4.340	-0.157	2.229	0.068	1.170	2.173	2.133	-0.201	-1.441	0.000	1.467	0.697	-0.532	1.274	1.457	0.583	-1.640	0.000	0.875	1.417	0.976	2.512	2.390	1.794	1.665
-1	1.731	-0.803	-1.013	1.492	-0.020	1.646	-0.541	1.121	2.173	0.459	-1.251	-1.495	2.215	0.605	-1.711	-0.232	0.000	0.658	0.634	-0.068	0.000	1.214	0.886	1.738	1.833	1.024	1.192	1.034
-0	0.515	1.416	-1.089	1.697	1.426	1.414	0.941	0.027	0.000	1.480	0.133	-1.595	2.215	1.110	0.752	0.760	2.548	1.062	0.697	-0.492	0.000	0.851	0.955	0.994	1.105	1.255	1.175	1.095
-0	1.261	0.858	1.465	0.757	0.305	2.310	0.679	1.080	2.173	1.544	2.518	-0.464	0.000	2.326	0.270	-0.841	0.000	2.163	0.839	-0.500	3.102	0.715	0.825	1.170	0.980	2.371	1.527	1.221
-1	1.445	1.509	1.471	0.414	-1.285	0.767	0.864	-0.677	2.173	0.524	1.388	0.171	0.000	0.826	0.190	0.121	2.548	0.572	1.691	-1.603	0.000	0.870	0.935	0.994	0.968	0.735	0.783	0.777
-1	0.919	-0.264	-1.245	0.681	-1.722	1.022	1.010	0.097	2.173	0.685	0.403	-1.351	0.000	1.357	-0.429	1.262	1.274	0.687	1.021	-0.563	0.000	0.953	0.796	0.991	0.873	1.749	1.056	0.917
-1	0.293	-2.258	-1.427	1.191	1.202	0.394	-2.030	1.438	0.000	0.723	0.596	-0.024	2.215	0.525	-1.678	-0.290	0.000	0.788	-0.824	-1.029	3.102	0.821	0.626	0.976	1.080	0.810	0.842	0.771
-0	3.286	0.386	1.688	1.619	-1.620	1.392	-0.009	0.280	0.000	1.179	-0.776	-0.110	2.215	1.256	0.248	-1.114	2.548	0.777	0.825	-0.156	0.000	1.026	1.065	0.964	0.909	1.249	1.384	1.395
-1	1.075	0.603	0.561	0.656	-0.685	0.985	0.175	0.979	2.173	1.154	0.584	-0.886	0.000	1.084	-0.354	-1.004	2.548	0.865	1.224	1.269	0.000	1.346	1.073	1.048	0.873	1.310	1.003	0.865
-1	1.098	-0.091	1.466	1.558	0.915	0.649	1.314	-1.182	2.173	0.791	0.073	0.351	0.000	0.517	0.940	1.195	0.000	1.150	1.187	-0.692	3.102	0.866	0.822	0.980	1.311	0.394	1.119	0.890
-1	0.481	-1.042	0.148	1.135	-1.249	1.202	-0.344	0.308	1.087	0.779	-1.431	1.581	0.000	0.860	-0.860	-1.125	0.000	0.785	0.303	1.199	3.102	0.878	0.853	0.988	1.072	0.827	0.936	0.815
-0	1.348	0.497	0.318	0.806	0.976	1.393	-0.152	0.632	2.173	2.130	0.515	-1.054	0.000	0.908	0.062	-0.780	0.000	1.185	0.687	1.668	1.551	0.720	0.898	0.985	0.683	1.292	1.320	1.131
-0	2.677	-0.420	-1.685	1.828	1.433	2.040	-0.718	-0.039	0.000	0.400	-0.873	0.472	0.000	0.444	0.340	-0.830	2.548	0.431	0.768	-1.417	3.102	0.869	0.917	0.996	0.707	0.193	0.728	1.154
-1	1.300	0.586	-0.122	1.306	0.609	0.727	-0.556	-1.652	2.173	0.636	0.720	1.393	2.215	0.328	1.280	-0.390	0.000	0.386	0.752	-0.905	0.000	0.202	0.751	1.106	0.864	0.799	0.928	0.717
-0	0.637	-0.176	1.737	1.322	-0.414	0.702	-0.964	-0.680	0.000	1.054	-0.461	0.889	2.215	0.861	-0.267	0.225	0.000	1.910	-1.888	1.027	0.000	0.919	0.899	1.186	0.993	1.109	0.862	0.775
-1	0.723	-0.104	1.572	0.428	-0.840	0.655	0.544	1.401	2.173	1.522	-0.154	-0.452	2.215	0.996	0.190	0.273	0.000	1.906	-0.176	0.966	0.000	0.945	0.894	0.990	0.981	1.555	0.988	0.893
-0	2.016	-0.570	1.612	0.798	0.441	0.334	0.191	-0.909	0.000	0.939	0.146	0.021	2.215	0.553	-0.444	1.156	2.548	0.781	-1.545	-0.520	0.000	0.922	0.956	1.528	0.722	0.699	0.778	0.901
-0	1.352	-0.707	1.284	0.665	0.580	0.694	-1.040	-0.899	2.173	0.692	-2.048	0.029	0.000	0.545	-2.042	1.259	0.000	0.661	-0.808	-1.251	3.102	0.845	0.991	0.979	0.662	0.225	0.685	0.769
-1	1.057	-1.561	-0.411	0.952	-0.681	1.236	-1.107	1.045	2.173	1.288	-2.521	-0.521	0.000	1.361	-1.239	1.546	0.000	0.373	-1.540	0.028	0.000	0.794	0.782	0.987	0.889	0.832	0.972	0.828
-0	1.118	-0.017	-1.227	1.077	1.256	0.714	0.624	-0.811	0.000	0.800	0.704	0.387	1.107	0.604	0.234	0.986	0.000	1.306	-0.456	0.094	3.102	0.828	0.984	1.195	0.987	0.672	0.774	0.748
-1	0.602	2.201	0.212	0.119	0.182	0.474	2.130	1.270	0.000	0.370	2.088	-0.573	0.000	0.780	-0.725	-1.033	0.000	1.642	0.598	0.303	3.102	0.886	0.988	0.985	0.644	0.756	0.651	0.599
-0	1.677	-0.844	1.581	0.585	0.887	1.012	-2.315	0.752	0.000	1.077	0.748	-0.195	0.000	0.718	0.832	-1.337	1.274	1.181	-0.557	-1.006	3.102	1.018	1.247	0.988	0.908	0.651	1.311	1.120
-1	1.695	0.259	1.224	1.344	1.067	0.718	-1.752	-0.215	0.000	0.473	0.991	-0.993	0.000	0.891	1.285	-1.500	2.548	0.908	-0.131	0.288	0.000	0.945	0.824	0.979	1.009	0.951	0.934	0.833
-0	0.793	0.628	0.432	1.707	0.302	0.919	1.045	-0.784	0.000	1.472	0.175	-1.284	2.215	1.569	0.155	0.971	2.548	0.435	0.735	1.625	0.000	0.801	0.907	0.992	0.831	1.446	1.082	1.051
-1	0.537	-0.664	-0.244	1.104	1.272	1.154	0.394	1.633	0.000	1.527	0.963	0.559	2.215	1.744	0.650	-0.912	0.000	1.097	0.730	-0.368	3.102	1.953	1.319	1.045	1.309	0.869	1.196	1.126
-1	0.585	-1.469	1.005	0.749	-1.060	1.224	-0.717	-0.323	2.173	1.012	-0.201	1.268	0.000	0.359	-0.567	0.476	0.000	1.117	-1.124	1.557	3.102	0.636	1.281	0.986	0.616	1.289	0.890	0.881
-1	0.354	-1.517	0.667	2.534	-1.298	1.020	-0.375	1.254	0.000	1.119	-0.060	-1.538	2.215	1.059	-0.395	-0.140	0.000	2.609	0.199	-0.778	1.551	0.957	0.975	1.286	1.666	1.003	1.224	1.135
-1	0.691	-1.619	-1.380	0.361	1.727	1.493	-1.093	-0.289	0.000	1.447	-0.640	1.341	0.000	1.453	-0.617	-1.456	1.274	1.061	-1.481	-0.091	0.000	0.744	0.649	0.987	0.596	0.727	0.856	0.797
-0	1.336	1.293	-1.359	0.357	0.067	1.110	-0.058	-0.515	0.000	0.976	1.498	1.207	0.000	1.133	0.437	1.053	2.548	0.543	1.374	0.171	0.000	0.764	0.761	0.984	0.827	0.553	0.607	0.612
-0	0.417	-1.111	1.661	2.209	-0.683	1.931	-0.642	0.959	1.087	1.514	-2.032	-0.686	0.000	1.521	-0.539	1.344	0.000	0.978	-0.866	0.363	1.551	2.813	1.850	1.140	1.854	0.799	1.600	1.556
-0	1.058	0.390	-0.591	0.134	1.149	0.346	-1.550	0.186	0.000	1.108	-0.999	0.843	1.107	1.124	0.415	-1.514	0.000	1.067	-0.426	-1.000	3.102	1.744	1.050	0.985	1.006	1.010	0.883	0.789
-1	1.655	0.253	1.216	0.270	1.703	0.500	-0.006	-1.418	2.173	0.690	-0.350	0.170	2.215	1.045	-0.924	-0.774	0.000	0.996	-0.745	-0.123	0.000	0.839	0.820	0.993	0.921	0.869	0.725	0.708
-0	1.603	-0.850	0.564	0.829	0.093	1.270	-1.113	-1.155	2.173	0.853	-1.021	1.248	2.215	0.617	-1.270	1.733	0.000	0.935	-0.092	0.136	0.000	1.011	1.074	0.977	0.823	1.269	1.054	0.878
-0	1.568	-0.792	1.005	0.545	0.896	0.895	-1.698	-0.988	0.000	0.608	-1.634	1.705	0.000	0.826	0.208	0.618	1.274	2.063	-1.743	-0.520	0.000	0.939	0.986	0.990	0.600	0.435	1.033	1.087
-0	0.489	-1.335	-1.102	1.738	1.028	0.628	-0.992	-0.627	0.000	0.652	-0.064	-0.215	0.000	1.072	0.173	-1.251	2.548	1.042	0.057	0.841	3.102	0.823	0.895	1.200	1.164	0.770	0.837	0.846
-1	1.876	0.870	1.234	0.556	-1.262	1.764	0.855	-0.467	2.173	1.079	1.351	0.852	0.000	0.773	0.383	0.874	0.000	1.292	0.829	-1.228	3.102	0.707	0.969	1.102	1.601	1.017	1.112	1.028
-0	1.033	0.407	-0.374	0.705	-1.254	0.690	-0.231	1.502	2.173	0.433	-2.009	-0.057	0.000	0.861	1.151	0.334	0.000	0.960	-0.839	1.299	3.102	2.411	1.480	0.982	0.995	0.377	1.012	0.994
-0	1.092	0.653	-0.801	0.463	0.426	0.529	-1.055	0.040	0.000	0.663	0.999	1.255	1.107	0.749	-1.106	1.185	2.548	0.841	-0.745	-1.029	0.000	0.841	0.743	0.988	0.750	1.028	0.831	0.868
-1	0.799	-0.285	-0.011	0.531	1.392	1.063	0.854	0.494	2.173	1.187	-1.065	-0.851	0.000	0.429	-0.296	1.072	0.000	0.942	-1.985	1.172	0.000	0.873	0.693	0.992	0.819	0.689	1.131	0.913
-0	0.503	1.973	-0.377	1.515	-1.514	0.708	1.081	-0.313	2.173	1.110	-0.417	0.839	0.000	0.712	-1.153	1.165	0.000	0.675	-0.303	-0.930	1.551	0.709	0.761	1.032	0.986	0.698	0.963	1.291
-0	0.690	-0.574	-1.608	1.182	1.118	0.557	-2.243	0.144	0.000	0.969	0.216	-1.383	1.107	1.054	0.888	-0.709	2.548	0.566	1.663	-0.550	0.000	0.752	1.528	0.987	1.408	0.740	1.290	1.123
-1	0.890	1.501	0.786	0.779	-0.615	1.126	0.716	1.541	2.173	0.887	0.728	-0.673	2.215	1.216	0.332	-0.020	0.000	0.965	1.828	0.101	0.000	0.827	0.715	1.099	1.088	1.339	0.924	0.878
-0	0.566	0.883	0.655	1.600	0.034	1.155	2.028	-1.499	0.000	0.723	-0.871	0.763	0.000	1.286	-0.696	-0.676	2.548	1.134	-0.113	1.207	3.102	4.366	2.493	0.984	0.960	0.962	1.843	1.511
-0	1.146	1.086	-0.911	0.838	1.298	0.821	0.127	-0.145	0.000	1.352	0.474	-1.580	2.215	1.619	-0.081	0.675	2.548	1.382	-0.748	0.127	0.000	0.958	0.976	1.239	0.876	1.481	1.116	1.076
-0	1.739	-0.326	-1.661	0.420	-1.705	1.193	-0.031	-1.212	2.173	1.783	-0.442	0.522	0.000	1.064	-0.692	0.027	0.000	1.314	0.359	-0.037	3.102	0.968	0.897	0.986	0.907	1.196	1.175	1.112
-1	0.669	0.194	-0.703	0.657	-0.260	0.899	-2.511	0.311	0.000	1.482	0.773	0.974	2.215	3.459	0.037	-1.299	1.274	2.113	0.067	1.516	0.000	0.740	0.871	0.979	1.361	2.330	1.322	1.046
-1	1.355	-1.033	-1.173	0.552	-0.048	0.899	-0.482	-1.287	2.173	1.422	-1.227	0.390	1.107	1.937	-0.028	0.914	0.000	0.849	-0.230	-1.734	0.000	0.986	1.224	1.017	1.051	1.788	1.150	1.009
-1	0.511	-0.202	1.029	0.780	1.154	0.816	0.532	-0.731	0.000	0.757	0.517	0.749	2.215	1.302	0.289	-1.188	0.000	0.584	1.211	-0.350	0.000	0.876	0.943	0.995	0.963	0.256	0.808	0.891
-1	1.109	0.572	1.484	0.753	1.543	1.711	-0.145	-0.746	1.087	1.759	0.631	0.845	2.215	0.945	0.542	0.003	0.000	0.378	-1.150	-0.044	0.000	0.764	1.042	0.992	1.045	2.736	1.441	1.140
-0	0.712	-0.025	0.553	0.928	-0.711	1.304	0.045	-0.300	0.000	0.477	0.720	0.969	0.000	1.727	-0.474	1.328	1.274	1.282	2.222	1.684	0.000	0.819	0.765	1.023	0.961	0.657	0.799	0.744
-1	1.131	-0.302	1.079	0.901	0.236	0.904	-0.249	1.694	2.173	1.507	-0.702	-1.128	0.000	0.774	0.565	0.284	2.548	1.802	1.446	-0.192	0.000	3.720	2.108	0.986	0.930	1.101	1.484	1.238
-0	1.392	1.253	0.118	0.864	-1.358	0.922	-0.447	-1.243	1.087	1.969	1.031	0.774	2.215	1.333	-0.359	-0.681	0.000	1.099	-0.257	1.473	0.000	1.246	0.909	1.475	1.234	2.531	1.449	1.306
-0	1.374	2.291	-0.479	1.339	-0.243	0.687	2.345	1.310	0.000	0.467	1.081	0.772	0.000	0.656	1.155	-1.636	2.548	0.592	0.536	-1.269	3.102	0.981	0.821	1.010	0.877	0.217	0.638	0.758
-1	0.401	-1.516	0.909	2.738	0.519	0.887	0.566	-1.202	0.000	0.909	-0.176	1.682	0.000	2.149	-0.878	-0.514	2.548	0.929	-0.563	-1.555	3.102	1.228	0.803	0.980	1.382	0.884	1.025	1.172
-1	0.430	-1.589	1.417	2.158	1.226	1.180	-0.829	-0.781	2.173	0.798	1.400	-0.111	0.000	0.939	-0.878	1.076	2.548	0.576	1.335	-0.826	0.000	0.861	0.970	0.982	1.489	1.308	1.015	0.992
-1	1.943	-0.391	-0.840	0.621	-1.613	2.026	1.734	1.025	0.000	0.930	0.573	-0.912	0.000	1.326	0.847	-0.220	1.274	1.181	0.079	0.709	3.102	1.164	1.007	0.987	1.094	0.821	0.857	0.786
-1	0.499	0.436	0.887	0.859	1.509	0.733	-0.559	1.111	1.087	1.011	-0.796	0.279	2.215	1.472	-0.510	-0.982	0.000	1.952	0.379	-0.733	0.000	1.076	1.358	0.991	0.589	0.879	1.068	0.922
-0	0.998	-0.407	-1.711	0.139	0.652	0.810	-0.331	-0.721	0.000	0.471	-0.533	0.442	0.000	0.531	-1.405	0.120	2.548	0.707	0.098	-1.176	1.551	1.145	0.809	0.988	0.529	0.612	0.562	0.609
-1	1.482	0.872	0.638	1.288	0.362	0.856	0.900	-0.511	1.087	1.072	1.061	-1.432	2.215	1.770	-2.292	-1.547	0.000	1.131	1.374	0.783	0.000	6.316	4.381	1.002	1.317	1.048	2.903	2.351
-1	2.084	-0.422	1.289	1.125	0.735	1.104	-0.518	-0.326	2.173	0.413	-0.719	-0.699	0.000	0.857	0.108	-1.631	0.000	0.527	0.641	-1.362	3.102	0.791	0.952	1.016	0.776	0.856	0.987	0.836
-0	0.464	0.674	0.025	0.430	-1.703	0.982	-1.311	-0.808	2.173	1.875	1.060	0.821	2.215	0.954	-0.480	-1.677	0.000	0.567	0.702	-0.939	0.000	0.781	1.076	0.989	1.256	3.632	1.652	1.252
-1	0.457	-1.944	-1.010	1.409	0.931	1.098	-0.742	-0.415	0.000	1.537	-0.834	0.945	2.215	1.752	-0.287	-1.269	2.548	0.692	-1.537	-0.223	0.000	0.801	1.192	1.094	1.006	1.659	1.175	1.122
-0	3.260	-0.943	1.737	0.920	1.309	0.946	-0.139	-0.271	2.173	0.994	-0.952	-0.311	0.000	0.563	-0.136	-0.881	0.000	1.236	-0.507	0.906	1.551	0.747	0.869	0.985	1.769	1.034	1.179	1.042
-0	0.615	-0.778	0.246	1.861	1.619	0.560	-0.943	-0.204	2.173	0.550	-0.759	-1.342	2.215	0.578	0.076	-0.973	0.000	0.939	0.035	0.680	0.000	0.810	0.747	1.401	0.772	0.702	0.719	0.662
-1	2.370	-0.064	-0.237	1.737	0.154	2.319	-1.838	-1.673	0.000	1.053	-1.305	-0.075	0.000	0.925	0.149	0.318	1.274	0.851	-0.922	0.981	3.102	0.919	0.940	0.989	0.612	0.598	1.219	1.626
-1	1.486	0.311	-1.262	1.354	-0.847	0.886	-0.158	1.213	2.173	1.160	-0.218	0.239	0.000	1.166	0.494	0.278	2.548	0.575	1.454	-1.701	0.000	0.429	1.129	0.983	1.111	1.049	1.006	0.920
-1	1.294	1.587	-0.864	0.487	-0.312	0.828	1.051	-0.031	1.087	2.443	1.216	1.609	2.215	1.167	0.813	0.921	0.000	1.751	-0.415	0.119	0.000	1.015	1.091	0.974	1.357	2.093	1.178	1.059
-1	0.984	0.465	-1.661	0.379	-0.554	0.977	0.237	0.365	0.000	0.510	0.143	1.101	0.000	1.099	-0.662	-1.593	2.548	1.104	-0.197	-0.648	3.102	0.925	0.922	0.986	0.642	0.667	0.806	0.722
-1	0.930	-0.009	0.047	0.667	1.367	1.065	-0.231	0.815	0.000	1.199	-1.114	-0.877	2.215	0.940	0.824	-1.583	0.000	1.052	-0.407	-0.076	1.551	1.843	1.257	1.013	1.047	0.751	1.158	0.941
-0	0.767	-0.011	-0.637	0.341	-1.437	1.438	-0.425	-0.450	2.173	1.073	-0.718	1.341	2.215	0.633	-1.394	0.486	0.000	0.603	-1.945	-1.626	0.000	0.703	0.790	0.984	1.111	1.848	1.129	1.072
-1	1.779	0.017	0.432	0.402	1.022	0.959	1.480	1.595	2.173	1.252	1.365	0.006	0.000	1.188	-0.174	-1.107	0.000	1.181	0.518	-0.258	0.000	1.057	0.910	0.991	1.616	0.779	1.158	1.053
-0	0.881	0.630	1.029	1.990	0.508	1.102	0.742	-1.298	2.173	1.565	1.085	0.686	2.215	2.691	1.391	-0.904	0.000	0.499	1.388	-1.199	0.000	0.347	0.861	0.997	0.881	1.920	1.233	1.310
-0	1.754	-0.266	0.389	0.347	-0.030	0.462	-1.408	-0.957	2.173	0.515	-2.341	-1.700	0.000	0.588	-0.797	1.355	2.548	0.608	0.329	-1.389	0.000	1.406	0.909	0.988	0.760	0.593	0.768	0.847
-0	1.087	0.311	-1.447	0.173	0.567	0.854	0.362	0.584	0.000	1.416	-0.716	-1.211	2.215	0.648	-0.358	-0.692	1.274	0.867	-0.513	0.206	0.000	0.803	0.813	0.984	1.110	0.491	0.921	0.873
-0	0.279	1.114	-1.190	3.004	-0.738	1.233	0.896	1.092	2.173	0.454	-0.374	0.117	2.215	0.357	0.119	1.270	0.000	0.458	1.343	0.316	0.000	0.495	0.540	0.988	1.715	1.139	1.618	1.183
-1	1.773	-0.694	-1.518	2.306	-1.200	3.104	0.749	0.362	0.000	1.871	0.230	-1.686	2.215	0.805	-0.179	-0.871	1.274	0.910	0.607	-0.246	0.000	1.338	1.598	0.984	1.050	0.919	1.678	1.807
-0	0.553	0.683	0.827	0.973	-0.706	1.488	0.149	1.140	2.173	1.788	0.447	-0.478	0.000	0.596	1.043	1.607	0.000	0.373	-0.868	-1.308	1.551	1.607	1.026	0.998	1.134	0.808	1.142	0.936
-1	0.397	1.101	-1.139	1.688	0.146	0.972	0.541	1.518	0.000	1.549	-0.873	-1.012	0.000	2.282	-0.151	0.314	2.548	1.174	0.033	-1.368	0.000	0.937	0.776	1.039	1.143	0.959	0.986	1.013
-1	0.840	1.906	-0.959	0.869	0.576	0.642	0.554	-1.351	0.000	0.756	0.923	-0.823	2.215	1.251	1.130	0.545	2.548	1.513	0.410	1.073	0.000	1.231	0.985	1.163	0.812	0.987	0.816	0.822
-1	0.477	1.665	0.814	0.763	-0.382	0.828	-0.008	0.280	2.173	1.213	-0.001	1.560	0.000	1.136	0.311	-1.289	0.000	0.797	1.091	-0.616	3.102	1.026	0.964	0.992	0.772	0.869	0.916	0.803
-0	2.655	0.020	0.273	1.464	0.482	1.709	-0.107	-1.456	2.173	0.825	0.141	-0.386	0.000	1.342	-0.592	1.635	1.274	0.859	-0.175	-0.874	0.000	0.829	0.946	1.003	2.179	0.836	1.505	1.176
-0	0.771	-1.992	-0.720	0.732	-1.464	0.869	-1.290	0.388	2.173	0.926	-1.072	-1.489	2.215	0.640	-1.232	0.840	0.000	0.528	-2.440	-0.446	0.000	0.811	0.868	0.993	0.995	1.317	0.809	0.714
-0	1.357	1.302	0.076	0.283	-1.060	0.783	1.559	-0.994	0.000	0.947	1.212	1.617	0.000	1.127	0.311	0.442	2.548	0.582	-0.052	1.186	1.551	1.330	0.995	0.985	0.846	0.404	0.858	0.815
-0	0.442	-0.381	-0.424	1.244	0.591	0.731	0.605	-0.713	2.173	0.629	2.762	1.040	0.000	0.476	2.693	-0.617	0.000	0.399	0.442	1.486	3.102	0.839	0.755	0.988	0.869	0.524	0.877	0.918
-0	0.884	0.422	0.055	0.818	0.624	0.950	-0.763	1.624	0.000	0.818	-0.609	-1.166	0.000	1.057	-0.528	1.070	2.548	1.691	-0.124	-0.335	3.102	1.104	0.933	0.985	0.913	1.000	0.863	1.056
-0	1.276	0.156	1.714	1.053	-1.189	0.672	-0.464	-0.030	2.173	0.469	-2.483	0.442	0.000	0.564	2.580	-0.253	0.000	0.444	-0.628	1.080	1.551	5.832	2.983	0.985	1.162	0.494	1.809	1.513
-0	1.106	-0.556	0.406	0.573	-1.400	0.769	-0.518	1.457	2.173	0.743	-0.352	-0.010	0.000	1.469	-0.550	-0.930	2.548	0.540	1.236	-0.571	0.000	0.962	0.970	1.101	0.805	1.107	0.873	0.773
-0	0.539	-0.964	-0.464	1.371	-1.606	0.667	-0.160	0.655	0.000	0.952	0.352	-0.740	2.215	0.952	0.007	1.123	0.000	1.061	-0.505	1.389	3.102	1.063	0.991	1.019	0.633	0.967	0.732	0.799
-1	0.533	-0.989	-1.608	0.462	-1.723	1.204	-0.598	-0.098	2.173	1.343	-0.460	1.632	2.215	0.577	0.221	-0.492	0.000	0.628	-0.073	0.472	0.000	0.518	0.880	0.988	1.179	1.874	1.041	0.813
-1	1.024	1.075	-0.795	0.286	-1.436	1.365	0.857	-0.309	2.173	0.804	1.532	1.435	0.000	1.511	0.722	1.494	0.000	1.778	0.903	0.753	1.551	0.686	0.810	0.999	0.900	1.360	1.133	0.978
-1	2.085	-0.269	-1.423	0.789	1.298	0.281	1.652	0.187	0.000	0.658	-0.760	-0.042	2.215	0.663	0.024	0.120	0.000	0.552	-0.299	-0.428	3.102	0.713	0.811	1.130	0.705	0.218	0.675	0.743
-1	0.980	-0.443	0.813	0.785	-1.253	0.719	0.448	-1.458	0.000	1.087	0.595	0.635	1.107	1.428	0.029	-0.995	0.000	1.083	1.562	-0.092	0.000	0.834	0.891	1.165	0.967	0.661	0.880	0.817
-1	0.903	-0.733	-0.980	0.634	-0.639	0.780	0.266	-0.287	2.173	1.264	-0.936	1.004	0.000	1.002	-0.056	-1.344	2.548	1.183	-0.098	1.169	0.000	0.733	1.002	0.985	0.711	0.916	0.966	0.875
-0	0.734	-0.304	-1.175	2.851	1.674	0.904	-0.634	0.412	2.173	1.363	-1.050	-0.282	0.000	1.476	-1.603	0.103	0.000	2.231	-0.718	1.708	3.102	0.813	0.896	1.088	0.686	1.392	1.033	1.078
-1	1.680	0.591	-0.243	0.111	-0.478	0.326	-0.079	-1.555	2.173	0.711	0.714	0.922	2.215	0.355	0.858	1.682	0.000	0.727	1.620	1.360	0.000	0.334	0.526	1.001	0.862	0.633	0.660	0.619
-1	1.163	0.225	-0.202	0.501	-0.979	1.609	-0.938	1.424	0.000	1.224	-0.118	-1.274	0.000	2.034	1.241	-0.254	0.000	1.765	0.536	0.237	3.102	0.894	0.838	0.988	0.693	0.579	0.762	0.726
-0	1.223	1.232	1.471	0.489	1.728	0.703	-0.111	0.411	0.000	1.367	1.014	-1.294	1.107	1.524	-0.414	-0.164	2.548	1.292	0.833	0.316	0.000	0.861	0.752	0.994	0.836	1.814	1.089	0.950
-0	0.816	1.637	-1.557	1.036	-0.342	0.913	1.333	0.949	2.173	0.812	0.756	-0.628	2.215	1.333	0.470	1.495	0.000	1.204	-2.222	-1.675	0.000	1.013	0.924	1.133	0.758	1.304	0.855	0.860
-0	0.851	-0.564	-0.691	0.692	1.345	1.219	1.014	0.318	0.000	1.422	-0.262	-1.635	2.215	0.531	1.802	0.008	0.000	0.508	0.515	-1.267	3.102	0.821	0.787	1.026	0.783	0.432	1.149	1.034
-0	0.800	-0.599	0.204	0.552	-0.484	0.974	0.413	0.961	2.173	1.269	-0.984	-1.039	2.215	0.380	-1.213	1.371	0.000	0.551	0.332	-0.659	0.000	0.694	0.852	0.984	1.057	2.037	1.096	0.846
-0	0.744	-0.071	-0.255	0.638	0.512	1.125	0.407	0.844	2.173	0.860	-0.481	-0.677	0.000	1.102	0.181	-1.194	0.000	1.011	-1.081	-1.713	3.102	0.854	0.862	0.982	1.111	1.372	1.042	0.920
-1	0.400	1.049	-0.625	0.880	-0.407	1.040	2.150	-1.359	0.000	0.747	-0.144	0.847	2.215	0.560	-1.829	0.698	0.000	1.663	-0.668	0.267	0.000	0.845	0.964	0.996	0.820	0.789	0.668	0.668
-0	1.659	-0.705	-1.057	1.803	-1.436	1.008	0.693	0.005	0.000	0.895	-0.007	0.681	1.107	1.085	0.125	1.476	2.548	1.214	1.068	0.486	0.000	0.867	0.919	0.986	1.069	0.692	1.026	1.313
-0	0.829	-0.153	0.861	0.615	-0.548	0.589	1.077	-0.041	2.173	1.056	0.763	-1.737	0.000	0.639	0.970	0.725	0.000	0.955	1.227	-0.799	3.102	1.020	1.024	0.985	0.750	0.525	0.685	0.671
-1	0.920	-0.806	-0.840	1.048	0.278	0.973	-0.077	-1.364	2.173	1.029	0.309	0.133	0.000	1.444	1.484	1.618	1.274	1.419	-0.482	0.417	0.000	0.831	1.430	1.151	1.829	1.560	1.343	1.224
-1	0.686	0.249	-0.905	0.343	-1.731	0.724	-2.823	-0.901	0.000	0.982	0.303	1.312	1.107	1.016	0.245	0.610	0.000	1.303	-0.557	-0.360	3.102	1.384	1.030	0.984	0.862	1.144	0.866	0.779
-0	1.603	0.444	0.508	0.586	0.401	0.610	0.467	-1.735	2.173	0.914	0.626	-1.019	0.000	0.812	0.422	-0.408	2.548	0.902	1.679	1.490	0.000	1.265	0.929	0.990	1.004	0.816	0.753	0.851
-1	0.623	0.780	-0.203	0.056	0.015	0.899	0.793	1.326	1.087	0.803	1.478	-1.499	2.215	1.561	1.492	-0.120	0.000	0.904	0.795	0.137	0.000	0.548	1.009	0.850	0.924	0.838	0.914	0.860
-0	1.654	-2.032	-1.160	0.859	-1.583	0.689	-1.965	0.891	0.000	0.646	-1.014	-0.288	2.215	0.630	-0.815	0.402	0.000	0.638	0.316	0.655	3.102	0.845	0.879	0.993	1.067	0.625	1.041	0.958
-1	0.828	-1.269	-1.203	0.744	-0.213	0.626	-1.017	-0.404	0.000	1.281	-0.931	1.733	2.215	0.699	-0.351	1.287	0.000	1.251	-1.171	0.197	0.000	0.976	1.186	0.987	0.646	0.655	0.733	0.671
-1	0.677	0.111	1.090	1.580	1.591	1.560	0.654	-0.341	2.173	0.794	-0.266	0.702	0.000	0.823	0.651	-1.239	2.548	0.730	1.467	-1.530	0.000	1.492	1.023	0.983	1.909	1.022	1.265	1.127
-1	0.736	0.882	-1.060	0.589	0.168	1.663	0.781	1.022	2.173	2.025	1.648	-1.292	0.000	1.240	0.924	-0.421	1.274	1.354	0.065	0.501	0.000	0.316	0.925	0.988	0.664	1.736	0.992	0.807
-1	1.040	-0.822	1.638	0.974	-0.674	0.393	0.830	0.011	2.173	0.770	-0.140	-0.402	0.000	0.294	-0.133	0.030	0.000	1.220	0.807	0.638	0.000	0.826	1.063	1.216	1.026	0.705	0.934	0.823
-1	0.711	0.602	0.048	1.145	0.966	0.934	0.263	-1.589	2.173	0.971	-0.496	-0.421	1.107	0.628	-0.865	0.845	0.000	0.661	-0.008	-0.565	0.000	0.893	0.705	0.988	0.998	1.339	0.908	0.872
-1	0.953	-1.651	-0.167	0.885	1.053	1.013	-1.239	0.133	0.000	1.884	-1.122	1.222	2.215	1.906	-0.860	-1.184	1.274	1.413	-0.668	-1.647	0.000	1.873	1.510	1.133	1.050	1.678	1.246	1.061
-1	0.986	-0.892	-1.380	0.917	1.134	0.950	-1.162	-0.469	0.000	0.569	-1.393	0.215	0.000	0.320	2.667	1.712	0.000	1.570	-0.375	1.457	3.102	0.925	1.128	1.011	0.598	0.824	0.913	0.833
-1	1.067	0.099	1.154	0.527	-0.789	1.085	0.623	-1.602	2.173	1.511	-0.230	0.022	2.215	0.269	-0.377	0.883	0.000	0.571	-0.540	-0.512	0.000	0.414	0.803	1.022	0.959	2.053	1.041	0.780
-0	0.825	-2.118	0.217	1.453	-0.493	0.819	0.313	-0.942	0.000	2.098	-0.725	1.096	2.215	0.484	1.336	1.458	0.000	0.482	0.100	1.163	0.000	0.913	0.536	0.990	1.679	0.957	1.095	1.143
-1	1.507	0.054	1.120	0.698	-1.340	0.912	0.384	0.015	1.087	0.720	0.247	-0.820	0.000	0.286	0.154	1.578	2.548	0.629	1.582	-0.576	0.000	0.828	0.893	1.136	0.514	0.632	0.699	0.709
-1	0.610	1.180	-0.993	0.816	0.301	0.932	0.758	1.539	0.000	0.726	-0.830	0.248	2.215	0.883	0.857	-1.305	0.000	1.338	1.009	-0.252	3.102	0.901	1.074	0.987	0.875	1.159	1.035	0.858
-1	1.247	-1.360	1.502	1.525	-1.332	0.618	1.063	0.755	0.000	0.582	-0.155	0.473	2.215	1.214	-0.422	-0.551	2.548	0.838	-1.171	-1.166	0.000	2.051	1.215	1.062	1.091	0.725	0.896	1.091
-0	0.373	-0.600	1.291	2.573	0.207	0.765	-0.209	1.667	0.000	0.668	0.724	-1.499	0.000	1.045	-0.338	-0.754	2.548	0.558	-0.469	0.029	3.102	0.868	0.939	1.124	0.519	0.383	0.636	0.838
-0	0.791	0.336	-0.307	0.494	1.213	1.158	0.336	1.081	2.173	0.918	1.289	-0.449	0.000	0.735	-0.521	-0.969	0.000	1.052	0.499	-1.188	3.102	0.699	1.013	0.987	0.622	1.050	0.712	0.661
-0	1.321	0.856	0.464	0.202	0.901	1.144	0.120	-1.651	0.000	0.803	0.577	-0.509	2.215	0.695	-0.114	0.423	2.548	0.621	1.852	-0.420	0.000	0.697	0.964	0.983	0.527	0.659	0.719	0.729
-0	0.563	2.081	0.913	0.982	-0.533	0.549	-0.481	-1.730	0.000	0.962	0.921	0.569	2.215	0.731	1.184	-0.679	1.274	0.918	0.931	-1.432	0.000	1.008	0.919	0.993	0.895	0.819	0.810	0.878
-1	1.148	0.345	0.953	0.921	0.617	0.991	1.103	-0.484	0.000	0.970	1.978	1.525	0.000	1.150	0.689	-0.757	2.548	0.517	0.995	1.245	0.000	1.093	1.140	0.998	1.006	0.756	0.864	0.838
-1	1.400	0.128	-1.695	1.169	1.070	1.094	-0.345	-0.249	0.000	1.224	0.364	-0.036	2.215	1.178	0.530	-1.544	0.000	1.334	0.933	1.604	0.000	0.560	1.267	1.073	0.716	0.780	0.832	0.792
-0	0.330	-2.133	1.403	0.628	0.379	1.686	-0.995	0.030	1.087	2.071	0.127	-0.457	0.000	4.662	-0.855	1.477	0.000	2.072	-0.917	-1.416	3.102	5.403	3.074	0.977	0.936	1.910	2.325	1.702
-0	0.989	0.473	0.968	1.970	1.368	0.844	0.574	-0.290	2.173	0.866	-0.345	-1.019	0.000	1.130	0.605	-0.752	0.000	0.956	-0.888	0.870	3.102	0.885	0.886	0.982	1.157	1.201	1.100	1.068
-1	0.773	0.418	0.753	1.388	1.070	1.104	-0.378	-0.758	0.000	1.027	0.397	-0.496	2.215	1.234	0.027	1.084	2.548	0.936	0.209	1.677	0.000	1.355	1.020	0.983	0.550	1.206	0.916	0.931
-0	0.319	2.015	1.534	0.570	-1.134	0.632	0.124	0.757	0.000	0.477	0.598	-1.109	1.107	0.449	0.438	-0.755	2.548	0.574	-0.659	0.691	0.000	0.440	0.749	0.985	0.517	0.158	0.505	0.522
-0	1.215	1.453	-1.386	1.276	1.298	0.643	0.570	-0.196	2.173	0.588	2.104	0.498	0.000	0.617	-0.296	-0.801	2.548	0.452	0.110	0.313	0.000	0.815	0.953	1.141	1.166	0.547	0.892	0.807
-1	1.257	-1.869	-0.060	0.265	0.653	1.527	-0.346	1.163	2.173	0.758	-2.119	-0.604	0.000	1.473	-1.133	-1.290	2.548	0.477	-0.428	-0.066	0.000	0.818	0.841	0.984	1.446	1.729	1.211	1.054
-1	1.449	0.464	1.585	1.418	-1.488	1.540	0.942	0.087	0.000	0.898	0.402	-0.631	2.215	0.753	0.039	-1.729	0.000	0.859	0.849	-1.054	0.000	0.791	0.677	0.995	0.687	0.527	0.703	0.606
-1	1.084	-1.997	0.900	1.333	1.024	0.872	-0.864	-1.500	2.173	1.072	-0.813	-0.421	2.215	0.924	0.478	0.304	0.000	0.992	-0.398	-1.022	0.000	0.741	1.085	0.980	1.221	1.176	1.032	0.961
-0	1.712	1.129	0.125	1.120	-1.402	1.749	0.951	-1.575	2.173	1.711	0.445	0.578	0.000	1.114	0.234	-1.011	0.000	1.577	-0.088	0.086	3.102	2.108	1.312	1.882	1.597	2.009	1.441	1.308
-0	0.530	0.248	1.622	1.450	-1.012	1.221	-1.154	-0.763	2.173	1.698	-0.586	0.733	0.000	0.889	1.042	1.038	1.274	0.657	0.008	0.701	0.000	0.430	1.005	0.983	0.930	2.264	1.357	1.146
-1	0.921	1.735	0.883	0.699	-1.614	0.821	1.463	0.319	1.087	1.099	0.814	-1.600	2.215	1.375	0.702	-0.691	0.000	0.869	1.326	-0.790	0.000	0.980	0.900	0.988	0.832	1.452	0.816	0.709
-0	2.485	-0.823	-0.297	0.886	-1.404	0.989	0.835	1.615	2.173	0.382	0.588	-0.224	0.000	1.029	-0.456	1.546	2.548	0.613	-0.359	-0.789	0.000	0.768	0.977	1.726	2.007	0.913	1.338	1.180
-1	0.657	-0.069	-0.078	1.107	1.549	0.804	1.335	-1.630	2.173	1.271	0.481	0.153	1.107	1.028	0.144	-0.762	0.000	1.098	0.132	1.570	0.000	0.830	0.979	1.175	1.069	1.624	1.000	0.868
-1	2.032	0.329	-1.003	0.493	-0.136	1.159	-0.224	0.750	1.087	0.396	0.546	0.587	0.000	0.620	1.805	0.982	0.000	1.236	0.744	-1.621	0.000	0.930	1.200	0.988	0.482	0.771	0.887	0.779
-0	0.524	-1.319	0.634	0.471	1.221	0.599	-0.588	-0.461	0.000	1.230	-1.504	-1.517	1.107	1.436	-0.035	0.104	2.548	0.629	1.997	-1.282	0.000	2.084	1.450	0.984	1.084	1.827	1.547	1.213
-1	0.871	0.618	-1.544	0.718	0.186	1.041	-1.180	0.434	2.173	1.133	1.558	-1.301	0.000	0.452	-0.595	0.522	0.000	0.665	0.567	0.130	3.102	1.872	1.114	1.095	1.398	0.979	1.472	1.168
-1	3.308	1.037	-0.634	0.690	-0.619	1.975	0.949	1.280	0.000	0.826	0.546	-0.139	2.215	0.635	-0.045	0.427	0.000	1.224	0.112	1.339	3.102	1.756	1.050	0.992	0.738	0.903	0.968	1.238
-0	0.588	2.104	-0.872	1.136	1.743	0.842	0.638	0.015	0.000	0.481	0.928	1.000	2.215	0.595	0.125	1.429	0.000	0.951	-1.140	-0.511	3.102	1.031	1.057	0.979	0.673	1.064	1.001	0.891
-0	0.289	0.823	0.013	0.615	-1.601	0.177	2.403	-0.015	0.000	0.258	1.151	1.036	2.215	0.694	0.553	-1.326	2.548	0.411	0.366	0.106	0.000	0.482	0.562	0.989	0.670	0.404	0.516	0.561
-1	0.294	-0.660	-1.162	1.752	0.384	0.860	0.513	1.119	0.000	2.416	0.107	-1.342	0.000	1.398	0.361	-0.350	2.548	1.126	-0.902	0.040	1.551	0.650	1.125	0.988	0.531	0.843	0.912	0.911
-0	0.599	-0.616	1.526	1.381	0.507	0.955	-0.646	-0.085	2.173	0.775	-0.533	1.116	2.215	0.789	-0.136	-1.176	0.000	2.449	1.435	-1.433	0.000	1.692	1.699	1.000	0.869	1.119	1.508	1.303
-1	1.100	-1.174	-1.114	1.601	-1.576	1.056	-1.343	0.547	2.173	0.555	0.367	0.592	2.215	0.580	-1.862	-0.914	0.000	0.904	0.508	-0.444	0.000	1.439	1.105	0.986	1.408	1.104	1.190	1.094
-1	2.237	-0.701	1.470	0.719	-0.199	0.745	-0.132	-0.737	1.087	0.976	-0.227	0.093	2.215	0.699	0.057	1.133	0.000	0.661	0.573	-0.679	0.000	0.785	0.772	1.752	1.235	0.856	0.990	0.825
-1	0.455	-0.880	-1.482	1.260	-0.178	1.499	0.158	1.022	0.000	1.867	-0.435	-0.675	2.215	1.234	0.783	1.586	0.000	0.641	-0.454	-0.409	3.102	1.002	0.964	0.986	0.761	0.240	1.190	0.995
-1	1.158	-0.778	-0.159	0.823	1.641	1.341	-0.830	-1.169	2.173	0.840	-1.554	0.934	0.000	0.693	0.488	-1.218	2.548	1.042	1.395	0.276	0.000	0.946	0.785	1.350	1.079	0.893	1.267	1.151
-1	0.902	-0.078	-0.055	0.872	-0.012	0.843	1.276	1.739	2.173	0.838	1.492	0.918	0.000	0.626	0.904	-0.648	2.548	0.412	-2.027	-0.883	0.000	2.838	1.664	0.988	1.803	0.768	1.244	1.280
-1	0.649	-1.028	-1.521	1.097	0.774	1.216	-0.383	-0.318	2.173	1.643	-0.285	-1.705	0.000	0.911	-0.091	0.341	0.000	0.592	0.537	0.732	3.102	0.911	0.856	1.027	1.160	0.874	0.986	0.893
-1	1.192	1.846	-0.781	1.326	-0.747	1.550	1.177	1.366	0.000	1.196	0.151	0.387	2.215	0.527	2.261	-0.190	0.000	0.390	1.474	0.381	0.000	0.986	1.025	1.004	1.392	0.761	0.965	1.043
-0	0.438	-0.358	-1.549	0.836	0.436	0.818	0.276	-0.708	2.173	0.707	0.826	0.392	0.000	1.050	1.741	-1.066	0.000	1.276	-1.583	0.842	0.000	1.475	1.273	0.986	0.853	1.593	1.255	1.226
-1	1.083	0.142	1.701	0.605	-0.253	1.237	0.791	1.183	2.173	0.842	2.850	-0.082	0.000	0.724	-0.464	-0.694	0.000	1.499	0.456	-0.226	3.102	0.601	0.799	1.102	0.995	1.389	1.013	0.851
-0	0.828	1.897	-0.615	0.572	-0.545	0.572	0.461	0.464	2.173	0.393	0.356	1.069	2.215	1.840	0.088	1.500	0.000	0.407	-0.663	-0.787	0.000	0.950	0.965	0.979	0.733	0.363	0.618	0.733
-0	0.735	1.438	1.197	1.123	-0.214	0.641	0.949	0.858	0.000	1.162	0.524	-0.896	2.215	0.992	0.454	-1.475	2.548	0.902	1.079	0.019	0.000	0.822	0.917	1.203	1.032	0.569	0.780	0.764
-0	0.437	-2.102	0.044	1.779	-1.042	1.231	-0.181	-0.515	1.087	2.666	0.863	1.466	2.215	1.370	0.345	-1.371	0.000	0.906	0.363	1.611	0.000	1.140	1.362	1.013	3.931	3.004	2.724	2.028
-1	0.881	1.814	-0.987	0.384	0.800	2.384	1.422	0.640	0.000	1.528	0.292	-0.962	1.107	2.126	-0.371	-1.401	2.548	0.700	0.109	0.203	0.000	0.450	0.813	0.985	0.956	1.013	0.993	0.774
-1	0.630	0.408	0.152	0.194	0.316	0.710	-0.824	-0.358	2.173	0.741	0.535	-0.851	2.215	0.933	0.406	1.148	0.000	0.523	-0.479	-0.625	0.000	0.873	0.960	0.988	0.830	0.921	0.711	0.661
-1	0.870	-0.448	-1.134	0.616	0.135	0.600	0.649	-0.622	2.173	0.768	0.709	-0.123	0.000	1.308	0.500	1.468	0.000	1.973	-0.286	1.462	3.102	0.909	0.944	0.990	0.835	1.250	0.798	0.776
-0	1.290	0.552	1.330	0.615	-1.353	0.661	0.240	-0.393	0.000	0.531	0.053	-1.588	0.000	0.675	0.839	-0.345	1.274	1.597	0.020	0.536	3.102	1.114	0.964	0.987	0.783	0.675	0.662	0.675
-1	0.943	0.936	1.068	1.373	0.671	2.170	-2.011	-1.032	0.000	0.640	0.361	-0.806	0.000	2.239	-0.083	0.590	2.548	1.224	0.646	-1.723	0.000	0.879	0.834	0.981	1.436	0.568	0.916	0.931
-1	0.431	1.686	-1.053	0.388	1.739	0.457	-0.471	-0.743	2.173	0.786	1.432	-0.547	2.215	0.537	-0.413	1.256	0.000	0.413	2.311	-0.408	0.000	1.355	1.017	0.982	0.689	1.014	0.821	0.715
-0	1.620	-0.055	-0.862	1.341	-1.571	0.634	-0.906	0.935	2.173	0.501	-2.198	-0.525	0.000	0.778	-0.708	-0.060	0.000	0.988	-0.621	0.489	3.102	0.870	0.956	1.216	0.992	0.336	0.871	0.889
-1	0.549	0.304	-1.443	1.309	-0.312	1.116	0.644	1.519	2.173	1.078	-0.303	-0.736	0.000	1.261	0.387	0.628	2.548	0.945	-0.190	0.090	0.000	0.893	1.043	1.000	1.124	1.077	1.026	0.886
-0	0.412	-0.618	-1.486	1.133	-0.665	0.646	0.436	1.520	0.000	0.993	0.976	0.106	2.215	0.832	0.091	0.164	2.548	0.672	-0.650	1.256	0.000	0.695	1.131	0.991	1.017	0.455	1.226	1.087
-0	1.183	-0.084	1.644	1.389	0.967	0.843	0.938	-0.670	0.000	0.480	0.256	0.123	2.215	0.437	1.644	0.491	0.000	0.501	-0.416	0.101	3.102	1.060	0.804	1.017	0.775	0.173	0.535	0.760
-0	1.629	-1.486	-0.683	2.786	-0.492	1.347	-2.638	1.453	0.000	1.857	0.208	0.873	0.000	0.519	-1.265	-1.602	1.274	0.903	-1.102	-0.329	1.551	6.892	3.522	0.998	0.570	0.477	2.039	2.006
-1	2.045	-0.671	-1.235	0.490	-0.952	0.525	-1.252	1.289	0.000	1.088	-0.993	0.648	2.215	0.975	-0.109	-0.254	2.548	0.556	-1.095	-0.194	0.000	0.803	0.861	0.980	1.282	0.945	0.925	0.811
-0	0.448	-0.058	-0.974	0.945	-1.633	1.181	-1.139	0.266	2.173	1.118	-0.761	1.502	1.107	1.706	0.585	-0.680	0.000	0.487	-1.951	0.945	0.000	2.347	1.754	0.993	1.161	1.549	1.414	1.176
-0	0.551	0.519	0.448	2.183	1.293	1.220	0.628	-0.627	2.173	1.019	-0.002	-0.652	0.000	1.843	-0.386	1.042	2.548	0.400	-1.102	-1.014	0.000	0.648	0.792	1.049	0.888	2.132	1.262	1.096
-0	1.624	0.488	1.403	0.760	0.559	0.812	0.777	-1.244	2.173	0.613	0.589	-0.030	2.215	0.692	1.058	0.683	0.000	1.054	1.165	-0.765	0.000	0.915	0.875	1.059	0.821	0.927	0.792	0.721
-1	0.774	0.444	1.257	0.515	-0.689	0.515	1.448	-1.271	0.000	0.793	0.118	0.811	1.107	0.679	0.326	-0.426	0.000	1.066	-0.865	-0.049	3.102	0.960	1.046	0.986	0.716	0.772	0.855	0.732
-1	2.093	-1.240	1.615	0.918	-1.202	1.412	-0.541	0.640	1.087	2.019	0.872	-0.639	0.000	0.672	-0.936	0.972	0.000	0.896	0.235	0.212	0.000	0.810	0.700	1.090	0.797	0.862	1.049	0.874
-1	0.908	1.069	0.283	0.400	1.293	0.609	1.452	-1.136	0.000	0.623	0.417	-0.098	2.215	1.023	0.775	1.054	1.274	0.706	2.346	-1.305	0.000	0.744	1.006	0.991	0.606	0.753	0.796	0.753
-0	0.403	-1.328	-0.065	0.901	1.052	0.708	-0.354	-0.718	2.173	0.892	0.633	1.684	2.215	0.999	-1.205	0.941	0.000	0.930	1.072	-0.809	0.000	2.105	1.430	0.989	0.838	1.147	1.042	0.883
-0	1.447	0.453	0.118	1.731	0.650	0.771	0.446	-1.564	0.000	0.973	-2.014	0.354	0.000	1.949	-0.643	-1.531	1.274	1.106	-0.334	-1.163	0.000	0.795	0.821	1.013	1.699	0.918	1.118	1.018
-1	1.794	0.123	-0.454	0.057	1.489	0.966	-1.190	1.090	1.087	0.539	-0.535	1.035	0.000	1.096	-1.069	-1.236	2.548	0.659	-1.196	-0.283	0.000	0.803	0.756	0.985	1.343	1.109	0.993	0.806
-0	1.484	-2.047	0.813	0.591	-0.295	0.923	0.312	-1.164	2.173	0.654	-0.316	0.752	2.215	0.599	1.966	-1.128	0.000	0.626	-0.304	-1.431	0.000	1.112	0.910	1.090	0.986	1.189	1.350	1.472
-0	0.417	-2.016	0.849	1.817	0.040	1.201	-1.676	-1.394	0.000	0.792	0.537	0.641	2.215	0.794	-1.222	0.187	0.000	0.825	-0.217	1.334	3.102	1.470	0.931	0.987	1.203	0.525	0.833	0.827
-1	0.603	1.009	0.033	0.486	1.225	0.884	-0.617	-1.058	0.000	0.500	-1.407	-0.567	0.000	1.476	-0.876	0.605	2.548	0.970	0.560	1.092	3.102	0.853	1.153	0.988	0.846	0.920	0.944	0.835
-1	1.381	-0.326	0.552	0.417	-0.027	1.030	-0.835	-1.287	2.173	0.941	-0.421	1.519	2.215	0.615	-1.650	0.377	0.000	0.606	0.644	0.650	0.000	1.146	0.970	0.990	1.191	0.884	0.897	0.826
-1	0.632	1.200	-0.703	0.438	-1.700	0.779	-0.731	0.958	1.087	0.605	0.393	-1.376	0.000	0.670	-0.827	-1.315	2.548	0.626	-0.501	0.417	0.000	0.904	0.903	0.998	0.673	0.803	0.722	0.640
-1	1.561	-0.569	1.580	0.329	0.237	1.059	0.731	0.415	2.173	0.454	0.016	-0.828	0.000	0.587	0.008	-0.291	1.274	0.597	1.119	1.191	0.000	0.815	0.908	0.988	0.733	0.690	0.892	0.764
-1	2.102	0.087	0.449	1.164	-0.390	1.085	-0.408	-1.116	2.173	0.578	0.197	-0.137	0.000	1.202	0.917	1.523	0.000	0.959	-0.832	1.404	3.102	1.380	1.109	1.486	1.496	0.886	1.066	1.025
-1	1.698	-0.489	-0.552	0.976	-1.009	1.620	-0.721	0.648	1.087	1.481	-1.860	-1.354	0.000	1.142	-1.140	1.401	2.548	1.000	-1.274	-0.158	0.000	1.430	1.130	0.987	1.629	1.154	1.303	1.223
-1	1.111	-0.249	-1.457	0.421	0.939	0.646	-2.076	0.362	0.000	1.315	0.796	-1.436	2.215	0.780	0.130	0.055	0.000	1.662	-0.834	0.461	0.000	0.920	0.948	0.990	1.046	0.905	1.493	1.169
-1	0.945	0.390	-1.159	1.675	0.437	0.356	0.261	0.543	1.087	0.574	0.838	1.599	2.215	0.496	-1.220	-0.022	0.000	0.558	-2.454	1.440	0.000	0.763	0.983	1.728	1.000	0.578	0.922	1.003
-1	2.076	0.014	-1.314	0.854	-0.306	3.446	1.341	0.598	0.000	2.086	0.227	-0.747	2.215	1.564	-0.216	1.649	2.548	0.965	-0.857	-1.062	0.000	0.477	0.734	1.456	1.003	1.660	1.001	0.908
-1	1.992	0.192	-0.103	0.108	-1.599	0.938	0.595	-1.360	2.173	0.869	-1.012	1.432	0.000	1.302	0.850	0.436	2.548	0.487	1.051	-1.027	0.000	0.502	0.829	0.983	1.110	1.394	0.904	0.836
-0	0.460	1.625	1.485	1.331	1.242	0.675	-0.329	-1.039	1.087	0.671	-1.028	-0.514	0.000	1.265	-0.788	0.415	1.274	0.570	-0.683	-1.738	0.000	0.725	0.758	1.004	1.024	1.156	0.944	0.833
-0	0.871	0.839	-1.536	0.428	1.198	0.875	-1.256	-0.466	1.087	0.684	-0.768	0.150	0.000	0.556	-1.793	0.389	0.000	0.942	-1.126	1.339	1.551	0.624	0.734	0.986	1.357	0.960	1.474	1.294
-1	0.951	1.651	0.576	1.273	1.495	0.834	0.048	-0.578	2.173	0.386	-0.056	-1.448	0.000	0.597	-0.196	0.162	2.548	0.524	1.649	1.625	0.000	0.737	0.901	1.124	1.014	0.556	1.039	0.845
-1	1.049	-0.223	0.685	0.256	-1.191	2.506	0.238	-0.359	0.000	1.510	-0.904	1.158	1.107	2.733	-0.902	1.679	2.548	0.407	-0.474	-1.572	0.000	1.513	2.472	0.982	1.238	0.978	1.985	1.510
-0	0.455	-0.028	0.265	1.286	1.373	0.459	0.331	-0.922	0.000	0.343	0.634	0.430	0.000	0.279	-0.084	-0.272	0.000	0.475	0.926	-0.123	3.102	0.803	0.495	0.987	0.587	0.211	0.417	0.445
-1	2.074	0.388	0.878	1.110	1.557	1.077	-0.226	-0.295	2.173	0.865	-0.319	-1.116	2.215	0.707	-0.835	0.722	0.000	0.632	-0.608	-0.728	0.000	0.715	0.802	1.207	1.190	0.960	1.143	0.926
-1	1.390	0.265	1.196	0.919	-1.371	1.858	0.506	0.786	0.000	1.280	-1.367	-0.720	2.215	1.483	-0.441	-0.675	2.548	1.076	0.294	-0.539	0.000	1.126	0.830	1.155	1.551	0.702	1.103	0.933
-1	1.014	-0.079	1.597	1.038	-0.281	1.135	-0.722	-0.177	2.173	0.544	-1.475	-1.501	0.000	1.257	-1.315	1.212	0.000	0.496	-0.060	1.180	1.551	0.815	0.611	1.411	1.110	0.792	0.846	0.853
-0	0.335	1.267	-1.154	2.011	-0.574	0.753	0.618	1.411	0.000	0.474	0.748	0.681	2.215	0.608	-0.446	-0.354	2.548	0.399	1.295	-0.581	0.000	0.911	0.882	0.975	0.832	0.598	0.580	0.678
-1	0.729	-0.189	1.182	0.293	1.310	0.412	0.459	-0.632	0.000	0.869	-1.128	-0.625	2.215	1.173	-0.893	0.478	2.548	0.584	-2.394	-1.727	0.000	2.016	1.272	0.995	1.034	0.905	0.966	1.038
-1	1.225	-1.215	-0.088	0.881	-0.237	0.600	-0.976	1.462	2.173	0.876	0.506	1.583	2.215	0.718	1.228	-0.031	0.000	0.653	-1.292	1.216	0.000	0.838	1.108	0.981	1.805	0.890	1.251	1.197
-1	2.685	-0.444	0.847	0.253	0.183	0.641	-1.541	-0.873	2.173	0.417	2.874	-0.551	0.000	0.706	-1.431	0.764	0.000	1.390	-0.596	-1.397	0.000	0.894	0.829	0.993	0.789	0.654	0.883	0.746
-0	0.638	-0.481	0.683	1.457	-1.024	0.707	-1.338	1.498	0.000	0.980	0.518	0.289	2.215	0.964	-0.531	-0.423	0.000	0.694	-0.654	-1.314	3.102	0.807	1.283	1.335	0.658	0.907	0.797	0.772
-1	1.789	-0.765	-0.732	0.421	-0.020	1.142	-1.353	1.439	2.173	0.725	-1.518	-1.261	0.000	0.812	-2.597	-0.463	0.000	1.203	-0.120	1.001	0.000	0.978	0.673	0.985	1.303	1.400	1.078	0.983
-1	0.784	-1.431	1.724	0.848	0.559	0.615	-1.643	-1.456	0.000	1.339	-0.513	0.040	2.215	0.394	-2.483	1.304	0.000	0.987	0.889	-0.339	0.000	0.732	0.713	0.987	0.973	0.705	0.875	0.759
-1	0.911	1.098	-1.289	0.421	0.823	1.218	-0.503	0.431	0.000	0.775	0.432	-1.680	0.000	0.855	-0.226	-0.460	2.548	0.646	-0.947	-1.243	1.551	2.201	1.349	0.985	0.730	0.451	0.877	0.825
-1	0.959	0.372	-0.269	1.255	0.702	1.151	0.097	0.805	2.173	0.993	1.011	0.767	2.215	1.096	0.185	0.381	0.000	1.001	-0.205	0.059	0.000	0.979	0.997	1.168	0.796	0.771	0.839	0.776
-0	0.283	-1.864	-1.663	0.219	1.624	0.955	-1.213	0.932	2.173	0.889	0.395	-0.268	0.000	0.597	-1.083	-0.921	2.548	0.584	1.325	-1.072	0.000	0.856	0.927	0.996	0.937	0.936	1.095	0.892
-0	2.017	-0.488	-0.466	1.029	-0.870	3.157	0.059	-0.343	2.173	3.881	0.872	1.502	1.107	3.631	1.720	0.963	0.000	0.633	-1.264	-1.734	0.000	4.572	3.339	1.005	1.407	5.590	3.614	3.110
-1	1.088	0.414	-0.841	0.485	0.605	0.860	1.110	-0.568	0.000	1.152	-0.325	1.203	2.215	0.324	1.652	-0.104	0.000	0.510	1.095	-1.728	0.000	0.880	0.722	0.989	0.977	0.711	0.888	0.762
-0	0.409	-1.717	0.712	0.809	-1.301	0.701	-1.529	-1.411	0.000	1.191	-0.582	0.438	2.215	1.147	0.813	-0.571	2.548	1.039	0.543	0.892	0.000	0.636	0.810	0.986	0.861	1.411	0.907	0.756
-1	1.094	1.577	-0.988	0.497	-0.149	0.891	-2.459	1.034	0.000	0.646	0.792	-1.022	0.000	1.573	0.254	-0.053	2.548	1.428	0.190	-1.641	3.102	4.322	2.687	0.985	0.881	1.135	1.907	1.831
-1	0.613	1.993	-0.280	0.544	0.931	0.909	1.526	1.559	0.000	0.840	1.473	-0.483	2.215	0.856	0.352	0.408	2.548	1.058	1.733	-1.396	0.000	0.801	1.066	0.984	0.639	0.841	0.871	0.748
-0	0.958	-1.202	0.600	0.434	0.170	0.783	-0.214	1.319	0.000	0.835	-0.454	-0.615	2.215	0.658	-1.858	-0.891	0.000	0.640	0.172	-1.204	3.102	1.790	1.086	0.997	0.804	0.403	0.793	0.756
-1	1.998	-0.238	0.972	0.058	0.266	0.759	1.576	-0.357	2.173	1.004	-0.349	-0.747	2.215	0.962	0.490	-0.453	0.000	1.592	0.661	-1.405	0.000	0.874	1.086	0.990	1.436	1.527	1.177	0.993
-1	0.796	-0.171	-0.818	0.574	-1.625	1.201	-0.737	1.451	2.173	0.651	0.404	-0.452	0.000	1.150	-0.652	-0.120	0.000	1.008	-0.093	0.531	3.102	0.884	0.706	0.979	1.193	0.937	0.943	0.881
-1	0.773	1.023	0.527	1.537	-0.201	2.967	-0.574	-1.534	2.173	2.346	-0.307	0.394	2.215	1.393	0.135	-0.027	0.000	3.015	0.187	0.516	0.000	0.819	1.260	0.982	2.552	3.862	2.179	1.786
-0	1.823	1.008	-1.489	0.234	-0.962	0.591	0.461	0.996	2.173	0.568	-1.297	-0.410	0.000	0.887	2.157	1.194	0.000	2.079	0.369	-0.085	3.102	0.770	0.945	0.995	1.179	0.971	0.925	0.983
-0	0.780	0.640	0.490	0.680	-1.301	0.715	-0.137	0.152	2.173	0.616	-0.831	1.668	0.000	1.958	0.528	-0.982	2.548	0.966	-1.551	0.462	0.000	1.034	1.079	1.008	0.827	1.369	1.152	0.983
-1	0.543	0.801	1.543	1.134	-0.772	0.954	-0.849	0.410	1.087	0.851	-1.988	1.686	0.000	0.799	-0.912	-1.156	0.000	0.479	0.097	1.334	0.000	0.923	0.597	0.989	1.231	0.759	0.975	0.867
-0	1.241	-0.014	0.129	1.158	0.670	0.445	-0.732	1.739	2.173	0.918	0.659	-1.340	2.215	0.557	2.410	-1.404	0.000	0.966	-1.545	-1.120	0.000	0.874	0.918	0.987	1.001	0.798	0.904	0.937
-0	1.751	-0.266	-1.575	0.489	1.292	1.112	1.533	0.137	2.173	1.204	-0.414	-0.928	0.000	0.879	1.237	-0.415	2.548	1.479	1.469	0.913	0.000	2.884	1.747	0.989	1.742	0.600	1.363	1.293
-1	1.505	1.208	-1.476	0.995	-0.836	2.800	-1.600	0.111	0.000	2.157	1.241	1.110	2.215	1.076	2.619	-0.913	0.000	1.678	2.204	-1.575	0.000	0.849	1.224	0.990	1.412	0.976	1.271	1.105
-0	0.816	0.611	0.779	1.694	0.278	0.575	-0.787	1.592	2.173	1.148	1.076	-0.831	2.215	0.421	1.316	0.632	0.000	0.589	0.452	-1.466	0.000	0.779	0.909	0.990	1.146	1.639	1.236	0.949
-1	0.551	-0.808	0.330	1.188	-0.294	0.447	-0.035	-0.993	0.000	0.432	-0.276	-0.481	2.215	1.959	-0.288	1.195	2.548	0.638	0.583	1.107	0.000	0.832	0.924	0.993	0.723	0.976	0.968	0.895
-0	1.316	-0.093	0.995	0.860	-0.621	0.593	-0.560	-1.599	2.173	0.524	-0.318	-0.240	2.215	0.566	0.759	-0.368	0.000	0.483	-2.030	-1.104	0.000	1.468	1.041	1.464	0.811	0.778	0.690	0.722
-1	1.528	0.067	-0.855	0.959	-1.464	1.143	-0.082	1.023	0.000	0.702	-0.763	-0.244	0.000	0.935	-0.881	0.206	2.548	0.614	-0.831	1.657	3.102	1.680	1.105	0.983	1.078	0.559	0.801	0.809
-0	0.558	-0.833	-0.598	1.436	-1.724	1.316	-0.661	1.593	2.173	1.148	-0.503	-0.132	1.107	1.584	-0.125	0.380	0.000	1.110	-1.216	-0.181	0.000	1.258	0.860	1.053	0.790	1.814	1.159	1.007
-1	0.819	0.879	1.221	0.598	-1.450	0.754	0.417	-0.369	2.173	0.477	1.199	0.274	0.000	1.073	0.368	0.273	2.548	1.599	2.047	1.690	0.000	0.933	0.984	0.983	0.788	0.613	0.728	0.717
-0	0.981	-1.007	0.489	0.923	1.261	0.436	-0.698	-0.506	2.173	0.764	-1.105	-1.241	2.215	0.577	-2.573	-0.036	0.000	0.565	-1.628	1.610	0.000	0.688	0.801	0.991	0.871	0.554	0.691	0.656
-0	2.888	0.568	-1.416	1.461	-1.157	1.756	-0.900	0.522	0.000	0.657	0.409	1.076	2.215	1.419	0.672	-0.019	0.000	1.436	-0.184	-0.980	3.102	0.946	0.919	0.995	1.069	0.890	0.834	0.856
-1	0.522	1.805	-0.963	1.136	0.418	0.727	-0.195	-1.695	2.173	0.309	2.559	-0.178	0.000	0.521	1.794	0.919	0.000	0.788	0.174	-0.406	3.102	0.555	0.729	1.011	1.385	0.753	0.927	0.832
-1	0.793	-0.162	-1.643	0.634	0.337	0.898	-0.633	1.689	0.000	0.806	-0.826	-0.356	2.215	0.890	-0.142	-1.268	0.000	1.293	0.574	0.725	0.000	0.833	1.077	0.988	0.721	0.679	0.867	0.753
-0	1.298	1.098	0.280	0.371	-0.373	0.855	-0.306	-1.186	0.000	0.977	-0.421	1.003	0.000	0.978	0.956	-1.249	2.548	0.735	0.577	-0.037	3.102	0.974	1.002	0.992	0.549	0.587	0.725	0.954
-1	0.751	-0.520	-1.653	0.168	-0.419	0.878	-1.023	-1.364	2.173	1.310	-0.667	0.863	0.000	1.196	-0.827	0.358	0.000	1.154	-0.165	-0.360	1.551	0.871	0.950	0.983	0.907	0.955	0.959	0.874
-0	1.730	0.666	-1.432	0.446	1.302	0.921	-0.203	0.621	0.000	1.171	-0.365	-0.611	1.107	0.585	0.807	1.150	0.000	0.415	-0.843	1.311	0.000	0.968	0.786	0.986	1.059	0.371	0.790	0.848
-1	0.596	-1.486	0.690	1.045	-1.344	0.928	0.867	0.820	2.173	0.610	0.999	-1.329	2.215	0.883	-0.001	-0.106	0.000	1.145	2.184	-0.808	0.000	2.019	1.256	1.056	1.751	1.037	1.298	1.518
-1	0.656	-1.993	-0.519	1.643	-0.143	0.815	0.256	1.220	1.087	0.399	-1.184	-1.458	0.000	0.738	1.361	-1.443	0.000	0.842	0.033	0.293	0.000	0.910	0.891	0.993	0.668	0.562	0.958	0.787
-1	1.127	-0.542	0.645	0.318	-1.496	0.661	-0.640	0.369	2.173	0.992	0.358	1.702	0.000	1.004	0.316	-1.109	0.000	1.616	-0.936	-0.707	1.551	0.875	1.191	0.985	0.651	0.940	0.969	0.834
-0	0.916	-1.423	-1.490	1.248	-0.538	0.625	-0.535	-0.174	0.000	0.769	-0.389	1.608	2.215	0.667	-1.138	-1.738	1.274	0.877	-0.019	0.482	0.000	0.696	0.917	1.121	0.678	0.347	0.647	0.722
-1	2.756	-0.637	-1.715	1.331	1.124	0.913	-0.296	-0.491	0.000	0.983	-0.831	0.000	2.215	1.180	-0.428	0.742	0.000	1.113	0.005	-1.157	1.551	1.681	1.096	1.462	0.976	0.917	1.009	1.040
-0	0.755	1.754	0.701	2.111	0.256	1.243	0.057	-1.502	2.173	0.565	-0.034	-1.078	1.107	0.529	1.696	-1.090	0.000	0.665	0.292	0.107	0.000	0.870	0.780	0.990	2.775	0.465	1.876	1.758
-1	0.593	-0.762	1.743	0.908	0.442	0.773	-1.357	-0.768	2.173	0.432	1.421	1.236	0.000	0.579	0.291	-0.403	0.000	0.966	-0.309	1.016	3.102	0.893	0.743	0.989	0.857	1.030	0.943	0.854
-1	0.891	-1.151	-1.269	0.504	-0.622	0.893	-0.549	0.700	0.000	0.828	-0.825	0.154	2.215	1.083	0.632	-1.141	0.000	1.059	-0.557	1.526	3.102	2.117	1.281	0.987	0.819	0.802	0.917	0.828
-1	2.358	-0.248	0.080	0.747	-0.975	1.019	1.374	1.363	0.000	0.935	0.127	-1.707	2.215	0.312	-0.827	0.017	0.000	0.737	1.059	-0.327	0.000	0.716	0.828	1.495	0.953	0.704	0.880	0.745
-0	0.660	-0.017	-1.138	0.453	1.002	0.645	0.518	0.703	2.173	0.751	0.705	-0.592	2.215	0.744	-0.909	-1.596	0.000	0.410	-1.135	0.481	0.000	0.592	0.922	0.989	0.897	0.948	0.777	0.701
-1	0.718	0.518	0.225	1.710	-0.022	1.888	-0.424	1.092	0.000	4.134	0.185	-1.366	0.000	1.415	1.293	0.242	2.548	2.351	0.264	-0.057	3.102	0.830	1.630	0.976	1.215	0.890	1.422	1.215
-1	1.160	0.203	0.941	0.594	0.212	0.636	-0.556	0.679	2.173	1.089	-0.481	-1.008	1.107	1.245	-0.056	-1.357	0.000	0.587	1.007	0.056	0.000	1.106	0.901	0.987	0.786	1.224	0.914	0.837
-1	0.697	0.542	0.619	0.985	1.481	0.745	0.415	1.644	2.173	0.903	0.495	-0.958	2.215	1.165	1.195	0.346	0.000	1.067	-0.881	-0.264	0.000	0.830	1.025	0.987	0.690	0.863	0.894	0.867
-0	1.430	0.190	-0.700	0.246	0.518	1.302	0.660	-0.247	2.173	1.185	-0.539	1.504	0.000	1.976	-0.401	1.079	0.000	0.855	-0.958	-1.110	3.102	0.886	0.953	0.993	0.889	1.400	1.376	1.119
-1	1.122	-0.795	0.202	0.397	-1.553	0.597	-1.459	-0.734	2.173	0.522	1.044	1.027	2.215	0.783	-1.243	1.701	0.000	0.371	1.737	0.199	0.000	1.719	1.176	0.988	0.723	1.583	1.063	0.914
-0	1.153	0.526	1.236	0.266	0.001	1.139	-1.236	-0.585	2.173	1.337	-0.215	-1.356	2.215	1.780	1.129	0.902	0.000	1.608	-0.391	-0.161	0.000	1.441	1.633	0.990	1.838	1.516	1.635	1.373
-1	0.760	1.012	0.758	0.937	0.051	0.941	0.687	-1.247	2.173	1.288	-0.743	0.822	0.000	1.552	1.782	-1.533	0.000	0.767	1.349	0.168	0.000	0.716	0.862	0.988	0.595	0.359	0.697	0.623
-1	1.756	-1.469	1.395	1.345	-1.595	0.817	0.017	-0.741	2.173	0.483	-0.008	0.293	0.000	1.768	-0.663	0.438	1.274	1.202	-1.387	-0.222	0.000	1.022	1.058	0.992	1.407	1.427	1.356	1.133
-0	0.397	0.582	-0.758	1.260	-1.735	0.889	-0.515	1.139	2.173	0.973	1.616	0.460	0.000	1.308	1.001	-0.709	2.548	0.858	0.995	-0.231	0.000	0.749	0.888	0.979	1.487	1.804	1.208	1.079
-0	0.515	-0.984	0.425	1.114	-0.439	1.999	0.818	1.561	0.000	1.407	0.009	-0.380	0.000	1.332	0.230	0.397	0.000	1.356	-0.616	-1.057	3.102	0.978	1.017	0.990	1.118	0.862	0.835	0.919
-1	1.368	-0.921	-0.866	0.842	-0.598	0.456	-1.176	1.219	1.087	0.419	-1.974	-0.819	0.000	0.791	-1.640	0.881	0.000	1.295	-0.782	0.442	3.102	0.945	0.761	0.974	0.915	0.535	0.733	0.651
-0	2.276	0.134	0.399	2.525	0.376	1.111	-1.078	-1.571	0.000	0.657	2.215	-0.900	0.000	1.183	-0.662	-0.508	2.548	1.436	-0.517	0.960	3.102	0.569	0.931	0.993	1.170	0.967	0.879	1.207
-0	0.849	0.907	0.124	0.652	1.585	0.715	0.355	-1.200	0.000	0.599	-0.892	1.301	0.000	1.106	1.151	0.582	0.000	1.895	-0.279	-0.568	3.102	0.881	0.945	0.998	0.559	0.649	0.638	0.660
-1	2.105	0.248	-0.797	0.530	0.206	1.957	-2.175	0.797	0.000	1.193	0.637	-1.646	2.215	0.881	1.111	-1.046	0.000	0.872	-0.185	1.085	1.551	0.986	1.343	1.151	1.069	0.714	2.063	1.951
-1	1.838	1.060	1.637	1.017	1.370	0.913	0.461	-0.609	1.087	0.766	-0.461	0.303	2.215	0.724	-0.061	0.886	0.000	0.941	1.123	-0.745	0.000	0.858	0.847	0.979	1.313	1.083	1.094	0.910
-0	0.364	1.274	1.066	1.570	-0.394	0.485	0.012	-1.716	0.000	0.317	-1.233	0.534	2.215	0.548	-2.165	0.762	0.000	0.729	0.169	-0.318	3.102	0.892	0.944	1.013	0.594	0.461	0.688	0.715
-1	0.503	1.343	-0.031	1.134	-1.204	0.590	-0.309	0.174	2.173	0.408	2.372	-0.628	0.000	1.850	0.400	1.147	2.548	0.664	-0.458	-0.885	0.000	1.445	1.283	0.989	1.280	1.118	1.127	1.026
-0	1.873	0.258	0.103	2.491	0.530	1.678	0.644	-1.738	2.173	1.432	0.848	-1.340	0.000	0.621	1.323	-1.316	0.000	0.628	0.789	-0.206	1.551	0.426	0.802	1.125	0.688	1.079	1.338	1.239
-1	0.826	-0.732	1.587	0.582	-1.236	0.495	0.757	-0.741	2.173	0.940	1.474	0.354	2.215	0.474	1.055	-1.657	0.000	0.415	1.758	0.841	0.000	0.451	0.578	0.984	0.757	0.922	0.860	0.696
-0	0.935	-1.614	-0.597	0.299	1.223	0.707	-0.853	-1.026	0.000	0.751	0.007	-1.691	0.000	1.062	-0.125	0.976	2.548	0.877	1.275	0.646	0.000	0.962	1.074	0.980	0.608	0.726	0.741	0.662
-1	0.643	0.542	-1.285	0.474	-0.366	0.667	-0.446	1.195	2.173	1.076	0.145	-0.126	0.000	0.970	-0.661	0.394	1.274	1.218	-0.184	-1.722	0.000	1.331	1.019	0.985	1.192	0.677	0.973	0.910
-0	0.713	0.164	1.080	1.427	-0.460	0.960	-0.152	-0.940	2.173	1.427	-0.901	1.036	1.107	0.440	-1.269	-0.194	0.000	0.452	1.932	-0.532	0.000	1.542	1.210	1.374	1.319	1.818	1.220	1.050
-0	0.876	-0.463	-1.224	2.458	-1.689	1.007	-0.752	0.398	0.000	2.456	-1.285	-0.152	1.107	1.641	1.838	1.717	0.000	0.458	0.194	0.488	3.102	4.848	2.463	0.986	1.981	0.974	2.642	2.258
-1	0.384	-0.275	0.387	1.403	-0.994	0.620	-1.529	1.685	0.000	1.091	-1.644	1.078	0.000	0.781	-1.311	0.326	2.548	1.228	-0.728	-0.633	1.551	0.920	0.854	0.987	0.646	0.609	0.740	0.884
-0	0.318	-1.818	-1.008	0.977	1.268	0.457	2.451	-1.522	0.000	0.881	1.351	0.461	2.215	0.929	0.239	-0.380	2.548	0.382	-0.613	1.330	0.000	1.563	1.193	0.994	0.829	0.874	0.901	1.026
-1	0.612	-1.120	1.098	0.402	-0.480	0.818	0.188	1.511	0.000	0.800	-0.253	0.977	0.000	1.175	0.271	-1.289	1.274	2.531	0.226	-0.409	3.102	0.889	0.947	0.979	1.486	0.940	1.152	1.119
-1	0.587	-0.737	-0.228	0.970	1.119	0.823	0.184	1.594	0.000	1.104	0.301	-0.818	2.215	0.819	0.712	-0.560	0.000	2.240	-0.419	0.340	3.102	1.445	1.103	0.988	0.715	1.363	1.019	0.926
-0	1.030	-0.694	-1.638	0.893	-1.074	1.160	-0.766	0.485	0.000	1.632	-0.698	-1.142	2.215	1.050	-1.092	0.952	0.000	1.475	0.286	0.125	3.102	0.914	1.075	0.982	0.732	1.493	1.219	1.079
-1	2.142	0.617	1.517	0.387	-0.862	0.345	1.203	-1.014	2.173	0.609	1.092	0.275	0.000	1.331	0.582	-0.183	2.548	0.557	1.540	-1.642	0.000	0.801	0.737	1.060	0.715	0.626	0.749	0.674
-0	1.076	0.240	-0.246	0.871	-1.241	0.496	0.282	0.746	2.173	1.095	-0.648	1.100	2.215	0.446	-1.756	0.764	0.000	0.434	0.788	-0.991	0.000	1.079	0.868	1.047	0.818	0.634	0.795	0.733
-0	1.400	0.901	-1.617	0.625	-0.163	0.661	-0.411	-1.616	2.173	0.685	0.524	0.425	0.000	0.881	-0.766	0.312	0.000	0.979	0.255	-0.667	3.102	0.898	1.105	1.253	0.730	0.716	0.738	0.795
-0	3.302	1.132	1.051	0.658	0.768	1.308	0.251	-0.374	1.087	1.673	0.015	-0.898	0.000	0.688	-0.535	1.363	1.274	0.871	1.325	-1.583	0.000	1.646	1.249	0.995	1.919	1.288	1.330	1.329
-0	1.757	0.202	0.750	0.767	-0.362	0.932	-1.033	-1.366	0.000	1.529	-1.012	-0.771	0.000	1.161	-0.287	0.059	0.000	2.185	1.147	1.099	3.102	0.795	0.529	1.354	1.144	1.491	1.319	1.161
-0	1.290	0.905	-1.711	1.017	-0.695	1.008	-1.038	0.693	2.173	1.202	-0.595	0.187	0.000	1.011	0.139	-1.607	0.000	0.789	-0.613	-1.041	3.102	1.304	0.895	1.259	1.866	0.955	1.211	1.200
-1	1.125	-0.004	1.694	0.373	0.329	0.978	0.640	-0.391	0.000	1.122	-0.376	1.521	2.215	0.432	2.413	-1.259	0.000	0.969	0.730	0.512	3.102	0.716	0.773	0.991	0.624	0.977	0.981	0.875
-0	1.081	0.861	1.252	1.621	1.474	1.293	0.600	0.630	0.000	1.991	-0.090	-0.675	2.215	0.861	1.105	-0.201	0.000	1.135	2.489	-1.659	0.000	1.089	0.657	0.991	2.179	0.412	1.334	1.071
-1	0.652	-0.294	1.241	1.034	0.490	1.033	0.551	-0.963	2.173	0.661	1.031	-1.654	2.215	1.376	-0.018	0.843	0.000	0.943	-0.329	-0.269	0.000	1.085	1.067	0.991	1.504	0.773	1.135	0.993
-1	1.408	-1.028	-1.018	0.252	-0.242	0.465	-0.364	-0.200	0.000	1.466	0.669	0.739	1.107	1.031	0.415	-1.468	2.548	0.457	-1.091	-1.722	0.000	0.771	0.811	0.979	1.459	1.204	1.041	0.866
-1	0.781	-1.143	-0.659	0.961	1.266	1.183	-0.686	0.119	2.173	1.126	-0.064	1.447	0.000	0.730	1.430	-1.535	0.000	1.601	0.513	1.658	0.000	0.871	1.345	1.184	1.058	0.620	1.107	0.978
-1	1.300	-0.616	1.032	0.751	-0.731	0.961	-0.716	1.592	0.000	2.079	-1.063	-0.271	2.215	0.475	0.518	1.695	1.274	0.395	-2.204	0.349	0.000	1.350	0.983	1.369	1.265	1.428	1.135	0.982
-1	0.833	0.809	1.657	1.637	1.019	0.705	1.077	-0.968	2.173	1.261	0.114	-0.298	1.107	1.032	0.017	0.236	0.000	0.640	-0.026	-1.598	0.000	0.894	0.982	0.981	1.250	1.054	1.018	0.853
-1	1.686	-1.090	-0.301	0.890	0.557	1.304	-0.284	-1.393	2.173	0.388	2.118	0.513	0.000	0.514	-0.015	0.891	0.000	0.460	0.547	0.627	3.102	0.942	0.524	1.186	1.528	0.889	1.015	1.122
-1	0.551	0.911	0.879	0.379	-0.796	1.154	-0.808	-0.966	0.000	1.168	-0.513	0.355	2.215	0.646	-1.309	0.773	0.000	0.544	-0.283	1.301	3.102	0.847	0.705	0.990	0.772	0.546	0.790	0.719
-1	1.597	0.793	-1.119	0.691	-1.455	0.370	0.337	1.354	0.000	0.646	-1.005	0.732	2.215	1.019	0.040	0.209	0.000	0.545	0.958	0.239	3.102	0.962	0.793	0.994	0.719	0.745	0.812	0.739
-0	1.033	-1.193	-0.452	0.247	0.970	0.503	-1.424	1.362	0.000	1.062	-0.416	-1.156	2.215	0.935	-0.023	0.555	2.548	0.410	-1.766	0.379	0.000	0.590	0.953	0.991	0.717	1.081	0.763	0.690
-1	0.859	-1.004	1.521	0.781	-0.993	0.677	0.643	-0.338	2.173	0.486	0.409	1.283	0.000	0.679	0.110	0.285	0.000	0.715	-0.735	-0.157	1.551	0.702	0.773	0.984	0.627	0.633	0.694	0.643
-0	0.612	-1.127	1.074	1.225	-0.426	0.927	-2.141	-0.473	0.000	1.290	-0.927	-1.085	2.215	1.183	1.981	-1.687	0.000	2.176	0.406	-1.581	0.000	0.945	0.651	1.170	0.895	1.604	1.179	1.142
-1	0.535	0.321	-1.095	0.281	-0.960	0.876	-0.709	-0.076	0.000	1.563	-0.666	1.536	2.215	0.773	-0.321	0.435	0.000	0.682	-0.801	-0.952	3.102	0.711	0.667	0.985	0.888	0.741	0.872	0.758
-1	0.745	1.586	1.578	0.863	-1.423	0.530	1.714	1.085	0.000	1.174	0.679	1.015	0.000	1.158	0.609	-1.186	2.548	1.851	0.832	-0.248	3.102	0.910	1.164	0.983	0.947	0.858	0.928	0.823
-0	0.677	-1.014	-1.648	1.455	1.461	0.596	-2.358	0.517	0.000	0.800	0.849	-0.743	2.215	1.024	-0.282	-1.004	0.000	1.846	-0.977	0.378	3.102	2.210	1.423	0.982	1.074	1.623	1.417	1.258
-1	0.815	-1.263	0.057	1.018	-0.208	0.339	-0.347	-1.646	2.173	1.223	0.600	-1.658	2.215	1.435	0.042	0.926	0.000	0.777	1.698	-0.698	0.000	1.022	1.058	1.000	0.784	0.477	0.886	0.836
-0	3.512	-1.094	-0.220	0.338	-0.328	1.962	-1.099	1.544	1.087	1.461	-1.305	-0.922	2.215	1.219	-1.289	0.400	0.000	0.731	0.155	1.249	0.000	1.173	1.366	0.993	2.259	2.000	1.626	1.349
-0	0.904	1.248	0.325	0.317	-1.624	0.685	-0.538	1.665	2.173	0.685	-2.145	-1.106	0.000	0.632	-1.460	1.017	0.000	1.085	-0.182	0.162	3.102	0.885	0.801	0.989	0.930	0.904	1.012	0.961

contrib/gbdt/lightgbm/binary1.test

@@ -1,500 +0,0 @@
-1	0.644	0.247	-0.447	0.862	0.374	0.854	-1.126	-0.790	2.173	1.015	-0.201	1.400	0.000	1.575	1.807	1.607	0.000	1.585	-0.190	-0.744	3.102	0.958	1.061	0.980	0.875	0.581	0.905	0.796
-0	0.385	1.800	1.037	1.044	0.349	1.502	-0.966	1.734	0.000	0.966	-1.960	-0.249	0.000	1.501	0.465	-0.354	2.548	0.834	-0.440	0.638	3.102	0.695	0.909	0.981	0.803	0.813	1.149	1.116
-0	1.214	-0.166	0.004	0.505	1.434	0.628	-1.174	-1.230	1.087	0.579	-1.047	-0.118	0.000	0.835	0.340	1.234	2.548	0.711	-1.383	1.355	0.000	0.848	0.911	1.043	0.931	1.058	0.744	0.696
-1	0.420	1.111	0.137	1.516	-1.657	0.854	0.623	1.605	1.087	1.511	-1.297	0.251	0.000	0.872	-0.368	-0.721	0.000	0.543	0.731	1.424	3.102	1.597	1.282	1.105	0.730	0.148	1.231	1.234
-0	0.897	-1.703	-1.306	1.022	-0.729	0.836	0.859	-0.333	2.173	1.336	-0.965	0.972	2.215	0.671	1.021	-1.439	0.000	0.493	-2.019	-0.289	0.000	0.805	0.930	0.984	1.430	2.198	1.934	1.684
-0	0.756	1.126	-0.945	2.355	-0.555	0.889	0.800	1.440	0.000	0.585	0.271	0.631	2.215	0.722	1.744	1.051	0.000	0.618	0.924	0.698	1.551	0.976	0.864	0.988	0.803	0.234	0.822	0.911
-0	1.141	-0.741	0.953	1.478	-0.524	1.197	-0.871	1.689	2.173	0.875	1.321	-0.518	1.107	0.540	0.037	-0.987	0.000	0.879	1.187	0.245	0.000	0.888	0.701	1.747	1.358	2.479	1.491	1.223
-1	0.606	-0.936	-0.384	1.257	-1.162	2.719	-0.600	0.100	2.173	3.303	-0.284	1.561	1.107	0.689	1.786	-0.326	0.000	0.780	-0.532	1.216	0.000	0.936	2.022	0.985	1.574	4.323	2.263	1.742
-1	0.603	0.429	-0.279	1.448	1.301	1.008	2.423	-1.295	0.000	0.452	1.305	0.533	0.000	1.076	1.011	1.256	2.548	2.021	1.260	-0.343	0.000	0.890	0.969	1.281	0.763	0.652	0.827	0.785
-0	1.171	-0.962	0.521	0.841	-0.315	1.196	-0.744	-0.882	2.173	0.726	-1.305	1.377	1.107	0.643	-1.790	-1.264	0.000	1.257	0.222	0.817	0.000	0.862	0.911	0.987	0.846	1.293	0.899	0.756
-1	1.392	-0.358	0.235	1.494	-0.461	0.895	-0.848	1.549	2.173	0.841	-0.384	0.666	1.107	1.199	2.509	-0.891	0.000	1.109	-0.364	-0.945	0.000	0.693	2.135	1.170	1.362	0.959	2.056	1.842
-1	1.024	1.076	-0.886	0.851	1.530	0.673	-0.449	0.187	1.087	0.628	-0.895	1.176	2.215	0.696	-0.232	-0.875	0.000	0.411	1.501	0.048	0.000	0.842	0.919	1.063	1.193	0.777	0.964	0.807
-1	0.890	-0.760	1.182	1.369	0.751	0.696	-0.959	-0.710	1.087	0.775	-0.130	-1.409	2.215	0.701	-0.110	-0.739	0.000	0.508	-0.451	0.390	0.000	0.762	0.738	0.998	1.126	0.788	0.940	0.790
-1	0.460	0.537	0.636	1.442	-0.269	0.585	0.323	-1.731	2.173	0.503	1.034	-0.927	0.000	0.928	-1.024	1.006	2.548	0.513	-0.618	-1.336	0.000	0.802	0.831	0.992	1.019	0.925	1.056	0.833
-1	0.364	1.648	0.560	1.720	0.829	1.110	0.811	-0.588	0.000	0.408	1.045	1.054	2.215	0.319	-1.138	1.545	0.000	0.423	1.025	-1.265	3.102	1.656	0.928	1.003	0.544	0.327	0.670	0.746
-1	0.525	-0.096	1.206	0.948	-1.103	1.519	-0.582	0.606	2.173	1.274	-0.572	-0.934	0.000	0.855	-1.028	-1.222	0.000	0.578	-1.000	-1.725	3.102	0.896	0.878	0.981	0.498	0.909	0.772	0.668
-0	0.536	-0.821	-1.029	0.703	1.113	0.363	-0.711	0.022	1.087	0.325	1.503	1.249	2.215	0.673	1.041	-0.401	0.000	0.480	2.127	1.681	0.000	0.767	1.034	0.990	0.671	0.836	0.669	0.663
-1	1.789	-0.583	1.641	0.897	0.799	0.515	-0.100	-1.483	0.000	1.101	0.031	-0.326	2.215	1.195	0.001	0.126	2.548	0.768	-0.148	0.601	0.000	0.916	0.921	1.207	1.069	0.483	0.934	0.795
-1	1.332	-0.571	0.986	0.580	1.508	0.582	0.634	-0.746	1.087	1.084	-0.964	-0.489	0.000	0.785	0.274	0.343	2.548	0.779	0.721	1.489	0.000	1.733	1.145	0.990	1.270	0.715	0.897	0.915
-0	1.123	0.629	-1.708	0.597	-0.882	0.752	0.195	1.522	2.173	1.671	1.515	-0.003	0.000	0.778	0.514	0.139	1.274	0.801	1.260	1.600	0.000	1.495	0.976	0.988	0.676	0.921	1.010	0.943
-0	1.816	-0.515	0.171	0.980	-0.454	0.870	0.202	-1.399	2.173	1.130	1.066	-1.593	0.000	0.844	0.735	1.275	2.548	1.125	-1.133	0.348	0.000	0.837	0.693	0.988	1.112	0.784	1.009	0.974
-1	0.364	0.694	0.445	1.862	0.159	0.963	-1.356	1.260	1.087	0.887	-0.540	-1.533	2.215	0.658	-2.544	-1.236	0.000	0.516	-0.807	0.039	0.000	0.891	1.004	0.991	1.092	0.976	1.000	0.953
-1	0.790	-1.175	0.475	1.846	0.094	0.999	-1.090	0.257	0.000	1.422	0.854	1.112	2.215	1.302	1.004	-1.702	1.274	2.557	-0.787	-1.048	0.000	0.890	1.429	0.993	2.807	0.840	2.248	1.821
-1	0.765	-0.500	-0.603	1.843	-0.560	1.068	0.007	0.746	2.173	1.154	-0.017	1.329	0.000	1.165	1.791	-1.585	0.000	1.116	0.441	-0.886	0.000	0.774	0.982	0.989	1.102	0.633	1.178	1.021
-1	1.407	1.293	-1.418	0.502	-1.527	2.005	-2.122	0.622	0.000	1.699	1.508	-0.649	2.215	1.665	0.748	-0.755	0.000	2.555	0.811	1.423	1.551	7.531	5.520	0.985	1.115	1.881	4.487	3.379
-1	0.772	-0.186	-1.372	0.823	-0.140	0.781	0.763	0.046	2.173	1.128	0.516	1.380	0.000	0.797	-0.640	-0.134	2.548	2.019	-0.972	-1.670	0.000	2.022	1.466	0.989	0.856	0.808	1.230	0.991
-1	0.546	-0.954	0.715	1.335	-1.689	0.783	-0.443	-1.735	2.173	1.081	0.185	-0.435	0.000	1.433	-0.662	-0.389	0.000	0.969	0.924	1.099	0.000	0.910	0.879	0.988	0.683	0.753	0.878	0.865
-1	0.596	0.276	-1.054	1.358	1.355	1.444	1.813	-0.208	0.000	1.175	-0.949	-1.573	0.000	0.855	-1.228	-0.925	2.548	1.837	-0.400	0.913	0.000	0.637	0.901	1.028	0.553	0.790	0.679	0.677
-0	0.458	2.292	1.530	0.291	1.283	0.749	-0.930	-0.198	0.000	0.300	-1.560	0.990	0.000	0.811	-0.176	0.995	2.548	1.085	-0.178	-1.213	3.102	0.891	0.648	0.999	0.732	0.655	0.619	0.620
-0	0.638	-0.575	-1.048	0.125	0.178	0.846	-0.753	-0.339	1.087	0.799	-0.727	1.182	0.000	0.888	0.283	0.717	0.000	1.051	-1.046	-1.557	3.102	0.889	0.871	0.989	0.884	0.923	0.836	0.779
-1	0.434	-1.119	-0.313	2.427	0.461	0.497	0.261	-1.177	2.173	0.618	-0.737	-0.688	0.000	1.150	-1.237	-1.652	2.548	0.757	-0.054	1.700	0.000	0.809	0.741	0.982	1.450	0.936	1.086	0.910
-1	0.431	-1.144	-1.030	0.778	-0.655	0.490	0.047	-1.546	0.000	1.583	-0.014	0.891	2.215	0.516	0.956	0.567	2.548	0.935	-1.123	-0.082	0.000	0.707	0.995	0.995	0.700	0.602	0.770	0.685
-1	1.894	0.222	1.224	1.578	1.715	0.966	2.890	-0.013	0.000	0.922	-0.703	-0.844	0.000	0.691	2.056	1.039	0.000	0.900	-0.733	-1.240	3.102	1.292	1.992	1.026	0.881	0.684	1.759	1.755
-0	0.985	-0.316	0.141	1.067	-0.946	0.819	-1.177	1.307	2.173	1.080	-0.429	0.557	1.107	1.726	1.435	-1.075	0.000	1.100	1.547	-0.647	0.000	0.873	1.696	1.179	1.146	1.015	1.538	1.270
-0	0.998	-0.187	-0.236	0.882	0.755	0.468	0.950	-0.439	2.173	0.579	-0.550	-0.624	0.000	1.847	1.196	1.384	1.274	0.846	1.273	-1.072	0.000	1.194	0.797	1.013	1.319	1.174	0.963	0.898
-0	0.515	0.246	-0.593	1.082	1.591	0.912	-0.623	-0.957	2.173	0.858	0.418	0.844	0.000	0.948	2.519	1.599	0.000	1.158	1.385	-0.095	3.102	0.973	1.033	0.988	0.998	1.716	1.054	0.901
-0	0.919	-1.001	1.506	1.389	0.653	0.507	-0.616	-0.689	2.173	0.808	0.536	-0.467	2.215	0.496	2.187	-0.859	0.000	0.822	0.807	1.163	0.000	0.876	0.861	1.088	0.947	0.614	0.911	1.087
-0	0.794	0.051	1.477	1.504	-1.695	0.716	0.315	0.264	1.087	0.879	-0.135	-1.094	2.215	1.433	-0.741	0.201	0.000	1.566	0.534	-0.989	0.000	0.627	0.882	0.974	0.807	1.130	0.929	0.925
-1	0.455	-0.946	-1.175	1.453	-0.580	0.763	-0.856	0.840	0.000	0.829	1.223	1.174	2.215	0.714	0.638	-0.466	0.000	1.182	0.223	-1.333	0.000	0.977	0.938	0.986	0.713	0.714	0.796	0.843
-1	0.662	-0.296	-1.287	1.212	-0.707	0.641	1.457	0.222	0.000	0.600	0.525	-1.700	2.215	0.784	-0.835	-0.961	2.548	0.865	1.131	1.162	0.000	0.854	0.877	0.978	0.740	0.734	0.888	0.811
-0	0.390	0.698	-1.629	1.888	0.298	0.990	1.614	-1.572	0.000	1.666	0.170	0.719	2.215	1.590	1.064	-0.886	1.274	0.952	0.305	-1.216	0.000	1.048	0.897	1.173	0.891	1.936	1.273	1.102
-0	1.014	0.117	1.384	0.686	-1.047	0.609	-1.245	-0.850	0.000	1.076	-1.158	0.814	1.107	1.598	-0.389	-0.111	0.000	0.907	1.688	-1.673	0.000	1.333	0.866	0.989	0.975	0.442	0.797	0.788
-0	1.530	-1.408	-0.207	0.440	-1.357	0.902	-0.647	1.325	1.087	1.320	-0.819	0.246	1.107	0.503	1.407	-1.683	0.000	1.189	-0.972	-0.925	0.000	0.386	1.273	0.988	0.829	1.335	1.173	1.149
-1	1.689	-0.590	0.915	2.076	1.202	0.644	-0.478	-0.238	0.000	0.809	-1.660	-1.184	0.000	1.227	-0.224	-0.808	2.548	1.655	1.047	-0.623	0.000	0.621	1.192	0.988	1.309	0.866	0.924	1.012
-0	1.102	0.402	-1.622	1.262	1.022	0.576	0.271	-0.269	0.000	0.591	0.495	-1.278	0.000	1.271	0.209	0.575	2.548	0.941	0.964	-0.685	3.102	0.989	0.963	1.124	0.857	0.858	0.716	0.718
-0	2.491	0.825	0.581	1.593	0.205	0.782	-0.815	1.499	0.000	1.179	-0.999	-1.509	0.000	0.926	0.920	-0.522	2.548	2.068	-1.021	-1.050	3.102	0.874	0.943	0.980	0.945	1.525	1.570	1.652
-0	0.666	0.254	1.601	1.303	-0.250	1.236	-1.929	0.793	0.000	1.074	0.447	-0.871	0.000	0.991	1.059	-0.342	0.000	1.703	-0.393	-1.419	3.102	0.921	0.945	1.285	0.931	0.462	0.770	0.729
-0	0.937	-1.126	1.424	1.395	1.743	0.760	0.428	-0.238	2.173	0.846	0.494	1.320	2.215	0.872	-1.826	-0.507	0.000	0.612	1.860	1.403	0.000	3.402	2.109	0.985	1.298	1.165	1.404	1.240
-1	0.881	-1.086	-0.870	0.513	0.266	2.049	-1.870	1.160	0.000	2.259	-0.428	-0.935	2.215	1.321	-0.655	-0.449	2.548	1.350	-1.766	-0.108	0.000	0.911	1.852	0.987	1.167	0.820	1.903	1.443
-0	0.410	0.835	-0.819	1.257	1.112	0.871	-1.737	-0.401	0.000	0.927	0.158	1.253	0.000	1.183	0.405	-1.570	0.000	0.807	-0.704	-0.438	3.102	0.932	0.962	0.987	0.653	0.315	0.616	0.648
-1	0.634	0.196	-1.679	1.379	-0.967	2.260	-0.273	1.114	0.000	1.458	1.070	-0.278	1.107	1.195	0.110	-0.688	2.548	0.907	0.298	-1.359	0.000	0.949	1.129	0.984	0.675	0.877	0.938	0.824
-1	0.632	-1.254	1.201	0.496	-0.106	0.235	2.731	-0.955	0.000	0.615	-0.805	0.600	0.000	0.633	-0.934	1.641	0.000	1.407	-0.483	-0.962	1.551	0.778	0.797	0.989	0.578	0.722	0.576	0.539
-0	0.714	1.122	1.566	2.399	-1.431	1.665	0.299	0.323	0.000	1.489	1.087	-0.861	2.215	1.174	0.140	1.083	2.548	0.404	-0.968	1.105	0.000	0.867	0.969	0.981	1.039	1.552	1.157	1.173
-1	0.477	-0.321	-0.471	1.966	1.034	2.282	1.359	-0.874	0.000	1.672	-0.258	1.109	0.000	1.537	0.604	0.231	2.548	1.534	-0.640	0.827	0.000	0.746	1.337	1.311	0.653	0.721	0.795	0.742
-1	1.351	0.460	0.031	1.194	-1.185	0.670	-1.157	-1.637	2.173	0.599	-0.823	0.680	0.000	0.478	0.373	1.716	0.000	0.809	-0.919	0.010	1.551	0.859	0.839	1.564	0.994	0.777	0.971	0.826
-1	0.520	-1.442	-0.348	0.840	1.654	1.273	-0.760	1.317	0.000	0.861	2.579	-0.791	0.000	1.779	0.257	-0.703	0.000	2.154	1.928	0.457	0.000	1.629	3.194	0.992	0.730	1.107	2.447	2.747
-0	0.700	-0.308	0.920	0.438	-0.879	0.516	1.409	1.101	0.000	0.960	0.701	-0.049	2.215	1.442	-0.416	-1.439	2.548	0.628	1.009	-0.364	0.000	0.848	0.817	0.987	0.759	1.421	0.937	0.920
-1	0.720	1.061	-0.546	0.798	-1.521	1.066	0.173	0.271	1.087	1.453	0.114	1.336	1.107	0.702	0.616	-0.367	0.000	0.543	-0.386	-1.301	0.000	0.653	0.948	0.989	1.031	1.500	0.965	0.790
-1	0.735	-0.416	0.588	1.308	-0.382	1.042	0.344	1.609	0.000	0.926	0.163	-0.520	1.107	1.050	-0.427	1.159	2.548	0.834	0.613	0.948	0.000	0.848	1.189	1.042	0.844	1.099	0.829	0.843
-1	0.777	-0.396	1.540	1.608	0.638	0.955	0.040	0.918	2.173	1.315	1.116	-0.823	0.000	0.781	-0.762	0.564	2.548	0.945	-0.573	1.379	0.000	0.679	0.706	1.124	0.608	0.593	0.515	0.493
-1	0.934	0.319	-0.257	0.970	-0.980	0.726	0.774	0.731	0.000	0.896	0.038	-1.465	1.107	0.773	-0.055	-0.831	2.548	1.439	-0.229	0.698	0.000	0.964	1.031	0.995	0.845	0.480	0.810	0.762
-0	0.461	0.771	0.019	2.055	-1.288	1.043	0.147	0.261	2.173	0.833	-0.156	1.425	0.000	0.832	0.805	-0.491	2.548	0.589	1.252	1.414	0.000	0.850	0.906	1.245	1.364	0.850	0.908	0.863
-1	0.858	-0.116	-0.937	0.966	1.167	0.825	-0.108	1.111	1.087	0.733	1.163	-0.634	0.000	0.894	0.771	0.020	0.000	0.846	-1.124	-1.195	3.102	0.724	1.194	1.195	0.813	0.969	0.985	0.856
-0	0.720	-0.335	-0.307	1.445	0.540	1.108	-0.034	-1.691	1.087	0.883	-1.356	-0.678	2.215	0.440	1.093	0.253	0.000	0.389	-1.582	-1.097	0.000	1.113	1.034	0.988	1.256	1.572	1.062	0.904
-1	0.750	-0.811	-0.542	0.985	0.408	0.471	0.477	0.355	0.000	1.347	-0.875	-1.556	2.215	0.564	1.082	-0.724	0.000	0.793	-0.958	-0.020	3.102	0.836	0.825	0.986	1.066	0.924	0.927	0.883
-0	0.392	-0.468	-0.216	0.680	1.565	1.086	-0.765	-0.581	1.087	1.264	-1.035	1.189	2.215	0.986	-0.338	0.747	0.000	0.884	-1.328	-0.965	0.000	1.228	0.988	0.982	1.135	1.741	1.108	0.956
-1	0.434	-1.269	0.643	0.713	0.608	0.597	0.832	1.627	0.000	0.708	-0.422	0.079	2.215	1.533	-0.823	-1.127	2.548	0.408	-1.357	-0.828	0.000	1.331	1.087	0.999	1.075	1.015	0.875	0.809
-0	0.828	-1.803	0.342	0.847	-0.162	1.585	-1.128	-0.272	2.173	1.974	0.039	-1.717	0.000	0.900	0.764	-1.741	0.000	1.349	-0.079	1.035	3.102	0.984	0.815	0.985	0.780	1.661	1.403	1.184
-1	1.089	-0.350	-0.747	1.472	0.792	1.087	-0.069	-1.192	0.000	0.512	-0.841	-1.284	0.000	2.162	-0.821	0.545	2.548	1.360	2.243	-0.183	0.000	0.977	0.628	1.725	1.168	0.635	0.823	0.822
-1	0.444	0.451	-1.332	1.176	-0.247	0.898	0.194	0.007	0.000	1.958	0.576	-1.618	2.215	0.584	1.203	0.268	0.000	0.939	1.033	1.264	3.102	0.829	0.886	0.985	1.265	0.751	1.032	0.948
-0	0.629	0.114	1.177	0.917	-1.204	0.845	0.828	-0.088	0.000	0.962	-1.302	0.823	2.215	0.732	0.358	-1.334	2.548	0.538	0.582	1.561	0.000	1.028	0.834	0.988	0.904	1.205	1.039	0.885
-1	1.754	-1.259	-0.573	0.959	-1.483	0.358	0.448	-1.452	0.000	0.711	0.313	0.499	2.215	1.482	-0.390	1.474	2.548	1.879	-1.540	0.668	0.000	0.843	0.825	1.313	1.315	0.939	1.048	0.871
-1	0.549	0.706	-1.437	0.894	0.891	0.680	-0.762	-1.568	0.000	0.981	0.499	-0.425	2.215	1.332	0.678	0.485	1.274	0.803	0.022	-0.893	0.000	0.793	1.043	0.987	0.761	0.899	0.915	0.794
-0	0.475	0.542	-0.987	1.569	0.069	0.551	1.543	-1.488	0.000	0.608	0.301	1.734	2.215	0.277	0.499	-0.522	0.000	1.375	1.212	0.696	3.102	0.652	0.756	0.987	0.828	0.830	0.715	0.679
-1	0.723	0.049	-1.153	1.300	0.083	0.723	-0.749	0.630	0.000	1.126	0.412	-0.384	0.000	1.272	1.256	1.358	2.548	3.108	0.777	-1.486	3.102	0.733	1.096	1.206	1.269	0.899	1.015	0.903
-1	1.062	0.296	0.725	0.285	-0.531	0.819	1.277	-0.667	0.000	0.687	0.829	-0.092	0.000	1.158	0.447	1.047	2.548	1.444	-0.186	-1.491	3.102	0.863	1.171	0.986	0.769	0.828	0.919	0.840
-0	0.572	-0.349	1.396	2.023	0.795	0.577	0.457	-0.533	0.000	1.351	0.701	-1.091	0.000	0.724	-1.012	-0.182	2.548	0.923	-0.012	0.789	3.102	0.936	1.025	0.985	1.002	0.600	0.828	0.909
-1	0.563	0.387	0.412	0.553	1.050	0.723	-0.992	-0.447	0.000	0.748	0.948	0.546	2.215	1.761	-0.559	-1.183	0.000	1.114	-0.251	1.192	3.102	0.936	0.912	0.976	0.578	0.722	0.829	0.892
-1	1.632	1.577	-0.697	0.708	-1.263	0.863	0.012	1.197	2.173	0.498	0.990	-0.806	0.000	0.627	2.387	-1.283	0.000	0.607	1.290	-0.174	3.102	0.916	1.328	0.986	0.557	0.971	0.935	0.836
-1	0.562	-0.360	0.399	0.803	-1.334	1.443	-0.116	1.628	2.173	0.750	0.987	0.135	1.107	0.795	0.298	-0.556	0.000	1.150	-0.113	-0.093	0.000	0.493	1.332	0.985	1.001	1.750	1.013	0.886
-1	0.987	0.706	-0.492	0.861	0.607	0.593	0.088	-0.184	0.000	0.802	0.894	1.608	2.215	0.782	-0.471	1.500	2.548	0.521	0.772	-0.960	0.000	0.658	0.893	1.068	0.877	0.664	0.709	0.661
-1	1.052	0.883	-0.581	1.566	0.860	0.931	1.515	-0.873	0.000	0.493	0.145	-0.672	0.000	1.133	0.935	1.581	2.548	1.630	0.695	0.923	3.102	1.105	1.087	1.713	0.948	0.590	0.872	0.883
-1	2.130	-0.516	-0.291	0.776	-1.230	0.689	-0.257	0.800	2.173	0.730	-0.274	-1.437	0.000	0.615	0.241	1.083	0.000	0.834	0.757	1.613	3.102	0.836	0.806	1.333	1.061	0.730	0.889	0.783
-1	0.742	0.797	1.628	0.311	-0.418	0.620	0.685	-1.457	0.000	0.683	1.774	-1.082	0.000	1.700	1.104	0.225	2.548	0.382	-2.184	-1.307	0.000	0.945	1.228	0.984	0.864	0.931	0.988	0.838
-0	0.311	-1.249	-0.927	1.272	-1.262	0.642	-1.228	-0.136	0.000	1.220	-0.804	-1.558	2.215	0.950	-0.828	0.495	1.274	2.149	-1.672	0.634	0.000	1.346	0.887	0.981	0.856	1.101	1.001	1.106
-0	0.660	-1.834	-0.667	0.601	1.236	0.932	-0.933	-0.135	2.173	1.373	-0.122	1.429	0.000	0.654	-0.034	-0.847	2.548	0.711	0.911	0.703	0.000	1.144	0.942	0.984	0.822	0.739	0.992	0.895
-0	3.609	-0.590	0.851	0.615	0.455	1.280	0.003	-0.866	1.087	1.334	0.708	-1.131	0.000	0.669	0.480	0.092	0.000	0.975	0.983	-1.429	3.102	1.301	1.089	0.987	1.476	0.934	1.469	1.352
-1	0.905	-0.403	1.567	2.651	0.953	1.194	-0.241	-0.567	1.087	0.308	-0.384	-0.007	0.000	0.608	-0.175	-1.163	2.548	0.379	0.941	1.662	0.000	0.580	0.721	1.126	0.895	0.544	1.097	0.836
-1	0.983	0.255	1.093	0.905	-0.874	0.863	0.060	-0.368	0.000	0.824	-0.747	-0.633	0.000	0.614	0.961	1.052	0.000	0.792	-0.260	1.632	3.102	0.874	0.883	1.280	0.663	0.406	0.592	0.645
-1	1.160	-1.027	0.274	0.460	0.322	2.085	-1.623	-0.840	0.000	1.634	-1.046	1.182	2.215	0.492	-0.367	1.174	0.000	0.824	-0.998	1.617	0.000	0.943	0.884	1.001	1.209	1.313	1.034	0.866
-0	0.299	0.028	-1.372	1.930	-0.661	0.840	-0.979	0.664	1.087	0.535	-2.041	1.434	0.000	1.087	-1.797	0.344	0.000	0.485	-0.560	-1.105	3.102	0.951	0.890	0.980	0.483	0.684	0.730	0.706
-0	0.293	1.737	-1.418	2.074	0.794	0.679	1.024	-1.457	0.000	1.034	1.094	-0.168	1.107	0.506	1.680	-0.661	0.000	0.523	-0.042	-1.274	3.102	0.820	0.944	0.987	0.842	0.694	0.761	0.750
-0	0.457	-0.393	1.560	0.738	-0.007	0.475	-0.230	0.246	0.000	0.776	-1.264	-0.606	2.215	0.865	-0.731	-1.576	2.548	1.153	0.343	1.436	0.000	1.060	0.883	0.988	0.972	0.703	0.758	0.720
-0	0.935	-0.582	0.240	2.401	0.818	1.231	-0.618	-1.289	0.000	0.799	0.544	-0.228	2.215	0.525	-1.494	-0.969	0.000	0.609	-1.123	1.168	3.102	0.871	0.767	1.035	1.154	0.919	0.868	1.006
-1	0.902	-0.745	-1.215	1.174	-0.501	1.215	0.167	1.162	0.000	0.896	1.217	-0.976	0.000	0.585	-0.429	1.036	0.000	1.431	-0.416	0.151	3.102	0.524	0.952	0.990	0.707	0.271	0.592	0.826
-1	0.653	0.337	-0.320	1.118	-0.934	1.050	0.745	0.529	1.087	1.075	1.742	-1.538	0.000	0.585	1.090	0.973	0.000	1.091	-0.187	1.160	1.551	1.006	1.108	0.978	1.121	0.838	0.947	0.908
-0	1.157	1.401	0.340	0.395	-1.218	0.945	1.928	-0.876	0.000	1.384	0.320	1.002	1.107	1.900	1.177	-0.462	2.548	1.122	1.316	1.720	0.000	1.167	1.096	0.989	0.937	1.879	1.307	1.041
-0	0.960	0.355	-0.152	0.872	-0.338	0.391	0.348	0.956	1.087	0.469	2.664	1.409	0.000	0.756	-1.561	1.500	0.000	0.525	1.436	1.728	3.102	1.032	0.946	0.996	0.929	0.470	0.698	0.898
-1	1.038	0.274	0.825	1.198	0.963	1.078	-0.496	-1.014	2.173	0.739	-0.727	-0.151	2.215	1.035	-0.799	0.398	0.000	1.333	-0.872	-1.498	0.000	0.849	1.033	0.985	0.886	0.936	0.975	0.823
-0	0.490	0.277	0.318	1.303	0.694	1.333	-1.620	-0.563	0.000	1.459	-1.326	1.140	0.000	0.779	-0.673	-1.324	2.548	0.860	-1.247	0.043	0.000	0.857	0.932	0.992	0.792	0.278	0.841	1.498
-0	1.648	-0.688	-1.386	2.790	0.995	1.087	1.359	-0.687	0.000	1.050	-0.223	-0.261	2.215	0.613	-0.889	1.335	0.000	1.204	0.827	0.309	3.102	0.464	0.973	2.493	1.737	0.827	1.319	1.062
-0	1.510	-0.662	1.668	0.860	0.280	0.705	0.974	-1.647	1.087	0.662	-0.393	-0.225	0.000	0.610	-0.996	0.532	2.548	0.464	1.305	0.102	0.000	0.859	1.057	1.498	0.799	1.260	0.946	0.863
-1	0.850	-1.185	-0.117	0.943	-0.449	1.142	0.875	-0.030	0.000	2.223	-0.461	1.627	2.215	0.767	-1.761	-1.692	0.000	1.012	-0.727	0.639	3.102	3.649	2.062	0.985	1.478	1.087	1.659	1.358
-0	0.933	1.259	0.130	0.326	-0.890	0.306	1.136	1.142	0.000	0.964	0.705	-1.373	2.215	0.546	-0.196	-0.001	0.000	0.578	-1.169	1.004	3.102	0.830	0.836	0.988	0.837	1.031	0.749	0.655
-0	0.471	0.697	1.570	1.109	0.201	1.248	0.348	-1.448	0.000	2.103	0.773	0.686	2.215	1.451	-0.087	-0.453	2.548	1.197	-0.045	-1.026	0.000	0.793	1.094	0.987	0.851	1.804	1.378	1.089
-1	2.446	-0.701	-1.568	0.059	0.822	1.401	-0.600	-0.044	2.173	0.324	-0.001	1.344	2.215	0.913	-0.818	1.049	0.000	0.442	-1.088	-0.005	0.000	0.611	1.062	0.979	0.562	0.988	0.998	0.806
-0	0.619	2.029	0.933	0.528	-0.903	0.974	0.760	-0.311	2.173	0.825	0.658	-1.466	1.107	0.894	1.594	0.370	0.000	0.882	-0.258	1.661	0.000	1.498	1.088	0.987	0.867	1.139	0.900	0.779
-1	0.674	-0.131	-0.362	0.518	-1.574	0.876	0.442	0.145	1.087	0.497	-1.526	-1.704	0.000	0.680	2.514	-1.374	0.000	0.792	-0.479	0.773	1.551	0.573	1.198	0.984	0.800	0.667	0.987	0.832
-1	1.447	1.145	-0.937	0.307	-1.458	0.478	1.264	0.816	1.087	0.558	1.015	-0.101	2.215	0.937	-0.190	1.177	0.000	0.699	0.954	-1.512	0.000	0.877	0.838	0.990	0.873	0.566	0.646	0.713
-1	0.976	0.308	-0.844	0.436	0.610	1.253	0.149	-1.585	2.173	1.415	0.568	0.096	2.215	0.953	-0.855	0.441	0.000	0.867	-0.650	1.643	0.000	0.890	1.234	0.988	0.796	2.002	1.179	0.977
-0	0.697	0.401	-0.718	0.920	0.735	0.958	-0.172	0.168	2.173	0.872	-0.097	-1.335	0.000	0.513	-1.192	-1.710	1.274	0.426	-1.637	1.368	0.000	0.997	1.227	1.072	0.800	1.013	0.786	0.749
-1	1.305	-2.157	1.740	0.661	-0.912	0.705	-0.516	0.759	2.173	0.989	-0.716	-0.300	2.215	0.627	-1.052	-1.736	0.000	0.467	-2.467	0.568	0.000	0.807	0.964	0.988	1.427	1.012	1.165	0.926
-0	1.847	1.663	-0.618	0.280	1.258	1.462	-0.054	1.371	0.000	0.900	0.309	-0.544	0.000	0.331	-2.149	-0.341	0.000	1.091	-0.833	0.710	3.102	1.496	0.931	0.989	1.549	0.115	1.140	1.150
-0	0.410	-0.323	1.069	2.160	0.010	0.892	0.942	-1.640	2.173	0.946	0.938	1.314	0.000	1.213	-1.099	-0.794	2.548	0.650	0.053	0.056	0.000	1.041	0.916	1.063	0.985	1.910	1.246	1.107
-1	0.576	1.092	-0.088	0.777	-1.579	0.757	0.271	0.109	0.000	0.819	0.827	-1.554	2.215	1.313	2.341	-1.568	0.000	2.827	0.239	-0.338	0.000	0.876	0.759	0.986	0.692	0.457	0.796	0.791
-1	0.537	0.925	-1.406	0.306	-0.050	0.906	1.051	0.037	0.000	1.469	-0.177	-1.320	2.215	1.872	0.723	1.158	0.000	1.313	0.227	-0.501	3.102	0.953	0.727	0.978	0.755	0.892	0.932	0.781
-0	0.716	-0.065	-0.484	1.313	-1.563	0.596	-0.242	0.678	2.173	0.426	-1.909	0.616	0.000	0.885	-0.406	-1.343	2.548	0.501	-1.327	-0.340	0.000	0.470	0.728	1.109	0.919	0.881	0.665	0.692
-1	0.624	-0.389	0.128	1.636	-1.110	1.025	0.573	-0.843	2.173	0.646	-0.697	1.064	0.000	0.632	-1.442	0.961	0.000	0.863	-0.106	1.717	0.000	0.825	0.917	1.257	0.983	0.713	0.890	0.824
-0	0.484	2.101	1.714	1.131	-0.823	0.750	0.583	-1.304	1.087	0.894	0.421	0.559	2.215	0.921	-0.063	0.282	0.000	0.463	-0.474	-1.387	0.000	0.742	0.886	0.995	0.993	1.201	0.806	0.754
-0	0.570	0.339	-1.478	0.528	0.439	0.978	1.479	-1.411	2.173	0.763	1.541	-0.734	0.000	1.375	0.840	0.903	0.000	0.965	1.599	0.364	0.000	0.887	1.061	0.992	1.322	1.453	1.013	0.969
-0	0.940	1.303	1.636	0.851	-1.732	0.803	-0.030	-0.177	0.000	0.480	-0.125	-0.954	0.000	0.944	0.709	0.296	2.548	1.342	-0.418	1.197	3.102	0.853	0.989	0.979	0.873	0.858	0.719	0.786
-1	0.599	0.544	-0.238	0.816	1.043	0.857	0.660	1.128	2.173	0.864	-0.624	-0.843	0.000	1.159	0.367	0.174	0.000	1.520	-0.543	-1.508	0.000	0.842	0.828	0.984	0.759	0.895	0.918	0.791
-1	1.651	1.897	-0.914	0.423	0.315	0.453	0.619	-1.607	2.173	0.532	-0.424	0.209	1.107	0.369	2.479	0.034	0.000	0.701	0.217	0.984	0.000	0.976	0.951	1.035	0.879	0.825	0.915	0.798
-1	0.926	-0.574	-0.763	0.285	1.094	0.672	2.314	1.545	0.000	1.124	0.415	0.809	0.000	1.387	0.270	-0.949	2.548	1.547	-0.631	-0.200	3.102	0.719	0.920	0.986	0.889	0.933	0.797	0.777
-0	0.677	1.698	-0.890	0.641	-0.449	0.607	1.754	1.720	0.000	0.776	0.372	0.782	2.215	0.511	1.491	-0.480	0.000	0.547	-0.341	0.853	3.102	0.919	1.026	0.997	0.696	0.242	0.694	0.687
-0	1.266	0.602	0.958	0.487	1.256	0.709	0.843	-1.196	0.000	0.893	1.303	-0.594	1.107	1.090	1.320	0.354	0.000	0.797	1.846	1.139	0.000	0.780	0.896	0.986	0.661	0.709	0.790	0.806
-1	0.628	-0.616	-0.329	0.764	-1.150	0.477	-0.715	1.187	2.173	1.250	0.607	1.026	2.215	0.983	-0.023	-0.583	0.000	0.377	1.344	-1.015	0.000	0.744	0.954	0.987	0.837	0.841	0.795	0.694
-1	1.035	-0.828	-1.358	1.870	-1.060	1.075	0.130	0.448	2.173	0.660	0.697	0.641	0.000	0.425	1.006	-1.035	0.000	0.751	1.055	1.364	3.102	0.826	0.822	0.988	0.967	0.901	1.077	0.906
-1	0.830	0.265	-0.150	0.660	1.105	0.592	-0.557	0.908	2.173	0.670	-1.419	-0.671	0.000	1.323	-0.409	1.644	2.548	0.850	-0.033	-0.615	0.000	0.760	0.967	0.984	0.895	0.681	0.747	0.770
-1	1.395	1.100	1.167	1.088	0.218	0.400	-0.132	0.024	2.173	0.743	0.530	-1.361	2.215	0.341	-0.691	-0.238	0.000	0.396	-1.426	-0.933	0.000	0.363	0.472	1.287	0.922	0.810	0.792	0.656
-1	1.070	1.875	-1.298	1.215	-0.106	0.767	0.795	0.514	1.087	0.401	2.780	1.276	0.000	0.686	1.127	1.721	2.548	0.391	-0.259	-1.167	0.000	1.278	1.113	1.389	0.852	0.824	0.838	0.785
-0	1.114	-0.071	1.719	0.399	-1.383	0.849	0.254	0.481	0.000	0.958	-0.579	0.742	0.000	1.190	-0.140	-0.862	2.548	0.479	1.390	0.856	0.000	0.952	0.988	0.985	0.764	0.419	0.835	0.827
-0	0.714	0.376	-0.568	1.578	-1.165	0.648	0.141	0.639	2.173	0.472	0.569	1.449	1.107	0.783	1.483	0.361	0.000	0.540	-0.790	0.032	0.000	0.883	0.811	0.982	0.775	0.572	0.760	0.745
-0	0.401	-1.731	0.765	0.974	1.648	0.652	-1.024	0.191	0.000	0.544	-0.366	-1.246	2.215	0.627	0.140	1.008	2.548	0.810	0.409	0.429	0.000	0.950	0.934	0.977	0.621	0.580	0.677	0.650
-1	0.391	1.679	-1.298	0.605	-0.832	0.549	1.338	0.522	2.173	1.244	0.884	1.070	0.000	1.002	0.846	-1.345	2.548	0.783	-2.464	-0.237	0.000	4.515	2.854	0.981	0.877	0.939	1.942	1.489
-1	0.513	-0.220	-0.444	1.699	0.479	1.109	0.181	-0.999	2.173	0.883	-0.335	-1.716	2.215	1.075	-0.380	1.352	0.000	0.857	0.048	0.147	0.000	0.937	0.758	0.986	1.206	0.958	0.949	0.876
-0	1.367	-0.388	0.798	1.158	1.078	0.811	-1.024	-1.628	0.000	1.504	0.097	-0.999	2.215	1.652	-0.860	0.054	2.548	0.573	-0.142	-1.401	0.000	0.869	0.833	1.006	1.412	1.641	1.214	1.041
-1	1.545	-0.533	-1.517	1.177	1.289	2.331	-0.370	-0.073	0.000	1.295	-0.358	-0.891	2.215	0.476	0.756	0.985	0.000	1.945	-0.016	-1.651	3.102	1.962	1.692	1.073	0.656	0.941	1.312	1.242
-0	0.858	0.978	-1.258	0.286	0.161	0.729	1.230	1.087	2.173	0.561	2.670	-0.109	0.000	0.407	2.346	0.938	0.000	1.078	0.729	-0.658	3.102	0.597	0.921	0.982	0.579	0.954	0.733	0.769
-1	1.454	-1.384	0.870	0.067	0.394	1.033	-0.673	0.318	0.000	1.166	-0.763	-1.533	2.215	2.848	-0.045	-0.856	2.548	0.697	-0.140	1.134	0.000	0.931	1.293	0.977	1.541	1.326	1.201	1.078
-1	0.559	-0.913	0.486	1.104	-0.321	1.073	-0.348	1.345	0.000	0.901	-0.827	-0.842	0.000	0.739	0.047	-0.415	2.548	0.433	-1.132	1.268	0.000	0.797	0.695	0.985	0.868	0.346	0.674	0.623
-1	1.333	0.780	-0.964	0.916	1.202	1.822	-0.071	0.742	2.173	1.486	-0.399	-0.824	0.000	0.740	0.568	-0.134	0.000	0.971	-0.070	-1.589	3.102	1.278	0.929	1.421	1.608	1.214	1.215	1.137
-1	2.417	0.631	-0.317	0.323	0.581	0.841	1.524	-1.738	0.000	0.543	1.176	-0.325	0.000	0.827	0.700	0.866	0.000	0.834	-0.262	-1.702	3.102	0.932	0.820	0.988	0.646	0.287	0.595	0.589
-0	0.955	-1.242	0.938	1.104	0.474	0.798	-0.743	1.535	0.000	1.356	-1.357	-1.080	2.215	1.320	-1.396	-0.132	2.548	0.728	-0.529	-0.633	0.000	0.832	0.841	0.988	0.923	1.077	0.988	0.816
-1	1.305	-1.918	0.391	1.161	0.063	0.724	2.593	1.481	0.000	0.592	-1.207	-0.329	0.000	0.886	-0.836	-1.168	2.548	1.067	-1.481	-1.440	0.000	0.916	0.688	0.991	0.969	0.550	0.665	0.638
-0	1.201	0.071	-1.123	2.242	-1.533	0.702	-0.256	0.688	0.000	0.967	0.491	1.040	2.215	1.271	-0.558	0.095	0.000	1.504	0.676	-0.383	3.102	0.917	1.006	0.985	1.017	1.057	0.928	1.057
-0	0.994	-1.607	1.596	0.774	-1.391	0.625	-0.134	-0.862	2.173	0.746	-0.765	-0.316	2.215	1.131	-0.320	0.869	0.000	0.607	0.826	0.301	0.000	0.798	0.967	0.999	0.880	0.581	0.712	0.774
-1	0.482	-0.467	0.729	1.419	1.458	0.824	0.376	-0.242	0.000	1.368	0.023	1.459	2.215	0.826	0.669	-1.079	2.548	0.936	2.215	-0.309	0.000	1.883	1.216	0.997	1.065	0.946	1.224	1.526
-1	0.383	1.588	1.611	0.748	1.194	0.866	-0.279	-0.636	0.000	0.707	0.536	0.801	2.215	1.647	-1.155	0.367	0.000	1.292	0.303	-1.681	3.102	2.016	1.581	0.986	0.584	0.684	1.107	0.958
-0	0.629	0.203	0.736	0.671	-0.271	1.350	-0.486	0.761	2.173	0.496	-0.805	-1.718	0.000	2.393	0.044	-1.046	1.274	0.651	-0.116	-0.541	0.000	0.697	1.006	0.987	1.069	2.317	1.152	0.902
-0	0.905	-0.564	-0.570	0.263	1.096	1.219	-1.397	-1.414	1.087	1.164	-0.533	-0.208	0.000	1.459	1.965	0.784	0.000	2.220	-1.421	0.452	0.000	0.918	1.360	0.993	0.904	0.389	2.118	1.707
-1	1.676	1.804	1.171	0.529	1.175	1.664	0.354	-0.530	0.000	1.004	0.691	-1.280	2.215	0.838	0.373	0.626	2.548	1.094	1.774	0.501	0.000	0.806	1.100	0.991	0.769	0.976	0.807	0.740
-1	1.364	-1.936	0.020	1.327	0.428	1.021	-1.665	-0.907	2.173	0.818	-2.701	1.303	0.000	0.716	-0.590	-1.629	2.548	0.895	-2.280	-1.602	0.000	1.211	0.849	0.989	1.320	0.864	1.065	0.949
-0	0.629	-0.626	0.609	1.828	1.280	0.644	-0.856	-0.873	2.173	0.555	1.066	-0.640	0.000	0.477	-1.364	-1.021	2.548	1.017	0.036	0.380	0.000	0.947	0.941	0.994	1.128	0.241	0.793	0.815
-1	1.152	-0.843	0.926	1.802	0.800	2.493	-1.449	-1.127	0.000	1.737	0.833	0.488	0.000	1.026	0.929	-0.990	2.548	1.408	0.689	1.142	3.102	1.171	0.956	0.993	2.009	0.867	1.499	1.474
-0	2.204	0.081	0.008	1.021	-0.679	2.676	0.090	1.163	0.000	2.210	-1.686	-1.195	0.000	1.805	0.891	-0.148	2.548	0.450	-0.502	-1.295	3.102	6.959	3.492	1.205	0.908	0.845	2.690	2.183
-1	0.957	0.954	1.702	0.043	-0.503	1.113	0.033	-0.308	0.000	0.757	-0.363	-1.129	2.215	1.635	0.068	1.048	1.274	0.415	-2.098	0.061	0.000	1.010	0.979	0.992	0.704	1.125	0.761	0.715
-0	1.222	0.418	1.059	1.303	1.442	0.282	-1.499	-1.286	0.000	1.567	0.016	-0.164	2.215	0.451	2.229	-1.229	0.000	0.660	-0.513	-0.296	3.102	2.284	1.340	0.985	1.531	0.314	1.032	1.094
-1	0.603	1.675	-0.973	0.703	-1.709	1.023	0.652	1.296	2.173	1.078	0.363	-0.263	0.000	0.734	-0.457	-0.745	1.274	0.561	1.434	-0.042	0.000	0.888	0.771	0.984	0.847	1.234	0.874	0.777
-0	0.897	0.949	-0.848	1.115	-0.085	0.522	-1.267	-1.418	0.000	0.684	-0.599	1.474	0.000	1.176	0.922	0.641	2.548	0.470	0.103	0.148	3.102	0.775	0.697	0.984	0.839	0.358	0.847	1.008
-1	0.987	1.013	-1.504	0.468	-0.259	1.160	0.476	-0.971	2.173	1.266	0.919	0.780	0.000	0.634	1.695	0.233	0.000	0.487	-0.082	0.719	3.102	0.921	0.641	0.991	0.730	0.828	0.952	0.807
-1	0.847	1.581	-1.397	1.629	1.529	1.053	0.816	-0.344	2.173	0.895	0.779	0.332	0.000	0.750	1.311	0.419	2.548	1.604	0.844	1.367	0.000	1.265	0.798	0.989	1.328	0.783	0.930	0.879
-1	0.805	1.416	-1.327	0.397	0.589	0.488	0.982	0.843	0.000	0.664	-0.999	0.129	0.000	0.624	0.613	-0.558	0.000	1.431	-0.667	-1.561	3.102	0.959	1.103	0.989	0.590	0.632	0.926	0.798
-0	1.220	-0.313	-0.489	1.759	0.201	1.698	-0.220	0.241	2.173	1.294	1.390	-1.682	0.000	1.447	-1.623	-1.296	0.000	1.710	0.872	-1.356	3.102	1.198	0.981	1.184	0.859	2.165	1.807	1.661
-0	0.772	-0.611	-0.549	0.465	-1.528	1.103	-0.140	0.001	2.173	0.854	-0.406	1.655	0.000	0.733	-1.250	1.072	0.000	0.883	0.627	-1.132	3.102	0.856	0.927	0.987	1.094	1.013	0.938	0.870
-1	1.910	0.771	0.828	0.231	1.267	1.398	1.455	-0.295	2.173	0.837	-2.564	0.770	0.000	0.540	2.189	1.287	0.000	1.345	1.311	-1.151	0.000	0.861	0.869	0.984	1.359	1.562	1.105	0.963
-1	0.295	0.832	1.399	1.222	-0.517	2.480	0.013	1.591	0.000	2.289	0.436	0.287	2.215	1.995	-0.367	-0.409	1.274	0.375	1.367	-1.716	0.000	1.356	2.171	0.990	1.467	1.664	1.855	1.705
-1	1.228	0.339	-0.575	0.417	1.474	0.480	-1.416	-1.498	2.173	0.614	-0.933	-0.961	0.000	1.189	1.690	1.003	0.000	1.690	-1.065	0.106	3.102	0.963	1.147	0.987	1.086	0.948	0.930	0.866
-0	2.877	-1.014	1.440	0.782	0.483	1.134	-0.735	-0.196	2.173	1.123	0.084	-0.596	0.000	1.796	-0.356	1.044	2.548	1.406	1.582	-0.991	0.000	0.939	1.178	1.576	0.996	1.629	1.216	1.280
-1	2.178	0.259	1.107	0.256	1.222	0.979	-0.440	-0.538	1.087	0.496	-0.760	-0.049	0.000	1.471	1.683	-1.486	0.000	0.646	0.695	-1.577	3.102	1.093	1.070	0.984	0.608	0.889	0.962	0.866
-1	0.604	0.592	1.295	0.964	0.348	1.178	-0.016	0.832	2.173	1.626	-0.420	-0.760	0.000	0.748	0.461	-0.906	0.000	0.728	0.309	-1.269	1.551	0.852	0.604	0.989	0.678	0.949	1.021	0.878
-0	0.428	-1.352	-0.912	1.713	0.797	1.894	-1.452	0.191	2.173	2.378	2.113	-1.190	0.000	0.860	2.174	0.949	0.000	1.693	0.759	1.426	3.102	0.885	1.527	1.186	1.090	3.294	4.492	3.676
-0	0.473	0.485	0.154	1.433	-1.504	0.766	1.257	-1.302	2.173	0.414	0.119	0.238	0.000	0.805	0.242	-0.691	2.548	0.734	0.749	0.753	0.000	0.430	0.893	1.137	0.686	0.724	0.618	0.608
-1	0.763	-0.601	0.876	0.182	-1.678	0.818	0.599	0.481	2.173	0.658	-0.737	-0.553	0.000	0.857	-1.138	-1.435	0.000	1.540	-1.466	-0.447	0.000	0.870	0.566	0.989	0.728	0.658	0.821	0.726
-0	0.619	-0.273	-0.143	0.992	-1.267	0.566	0.876	-1.396	2.173	0.515	0.892	0.618	0.000	0.434	-0.902	0.862	2.548	0.490	-0.539	0.549	0.000	0.568	0.794	0.984	0.667	0.867	0.597	0.578
-0	0.793	0.970	0.324	0.570	0.816	0.761	-0.550	1.519	2.173	1.150	0.496	-0.447	0.000	0.925	0.724	1.008	1.274	1.135	-0.275	-0.843	0.000	0.829	1.068	0.978	1.603	0.892	1.041	1.059
-1	0.480	0.364	-0.067	1.906	-1.582	1.397	1.159	0.140	0.000	0.639	0.398	-1.102	0.000	1.597	-0.668	1.607	2.548	1.306	-0.797	0.288	3.102	0.856	1.259	1.297	1.022	1.032	1.049	0.939
-0	0.514	1.304	1.490	1.741	-0.220	0.648	0.155	0.535	0.000	0.562	-1.016	0.837	0.000	0.863	-0.780	-0.815	2.548	1.688	-0.130	-1.545	3.102	0.887	0.980	1.309	1.269	0.654	1.044	1.035
-0	1.225	0.333	0.656	0.893	0.859	1.037	-0.876	1.603	1.087	1.769	0.272	-0.227	2.215	1.000	0.579	-1.690	0.000	1.385	0.471	-0.860	0.000	0.884	1.207	0.995	1.097	2.336	1.282	1.145
-0	2.044	-1.472	-0.294	0.392	0.369	0.927	0.718	1.492	1.087	1.619	-0.736	0.047	2.215	1.884	-0.101	-1.540	0.000	0.548	-0.441	1.117	0.000	0.798	0.877	0.981	0.750	2.272	1.469	1.276
-0	1.037	-0.276	0.735	3.526	1.156	2.498	0.401	-0.590	1.087	0.714	-1.203	1.393	2.215	0.681	0.629	1.534	0.000	0.719	-0.355	-0.706	0.000	0.831	0.857	0.988	2.864	2.633	1.988	1.466
-1	0.651	-1.218	-0.791	0.770	-1.449	0.610	-0.535	0.960	2.173	0.380	-1.072	-0.031	2.215	0.415	2.123	-1.100	0.000	0.776	0.217	0.420	0.000	0.986	1.008	1.001	0.853	0.588	0.799	0.776
-0	1.586	-0.409	0.085	3.258	0.405	1.647	-0.674	-1.519	0.000	0.640	-1.027	-1.681	0.000	1.452	-0.444	-0.957	2.548	0.927	-0.017	1.215	3.102	0.519	0.866	0.992	0.881	0.847	1.018	1.278
-0	0.712	0.092	-0.466	0.688	1.236	0.921	-1.217	-1.022	2.173	2.236	-1.167	0.868	2.215	0.851	-1.892	-0.753	0.000	0.475	-1.216	-0.383	0.000	0.668	0.758	0.988	1.180	2.093	1.157	0.934
-0	0.419	0.471	0.974	2.805	0.235	1.473	-0.198	1.255	1.087	0.931	1.083	-0.712	0.000	1.569	1.358	-1.179	2.548	2.506	0.199	-0.842	0.000	0.929	0.991	0.992	1.732	2.367	1.549	1.430
-1	0.667	1.003	1.504	0.368	1.061	0.885	-0.318	-0.353	0.000	1.438	-1.939	0.710	0.000	1.851	0.277	-1.460	2.548	1.403	0.517	-0.157	0.000	0.883	1.019	1.000	0.790	0.859	0.938	0.841
-1	1.877	-0.492	0.372	0.441	0.955	1.034	-1.220	-0.846	1.087	0.952	-0.320	1.125	0.000	0.542	0.308	-1.261	2.548	1.018	-1.415	-1.547	0.000	1.280	0.932	0.991	1.273	0.878	0.921	0.906
-0	1.052	0.901	1.176	1.280	1.517	0.562	-1.150	-0.079	2.173	1.228	-0.308	-0.354	0.000	0.790	-1.492	-0.963	0.000	0.942	-0.672	-1.588	3.102	1.116	0.902	0.988	1.993	0.765	1.375	1.325
-1	0.518	-0.254	1.642	0.865	0.725	0.980	0.734	0.023	0.000	1.448	0.780	-1.736	2.215	0.955	0.513	-0.519	0.000	0.365	-0.444	-0.243	3.102	0.833	0.555	0.984	0.827	0.795	0.890	0.786
-0	0.870	0.815	-0.506	0.663	-0.518	0.935	0.289	-1.675	2.173	1.188	0.005	0.635	0.000	0.580	0.066	-1.455	2.548	0.580	-0.634	-0.199	0.000	0.852	0.788	0.979	1.283	0.208	0.856	0.950
-0	0.628	1.382	0.135	0.683	0.571	1.097	0.564	-0.950	2.173	0.617	-0.326	0.371	0.000	1.093	0.918	1.667	2.548	0.460	1.221	0.708	0.000	0.743	0.861	0.975	1.067	1.007	0.843	0.762
-0	4.357	0.816	-1.609	1.845	-1.288	3.292	0.726	0.324	2.173	1.528	0.583	-0.801	2.215	0.605	0.572	1.406	0.000	0.794	-0.791	0.122	0.000	0.967	1.132	1.124	3.602	2.811	2.460	1.861
-0	0.677	-1.265	1.559	0.866	-0.618	0.823	0.260	0.185	0.000	1.133	0.337	1.589	2.215	0.563	-0.830	0.510	0.000	0.777	0.117	-0.941	3.102	0.839	0.763	0.986	1.182	0.649	0.796	0.851
-0	2.466	-1.838	-1.648	1.717	1.533	1.676	-1.553	-0.109	2.173	0.670	-0.666	0.284	0.000	0.334	-2.480	0.316	0.000	0.366	-0.804	-1.298	3.102	0.875	0.894	0.997	0.548	0.770	1.302	1.079
-1	1.403	0.129	-1.307	0.688	0.306	0.579	0.753	0.814	1.087	0.474	0.694	-1.400	0.000	0.520	1.995	0.185	0.000	0.929	-0.504	1.270	3.102	0.972	0.998	1.353	0.948	0.650	0.688	0.724
-1	0.351	1.188	-0.360	0.254	-0.346	1.129	0.545	1.691	0.000	0.652	-0.039	-0.258	2.215	1.089	0.655	0.472	2.548	0.554	-0.493	1.366	0.000	0.808	1.045	0.992	0.570	0.649	0.809	0.744
-0	1.875	-0.013	-0.128	0.236	1.163	0.902	0.426	0.590	2.173	1.251	-1.210	-0.616	0.000	1.035	1.534	0.912	0.000	1.944	1.789	-1.691	0.000	0.974	1.113	0.990	0.925	1.120	0.956	0.912
-0	0.298	0.750	-0.507	1.555	1.463	0.804	1.200	-0.665	0.000	0.439	-0.829	-0.252	1.107	0.770	-1.090	0.947	2.548	1.165	-0.166	-0.763	0.000	1.140	0.997	0.988	1.330	0.555	1.005	1.012
-0	0.647	0.342	0.245	4.340	-0.157	2.229	0.068	1.170	2.173	2.133	-0.201	-1.441	0.000	1.467	0.697	-0.532	1.274	1.457	0.583	-1.640	0.000	0.875	1.417	0.976	2.512	2.390	1.794	1.665
-1	1.731	-0.803	-1.013	1.492	-0.020	1.646	-0.541	1.121	2.173	0.459	-1.251	-1.495	2.215	0.605	-1.711	-0.232	0.000	0.658	0.634	-0.068	0.000	1.214	0.886	1.738	1.833	1.024	1.192	1.034
-0	0.515	1.416	-1.089	1.697	1.426	1.414	0.941	0.027	0.000	1.480	0.133	-1.595	2.215	1.110	0.752	0.760	2.548	1.062	0.697	-0.492	0.000	0.851	0.955	0.994	1.105	1.255	1.175	1.095
-0	1.261	0.858	1.465	0.757	0.305	2.310	0.679	1.080	2.173	1.544	2.518	-0.464	0.000	2.326	0.270	-0.841	0.000	2.163	0.839	-0.500	3.102	0.715	0.825	1.170	0.980	2.371	1.527	1.221
-1	1.445	1.509	1.471	0.414	-1.285	0.767	0.864	-0.677	2.173	0.524	1.388	0.171	0.000	0.826	0.190	0.121	2.548	0.572	1.691	-1.603	0.000	0.870	0.935	0.994	0.968	0.735	0.783	0.777
-1	0.919	-0.264	-1.245	0.681	-1.722	1.022	1.010	0.097	2.173	0.685	0.403	-1.351	0.000	1.357	-0.429	1.262	1.274	0.687	1.021	-0.563	0.000	0.953	0.796	0.991	0.873	1.749	1.056	0.917
-1	0.293	-2.258	-1.427	1.191	1.202	0.394	-2.030	1.438	0.000	0.723	0.596	-0.024	2.215	0.525	-1.678	-0.290	0.000	0.788	-0.824	-1.029	3.102	0.821	0.626	0.976	1.080	0.810	0.842	0.771
-0	3.286	0.386	1.688	1.619	-1.620	1.392	-0.009	0.280	0.000	1.179	-0.776	-0.110	2.215	1.256	0.248	-1.114	2.548	0.777	0.825	-0.156	0.000	1.026	1.065	0.964	0.909	1.249	1.384	1.395
-1	1.075	0.603	0.561	0.656	-0.685	0.985	0.175	0.979	2.173	1.154	0.584	-0.886	0.000	1.084	-0.354	-1.004	2.548	0.865	1.224	1.269	0.000	1.346	1.073	1.048	0.873	1.310	1.003	0.865
-1	1.098	-0.091	1.466	1.558	0.915	0.649	1.314	-1.182	2.173	0.791	0.073	0.351	0.000	0.517	0.940	1.195	0.000	1.150	1.187	-0.692	3.102	0.866	0.822	0.980	1.311	0.394	1.119	0.890
-1	0.481	-1.042	0.148	1.135	-1.249	1.202	-0.344	0.308	1.087	0.779	-1.431	1.581	0.000	0.860	-0.860	-1.125	0.000	0.785	0.303	1.199	3.102	0.878	0.853	0.988	1.072	0.827	0.936	0.815
-0	1.348	0.497	0.318	0.806	0.976	1.393	-0.152	0.632	2.173	2.130	0.515	-1.054	0.000	0.908	0.062	-0.780	0.000	1.185	0.687	1.668	1.551	0.720	0.898	0.985	0.683	1.292	1.320	1.131
-0	2.677	-0.420	-1.685	1.828	1.433	2.040	-0.718	-0.039	0.000	0.400	-0.873	0.472	0.000	0.444	0.340	-0.830	2.548	0.431	0.768	-1.417	3.102	0.869	0.917	0.996	0.707	0.193	0.728	1.154
-1	1.300	0.586	-0.122	1.306	0.609	0.727	-0.556	-1.652	2.173	0.636	0.720	1.393	2.215	0.328	1.280	-0.390	0.000	0.386	0.752	-0.905	0.000	0.202	0.751	1.106	0.864	0.799	0.928	0.717
-0	0.637	-0.176	1.737	1.322	-0.414	0.702	-0.964	-0.680	0.000	1.054	-0.461	0.889	2.215	0.861	-0.267	0.225	0.000	1.910	-1.888	1.027	0.000	0.919	0.899	1.186	0.993	1.109	0.862	0.775
-1	0.723	-0.104	1.572	0.428	-0.840	0.655	0.544	1.401	2.173	1.522	-0.154	-0.452	2.215	0.996	0.190	0.273	0.000	1.906	-0.176	0.966	0.000	0.945	0.894	0.990	0.981	1.555	0.988	0.893
-0	2.016	-0.570	1.612	0.798	0.441	0.334	0.191	-0.909	0.000	0.939	0.146	0.021	2.215	0.553	-0.444	1.156	2.548	0.781	-1.545	-0.520	0.000	0.922	0.956	1.528	0.722	0.699	0.778	0.901
-0	1.352	-0.707	1.284	0.665	0.580	0.694	-1.040	-0.899	2.173	0.692	-2.048	0.029	0.000	0.545	-2.042	1.259	0.000	0.661	-0.808	-1.251	3.102	0.845	0.991	0.979	0.662	0.225	0.685	0.769
-1	1.057	-1.561	-0.411	0.952	-0.681	1.236	-1.107	1.045	2.173	1.288	-2.521	-0.521	0.000	1.361	-1.239	1.546	0.000	0.373	-1.540	0.028	0.000	0.794	0.782	0.987	0.889	0.832	0.972	0.828
-0	1.118	-0.017	-1.227	1.077	1.256	0.714	0.624	-0.811	0.000	0.800	0.704	0.387	1.107	0.604	0.234	0.986	0.000	1.306	-0.456	0.094	3.102	0.828	0.984	1.195	0.987	0.672	0.774	0.748
-1	0.602	2.201	0.212	0.119	0.182	0.474	2.130	1.270	0.000	0.370	2.088	-0.573	0.000	0.780	-0.725	-1.033	0.000	1.642	0.598	0.303	3.102	0.886	0.988	0.985	0.644	0.756	0.651	0.599
-0	1.677	-0.844	1.581	0.585	0.887	1.012	-2.315	0.752	0.000	1.077	0.748	-0.195	0.000	0.718	0.832	-1.337	1.274	1.181	-0.557	-1.006	3.102	1.018	1.247	0.988	0.908	0.651	1.311	1.120
-1	1.695	0.259	1.224	1.344	1.067	0.718	-1.752	-0.215	0.000	0.473	0.991	-0.993	0.000	0.891	1.285	-1.500	2.548	0.908	-0.131	0.288	0.000	0.945	0.824	0.979	1.009	0.951	0.934	0.833
-0	0.793	0.628	0.432	1.707	0.302	0.919	1.045	-0.784	0.000	1.472	0.175	-1.284	2.215	1.569	0.155	0.971	2.548	0.435	0.735	1.625	0.000	0.801	0.907	0.992	0.831	1.446	1.082	1.051
-1	0.537	-0.664	-0.244	1.104	1.272	1.154	0.394	1.633	0.000	1.527	0.963	0.559	2.215	1.744	0.650	-0.912	0.000	1.097	0.730	-0.368	3.102	1.953	1.319	1.045	1.309	0.869	1.196	1.126
-1	0.585	-1.469	1.005	0.749	-1.060	1.224	-0.717	-0.323	2.173	1.012	-0.201	1.268	0.000	0.359	-0.567	0.476	0.000	1.117	-1.124	1.557	3.102	0.636	1.281	0.986	0.616	1.289	0.890	0.881
-1	0.354	-1.517	0.667	2.534	-1.298	1.020	-0.375	1.254	0.000	1.119	-0.060	-1.538	2.215	1.059	-0.395	-0.140	0.000	2.609	0.199	-0.778	1.551	0.957	0.975	1.286	1.666	1.003	1.224	1.135
-1	0.691	-1.619	-1.380	0.361	1.727	1.493	-1.093	-0.289	0.000	1.447	-0.640	1.341	0.000	1.453	-0.617	-1.456	1.274	1.061	-1.481	-0.091	0.000	0.744	0.649	0.987	0.596	0.727	0.856	0.797
-0	1.336	1.293	-1.359	0.357	0.067	1.110	-0.058	-0.515	0.000	0.976	1.498	1.207	0.000	1.133	0.437	1.053	2.548	0.543	1.374	0.171	0.000	0.764	0.761	0.984	0.827	0.553	0.607	0.612
-0	0.417	-1.111	1.661	2.209	-0.683	1.931	-0.642	0.959	1.087	1.514	-2.032	-0.686	0.000	1.521	-0.539	1.344	0.000	0.978	-0.866	0.363	1.551	2.813	1.850	1.140	1.854	0.799	1.600	1.556
-0	1.058	0.390	-0.591	0.134	1.149	0.346	-1.550	0.186	0.000	1.108	-0.999	0.843	1.107	1.124	0.415	-1.514	0.000	1.067	-0.426	-1.000	3.102	1.744	1.050	0.985	1.006	1.010	0.883	0.789
-1	1.655	0.253	1.216	0.270	1.703	0.500	-0.006	-1.418	2.173	0.690	-0.350	0.170	2.215	1.045	-0.924	-0.774	0.000	0.996	-0.745	-0.123	0.000	0.839	0.820	0.993	0.921	0.869	0.725	0.708
-0	1.603	-0.850	0.564	0.829	0.093	1.270	-1.113	-1.155	2.173	0.853	-1.021	1.248	2.215	0.617	-1.270	1.733	0.000	0.935	-0.092	0.136	0.000	1.011	1.074	0.977	0.823	1.269	1.054	0.878
-0	1.568	-0.792	1.005	0.545	0.896	0.895	-1.698	-0.988	0.000	0.608	-1.634	1.705	0.000	0.826	0.208	0.618	1.274	2.063	-1.743	-0.520	0.000	0.939	0.986	0.990	0.600	0.435	1.033	1.087
-0	0.489	-1.335	-1.102	1.738	1.028	0.628	-0.992	-0.627	0.000	0.652	-0.064	-0.215	0.000	1.072	0.173	-1.251	2.548	1.042	0.057	0.841	3.102	0.823	0.895	1.200	1.164	0.770	0.837	0.846
-1	1.876	0.870	1.234	0.556	-1.262	1.764	0.855	-0.467	2.173	1.079	1.351	0.852	0.000	0.773	0.383	0.874	0.000	1.292	0.829	-1.228	3.102	0.707	0.969	1.102	1.601	1.017	1.112	1.028
-0	1.033	0.407	-0.374	0.705	-1.254	0.690	-0.231	1.502	2.173	0.433	-2.009	-0.057	0.000	0.861	1.151	0.334	0.000	0.960	-0.839	1.299	3.102	2.411	1.480	0.982	0.995	0.377	1.012	0.994
-0	1.092	0.653	-0.801	0.463	0.426	0.529	-1.055	0.040	0.000	0.663	0.999	1.255	1.107	0.749	-1.106	1.185	2.548	0.841	-0.745	-1.029	0.000	0.841	0.743	0.988	0.750	1.028	0.831	0.868
-1	0.799	-0.285	-0.011	0.531	1.392	1.063	0.854	0.494	2.173	1.187	-1.065	-0.851	0.000	0.429	-0.296	1.072	0.000	0.942	-1.985	1.172	0.000	0.873	0.693	0.992	0.819	0.689	1.131	0.913
-0	0.503	1.973	-0.377	1.515	-1.514	0.708	1.081	-0.313	2.173	1.110	-0.417	0.839	0.000	0.712	-1.153	1.165	0.000	0.675	-0.303	-0.930	1.551	0.709	0.761	1.032	0.986	0.698	0.963	1.291
-0	0.690	-0.574	-1.608	1.182	1.118	0.557	-2.243	0.144	0.000	0.969	0.216	-1.383	1.107	1.054	0.888	-0.709	2.548	0.566	1.663	-0.550	0.000	0.752	1.528	0.987	1.408	0.740	1.290	1.123
-1	0.890	1.501	0.786	0.779	-0.615	1.126	0.716	1.541	2.173	0.887	0.728	-0.673	2.215	1.216	0.332	-0.020	0.000	0.965	1.828	0.101	0.000	0.827	0.715	1.099	1.088	1.339	0.924	0.878
-0	0.566	0.883	0.655	1.600	0.034	1.155	2.028	-1.499	0.000	0.723	-0.871	0.763	0.000	1.286	-0.696	-0.676	2.548	1.134	-0.113	1.207	3.102	4.366	2.493	0.984	0.960	0.962	1.843	1.511
-0	1.146	1.086	-0.911	0.838	1.298	0.821	0.127	-0.145	0.000	1.352	0.474	-1.580	2.215	1.619	-0.081	0.675	2.548	1.382	-0.748	0.127	0.000	0.958	0.976	1.239	0.876	1.481	1.116	1.076
-0	1.739	-0.326	-1.661	0.420	-1.705	1.193	-0.031	-1.212	2.173	1.783	-0.442	0.522	0.000	1.064	-0.692	0.027	0.000	1.314	0.359	-0.037	3.102	0.968	0.897	0.986	0.907	1.196	1.175	1.112
-1	0.669	0.194	-0.703	0.657	-0.260	0.899	-2.511	0.311	0.000	1.482	0.773	0.974	2.215	3.459	0.037	-1.299	1.274	2.113	0.067	1.516	0.000	0.740	0.871	0.979	1.361	2.330	1.322	1.046
-1	1.355	-1.033	-1.173	0.552	-0.048	0.899	-0.482	-1.287	2.173	1.422	-1.227	0.390	1.107	1.937	-0.028	0.914	0.000	0.849	-0.230	-1.734	0.000	0.986	1.224	1.017	1.051	1.788	1.150	1.009
-1	0.511	-0.202	1.029	0.780	1.154	0.816	0.532	-0.731	0.000	0.757	0.517	0.749	2.215	1.302	0.289	-1.188	0.000	0.584	1.211	-0.350	0.000	0.876	0.943	0.995	0.963	0.256	0.808	0.891
-1	1.109	0.572	1.484	0.753	1.543	1.711	-0.145	-0.746	1.087	1.759	0.631	0.845	2.215	0.945	0.542	0.003	0.000	0.378	-1.150	-0.044	0.000	0.764	1.042	0.992	1.045	2.736	1.441	1.140
-0	0.712	-0.025	0.553	0.928	-0.711	1.304	0.045	-0.300	0.000	0.477	0.720	0.969	0.000	1.727	-0.474	1.328	1.274	1.282	2.222	1.684	0.000	0.819	0.765	1.023	0.961	0.657	0.799	0.744
-1	1.131	-0.302	1.079	0.901	0.236	0.904	-0.249	1.694	2.173	1.507	-0.702	-1.128	0.000	0.774	0.565	0.284	2.548	1.802	1.446	-0.192	0.000	3.720	2.108	0.986	0.930	1.101	1.484	1.238
-0	1.392	1.253	0.118	0.864	-1.358	0.922	-0.447	-1.243	1.087	1.969	1.031	0.774	2.215	1.333	-0.359	-0.681	0.000	1.099	-0.257	1.473	0.000	1.246	0.909	1.475	1.234	2.531	1.449	1.306
-0	1.374	2.291	-0.479	1.339	-0.243	0.687	2.345	1.310	0.000	0.467	1.081	0.772	0.000	0.656	1.155	-1.636	2.548	0.592	0.536	-1.269	3.102	0.981	0.821	1.010	0.877	0.217	0.638	0.758
-1	0.401	-1.516	0.909	2.738	0.519	0.887	0.566	-1.202	0.000	0.909	-0.176	1.682	0.000	2.149	-0.878	-0.514	2.548	0.929	-0.563	-1.555	3.102	1.228	0.803	0.980	1.382	0.884	1.025	1.172
-1	0.430	-1.589	1.417	2.158	1.226	1.180	-0.829	-0.781	2.173	0.798	1.400	-0.111	0.000	0.939	-0.878	1.076	2.548	0.576	1.335	-0.826	0.000	0.861	0.970	0.982	1.489	1.308	1.015	0.992
-1	1.943	-0.391	-0.840	0.621	-1.613	2.026	1.734	1.025	0.000	0.930	0.573	-0.912	0.000	1.326	0.847	-0.220	1.274	1.181	0.079	0.709	3.102	1.164	1.007	0.987	1.094	0.821	0.857	0.786
-1	0.499	0.436	0.887	0.859	1.509	0.733	-0.559	1.111	1.087	1.011	-0.796	0.279	2.215	1.472	-0.510	-0.982	0.000	1.952	0.379	-0.733	0.000	1.076	1.358	0.991	0.589	0.879	1.068	0.922
-0	0.998	-0.407	-1.711	0.139	0.652	0.810	-0.331	-0.721	0.000	0.471	-0.533	0.442	0.000	0.531	-1.405	0.120	2.548	0.707	0.098	-1.176	1.551	1.145	0.809	0.988	0.529	0.612	0.562	0.609
-1	1.482	0.872	0.638	1.288	0.362	0.856	0.900	-0.511	1.087	1.072	1.061	-1.432	2.215	1.770	-2.292	-1.547	0.000	1.131	1.374	0.783	0.000	6.316	4.381	1.002	1.317	1.048	2.903	2.351
-1	2.084	-0.422	1.289	1.125	0.735	1.104	-0.518	-0.326	2.173	0.413	-0.719	-0.699	0.000	0.857	0.108	-1.631	0.000	0.527	0.641	-1.362	3.102	0.791	0.952	1.016	0.776	0.856	0.987	0.836
-0	0.464	0.674	0.025	0.430	-1.703	0.982	-1.311	-0.808	2.173	1.875	1.060	0.821	2.215	0.954	-0.480	-1.677	0.000	0.567	0.702	-0.939	0.000	0.781	1.076	0.989	1.256	3.632	1.652	1.252
-1	0.457	-1.944	-1.010	1.409	0.931	1.098	-0.742	-0.415	0.000	1.537	-0.834	0.945	2.215	1.752	-0.287	-1.269	2.548	0.692	-1.537	-0.223	0.000	0.801	1.192	1.094	1.006	1.659	1.175	1.122
-0	3.260	-0.943	1.737	0.920	1.309	0.946	-0.139	-0.271	2.173	0.994	-0.952	-0.311	0.000	0.563	-0.136	-0.881	0.000	1.236	-0.507	0.906	1.551	0.747	0.869	0.985	1.769	1.034	1.179	1.042
-0	0.615	-0.778	0.246	1.861	1.619	0.560	-0.943	-0.204	2.173	0.550	-0.759	-1.342	2.215	0.578	0.076	-0.973	0.000	0.939	0.035	0.680	0.000	0.810	0.747	1.401	0.772	0.702	0.719	0.662
-1	2.370	-0.064	-0.237	1.737	0.154	2.319	-1.838	-1.673	0.000	1.053	-1.305	-0.075	0.000	0.925	0.149	0.318	1.274	0.851	-0.922	0.981	3.102	0.919	0.940	0.989	0.612	0.598	1.219	1.626
-1	1.486	0.311	-1.262	1.354	-0.847	0.886	-0.158	1.213	2.173	1.160	-0.218	0.239	0.000	1.166	0.494	0.278	2.548	0.575	1.454	-1.701	0.000	0.429	1.129	0.983	1.111	1.049	1.006	0.920
-1	1.294	1.587	-0.864	0.487	-0.312	0.828	1.051	-0.031	1.087	2.443	1.216	1.609	2.215	1.167	0.813	0.921	0.000	1.751	-0.415	0.119	0.000	1.015	1.091	0.974	1.357	2.093	1.178	1.059
-1	0.984	0.465	-1.661	0.379	-0.554	0.977	0.237	0.365	0.000	0.510	0.143	1.101	0.000	1.099	-0.662	-1.593	2.548	1.104	-0.197	-0.648	3.102	0.925	0.922	0.986	0.642	0.667	0.806	0.722
-1	0.930	-0.009	0.047	0.667	1.367	1.065	-0.231	0.815	0.000	1.199	-1.114	-0.877	2.215	0.940	0.824	-1.583	0.000	1.052	-0.407	-0.076	1.551	1.843	1.257	1.013	1.047	0.751	1.158	0.941
-0	0.767	-0.011	-0.637	0.341	-1.437	1.438	-0.425	-0.450	2.173	1.073	-0.718	1.341	2.215	0.633	-1.394	0.486	0.000	0.603	-1.945	-1.626	0.000	0.703	0.790	0.984	1.111	1.848	1.129	1.072
-1	1.779	0.017	0.432	0.402	1.022	0.959	1.480	1.595	2.173	1.252	1.365	0.006	0.000	1.188	-0.174	-1.107	0.000	1.181	0.518	-0.258	0.000	1.057	0.910	0.991	1.616	0.779	1.158	1.053
-0	0.881	0.630	1.029	1.990	0.508	1.102	0.742	-1.298	2.173	1.565	1.085	0.686	2.215	2.691	1.391	-0.904	0.000	0.499	1.388	-1.199	0.000	0.347	0.861	0.997	0.881	1.920	1.233	1.310
-0	1.754	-0.266	0.389	0.347	-0.030	0.462	-1.408	-0.957	2.173	0.515	-2.341	-1.700	0.000	0.588	-0.797	1.355	2.548	0.608	0.329	-1.389	0.000	1.406	0.909	0.988	0.760	0.593	0.768	0.847
-0	1.087	0.311	-1.447	0.173	0.567	0.854	0.362	0.584	0.000	1.416	-0.716	-1.211	2.215	0.648	-0.358	-0.692	1.274	0.867	-0.513	0.206	0.000	0.803	0.813	0.984	1.110	0.491	0.921	0.873
-0	0.279	1.114	-1.190	3.004	-0.738	1.233	0.896	1.092	2.173	0.454	-0.374	0.117	2.215	0.357	0.119	1.270	0.000	0.458	1.343	0.316	0.000	0.495	0.540	0.988	1.715	1.139	1.618	1.183
-1	1.773	-0.694	-1.518	2.306	-1.200	3.104	0.749	0.362	0.000	1.871	0.230	-1.686	2.215	0.805	-0.179	-0.871	1.274	0.910	0.607	-0.246	0.000	1.338	1.598	0.984	1.050	0.919	1.678	1.807
-0	0.553	0.683	0.827	0.973	-0.706	1.488	0.149	1.140	2.173	1.788	0.447	-0.478	0.000	0.596	1.043	1.607	0.000	0.373	-0.868	-1.308	1.551	1.607	1.026	0.998	1.134	0.808	1.142	0.936
-1	0.397	1.101	-1.139	1.688	0.146	0.972	0.541	1.518	0.000	1.549	-0.873	-1.012	0.000	2.282	-0.151	0.314	2.548	1.174	0.033	-1.368	0.000	0.937	0.776	1.039	1.143	0.959	0.986	1.013
-1	0.840	1.906	-0.959	0.869	0.576	0.642	0.554	-1.351	0.000	0.756	0.923	-0.823	2.215	1.251	1.130	0.545	2.548	1.513	0.410	1.073	0.000	1.231	0.985	1.163	0.812	0.987	0.816	0.822
-1	0.477	1.665	0.814	0.763	-0.382	0.828	-0.008	0.280	2.173	1.213	-0.001	1.560	0.000	1.136	0.311	-1.289	0.000	0.797	1.091	-0.616	3.102	1.026	0.964	0.992	0.772	0.869	0.916	0.803
-0	2.655	0.020	0.273	1.464	0.482	1.709	-0.107	-1.456	2.173	0.825	0.141	-0.386	0.000	1.342	-0.592	1.635	1.274	0.859	-0.175	-0.874	0.000	0.829	0.946	1.003	2.179	0.836	1.505	1.176
-0	0.771	-1.992	-0.720	0.732	-1.464	0.869	-1.290	0.388	2.173	0.926	-1.072	-1.489	2.215	0.640	-1.232	0.840	0.000	0.528	-2.440	-0.446	0.000	0.811	0.868	0.993	0.995	1.317	0.809	0.714
-0	1.357	1.302	0.076	0.283	-1.060	0.783	1.559	-0.994	0.000	0.947	1.212	1.617	0.000	1.127	0.311	0.442	2.548	0.582	-0.052	1.186	1.551	1.330	0.995	0.985	0.846	0.404	0.858	0.815
-0	0.442	-0.381	-0.424	1.244	0.591	0.731	0.605	-0.713	2.173	0.629	2.762	1.040	0.000	0.476	2.693	-0.617	0.000	0.399	0.442	1.486	3.102	0.839	0.755	0.988	0.869	0.524	0.877	0.918
-0	0.884	0.422	0.055	0.818	0.624	0.950	-0.763	1.624	0.000	0.818	-0.609	-1.166	0.000	1.057	-0.528	1.070	2.548	1.691	-0.124	-0.335	3.102	1.104	0.933	0.985	0.913	1.000	0.863	1.056
-0	1.276	0.156	1.714	1.053	-1.189	0.672	-0.464	-0.030	2.173	0.469	-2.483	0.442	0.000	0.564	2.580	-0.253	0.000	0.444	-0.628	1.080	1.551	5.832	2.983	0.985	1.162	0.494	1.809	1.513
-0	1.106	-0.556	0.406	0.573	-1.400	0.769	-0.518	1.457	2.173	0.743	-0.352	-0.010	0.000	1.469	-0.550	-0.930	2.548	0.540	1.236	-0.571	0.000	0.962	0.970	1.101	0.805	1.107	0.873	0.773
-0	0.539	-0.964	-0.464	1.371	-1.606	0.667	-0.160	0.655	0.000	0.952	0.352	-0.740	2.215	0.952	0.007	1.123	0.000	1.061	-0.505	1.389	3.102	1.063	0.991	1.019	0.633	0.967	0.732	0.799
-1	0.533	-0.989	-1.608	0.462	-1.723	1.204	-0.598	-0.098	2.173	1.343	-0.460	1.632	2.215	0.577	0.221	-0.492	0.000	0.628	-0.073	0.472	0.000	0.518	0.880	0.988	1.179	1.874	1.041	0.813
-1	1.024	1.075	-0.795	0.286	-1.436	1.365	0.857	-0.309	2.173	0.804	1.532	1.435	0.000	1.511	0.722	1.494	0.000	1.778	0.903	0.753	1.551	0.686	0.810	0.999	0.900	1.360	1.133	0.978
-1	2.085	-0.269	-1.423	0.789	1.298	0.281	1.652	0.187	0.000	0.658	-0.760	-0.042	2.215	0.663	0.024	0.120	0.000	0.552	-0.299	-0.428	3.102	0.713	0.811	1.130	0.705	0.218	0.675	0.743
-1	0.980	-0.443	0.813	0.785	-1.253	0.719	0.448	-1.458	0.000	1.087	0.595	0.635	1.107	1.428	0.029	-0.995	0.000	1.083	1.562	-0.092	0.000	0.834	0.891	1.165	0.967	0.661	0.880	0.817
-1	0.903	-0.733	-0.980	0.634	-0.639	0.780	0.266	-0.287	2.173	1.264	-0.936	1.004	0.000	1.002	-0.056	-1.344	2.548	1.183	-0.098	1.169	0.000	0.733	1.002	0.985	0.711	0.916	0.966	0.875
-0	0.734	-0.304	-1.175	2.851	1.674	0.904	-0.634	0.412	2.173	1.363	-1.050	-0.282	0.000	1.476	-1.603	0.103	0.000	2.231	-0.718	1.708	3.102	0.813	0.896	1.088	0.686	1.392	1.033	1.078
-1	1.680	0.591	-0.243	0.111	-0.478	0.326	-0.079	-1.555	2.173	0.711	0.714	0.922	2.215	0.355	0.858	1.682	0.000	0.727	1.620	1.360	0.000	0.334	0.526	1.001	0.862	0.633	0.660	0.619
-1	1.163	0.225	-0.202	0.501	-0.979	1.609	-0.938	1.424	0.000	1.224	-0.118	-1.274	0.000	2.034	1.241	-0.254	0.000	1.765	0.536	0.237	3.102	0.894	0.838	0.988	0.693	0.579	0.762	0.726
-0	1.223	1.232	1.471	0.489	1.728	0.703	-0.111	0.411	0.000	1.367	1.014	-1.294	1.107	1.524	-0.414	-0.164	2.548	1.292	0.833	0.316	0.000	0.861	0.752	0.994	0.836	1.814	1.089	0.950
-0	0.816	1.637	-1.557	1.036	-0.342	0.913	1.333	0.949	2.173	0.812	0.756	-0.628	2.215	1.333	0.470	1.495	0.000	1.204	-2.222	-1.675	0.000	1.013	0.924	1.133	0.758	1.304	0.855	0.860
-0	0.851	-0.564	-0.691	0.692	1.345	1.219	1.014	0.318	0.000	1.422	-0.262	-1.635	2.215	0.531	1.802	0.008	0.000	0.508	0.515	-1.267	3.102	0.821	0.787	1.026	0.783	0.432	1.149	1.034
-0	0.800	-0.599	0.204	0.552	-0.484	0.974	0.413	0.961	2.173	1.269	-0.984	-1.039	2.215	0.380	-1.213	1.371	0.000	0.551	0.332	-0.659	0.000	0.694	0.852	0.984	1.057	2.037	1.096	0.846
-0	0.744	-0.071	-0.255	0.638	0.512	1.125	0.407	0.844	2.173	0.860	-0.481	-0.677	0.000	1.102	0.181	-1.194	0.000	1.011	-1.081	-1.713	3.102	0.854	0.862	0.982	1.111	1.372	1.042	0.920
-1	0.400	1.049	-0.625	0.880	-0.407	1.040	2.150	-1.359	0.000	0.747	-0.144	0.847	2.215	0.560	-1.829	0.698	0.000	1.663	-0.668	0.267	0.000	0.845	0.964	0.996	0.820	0.789	0.668	0.668
-0	1.659	-0.705	-1.057	1.803	-1.436	1.008	0.693	0.005	0.000	0.895	-0.007	0.681	1.107	1.085	0.125	1.476	2.548	1.214	1.068	0.486	0.000	0.867	0.919	0.986	1.069	0.692	1.026	1.313
-0	0.829	-0.153	0.861	0.615	-0.548	0.589	1.077	-0.041	2.173	1.056	0.763	-1.737	0.000	0.639	0.970	0.725	0.000	0.955	1.227	-0.799	3.102	1.020	1.024	0.985	0.750	0.525	0.685	0.671
-1	0.920	-0.806	-0.840	1.048	0.278	0.973	-0.077	-1.364	2.173	1.029	0.309	0.133	0.000	1.444	1.484	1.618	1.274	1.419	-0.482	0.417	0.000	0.831	1.430	1.151	1.829	1.560	1.343	1.224
-1	0.686	0.249	-0.905	0.343	-1.731	0.724	-2.823	-0.901	0.000	0.982	0.303	1.312	1.107	1.016	0.245	0.610	0.000	1.303	-0.557	-0.360	3.102	1.384	1.030	0.984	0.862	1.144	0.866	0.779
-0	1.603	0.444	0.508	0.586	0.401	0.610	0.467	-1.735	2.173	0.914	0.626	-1.019	0.000	0.812	0.422	-0.408	2.548	0.902	1.679	1.490	0.000	1.265	0.929	0.990	1.004	0.816	0.753	0.851
-1	0.623	0.780	-0.203	0.056	0.015	0.899	0.793	1.326	1.087	0.803	1.478	-1.499	2.215	1.561	1.492	-0.120	0.000	0.904	0.795	0.137	0.000	0.548	1.009	0.850	0.924	0.838	0.914	0.860
-0	1.654	-2.032	-1.160	0.859	-1.583	0.689	-1.965	0.891	0.000	0.646	-1.014	-0.288	2.215	0.630	-0.815	0.402	0.000	0.638	0.316	0.655	3.102	0.845	0.879	0.993	1.067	0.625	1.041	0.958
-1	0.828	-1.269	-1.203	0.744	-0.213	0.626	-1.017	-0.404	0.000	1.281	-0.931	1.733	2.215	0.699	-0.351	1.287	0.000	1.251	-1.171	0.197	0.000	0.976	1.186	0.987	0.646	0.655	0.733	0.671
-1	0.677	0.111	1.090	1.580	1.591	1.560	0.654	-0.341	2.173	0.794	-0.266	0.702	0.000	0.823	0.651	-1.239	2.548	0.730	1.467	-1.530	0.000	1.492	1.023	0.983	1.909	1.022	1.265	1.127
-1	0.736	0.882	-1.060	0.589	0.168	1.663	0.781	1.022	2.173	2.025	1.648	-1.292	0.000	1.240	0.924	-0.421	1.274	1.354	0.065	0.501	0.000	0.316	0.925	0.988	0.664	1.736	0.992	0.807
-1	1.040	-0.822	1.638	0.974	-0.674	0.393	0.830	0.011	2.173	0.770	-0.140	-0.402	0.000	0.294	-0.133	0.030	0.000	1.220	0.807	0.638	0.000	0.826	1.063	1.216	1.026	0.705	0.934	0.823
-1	0.711	0.602	0.048	1.145	0.966	0.934	0.263	-1.589	2.173	0.971	-0.496	-0.421	1.107	0.628	-0.865	0.845	0.000	0.661	-0.008	-0.565	0.000	0.893	0.705	0.988	0.998	1.339	0.908	0.872
-1	0.953	-1.651	-0.167	0.885	1.053	1.013	-1.239	0.133	0.000	1.884	-1.122	1.222	2.215	1.906	-0.860	-1.184	1.274	1.413	-0.668	-1.647	0.000	1.873	1.510	1.133	1.050	1.678	1.246	1.061
-1	0.986	-0.892	-1.380	0.917	1.134	0.950	-1.162	-0.469	0.000	0.569	-1.393	0.215	0.000	0.320	2.667	1.712	0.000	1.570	-0.375	1.457	3.102	0.925	1.128	1.011	0.598	0.824	0.913	0.833
-1	1.067	0.099	1.154	0.527	-0.789	1.085	0.623	-1.602	2.173	1.511	-0.230	0.022	2.215	0.269	-0.377	0.883	0.000	0.571	-0.540	-0.512	0.000	0.414	0.803	1.022	0.959	2.053	1.041	0.780
-0	0.825	-2.118	0.217	1.453	-0.493	0.819	0.313	-0.942	0.000	2.098	-0.725	1.096	2.215	0.484	1.336	1.458	0.000	0.482	0.100	1.163	0.000	0.913	0.536	0.990	1.679	0.957	1.095	1.143
-1	1.507	0.054	1.120	0.698	-1.340	0.912	0.384	0.015	1.087	0.720	0.247	-0.820	0.000	0.286	0.154	1.578	2.548	0.629	1.582	-0.576	0.000	0.828	0.893	1.136	0.514	0.632	0.699	0.709
-1	0.610	1.180	-0.993	0.816	0.301	0.932	0.758	1.539	0.000	0.726	-0.830	0.248	2.215	0.883	0.857	-1.305	0.000	1.338	1.009	-0.252	3.102	0.901	1.074	0.987	0.875	1.159	1.035	0.858
-1	1.247	-1.360	1.502	1.525	-1.332	0.618	1.063	0.755	0.000	0.582	-0.155	0.473	2.215	1.214	-0.422	-0.551	2.548	0.838	-1.171	-1.166	0.000	2.051	1.215	1.062	1.091	0.725	0.896	1.091
-0	0.373	-0.600	1.291	2.573	0.207	0.765	-0.209	1.667	0.000	0.668	0.724	-1.499	0.000	1.045	-0.338	-0.754	2.548	0.558	-0.469	0.029	3.102	0.868	0.939	1.124	0.519	0.383	0.636	0.838
-0	0.791	0.336	-0.307	0.494	1.213	1.158	0.336	1.081	2.173	0.918	1.289	-0.449	0.000	0.735	-0.521	-0.969	0.000	1.052	0.499	-1.188	3.102	0.699	1.013	0.987	0.622	1.050	0.712	0.661
-0	1.321	0.856	0.464	0.202	0.901	1.144	0.120	-1.651	0.000	0.803	0.577	-0.509	2.215	0.695	-0.114	0.423	2.548	0.621	1.852	-0.420	0.000	0.697	0.964	0.983	0.527	0.659	0.719	0.729
-0	0.563	2.081	0.913	0.982	-0.533	0.549	-0.481	-1.730	0.000	0.962	0.921	0.569	2.215	0.731	1.184	-0.679	1.274	0.918	0.931	-1.432	0.000	1.008	0.919	0.993	0.895	0.819	0.810	0.878
-1	1.148	0.345	0.953	0.921	0.617	0.991	1.103	-0.484	0.000	0.970	1.978	1.525	0.000	1.150	0.689	-0.757	2.548	0.517	0.995	1.245	0.000	1.093	1.140	0.998	1.006	0.756	0.864	0.838
-1	1.400	0.128	-1.695	1.169	1.070	1.094	-0.345	-0.249	0.000	1.224	0.364	-0.036	2.215	1.178	0.530	-1.544	0.000	1.334	0.933	1.604	0.000	0.560	1.267	1.073	0.716	0.780	0.832	0.792
-0	0.330	-2.133	1.403	0.628	0.379	1.686	-0.995	0.030	1.087	2.071	0.127	-0.457	0.000	4.662	-0.855	1.477	0.000	2.072	-0.917	-1.416	3.102	5.403	3.074	0.977	0.936	1.910	2.325	1.702
-0	0.989	0.473	0.968	1.970	1.368	0.844	0.574	-0.290	2.173	0.866	-0.345	-1.019	0.000	1.130	0.605	-0.752	0.000	0.956	-0.888	0.870	3.102	0.885	0.886	0.982	1.157	1.201	1.100	1.068
-1	0.773	0.418	0.753	1.388	1.070	1.104	-0.378	-0.758	0.000	1.027	0.397	-0.496	2.215	1.234	0.027	1.084	2.548	0.936	0.209	1.677	0.000	1.355	1.020	0.983	0.550	1.206	0.916	0.931
-0	0.319	2.015	1.534	0.570	-1.134	0.632	0.124	0.757	0.000	0.477	0.598	-1.109	1.107	0.449	0.438	-0.755	2.548	0.574	-0.659	0.691	0.000	0.440	0.749	0.985	0.517	0.158	0.505	0.522
-0	1.215	1.453	-1.386	1.276	1.298	0.643	0.570	-0.196	2.173	0.588	2.104	0.498	0.000	0.617	-0.296	-0.801	2.548	0.452	0.110	0.313	0.000	0.815	0.953	1.141	1.166	0.547	0.892	0.807
-1	1.257	-1.869	-0.060	0.265	0.653	1.527	-0.346	1.163	2.173	0.758	-2.119	-0.604	0.000	1.473	-1.133	-1.290	2.548	0.477	-0.428	-0.066	0.000	0.818	0.841	0.984	1.446	1.729	1.211	1.054
-1	1.449	0.464	1.585	1.418	-1.488	1.540	0.942	0.087	0.000	0.898	0.402	-0.631	2.215	0.753	0.039	-1.729	0.000	0.859	0.849	-1.054	0.000	0.791	0.677	0.995	0.687	0.527	0.703	0.606
-1	1.084	-1.997	0.900	1.333	1.024	0.872	-0.864	-1.500	2.173	1.072	-0.813	-0.421	2.215	0.924	0.478	0.304	0.000	0.992	-0.398	-1.022	0.000	0.741	1.085	0.980	1.221	1.176	1.032	0.961
-0	1.712	1.129	0.125	1.120	-1.402	1.749	0.951	-1.575	2.173	1.711	0.445	0.578	0.000	1.114	0.234	-1.011	0.000	1.577	-0.088	0.086	3.102	2.108	1.312	1.882	1.597	2.009	1.441	1.308
-0	0.530	0.248	1.622	1.450	-1.012	1.221	-1.154	-0.763	2.173	1.698	-0.586	0.733	0.000	0.889	1.042	1.038	1.274	0.657	0.008	0.701	0.000	0.430	1.005	0.983	0.930	2.264	1.357	1.146
-1	0.921	1.735	0.883	0.699	-1.614	0.821	1.463	0.319	1.087	1.099	0.814	-1.600	2.215	1.375	0.702	-0.691	0.000	0.869	1.326	-0.790	0.000	0.980	0.900	0.988	0.832	1.452	0.816	0.709
-0	2.485	-0.823	-0.297	0.886	-1.404	0.989	0.835	1.615	2.173	0.382	0.588	-0.224	0.000	1.029	-0.456	1.546	2.548	0.613	-0.359	-0.789	0.000	0.768	0.977	1.726	2.007	0.913	1.338	1.180
-1	0.657	-0.069	-0.078	1.107	1.549	0.804	1.335	-1.630	2.173	1.271	0.481	0.153	1.107	1.028	0.144	-0.762	0.000	1.098	0.132	1.570	0.000	0.830	0.979	1.175	1.069	1.624	1.000	0.868
-1	2.032	0.329	-1.003	0.493	-0.136	1.159	-0.224	0.750	1.087	0.396	0.546	0.587	0.000	0.620	1.805	0.982	0.000	1.236	0.744	-1.621	0.000	0.930	1.200	0.988	0.482	0.771	0.887	0.779
-0	0.524	-1.319	0.634	0.471	1.221	0.599	-0.588	-0.461	0.000	1.230	-1.504	-1.517	1.107	1.436	-0.035	0.104	2.548	0.629	1.997	-1.282	0.000	2.084	1.450	0.984	1.084	1.827	1.547	1.213
-1	0.871	0.618	-1.544	0.718	0.186	1.041	-1.180	0.434	2.173	1.133	1.558	-1.301	0.000	0.452	-0.595	0.522	0.000	0.665	0.567	0.130	3.102	1.872	1.114	1.095	1.398	0.979	1.472	1.168
-1	3.308	1.037	-0.634	0.690	-0.619	1.975	0.949	1.280	0.000	0.826	0.546	-0.139	2.215	0.635	-0.045	0.427	0.000	1.224	0.112	1.339	3.102	1.756	1.050	0.992	0.738	0.903	0.968	1.238
-0	0.588	2.104	-0.872	1.136	1.743	0.842	0.638	0.015	0.000	0.481	0.928	1.000	2.215	0.595	0.125	1.429	0.000	0.951	-1.140	-0.511	3.102	1.031	1.057	0.979	0.673	1.064	1.001	0.891
-0	0.289	0.823	0.013	0.615	-1.601	0.177	2.403	-0.015	0.000	0.258	1.151	1.036	2.215	0.694	0.553	-1.326	2.548	0.411	0.366	0.106	0.000	0.482	0.562	0.989	0.670	0.404	0.516	0.561
-1	0.294	-0.660	-1.162	1.752	0.384	0.860	0.513	1.119	0.000	2.416	0.107	-1.342	0.000	1.398	0.361	-0.350	2.548	1.126	-0.902	0.040	1.551	0.650	1.125	0.988	0.531	0.843	0.912	0.911
-0	0.599	-0.616	1.526	1.381	0.507	0.955	-0.646	-0.085	2.173	0.775	-0.533	1.116	2.215	0.789	-0.136	-1.176	0.000	2.449	1.435	-1.433	0.000	1.692	1.699	1.000	0.869	1.119	1.508	1.303
-1	1.100	-1.174	-1.114	1.601	-1.576	1.056	-1.343	0.547	2.173	0.555	0.367	0.592	2.215	0.580	-1.862	-0.914	0.000	0.904	0.508	-0.444	0.000	1.439	1.105	0.986	1.408	1.104	1.190	1.094
-1	2.237	-0.701	1.470	0.719	-0.199	0.745	-0.132	-0.737	1.087	0.976	-0.227	0.093	2.215	0.699	0.057	1.133	0.000	0.661	0.573	-0.679	0.000	0.785	0.772	1.752	1.235	0.856	0.990	0.825
-1	0.455	-0.880	-1.482	1.260	-0.178	1.499	0.158	1.022	0.000	1.867	-0.435	-0.675	2.215	1.234	0.783	1.586	0.000	0.641	-0.454	-0.409	3.102	1.002	0.964	0.986	0.761	0.240	1.190	0.995
-1	1.158	-0.778	-0.159	0.823	1.641	1.341	-0.830	-1.169	2.173	0.840	-1.554	0.934	0.000	0.693	0.488	-1.218	2.548	1.042	1.395	0.276	0.000	0.946	0.785	1.350	1.079	0.893	1.267	1.151
-1	0.902	-0.078	-0.055	0.872	-0.012	0.843	1.276	1.739	2.173	0.838	1.492	0.918	0.000	0.626	0.904	-0.648	2.548	0.412	-2.027	-0.883	0.000	2.838	1.664	0.988	1.803	0.768	1.244	1.280
-1	0.649	-1.028	-1.521	1.097	0.774	1.216	-0.383	-0.318	2.173	1.643	-0.285	-1.705	0.000	0.911	-0.091	0.341	0.000	0.592	0.537	0.732	3.102	0.911	0.856	1.027	1.160	0.874	0.986	0.893
-1	1.192	1.846	-0.781	1.326	-0.747	1.550	1.177	1.366	0.000	1.196	0.151	0.387	2.215	0.527	2.261	-0.190	0.000	0.390	1.474	0.381	0.000	0.986	1.025	1.004	1.392	0.761	0.965	1.043
-0	0.438	-0.358	-1.549	0.836	0.436	0.818	0.276	-0.708	2.173	0.707	0.826	0.392	0.000	1.050	1.741	-1.066	0.000	1.276	-1.583	0.842	0.000	1.475	1.273	0.986	0.853	1.593	1.255	1.226
-1	1.083	0.142	1.701	0.605	-0.253	1.237	0.791	1.183	2.173	0.842	2.850	-0.082	0.000	0.724	-0.464	-0.694	0.000	1.499	0.456	-0.226	3.102	0.601	0.799	1.102	0.995	1.389	1.013	0.851
-0	0.828	1.897	-0.615	0.572	-0.545	0.572	0.461	0.464	2.173	0.393	0.356	1.069	2.215	1.840	0.088	1.500	0.000	0.407	-0.663	-0.787	0.000	0.950	0.965	0.979	0.733	0.363	0.618	0.733
-0	0.735	1.438	1.197	1.123	-0.214	0.641	0.949	0.858	0.000	1.162	0.524	-0.896	2.215	0.992	0.454	-1.475	2.548	0.902	1.079	0.019	0.000	0.822	0.917	1.203	1.032	0.569	0.780	0.764
-0	0.437	-2.102	0.044	1.779	-1.042	1.231	-0.181	-0.515	1.087	2.666	0.863	1.466	2.215	1.370	0.345	-1.371	0.000	0.906	0.363	1.611	0.000	1.140	1.362	1.013	3.931	3.004	2.724	2.028
-1	0.881	1.814	-0.987	0.384	0.800	2.384	1.422	0.640	0.000	1.528	0.292	-0.962	1.107	2.126	-0.371	-1.401	2.548	0.700	0.109	0.203	0.000	0.450	0.813	0.985	0.956	1.013	0.993	0.774
-1	0.630	0.408	0.152	0.194	0.316	0.710	-0.824	-0.358	2.173	0.741	0.535	-0.851	2.215	0.933	0.406	1.148	0.000	0.523	-0.479	-0.625	0.000	0.873	0.960	0.988	0.830	0.921	0.711	0.661
-1	0.870	-0.448	-1.134	0.616	0.135	0.600	0.649	-0.622	2.173	0.768	0.709	-0.123	0.000	1.308	0.500	1.468	0.000	1.973	-0.286	1.462	3.102	0.909	0.944	0.990	0.835	1.250	0.798	0.776
-0	1.290	0.552	1.330	0.615	-1.353	0.661	0.240	-0.393	0.000	0.531	0.053	-1.588	0.000	0.675	0.839	-0.345	1.274	1.597	0.020	0.536	3.102	1.114	0.964	0.987	0.783	0.675	0.662	0.675
-1	0.943	0.936	1.068	1.373	0.671	2.170	-2.011	-1.032	0.000	0.640	0.361	-0.806	0.000	2.239	-0.083	0.590	2.548	1.224	0.646	-1.723	0.000	0.879	0.834	0.981	1.436	0.568	0.916	0.931
-1	0.431	1.686	-1.053	0.388	1.739	0.457	-0.471	-0.743	2.173	0.786	1.432	-0.547	2.215	0.537	-0.413	1.256	0.000	0.413	2.311	-0.408	0.000	1.355	1.017	0.982	0.689	1.014	0.821	0.715
-0	1.620	-0.055	-0.862	1.341	-1.571	0.634	-0.906	0.935	2.173	0.501	-2.198	-0.525	0.000	0.778	-0.708	-0.060	0.000	0.988	-0.621	0.489	3.102	0.870	0.956	1.216	0.992	0.336	0.871	0.889
-1	0.549	0.304	-1.443	1.309	-0.312	1.116	0.644	1.519	2.173	1.078	-0.303	-0.736	0.000	1.261	0.387	0.628	2.548	0.945	-0.190	0.090	0.000	0.893	1.043	1.000	1.124	1.077	1.026	0.886
-0	0.412	-0.618	-1.486	1.133	-0.665	0.646	0.436	1.520	0.000	0.993	0.976	0.106	2.215	0.832	0.091	0.164	2.548	0.672	-0.650	1.256	0.000	0.695	1.131	0.991	1.017	0.455	1.226	1.087
-0	1.183	-0.084	1.644	1.389	0.967	0.843	0.938	-0.670	0.000	0.480	0.256	0.123	2.215	0.437	1.644	0.491	0.000	0.501	-0.416	0.101	3.102	1.060	0.804	1.017	0.775	0.173	0.535	0.760
-0	1.629	-1.486	-0.683	2.786	-0.492	1.347	-2.638	1.453	0.000	1.857	0.208	0.873	0.000	0.519	-1.265	-1.602	1.274	0.903	-1.102	-0.329	1.551	6.892	3.522	0.998	0.570	0.477	2.039	2.006
-1	2.045	-0.671	-1.235	0.490	-0.952	0.525	-1.252	1.289	0.000	1.088	-0.993	0.648	2.215	0.975	-0.109	-0.254	2.548	0.556	-1.095	-0.194	0.000	0.803	0.861	0.980	1.282	0.945	0.925	0.811
-0	0.448	-0.058	-0.974	0.945	-1.633	1.181	-1.139	0.266	2.173	1.118	-0.761	1.502	1.107	1.706	0.585	-0.680	0.000	0.487	-1.951	0.945	0.000	2.347	1.754	0.993	1.161	1.549	1.414	1.176
-0	0.551	0.519	0.448	2.183	1.293	1.220	0.628	-0.627	2.173	1.019	-0.002	-0.652	0.000	1.843	-0.386	1.042	2.548	0.400	-1.102	-1.014	0.000	0.648	0.792	1.049	0.888	2.132	1.262	1.096
-0	1.624	0.488	1.403	0.760	0.559	0.812	0.777	-1.244	2.173	0.613	0.589	-0.030	2.215	0.692	1.058	0.683	0.000	1.054	1.165	-0.765	0.000	0.915	0.875	1.059	0.821	0.927	0.792	0.721
-1	0.774	0.444	1.257	0.515	-0.689	0.515	1.448	-1.271	0.000	0.793	0.118	0.811	1.107	0.679	0.326	-0.426	0.000	1.066	-0.865	-0.049	3.102	0.960	1.046	0.986	0.716	0.772	0.855	0.732
-1	2.093	-1.240	1.615	0.918	-1.202	1.412	-0.541	0.640	1.087	2.019	0.872	-0.639	0.000	0.672	-0.936	0.972	0.000	0.896	0.235	0.212	0.000	0.810	0.700	1.090	0.797	0.862	1.049	0.874
-1	0.908	1.069	0.283	0.400	1.293	0.609	1.452	-1.136	0.000	0.623	0.417	-0.098	2.215	1.023	0.775	1.054	1.274	0.706	2.346	-1.305	0.000	0.744	1.006	0.991	0.606	0.753	0.796	0.753
-0	0.403	-1.328	-0.065	0.901	1.052	0.708	-0.354	-0.718	2.173	0.892	0.633	1.684	2.215	0.999	-1.205	0.941	0.000	0.930	1.072	-0.809	0.000	2.105	1.430	0.989	0.838	1.147	1.042	0.883
-0	1.447	0.453	0.118	1.731	0.650	0.771	0.446	-1.564	0.000	0.973	-2.014	0.354	0.000	1.949	-0.643	-1.531	1.274	1.106	-0.334	-1.163	0.000	0.795	0.821	1.013	1.699	0.918	1.118	1.018
-1	1.794	0.123	-0.454	0.057	1.489	0.966	-1.190	1.090	1.087	0.539	-0.535	1.035	0.000	1.096	-1.069	-1.236	2.548	0.659	-1.196	-0.283	0.000	0.803	0.756	0.985	1.343	1.109	0.993	0.806
-0	1.484	-2.047	0.813	0.591	-0.295	0.923	0.312	-1.164	2.173	0.654	-0.316	0.752	2.215	0.599	1.966	-1.128	0.000	0.626	-0.304	-1.431	0.000	1.112	0.910	1.090	0.986	1.189	1.350	1.472
-0	0.417	-2.016	0.849	1.817	0.040	1.201	-1.676	-1.394	0.000	0.792	0.537	0.641	2.215	0.794	-1.222	0.187	0.000	0.825	-0.217	1.334	3.102	1.470	0.931	0.987	1.203	0.525	0.833	0.827
-1	0.603	1.009	0.033	0.486	1.225	0.884	-0.617	-1.058	0.000	0.500	-1.407	-0.567	0.000	1.476	-0.876	0.605	2.548	0.970	0.560	1.092	3.102	0.853	1.153	0.988	0.846	0.920	0.944	0.835
-1	1.381	-0.326	0.552	0.417	-0.027	1.030	-0.835	-1.287	2.173	0.941	-0.421	1.519	2.215	0.615	-1.650	0.377	0.000	0.606	0.644	0.650	0.000	1.146	0.970	0.990	1.191	0.884	0.897	0.826
-1	0.632	1.200	-0.703	0.438	-1.700	0.779	-0.731	0.958	1.087	0.605	0.393	-1.376	0.000	0.670	-0.827	-1.315	2.548	0.626	-0.501	0.417	0.000	0.904	0.903	0.998	0.673	0.803	0.722	0.640
-1	1.561	-0.569	1.580	0.329	0.237	1.059	0.731	0.415	2.173	0.454	0.016	-0.828	0.000	0.587	0.008	-0.291	1.274	0.597	1.119	1.191	0.000	0.815	0.908	0.988	0.733	0.690	0.892	0.764
-1	2.102	0.087	0.449	1.164	-0.390	1.085	-0.408	-1.116	2.173	0.578	0.197	-0.137	0.000	1.202	0.917	1.523	0.000	0.959	-0.832	1.404	3.102	1.380	1.109	1.486	1.496	0.886	1.066	1.025
-1	1.698	-0.489	-0.552	0.976	-1.009	1.620	-0.721	0.648	1.087	1.481	-1.860	-1.354	0.000	1.142	-1.140	1.401	2.548	1.000	-1.274	-0.158	0.000	1.430	1.130	0.987	1.629	1.154	1.303	1.223
-1	1.111	-0.249	-1.457	0.421	0.939	0.646	-2.076	0.362	0.000	1.315	0.796	-1.436	2.215	0.780	0.130	0.055	0.000	1.662	-0.834	0.461	0.000	0.920	0.948	0.990	1.046	0.905	1.493	1.169
-1	0.945	0.390	-1.159	1.675	0.437	0.356	0.261	0.543	1.087	0.574	0.838	1.599	2.215	0.496	-1.220	-0.022	0.000	0.558	-2.454	1.440	0.000	0.763	0.983	1.728	1.000	0.578	0.922	1.003
-1	2.076	0.014	-1.314	0.854	-0.306	3.446	1.341	0.598	0.000	2.086	0.227	-0.747	2.215	1.564	-0.216	1.649	2.548	0.965	-0.857	-1.062	0.000	0.477	0.734	1.456	1.003	1.660	1.001	0.908
-1	1.992	0.192	-0.103	0.108	-1.599	0.938	0.595	-1.360	2.173	0.869	-1.012	1.432	0.000	1.302	0.850	0.436	2.548	0.487	1.051	-1.027	0.000	0.502	0.829	0.983	1.110	1.394	0.904	0.836
-0	0.460	1.625	1.485	1.331	1.242	0.675	-0.329	-1.039	1.087	0.671	-1.028	-0.514	0.000	1.265	-0.788	0.415	1.274	0.570	-0.683	-1.738	0.000	0.725	0.758	1.004	1.024	1.156	0.944	0.833
-0	0.871	0.839	-1.536	0.428	1.198	0.875	-1.256	-0.466	1.087	0.684	-0.768	0.150	0.000	0.556	-1.793	0.389	0.000	0.942	-1.126	1.339	1.551	0.624	0.734	0.986	1.357	0.960	1.474	1.294
-1	0.951	1.651	0.576	1.273	1.495	0.834	0.048	-0.578	2.173	0.386	-0.056	-1.448	0.000	0.597	-0.196	0.162	2.548	0.524	1.649	1.625	0.000	0.737	0.901	1.124	1.014	0.556	1.039	0.845
-1	1.049	-0.223	0.685	0.256	-1.191	2.506	0.238	-0.359	0.000	1.510	-0.904	1.158	1.107	2.733	-0.902	1.679	2.548	0.407	-0.474	-1.572	0.000	1.513	2.472	0.982	1.238	0.978	1.985	1.510
-0	0.455	-0.028	0.265	1.286	1.373	0.459	0.331	-0.922	0.000	0.343	0.634	0.430	0.000	0.279	-0.084	-0.272	0.000	0.475	0.926	-0.123	3.102	0.803	0.495	0.987	0.587	0.211	0.417	0.445
-1	2.074	0.388	0.878	1.110	1.557	1.077	-0.226	-0.295	2.173	0.865	-0.319	-1.116	2.215	0.707	-0.835	0.722	0.000	0.632	-0.608	-0.728	0.000	0.715	0.802	1.207	1.190	0.960	1.143	0.926
-1	1.390	0.265	1.196	0.919	-1.371	1.858	0.506	0.786	0.000	1.280	-1.367	-0.720	2.215	1.483	-0.441	-0.675	2.548	1.076	0.294	-0.539	0.000	1.126	0.830	1.155	1.551	0.702	1.103	0.933
-1	1.014	-0.079	1.597	1.038	-0.281	1.135	-0.722	-0.177	2.173	0.544	-1.475	-1.501	0.000	1.257	-1.315	1.212	0.000	0.496	-0.060	1.180	1.551	0.815	0.611	1.411	1.110	0.792	0.846	0.853
-0	0.335	1.267	-1.154	2.011	-0.574	0.753	0.618	1.411	0.000	0.474	0.748	0.681	2.215	0.608	-0.446	-0.354	2.548	0.399	1.295	-0.581	0.000	0.911	0.882	0.975	0.832	0.598	0.580	0.678
-1	0.729	-0.189	1.182	0.293	1.310	0.412	0.459	-0.632	0.000	0.869	-1.128	-0.625	2.215	1.173	-0.893	0.478	2.548	0.584	-2.394	-1.727	0.000	2.016	1.272	0.995	1.034	0.905	0.966	1.038
-1	1.225	-1.215	-0.088	0.881	-0.237	0.600	-0.976	1.462	2.173	0.876	0.506	1.583	2.215	0.718	1.228	-0.031	0.000	0.653	-1.292	1.216	0.000	0.838	1.108	0.981	1.805	0.890	1.251	1.197
-1	2.685	-0.444	0.847	0.253	0.183	0.641	-1.541	-0.873	2.173	0.417	2.874	-0.551	0.000	0.706	-1.431	0.764	0.000	1.390	-0.596	-1.397	0.000	0.894	0.829	0.993	0.789	0.654	0.883	0.746
-0	0.638	-0.481	0.683	1.457	-1.024	0.707	-1.338	1.498	0.000	0.980	0.518	0.289	2.215	0.964	-0.531	-0.423	0.000	0.694	-0.654	-1.314	3.102	0.807	1.283	1.335	0.658	0.907	0.797	0.772
-1	1.789	-0.765	-0.732	0.421	-0.020	1.142	-1.353	1.439	2.173	0.725	-1.518	-1.261	0.000	0.812	-2.597	-0.463	0.000	1.203	-0.120	1.001	0.000	0.978	0.673	0.985	1.303	1.400	1.078	0.983
-1	0.784	-1.431	1.724	0.848	0.559	0.615	-1.643	-1.456	0.000	1.339	-0.513	0.040	2.215	0.394	-2.483	1.304	0.000	0.987	0.889	-0.339	0.000	0.732	0.713	0.987	0.973	0.705	0.875	0.759
-1	0.911	1.098	-1.289	0.421	0.823	1.218	-0.503	0.431	0.000	0.775	0.432	-1.680	0.000	0.855	-0.226	-0.460	2.548	0.646	-0.947	-1.243	1.551	2.201	1.349	0.985	0.730	0.451	0.877	0.825
-1	0.959	0.372	-0.269	1.255	0.702	1.151	0.097	0.805	2.173	0.993	1.011	0.767	2.215	1.096	0.185	0.381	0.000	1.001	-0.205	0.059	0.000	0.979	0.997	1.168	0.796	0.771	0.839	0.776
-0	0.283	-1.864	-1.663	0.219	1.624	0.955	-1.213	0.932	2.173	0.889	0.395	-0.268	0.000	0.597	-1.083	-0.921	2.548	0.584	1.325	-1.072	0.000	0.856	0.927	0.996	0.937	0.936	1.095	0.892
-0	2.017	-0.488	-0.466	1.029	-0.870	3.157	0.059	-0.343	2.173	3.881	0.872	1.502	1.107	3.631	1.720	0.963	0.000	0.633	-1.264	-1.734	0.000	4.572	3.339	1.005	1.407	5.590	3.614	3.110
-1	1.088	0.414	-0.841	0.485	0.605	0.860	1.110	-0.568	0.000	1.152	-0.325	1.203	2.215	0.324	1.652	-0.104	0.000	0.510	1.095	-1.728	0.000	0.880	0.722	0.989	0.977	0.711	0.888	0.762
-0	0.409	-1.717	0.712	0.809	-1.301	0.701	-1.529	-1.411	0.000	1.191	-0.582	0.438	2.215	1.147	0.813	-0.571	2.548	1.039	0.543	0.892	0.000	0.636	0.810	0.986	0.861	1.411	0.907	0.756
-1	1.094	1.577	-0.988	0.497	-0.149	0.891	-2.459	1.034	0.000	0.646	0.792	-1.022	0.000	1.573	0.254	-0.053	2.548	1.428	0.190	-1.641	3.102	4.322	2.687	0.985	0.881	1.135	1.907	1.831
-1	0.613	1.993	-0.280	0.544	0.931	0.909	1.526	1.559	0.000	0.840	1.473	-0.483	2.215	0.856	0.352	0.408	2.548	1.058	1.733	-1.396	0.000	0.801	1.066	0.984	0.639	0.841	0.871	0.748
-0	0.958	-1.202	0.600	0.434	0.170	0.783	-0.214	1.319	0.000	0.835	-0.454	-0.615	2.215	0.658	-1.858	-0.891	0.000	0.640	0.172	-1.204	3.102	1.790	1.086	0.997	0.804	0.403	0.793	0.756
-1	1.998	-0.238	0.972	0.058	0.266	0.759	1.576	-0.357	2.173	1.004	-0.349	-0.747	2.215	0.962	0.490	-0.453	0.000	1.592	0.661	-1.405	0.000	0.874	1.086	0.990	1.436	1.527	1.177	0.993
-1	0.796	-0.171	-0.818	0.574	-1.625	1.201	-0.737	1.451	2.173	0.651	0.404	-0.452	0.000	1.150	-0.652	-0.120	0.000	1.008	-0.093	0.531	3.102	0.884	0.706	0.979	1.193	0.937	0.943	0.881
-1	0.773	1.023	0.527	1.537	-0.201	2.967	-0.574	-1.534	2.173	2.346	-0.307	0.394	2.215	1.393	0.135	-0.027	0.000	3.015	0.187	0.516	0.000	0.819	1.260	0.982	2.552	3.862	2.179	1.786
-0	1.823	1.008	-1.489	0.234	-0.962	0.591	0.461	0.996	2.173	0.568	-1.297	-0.410	0.000	0.887	2.157	1.194	0.000	2.079	0.369	-0.085	3.102	0.770	0.945	0.995	1.179	0.971	0.925	0.983
-0	0.780	0.640	0.490	0.680	-1.301	0.715	-0.137	0.152	2.173	0.616	-0.831	1.668	0.000	1.958	0.528	-0.982	2.548	0.966	-1.551	0.462	0.000	1.034	1.079	1.008	0.827	1.369	1.152	0.983
-1	0.543	0.801	1.543	1.134	-0.772	0.954	-0.849	0.410	1.087	0.851	-1.988	1.686	0.000	0.799	-0.912	-1.156	0.000	0.479	0.097	1.334	0.000	0.923	0.597	0.989	1.231	0.759	0.975	0.867
-0	1.241	-0.014	0.129	1.158	0.670	0.445	-0.732	1.739	2.173	0.918	0.659	-1.340	2.215	0.557	2.410	-1.404	0.000	0.966	-1.545	-1.120	0.000	0.874	0.918	0.987	1.001	0.798	0.904	0.937
-0	1.751	-0.266	-1.575	0.489	1.292	1.112	1.533	0.137	2.173	1.204	-0.414	-0.928	0.000	0.879	1.237	-0.415	2.548	1.479	1.469	0.913	0.000	2.884	1.747	0.989	1.742	0.600	1.363	1.293
-1	1.505	1.208	-1.476	0.995	-0.836	2.800	-1.600	0.111	0.000	2.157	1.241	1.110	2.215	1.076	2.619	-0.913	0.000	1.678	2.204	-1.575	0.000	0.849	1.224	0.990	1.412	0.976	1.271	1.105
-0	0.816	0.611	0.779	1.694	0.278	0.575	-0.787	1.592	2.173	1.148	1.076	-0.831	2.215	0.421	1.316	0.632	0.000	0.589	0.452	-1.466	0.000	0.779	0.909	0.990	1.146	1.639	1.236	0.949
-1	0.551	-0.808	0.330	1.188	-0.294	0.447	-0.035	-0.993	0.000	0.432	-0.276	-0.481	2.215	1.959	-0.288	1.195	2.548	0.638	0.583	1.107	0.000	0.832	0.924	0.993	0.723	0.976	0.968	0.895
-0	1.316	-0.093	0.995	0.860	-0.621	0.593	-0.560	-1.599	2.173	0.524	-0.318	-0.240	2.215	0.566	0.759	-0.368	0.000	0.483	-2.030	-1.104	0.000	1.468	1.041	1.464	0.811	0.778	0.690	0.722
-1	1.528	0.067	-0.855	0.959	-1.464	1.143	-0.082	1.023	0.000	0.702	-0.763	-0.244	0.000	0.935	-0.881	0.206	2.548	0.614	-0.831	1.657	3.102	1.680	1.105	0.983	1.078	0.559	0.801	0.809
-0	0.558	-0.833	-0.598	1.436	-1.724	1.316	-0.661	1.593	2.173	1.148	-0.503	-0.132	1.107	1.584	-0.125	0.380	0.000	1.110	-1.216	-0.181	0.000	1.258	0.860	1.053	0.790	1.814	1.159	1.007
-1	0.819	0.879	1.221	0.598	-1.450	0.754	0.417	-0.369	2.173	0.477	1.199	0.274	0.000	1.073	0.368	0.273	2.548	1.599	2.047	1.690	0.000	0.933	0.984	0.983	0.788	0.613	0.728	0.717
-0	0.981	-1.007	0.489	0.923	1.261	0.436	-0.698	-0.506	2.173	0.764	-1.105	-1.241	2.215	0.577	-2.573	-0.036	0.000	0.565	-1.628	1.610	0.000	0.688	0.801	0.991	0.871	0.554	0.691	0.656
-0	2.888	0.568	-1.416	1.461	-1.157	1.756	-0.900	0.522	0.000	0.657	0.409	1.076	2.215	1.419	0.672	-0.019	0.000	1.436	-0.184	-0.980	3.102	0.946	0.919	0.995	1.069	0.890	0.834	0.856
-1	0.522	1.805	-0.963	1.136	0.418	0.727	-0.195	-1.695	2.173	0.309	2.559	-0.178	0.000	0.521	1.794	0.919	0.000	0.788	0.174	-0.406	3.102	0.555	0.729	1.011	1.385	0.753	0.927	0.832
-1	0.793	-0.162	-1.643	0.634	0.337	0.898	-0.633	1.689	0.000	0.806	-0.826	-0.356	2.215	0.890	-0.142	-1.268	0.000	1.293	0.574	0.725	0.000	0.833	1.077	0.988	0.721	0.679	0.867	0.753
-0	1.298	1.098	0.280	0.371	-0.373	0.855	-0.306	-1.186	0.000	0.977	-0.421	1.003	0.000	0.978	0.956	-1.249	2.548	0.735	0.577	-0.037	3.102	0.974	1.002	0.992	0.549	0.587	0.725	0.954
-1	0.751	-0.520	-1.653	0.168	-0.419	0.878	-1.023	-1.364	2.173	1.310	-0.667	0.863	0.000	1.196	-0.827	0.358	0.000	1.154	-0.165	-0.360	1.551	0.871	0.950	0.983	0.907	0.955	0.959	0.874
-0	1.730	0.666	-1.432	0.446	1.302	0.921	-0.203	0.621	0.000	1.171	-0.365	-0.611	1.107	0.585	0.807	1.150	0.000	0.415	-0.843	1.311	0.000	0.968	0.786	0.986	1.059	0.371	0.790	0.848
-1	0.596	-1.486	0.690	1.045	-1.344	0.928	0.867	0.820	2.173	0.610	0.999	-1.329	2.215	0.883	-0.001	-0.106	0.000	1.145	2.184	-0.808	0.000	2.019	1.256	1.056	1.751	1.037	1.298	1.518
-1	0.656	-1.993	-0.519	1.643	-0.143	0.815	0.256	1.220	1.087	0.399	-1.184	-1.458	0.000	0.738	1.361	-1.443	0.000	0.842	0.033	0.293	0.000	0.910	0.891	0.993	0.668	0.562	0.958	0.787
-1	1.127	-0.542	0.645	0.318	-1.496	0.661	-0.640	0.369	2.173	0.992	0.358	1.702	0.000	1.004	0.316	-1.109	0.000	1.616	-0.936	-0.707	1.551	0.875	1.191	0.985	0.651	0.940	0.969	0.834
-0	0.916	-1.423	-1.490	1.248	-0.538	0.625	-0.535	-0.174	0.000	0.769	-0.389	1.608	2.215	0.667	-1.138	-1.738	1.274	0.877	-0.019	0.482	0.000	0.696	0.917	1.121	0.678	0.347	0.647	0.722
-1	2.756	-0.637	-1.715	1.331	1.124	0.913	-0.296	-0.491	0.000	0.983	-0.831	0.000	2.215	1.180	-0.428	0.742	0.000	1.113	0.005	-1.157	1.551	1.681	1.096	1.462	0.976	0.917	1.009	1.040
-0	0.755	1.754	0.701	2.111	0.256	1.243	0.057	-1.502	2.173	0.565	-0.034	-1.078	1.107	0.529	1.696	-1.090	0.000	0.665	0.292	0.107	0.000	0.870	0.780	0.990	2.775	0.465	1.876	1.758
-1	0.593	-0.762	1.743	0.908	0.442	0.773	-1.357	-0.768	2.173	0.432	1.421	1.236	0.000	0.579	0.291	-0.403	0.000	0.966	-0.309	1.016	3.102	0.893	0.743	0.989	0.857	1.030	0.943	0.854
-1	0.891	-1.151	-1.269	0.504	-0.622	0.893	-0.549	0.700	0.000	0.828	-0.825	0.154	2.215	1.083	0.632	-1.141	0.000	1.059	-0.557	1.526	3.102	2.117	1.281	0.987	0.819	0.802	0.917	0.828
-1	2.358	-0.248	0.080	0.747	-0.975	1.019	1.374	1.363	0.000	0.935	0.127	-1.707	2.215	0.312	-0.827	0.017	0.000	0.737	1.059	-0.327	0.000	0.716	0.828	1.495	0.953	0.704	0.880	0.745
-0	0.660	-0.017	-1.138	0.453	1.002	0.645	0.518	0.703	2.173	0.751	0.705	-0.592	2.215	0.744	-0.909	-1.596	0.000	0.410	-1.135	0.481	0.000	0.592	0.922	0.989	0.897	0.948	0.777	0.701
-1	0.718	0.518	0.225	1.710	-0.022	1.888	-0.424	1.092	0.000	4.134	0.185	-1.366	0.000	1.415	1.293	0.242	2.548	2.351	0.264	-0.057	3.102	0.830	1.630	0.976	1.215	0.890	1.422	1.215
-1	1.160	0.203	0.941	0.594	0.212	0.636	-0.556	0.679	2.173	1.089	-0.481	-1.008	1.107	1.245	-0.056	-1.357	0.000	0.587	1.007	0.056	0.000	1.106	0.901	0.987	0.786	1.224	0.914	0.837
-1	0.697	0.542	0.619	0.985	1.481	0.745	0.415	1.644	2.173	0.903	0.495	-0.958	2.215	1.165	1.195	0.346	0.000	1.067	-0.881	-0.264	0.000	0.830	1.025	0.987	0.690	0.863	0.894	0.867
-0	1.430	0.190	-0.700	0.246	0.518	1.302	0.660	-0.247	2.173	1.185	-0.539	1.504	0.000	1.976	-0.401	1.079	0.000	0.855	-0.958	-1.110	3.102	0.886	0.953	0.993	0.889	1.400	1.376	1.119
-1	1.122	-0.795	0.202	0.397	-1.553	0.597	-1.459	-0.734	2.173	0.522	1.044	1.027	2.215	0.783	-1.243	1.701	0.000	0.371	1.737	0.199	0.000	1.719	1.176	0.988	0.723	1.583	1.063	0.914
-0	1.153	0.526	1.236	0.266	0.001	1.139	-1.236	-0.585	2.173	1.337	-0.215	-1.356	2.215	1.780	1.129	0.902	0.000	1.608	-0.391	-0.161	0.000	1.441	1.633	0.990	1.838	1.516	1.635	1.373
-1	0.760	1.012	0.758	0.937	0.051	0.941	0.687	-1.247	2.173	1.288	-0.743	0.822	0.000	1.552	1.782	-1.533	0.000	0.767	1.349	0.168	0.000	0.716	0.862	0.988	0.595	0.359	0.697	0.623
-1	1.756	-1.469	1.395	1.345	-1.595	0.817	0.017	-0.741	2.173	0.483	-0.008	0.293	0.000	1.768	-0.663	0.438	1.274	1.202	-1.387	-0.222	0.000	1.022	1.058	0.992	1.407	1.427	1.356	1.133
-0	0.397	0.582	-0.758	1.260	-1.735	0.889	-0.515	1.139	2.173	0.973	1.616	0.460	0.000	1.308	1.001	-0.709	2.548	0.858	0.995	-0.231	0.000	0.749	0.888	0.979	1.487	1.804	1.208	1.079
-0	0.515	-0.984	0.425	1.114	-0.439	1.999	0.818	1.561	0.000	1.407	0.009	-0.380	0.000	1.332	0.230	0.397	0.000	1.356	-0.616	-1.057	3.102	0.978	1.017	0.990	1.118	0.862	0.835	0.919
-1	1.368	-0.921	-0.866	0.842	-0.598	0.456	-1.176	1.219	1.087	0.419	-1.974	-0.819	0.000	0.791	-1.640	0.881	0.000	1.295	-0.782	0.442	3.102	0.945	0.761	0.974	0.915	0.535	0.733	0.651
-0	2.276	0.134	0.399	2.525	0.376	1.111	-1.078	-1.571	0.000	0.657	2.215	-0.900	0.000	1.183	-0.662	-0.508	2.548	1.436	-0.517	0.960	3.102	0.569	0.931	0.993	1.170	0.967	0.879	1.207
-0	0.849	0.907	0.124	0.652	1.585	0.715	0.355	-1.200	0.000	0.599	-0.892	1.301	0.000	1.106	1.151	0.582	0.000	1.895	-0.279	-0.568	3.102	0.881	0.945	0.998	0.559	0.649	0.638	0.660
-1	2.105	0.248	-0.797	0.530	0.206	1.957	-2.175	0.797	0.000	1.193	0.637	-1.646	2.215	0.881	1.111	-1.046	0.000	0.872	-0.185	1.085	1.551	0.986	1.343	1.151	1.069	0.714	2.063	1.951
-1	1.838	1.060	1.637	1.017	1.370	0.913	0.461	-0.609	1.087	0.766	-0.461	0.303	2.215	0.724	-0.061	0.886	0.000	0.941	1.123	-0.745	0.000	0.858	0.847	0.979	1.313	1.083	1.094	0.910
-0	0.364	1.274	1.066	1.570	-0.394	0.485	0.012	-1.716	0.000	0.317	-1.233	0.534	2.215	0.548	-2.165	0.762	0.000	0.729	0.169	-0.318	3.102	0.892	0.944	1.013	0.594	0.461	0.688	0.715
-1	0.503	1.343	-0.031	1.134	-1.204	0.590	-0.309	0.174	2.173	0.408	2.372	-0.628	0.000	1.850	0.400	1.147	2.548	0.664	-0.458	-0.885	0.000	1.445	1.283	0.989	1.280	1.118	1.127	1.026
-0	1.873	0.258	0.103	2.491	0.530	1.678	0.644	-1.738	2.173	1.432	0.848	-1.340	0.000	0.621	1.323	-1.316	0.000	0.628	0.789	-0.206	1.551	0.426	0.802	1.125	0.688	1.079	1.338	1.239
-1	0.826	-0.732	1.587	0.582	-1.236	0.495	0.757	-0.741	2.173	0.940	1.474	0.354	2.215	0.474	1.055	-1.657	0.000	0.415	1.758	0.841	0.000	0.451	0.578	0.984	0.757	0.922	0.860	0.696
-0	0.935	-1.614	-0.597	0.299	1.223	0.707	-0.853	-1.026	0.000	0.751	0.007	-1.691	0.000	1.062	-0.125	0.976	2.548	0.877	1.275	0.646	0.000	0.962	1.074	0.980	0.608	0.726	0.741	0.662
-1	0.643	0.542	-1.285	0.474	-0.366	0.667	-0.446	1.195	2.173	1.076	0.145	-0.126	0.000	0.970	-0.661	0.394	1.274	1.218	-0.184	-1.722	0.000	1.331	1.019	0.985	1.192	0.677	0.973	0.910
-0	0.713	0.164	1.080	1.427	-0.460	0.960	-0.152	-0.940	2.173	1.427	-0.901	1.036	1.107	0.440	-1.269	-0.194	0.000	0.452	1.932	-0.532	0.000	1.542	1.210	1.374	1.319	1.818	1.220	1.050
-0	0.876	-0.463	-1.224	2.458	-1.689	1.007	-0.752	0.398	0.000	2.456	-1.285	-0.152	1.107	1.641	1.838	1.717	0.000	0.458	0.194	0.488	3.102	4.848	2.463	0.986	1.981	0.974	2.642	2.258
-1	0.384	-0.275	0.387	1.403	-0.994	0.620	-1.529	1.685	0.000	1.091	-1.644	1.078	0.000	0.781	-1.311	0.326	2.548	1.228	-0.728	-0.633	1.551	0.920	0.854	0.987	0.646	0.609	0.740	0.884
-0	0.318	-1.818	-1.008	0.977	1.268	0.457	2.451	-1.522	0.000	0.881	1.351	0.461	2.215	0.929	0.239	-0.380	2.548	0.382	-0.613	1.330	0.000	1.563	1.193	0.994	0.829	0.874	0.901	1.026
-1	0.612	-1.120	1.098	0.402	-0.480	0.818	0.188	1.511	0.000	0.800	-0.253	0.977	0.000	1.175	0.271	-1.289	1.274	2.531	0.226	-0.409	3.102	0.889	0.947	0.979	1.486	0.940	1.152	1.119
-1	0.587	-0.737	-0.228	0.970	1.119	0.823	0.184	1.594	0.000	1.104	0.301	-0.818	2.215	0.819	0.712	-0.560	0.000	2.240	-0.419	0.340	3.102	1.445	1.103	0.988	0.715	1.363	1.019	0.926
-0	1.030	-0.694	-1.638	0.893	-1.074	1.160	-0.766	0.485	0.000	1.632	-0.698	-1.142	2.215	1.050	-1.092	0.952	0.000	1.475	0.286	0.125	3.102	0.914	1.075	0.982	0.732	1.493	1.219	1.079
-1	2.142	0.617	1.517	0.387	-0.862	0.345	1.203	-1.014	2.173	0.609	1.092	0.275	0.000	1.331	0.582	-0.183	2.548	0.557	1.540	-1.642	0.000	0.801	0.737	1.060	0.715	0.626	0.749	0.674
-0	1.076	0.240	-0.246	0.871	-1.241	0.496	0.282	0.746	2.173	1.095	-0.648	1.100	2.215	0.446	-1.756	0.764	0.000	0.434	0.788	-0.991	0.000	1.079	0.868	1.047	0.818	0.634	0.795	0.733
-0	1.400	0.901	-1.617	0.625	-0.163	0.661	-0.411	-1.616	2.173	0.685	0.524	0.425	0.000	0.881	-0.766	0.312	0.000	0.979	0.255	-0.667	3.102	0.898	1.105	1.253	0.730	0.716	0.738	0.795
-0	3.302	1.132	1.051	0.658	0.768	1.308	0.251	-0.374	1.087	1.673	0.015	-0.898	0.000	0.688	-0.535	1.363	1.274	0.871	1.325	-1.583	0.000	1.646	1.249	0.995	1.919	1.288	1.330	1.329
-0	1.757	0.202	0.750	0.767	-0.362	0.932	-1.033	-1.366	0.000	1.529	-1.012	-0.771	0.000	1.161	-0.287	0.059	0.000	2.185	1.147	1.099	3.102	0.795	0.529	1.354	1.144	1.491	1.319	1.161
-0	1.290	0.905	-1.711	1.017	-0.695	1.008	-1.038	0.693	2.173	1.202	-0.595	0.187	0.000	1.011	0.139	-1.607	0.000	0.789	-0.613	-1.041	3.102	1.304	0.895	1.259	1.866	0.955	1.211	1.200
-1	1.125	-0.004	1.694	0.373	0.329	0.978	0.640	-0.391	0.000	1.122	-0.376	1.521	2.215	0.432	2.413	-1.259	0.000	0.969	0.730	0.512	3.102	0.716	0.773	0.991	0.624	0.977	0.981	0.875
-0	1.081	0.861	1.252	1.621	1.474	1.293	0.600	0.630	0.000	1.991	-0.090	-0.675	2.215	0.861	1.105	-0.201	0.000	1.135	2.489	-1.659	0.000	1.089	0.657	0.991	2.179	0.412	1.334	1.071
-1	0.652	-0.294	1.241	1.034	0.490	1.033	0.551	-0.963	2.173	0.661	1.031	-1.654	2.215	1.376	-0.018	0.843	0.000	0.943	-0.329	-0.269	0.000	1.085	1.067	0.991	1.504	0.773	1.135	0.993
-1	1.408	-1.028	-1.018	0.252	-0.242	0.465	-0.364	-0.200	0.000	1.466	0.669	0.739	1.107	1.031	0.415	-1.468	2.548	0.457	-1.091	-1.722	0.000	0.771	0.811	0.979	1.459	1.204	1.041	0.866
-1	0.781	-1.143	-0.659	0.961	1.266	1.183	-0.686	0.119	2.173	1.126	-0.064	1.447	0.000	0.730	1.430	-1.535	0.000	1.601	0.513	1.658	0.000	0.871	1.345	1.184	1.058	0.620	1.107	0.978
-1	1.300	-0.616	1.032	0.751	-0.731	0.961	-0.716	1.592	0.000	2.079	-1.063	-0.271	2.215	0.475	0.518	1.695	1.274	0.395	-2.204	0.349	0.000	1.350	0.983	1.369	1.265	1.428	1.135	0.982
-1	0.833	0.809	1.657	1.637	1.019	0.705	1.077	-0.968	2.173	1.261	0.114	-0.298	1.107	1.032	0.017	0.236	0.000	0.640	-0.026	-1.598	0.000	0.894	0.982	0.981	1.250	1.054	1.018	0.853
-1	1.686	-1.090	-0.301	0.890	0.557	1.304	-0.284	-1.393	2.173	0.388	2.118	0.513	0.000	0.514	-0.015	0.891	0.000	0.460	0.547	0.627	3.102	0.942	0.524	1.186	1.528	0.889	1.015	1.122
-1	0.551	0.911	0.879	0.379	-0.796	1.154	-0.808	-0.966	0.000	1.168	-0.513	0.355	2.215	0.646	-1.309	0.773	0.000	0.544	-0.283	1.301	3.102	0.847	0.705	0.990	0.772	0.546	0.790	0.719
-1	1.597	0.793	-1.119	0.691	-1.455	0.370	0.337	1.354	0.000	0.646	-1.005	0.732	2.215	1.019	0.040	0.209	0.000	0.545	0.958	0.239	3.102	0.962	0.793	0.994	0.719	0.745	0.812	0.739
-0	1.033	-1.193	-0.452	0.247	0.970	0.503	-1.424	1.362	0.000	1.062	-0.416	-1.156	2.215	0.935	-0.023	0.555	2.548	0.410	-1.766	0.379	0.000	0.590	0.953	0.991	0.717	1.081	0.763	0.690
-1	0.859	-1.004	1.521	0.781	-0.993	0.677	0.643	-0.338	2.173	0.486	0.409	1.283	0.000	0.679	0.110	0.285	0.000	0.715	-0.735	-0.157	1.551	0.702	0.773	0.984	0.627	0.633	0.694	0.643
-0	0.612	-1.127	1.074	1.225	-0.426	0.927	-2.141	-0.473	0.000	1.290	-0.927	-1.085	2.215	1.183	1.981	-1.687	0.000	2.176	0.406	-1.581	0.000	0.945	0.651	1.170	0.895	1.604	1.179	1.142
-1	0.535	0.321	-1.095	0.281	-0.960	0.876	-0.709	-0.076	0.000	1.563	-0.666	1.536	2.215	0.773	-0.321	0.435	0.000	0.682	-0.801	-0.952	3.102	0.711	0.667	0.985	0.888	0.741	0.872	0.758
-1	0.745	1.586	1.578	0.863	-1.423	0.530	1.714	1.085	0.000	1.174	0.679	1.015	0.000	1.158	0.609	-1.186	2.548	1.851	0.832	-0.248	3.102	0.910	1.164	0.983	0.947	0.858	0.928	0.823
-0	0.677	-1.014	-1.648	1.455	1.461	0.596	-2.358	0.517	0.000	0.800	0.849	-0.743	2.215	1.024	-0.282	-1.004	0.000	1.846	-0.977	0.378	3.102	2.210	1.423	0.982	1.074	1.623	1.417	1.258
-1	0.815	-1.263	0.057	1.018	-0.208	0.339	-0.347	-1.646	2.173	1.223	0.600	-1.658	2.215	1.435	0.042	0.926	0.000	0.777	1.698	-0.698	0.000	1.022	1.058	1.000	0.784	0.477	0.886	0.836
-0	3.512	-1.094	-0.220	0.338	-0.328	1.962	-1.099	1.544	1.087	1.461	-1.305	-0.922	2.215	1.219	-1.289	0.400	0.000	0.731	0.155	1.249	0.000	1.173	1.366	0.993	2.259	2.000	1.626	1.349
-0	0.904	1.248	0.325	0.317	-1.624	0.685	-0.538	1.665	2.173	0.685	-2.145	-1.106	0.000	0.632	-1.460	1.017	0.000	1.085	-0.182	0.162	3.102	0.885	0.801	0.989	0.930	0.904	1.012	0.961

contrib/gbdt/lightgbm/lightgbm-example.ipynb

@@ -1,270 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/contrib/gbdt/lightgbm/lightgbm-example.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Use LightGBM Estimator in Azure Machine Learning\n",
-        "In this notebook we will demonstrate how to run a training job using LightGBM Estimator. [LightGBM](https://lightgbm.readthedocs.io/en/latest/) is a gradient boosting framework that uses tree based learning algorithms. "
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Prerequisites\n",
-        "This notebook uses azureml-contrib-gbdt package, if you don't already have the package, please install by uncommenting below cell."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#!pip install azureml-contrib-gbdt"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace, Run, Experiment\n",
-        "import shutil, os\n",
-        "from azureml.widgets import RunDetails\n",
-        "from azureml.contrib.gbdt import LightGBM\n",
-        "from azureml.train.dnn import Mpi\n",
-        "from azureml.core.compute import AmlCompute, ComputeTarget\n",
-        "from azureml.core.compute_target import ComputeTargetException"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If you are using an AzureML Compute Instance, you are all set. Otherwise, go through the [configuration.ipynb](../../../configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML Workspace"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Set up machine learning resources"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "\n",
-        "print('Workspace name: ' + ws.name, \n",
-        "      'Azure region: ' + ws.location, \n",
-        "      'Subscription id: ' + ws.subscription_id, \n",
-        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "cluster_vm_size = \"STANDARD_DS14_V2\"\n",
-        "cluster_min_nodes = 0\n",
-        "cluster_max_nodes = 20\n",
-        "cpu_cluster_name = 'TrainingCompute2' \n",
-        "\n",
-        "try:\n",
-        "    cpu_cluster = AmlCompute(ws, cpu_cluster_name)\n",
-        "    if cpu_cluster and type(cpu_cluster) is AmlCompute:\n",
-        "        print('found compute target: ' + cpu_cluster_name)\n",
-        "except ComputeTargetException:\n",
-        "    print('creating a new compute target...')\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = cluster_vm_size, \n",
-        "                                                                vm_priority = 'lowpriority', \n",
-        "                                                                min_nodes = cluster_min_nodes, \n",
-        "                                                                max_nodes = cluster_max_nodes)\n",
-        "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, provisioning_config)\n",
-        "    \n",
-        "    # can poll for a minimum number of nodes and for a specific timeout. \n",
-        "    # if no min node count is provided it will use the scale settings for the cluster\n",
-        "    cpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
-        "    \n",
-        "     # For a more detailed view of current Azure Machine Learning Compute  status, use get_status()\n",
-        "    print(cpu_cluster.get_status().serialize())"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "From this point, you can either upload training data file directly or use Datastore for training data storage\n",
-        "## Upload training file from local"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "scripts_folder = \"scripts_folder\"\n",
-        "if not os.path.isdir(scripts_folder):\n",
-        "    os.mkdir(scripts_folder)\n",
-        "shutil.copy('./train.conf', os.path.join(scripts_folder, 'train.conf'))\n",
-        "shutil.copy('./binary0.train', os.path.join(scripts_folder, 'binary0.train'))\n",
-        "shutil.copy('./binary1.train', os.path.join(scripts_folder, 'binary1.train'))\n",
-        "shutil.copy('./binary0.test', os.path.join(scripts_folder, 'binary0.test'))\n",
-        "shutil.copy('./binary1.test', os.path.join(scripts_folder, 'binary1.test'))"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_data_list=[\"binary0.train\", \"binary1.train\"]\n",
-        "validation_data_list = [\"binary0.test\", \"binary1.test\"]\n",
-        "lgbm = LightGBM(source_directory=scripts_folder, \n",
-        "                compute_target=cpu_cluster, \n",
-        "                distributed_training=Mpi(),\n",
-        "                node_count=2,\n",
-        "                lightgbm_config='train.conf',\n",
-        "                data=training_data_list,\n",
-        "                valid=validation_data_list\n",
-        "               )\n",
-        "experiment_name = 'lightgbm-estimator-test'\n",
-        "experiment = Experiment(ws, name=experiment_name)\n",
-        "run = experiment.submit(lgbm, tags={\"test public docker image\": None})\n",
-        "RunDetails(run).show()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "run.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Use data reference"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.datastore import Datastore\n",
-        "from azureml.data.data_reference import DataReference\n",
-        "datastore = ws.get_default_datastore()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "datastore.upload(src_dir='.',\n",
-        "                 target_path='.',\n",
-        "                 show_progress=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_data_list=[\"binary0.train\", \"binary1.train\"]\n",
-        "validation_data_list = [\"binary0.test\", \"binary1.test\"]\n",
-        "lgbm = LightGBM(source_directory='.', \n",
-        "                compute_target=cpu_cluster, \n",
-        "                distributed_training=Mpi(),\n",
-        "                node_count=2,\n",
-        "                inputs=[datastore.as_mount()],\n",
-        "                lightgbm_config='train.conf',\n",
-        "                data=training_data_list,\n",
-        "                valid=validation_data_list\n",
-        "               )\n",
-        "experiment_name = 'lightgbm-estimator-test'\n",
-        "experiment = Experiment(ws, name=experiment_name)\n",
-        "run = experiment.submit(lgbm, tags={\"use datastore.as_mount()\": None})\n",
-        "RunDetails(run).show()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "run.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# uncomment below and run if compute resources are no longer needed\n",
-        "# cpu_cluster.delete() "
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "jingywa"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.9"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}

contrib/gbdt/lightgbm/lightgbm-example.yml

@@ -1,7 +0,0 @@
-name: lightgbm-example
-dependencies:
-- pip:
-  - azureml-sdk
-  - azureml-contrib-gbdt
-  - azureml-widgets
-  - azureml-core

contrib/gbdt/lightgbm/train.conf

@@ -1,111 +0,0 @@
-# task type, support train and predict
-task = train
-
-# boosting type, support gbdt for now, alias: boosting, boost
-boosting_type = gbdt
-
-# application type, support following application
-# regression , regression task
-# binary , binary classification task
-# lambdarank , lambdarank task
-# alias: application, app
-objective = binary
-
-# eval metrics, support multi metric, delimite by ',' , support following metrics
-# l1 
-# l2 , default metric for regression
-# ndcg , default metric for lambdarank
-# auc 
-# binary_logloss , default metric for binary
-# binary_error
-metric = binary_logloss,auc
-
-# frequence for metric output
-metric_freq = 1
-
-# true if need output metric for training data, alias: tranining_metric, train_metric
-is_training_metric = true
-
-# number of bins for feature bucket, 255 is a recommend setting, it can save memories, and also has good accuracy. 
-max_bin = 255
-
-# training data
-# if exsting weight file, should name to "binary.train.weight"
-# alias: train_data, train
-data = binary.train
-
-# validation data, support multi validation data, separated by ','
-# if exsting weight file, should name to "binary.test.weight"
-# alias: valid, test, test_data, 
-valid_data = binary.test
-
-# number of trees(iterations), alias: num_tree, num_iteration, num_iterations, num_round, num_rounds
-num_trees = 100
-
-# shrinkage rate , alias: shrinkage_rate
-learning_rate = 0.1
-
-# number of leaves for one tree, alias: num_leaf
-num_leaves = 63
-
-# type of tree learner, support following types:
-# serial , single machine version
-# feature , use feature parallel to train
-# data , use data parallel to train
-# voting , use voting based parallel to train
-# alias: tree
-tree_learner = feature
-
-# number of threads for multi-threading. One thread will use one CPU, defalut is setted to #cpu. 
-# num_threads = 8
-
-# feature sub-sample, will random select 80% feature to train on each iteration 
-# alias: sub_feature
-feature_fraction = 0.8
-
-# Support bagging (data sub-sample), will perform bagging every 5 iterations
-bagging_freq = 5
-
-# Bagging farction, will random select 80% data on bagging
-# alias: sub_row
-bagging_fraction = 0.8
-
-# minimal number data for one leaf, use this to deal with over-fit
-# alias : min_data_per_leaf, min_data
-min_data_in_leaf = 50
-
-# minimal sum hessians for one leaf, use this to deal with over-fit
-min_sum_hessian_in_leaf = 5.0
-
-# save memory and faster speed for sparse feature, alias: is_sparse
-is_enable_sparse = true
-
-# when data is bigger than memory size, set this to true. otherwise set false will have faster speed
-# alias: two_round_loading, two_round
-use_two_round_loading = false
-
-# true if need to save data to binary file and application will auto load data from binary file next time
-# alias: is_save_binary, save_binary
-is_save_binary_file = false
-
-# output model file
-output_model = LightGBM_model.txt
-
-# support continuous train from trained gbdt model
-# input_model= trained_model.txt
-
-# output prediction file for predict task
-# output_result= prediction.txt
-
-# support continuous train from initial score file
-# input_init_score= init_score.txt
-
-
-# number of machines in parallel training, alias: num_machine
-num_machines = 2
-
-# local listening port in parallel training, alias: local_port
-local_listen_port = 12400
-
-# machines list file for parallel training, alias: mlist
-machine_list_file = mlist.txt

how-to-use-azureml/README.md

@@ -4,7 +4,7 @@ Learn how to use Azure Machine Learning services for experimentation and model m
 
 As a pre-requisite, run the [configuration Notebook](../configuration.ipynb) notebook first to set up your Azure ML Workspace. Then, run the notebooks in following recommended order.
 
-* [train-within-notebook](./training/train-within-notebook): Train a model hile tracking run history, and learn how to deploy the model as web service to Azure Container Instance.
+* [train-within-notebook](./training/train-within-notebook): Train a model while tracking run history, and learn how to deploy the model as web service to Azure Container Instance.
 * [train-on-local](./training/train-on-local): Learn how to submit a run to local computer and use Azure ML managed run configuration.
 * [train-on-amlcompute](./training/train-on-amlcompute): Use a 1-n node Azure ML managed compute cluster for remote runs on Azure CPU or GPU infrastructure.
 * [train-on-remote-vm](./training/train-on-remote-vm): Use Data Science Virtual Machine as a target for remote runs.

how-to-use-azureml/automated-machine-learning/README.md

@@ -97,68 +97,96 @@ jupyter notebook
  <a name="databricks"></a>
 ## Setup using Azure Databricks
 
-**NOTE**: Please create your Azure Databricks cluster as v6.0 (high concurrency preferred) with **Python 3** (dropdown).
+**NOTE**: Please create your Azure Databricks cluster as v7.1 (high concurrency preferred) with **Python 3** (dropdown).
 **NOTE**: You should at least have contributor access to your Azure subcription to run the notebook.
-- Please remove the previous SDK version if there is any and install the latest SDK by installing **azureml-sdk[automl]** as a PyPi library in Azure Databricks workspace.
-- You can find the detail Readme instructions at [GitHub](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks).
-- Download the sample notebook automl-databricks-local-01.ipynb from [GitHub](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks) and import into the Azure databricks workspace.
+- You can find the detail Readme instructions at [GitHub](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks/automl).
+- Download the sample notebook automl-databricks-local-01.ipynb from [GitHub](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks/automl) and import into the Azure databricks workspace.
 - Attach the notebook to the cluster.
 
 <a name="samples"></a>
 # Automated ML SDK Sample Notebooks
 
-- [auto-ml-classification-credit-card-fraud.ipynb](classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb)
-    - Dataset: Kaggle's [credit card fraud detection dataset](https://www.kaggle.com/mlg-ulb/creditcardfraud)
-    - Simple example of using automated ML for classification to fraudulent credit card transactions
-    - Uses azure compute for training
+## Classification
+- **Classify Credit Card Fraud**
+    - Dataset: [Kaggle's credit card fraud detection dataset](https://www.kaggle.com/mlg-ulb/creditcardfraud)
+      - **[Jupyter Notebook (remote run)](classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb)**
+          - run the experiment remotely on AML Compute cluster
+          - test the performance of the best model in the local environment
+      - **[Jupyter Notebook (local run)](local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb)**
+          - run experiment in the local environment
+          - use Mimic Explainer for computing feature importance
+          - deploy the best model along with the explainer to an Azure Kubernetes (AKS) cluster, which will compute the raw and engineered feature importances at inference time
+- **Predict Term Deposit Subscriptions in a Bank**
+    - Dataset: [UCI's bank marketing dataset](https://www.kaggle.com/janiobachmann/bank-marketing-dataset)
+        - **[Jupyter Notebook](classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb)**
+          - run experiment remotely on AML Compute cluster to generate ONNX compatible models
+          - view the featurization steps that were applied during training
+          - view feature importance for the best model
+          - download the best model in ONNX format and use it for inferencing using ONNXRuntime
+          - deploy the best model in PKL format to Azure Container Instance (ACI)
+- **Predict Newsgroup based on Text from News Article**
+    - Dataset: [20 newsgroups text dataset](https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html)
+        - **[Jupyter Notebook](classification-text-dnn/auto-ml-classification-text-dnn.ipynb)**
+          - AutoML highlights here include using deep neural networks (DNNs) to create embedded features from text data
+          - AutoML will use Bidirectional Encoder Representations from Transformers (BERT) when a GPU compute is used
+          - Bidirectional Long-Short Term neural network (BiLSTM) will be utilized when a CPU compute is used, thereby optimizing the choice of DNN
 
-- [auto-ml-regression.ipynb](regression/auto-ml-regression.ipynb)
+## Regression
+- **Predict Performance of Hardware Parts**
     - Dataset: Hardware Performance Dataset
-    - Simple example of using automated ML for regression
-    - Uses azure compute for training
+        - **[Jupyter Notebook](regression/auto-ml-regression.ipynb)**
+            - run the experiment remotely on AML Compute cluster
+            - get best trained model for a different metric than the one the experiment was optimized for
+            - test the performance of the best model in the local environment
+        - **[Jupyter Notebook (advanced)](regression/auto-ml-regression.ipynb)**
+            - run the experiment remotely on AML Compute cluster
+            - customize featurization: override column purpose within the dataset, configure transformer parameters
+            - get best trained model for a different metric than the one the experiment was optimized for
+            - run a model explanation experiment on the remote cluster
+            - deploy the model along the explainer and run online inferencing
 
-- [auto-ml-regression-explanation-featurization.ipynb](regression-explanation-featurization/auto-ml-regression-explanation-featurization.ipynb)
-    - Dataset: Hardware Performance Dataset
-    - Shows featurization and excplanation
-    - Uses azure compute for training
-
-- [auto-ml-forecasting-energy-demand.ipynb](forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb)
-    - Dataset: [NYC energy demand data](forecasting-a/nyc_energy.csv)
-    - Example of using automated ML for training a forecasting model
-
-- [auto-ml-classification-credit-card-fraud-local.ipynb](local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb)
-    - Dataset: Kaggle's [credit card fraud detection dataset](https://www.kaggle.com/mlg-ulb/creditcardfraud)
-    - Simple example of using automated ML for classification to fraudulent credit card transactions
-    - Uses local compute for training
-
-- [auto-ml-classification-bank-marketing-all-features.ipynb](classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb)
-    - Dataset: UCI's [bank marketing dataset](https://www.kaggle.com/janiobachmann/bank-marketing-dataset)
-    - Simple example of using automated ML for classification to predict term deposit subscriptions for a bank
-    - Uses azure compute for training
-
-- [auto-ml-forecasting-orange-juice-sales.ipynb](forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb)
-    - Dataset: [Dominick's grocery sales of orange juice](forecasting-b/dominicks_OJ.csv)
-    - Example of training an automated ML forecasting model on multiple time-series
-
-- [auto-ml-forecasting-bike-share.ipynb](forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb)
-    - Dataset: forecasting for a bike-sharing
-    - Example of training an automated ML forecasting model on multiple time-series
-
-- [auto-ml-forecasting-function.ipynb](forecasting-forecast-function/auto-ml-forecasting-function.ipynb)
-    - Example of training an automated ML forecasting model on multiple time-series
-
-- [auto-ml-forecasting-beer-remote.ipynb](forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb)
-    - Example of training an automated ML forecasting model on multiple time-series
-    - Beer Production Forecasting
-
-- [auto-ml-continuous-retraining.ipynb](continuous-retraining/auto-ml-continuous-retraining.ipynb)
-    - Continuous retraining using Pipelines and Time-Series TabularDataset
-
-- [auto-ml-classification-text-dnn.ipynb](classification-text-dnn/auto-ml-classification-text-dnn.ipynb)
-    - Classification with text data using deep learning in AutoML
-    - AutoML highlights here include using deep neural networks (DNNs) to create embedded features from text data.
-    - Depending on the compute cluster the user provides, AutoML tried out Bidirectional Encoder Representations from Transformers (BERT) when a GPU compute is used.
-    - Bidirectional Long-Short Term neural network (BiLSTM) when a CPU compute is used, thereby optimizing the choice of DNN for the uesr's setup.
+## Time Series Forecasting
+- **Forecast Energy Demand**
+    - Dataset: [NYC energy demand data](http://mis.nyiso.com/public/P-58Blist.htm)
+        - **[Jupyter Notebook](forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb)**
+          - run experiment remotely on AML Compute cluster
+          - use lags and rolling window features
+          - view the featurization steps that were applied during training
+          - get the best model, use it to forecast on test data and compare the accuracy of predictions against real data
+- **Forecast Orange Juice Sales (Multi-Series)**
+    - Dataset: [Dominick's grocery sales of orange juice](forecasting-orange-juice-sales/dominicks_OJ.csv)
+        - **[Jupyter Notebook](forecasting-orange-juice-sales/dominicks_OJ.csv)**
+          - run experiment remotely on AML Compute cluster
+          - customize time-series featurization, change column purpose and override transformer hyper parameters
+          - evaluate locally the performance of the generated best model
+          - deploy the best model as a webservice on Azure Container Instance (ACI)
+          - get online predictions from the deployed model
+- **Forecast Demand of a Bike-Sharing Service**
+    - Dataset: [Bike demand data](forecasting-bike-share/bike-no.csv)
+        - **[Jupyter Notebook](forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb)**
+          - run experiment remotely on AML Compute cluster
+          - integrate holiday features
+          - run rolling forecast for test set that is longer than the forecast horizon
+          - compute metrics on the predictions from the remote forecast
+- **The Forecast Function Interface**
+    - Dataset: Generated for sample purposes
+        - **[Jupyter Notebook](forecasting-forecast-function/auto-ml-forecasting-function.ipynb)**
+          - train a forecaster using a remote AML Compute cluster
+          - capabilities of forecast function (e.g. forecast farther into the horizon)
+          - generate confidence intervals
+- **Forecast Beverage Production**
+    - Dataset: [Monthly beer production data](forecasting-beer-remote/Beer_no_valid_split_train.csv)
+        - **[Jupyter Notebook](forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb)**
+          - train using a remote AML Compute cluster
+          - enable the DNN learning model
+          - forecast on a remote compute cluster and compare different model performance
+- **Continuous Retraining with NOAA Weather Data**
+    - Dataset: [NOAA weather data from Azure Open Datasets](https://azure.microsoft.com/en-us/services/open-datasets/)
+        - **[Jupyter Notebook](continuous-retraining/auto-ml-continuous-retraining.ipynb)**
+          - continuously retrain a model using Pipelines and AutoML
+          - create a Pipeline to upload a time series dataset to an Azure blob
+          - create a Pipeline to run an AutoML experiment and register the best resulting model in the Workspace
+          - publish the training pipeline created and schedule it to run daily
 
 <a name="documentation"></a>
 See [Configure automated machine learning experiments](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-auto-train) to learn how more about the the settings and features available for automated machine learning experiments.
@@ -179,7 +207,7 @@ The main code of the file must be indented so that it is under this condition.
 ## automl_setup fails
 1. On Windows, make sure that you are running automl_setup from an Anconda Prompt window rather than a regular cmd window.  You can launch the "Anaconda Prompt" window by hitting the Start button and typing "Anaconda Prompt".  If you don't see the application "Anaconda Prompt", you might not have conda or mini conda installed.  In that case, you can install it [here](https://conda.io/miniconda.html)
 2. Check that you have conda 64-bit installed rather than 32-bit.  You can check this with the command `conda info`.  The `platform` should be `win-64` for Windows or `osx-64` for Mac.
-3. Check that you have conda 4.4.10 or later.  You can check the version with the command `conda -V`.  If you have a previous version installed, you can update it using the command: `conda update conda`.
+3. Check that you have conda 4.7.8 or later.  You can check the version with the command `conda -V`.  If you have a previous version installed, you can update it using the command: `conda update conda`.
 4. On Linux, if the error is `gcc: error trying to exec 'cc1plus': execvp: No such file or directory`, install build essentials using the command `sudo apt-get install build-essential`.
 5. Pass a new name as the first parameter to automl_setup so that it creates a new conda environment. You can view existing conda environments using `conda env list` and remove them with `conda env remove -n <environmentname>`.
 
@@ -230,6 +258,15 @@ You may check the version of tensorflow and uninstall as follows
 2) enter `pip freeze` and look for `tensorflow` , if found, the version listed should be < 1.13
 3) If the listed version is a not a supported version,  `pip uninstall tensorflow` in the command shell and enter y for confirmation.
 
+## KeyError: 'brand' when running AutoML on local compute or Azure Databricks cluster**
+If a new environment was created after 10 June 2020 using SDK 1.7.0 or lower, training may fail with the above error due to an update in the py-cpuinfo package. (Environments created on or before 10 June 2020 are unaffected, as well as experiments run on remote compute as cached training images are used.) To work around this issue, either of the two following steps can be taken:
+
+1) Update the SDK version to 1.8.0 or higher (this will also downgrade py-cpuinfo to 5.0.0):
+`pip install --upgrade azureml-sdk[automl]`
+
+2) Downgrade the installed version of py-cpuinfo to 5.0.0:
+`pip install py-cpuinfo==5.0.0`
+
 ## Remote run: DsvmCompute.create fails
 There are several reasons why the DsvmCompute.create can fail.  The reason is usually in the error message but you have to look at the end of the error message for the detailed reason.  Some common reasons are:
 1) `Compute name is invalid, it should start with a letter, be between 2 and 16 character, and only include letters (a-zA-Z), numbers (0-9) and \'-\'.`  Note that underscore is not allowed in the name.

how-to-use-azureml/automated-machine-learning/automl_env.yml

@@ -2,30 +2,27 @@ name: azure_automl
 dependencies:
   # The python interpreter version.
   # Currently Azure ML only supports 3.5.2 and later.
-- pip<=19.3.1
-- python>=3.5.2,<3.6.8
+- pip==21.1.2
+- python>=3.5.2,<3.8
 - nb_conda
+- boto3==1.15.18
 - matplotlib==2.1.0
-- numpy>=1.16.0,<=1.16.2
+- numpy==1.18.5
 - cython
 - urllib3<1.24
-- scipy==1.4.1
-- scikit-learn>=0.19.0,<=0.20.3
-- pandas>=0.22.0,<=0.23.4
+- scipy>=1.4.1,<=1.5.2
+- scikit-learn==0.22.1
+- pandas==0.25.1
 - py-xgboost<=0.90
 - conda-forge::fbprophet==0.5
+- holidays==0.9.11
 - pytorch::pytorch=1.4.0
 - cudatoolkit=10.1.243
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azureml-defaults
-  - azureml-train-automl
-  - azureml-train
-  - azureml-widgets
-  - azureml-pipeline
+  - azureml-widgets~=1.32.0
   - pytorch-transformers==1.0.0
   - spacy==2.1.8
-  - pyarrow==0.17.0
   - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
-
+  - -r https://automlresources-prod.azureedge.net/validated-requirements/1.32.0/validated_win32_requirements.txt [--no-deps]

how-to-use-azureml/automated-machine-learning/automl_env_linux.yml

@@ -0,0 +1,28 @@
+name: azure_automl
+dependencies:
+  # The python interpreter version.
+  # Currently Azure ML only supports 3.5.2 and later.
+- pip==21.1.2
+- python>=3.5.2,<3.8
+- nb_conda
+- boto3==1.15.18
+- matplotlib==2.1.0
+- numpy==1.18.5
+- cython
+- urllib3<1.24
+- scipy>=1.4.1,<=1.5.2
+- scikit-learn==0.22.1
+- pandas==0.25.1
+- py-xgboost<=0.90
+- conda-forge::fbprophet==0.5
+- holidays==0.9.11
+- pytorch::pytorch=1.4.0
+- cudatoolkit=10.1.243
+
+- pip:
+  # Required packages for AzureML execution, history, and data preparation.
+  - azureml-widgets~=1.32.0
+  - pytorch-transformers==1.0.0
+  - spacy==2.1.8
+  - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
+  - -r https://automlresources-prod.azureedge.net/validated-requirements/1.32.0/validated_linux_requirements.txt [--no-deps]

how-to-use-azureml/automated-machine-learning/automl_env_mac.yml

@@ -2,30 +2,28 @@ name: azure_automl
 dependencies:
   # The python interpreter version.
   # Currently Azure ML only supports 3.5.2 and later.
-- pip<=19.3.1
+- pip==21.1.2
 - nomkl
-- python>=3.5.2,<3.6.8
+- python>=3.5.2,<3.8
 - nb_conda
+- boto3==1.15.18
 - matplotlib==2.1.0
-- numpy>=1.16.0,<=1.16.2
+- numpy==1.18.5
 - cython
 - urllib3<1.24
-- scipy==1.4.1
-- scikit-learn>=0.19.0,<=0.20.3
-- pandas>=0.22.0,<=0.23.4
+- scipy>=1.4.1,<=1.5.2
+- scikit-learn==0.22.1
+- pandas==0.25.1
 - py-xgboost<=0.90
 - conda-forge::fbprophet==0.5
+- holidays==0.9.11
 - pytorch::pytorch=1.4.0
 - cudatoolkit=9.0
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azureml-defaults
-  - azureml-train-automl
-  - azureml-train
-  - azureml-widgets
-  - azureml-pipeline
+  - azureml-widgets~=1.32.0
   - pytorch-transformers==1.0.0
   - spacy==2.1.8
-  - pyarrow==0.17.0
   - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
+  - -r https://automlresources-prod.azureedge.net/validated-requirements/1.32.0/validated_darwin_requirements.txt [--no-deps]

how-to-use-azureml/automated-machine-learning/automl_setup.cmd

@@ -6,11 +6,22 @@ set PIP_NO_WARN_SCRIPT_LOCATION=0
 
 IF "%conda_env_name%"=="" SET conda_env_name="azure_automl"
 IF "%automl_env_file%"=="" SET automl_env_file="automl_env.yml"
+SET check_conda_version_script="check_conda_version.py"
 
 IF NOT EXIST %automl_env_file% GOTO YmlMissing
 
 IF "%CONDA_EXE%"=="" GOTO CondaMissing
 
+IF NOT EXIST %check_conda_version_script% GOTO VersionCheckMissing
+
+python "%check_conda_version_script%"
+IF errorlevel 1 GOTO ErrorExit:
+
+SET replace_version_script="replace_latest_version.ps1"
+IF EXIST %replace_version_script% (
+  powershell -file %replace_version_script% %automl_env_file%
+)
+
 call conda activate %conda_env_name% 2>nul:
 
 if not errorlevel 1 (
@@ -54,6 +65,10 @@ echo If you are running an older version of Miniconda or Anaconda,
 echo you can upgrade using the command: conda update conda
 goto End
 
+:VersionCheckMissing
+echo File %check_conda_version_script% not found.
+goto End
+
 :YmlMissing
 echo File %automl_env_file% not found.
 

how-to-use-azureml/automated-machine-learning/automl_setup_linux.sh

@@ -4,6 +4,7 @@ CONDA_ENV_NAME=$1
 AUTOML_ENV_FILE=$2
 OPTIONS=$3
 PIP_NO_WARN_SCRIPT_LOCATION=0
+CHECK_CONDA_VERSION_SCRIPT="check_conda_version.py"
 
 if [ "$CONDA_ENV_NAME" == "" ]
 then
@@ -12,7 +13,7 @@ fi
 
 if [ "$AUTOML_ENV_FILE" == "" ]
 then
-  AUTOML_ENV_FILE="automl_env.yml"
+  AUTOML_ENV_FILE="automl_env_linux.yml"
 fi
 
 if [ ! -f $AUTOML_ENV_FILE ]; then
@@ -20,6 +21,18 @@ if [ ! -f $AUTOML_ENV_FILE ]; then
     exit 1
 fi
 
+if [ ! -f $CHECK_CONDA_VERSION_SCRIPT ]; then
+    echo "File $CHECK_CONDA_VERSION_SCRIPT not found"
+    exit 1
+fi
+
+python "$CHECK_CONDA_VERSION_SCRIPT"
+if [ $? -ne 0 ]; then
+    exit 1
+fi
+
+sed -i 's/AZUREML-SDK-VERSION/latest/' $AUTOML_ENV_FILE
+
 if source activate $CONDA_ENV_NAME 2> /dev/null
 then
    echo "Upgrading existing conda environment" $CONDA_ENV_NAME

how-to-use-azureml/automated-machine-learning/automl_setup_mac.sh

@@ -4,6 +4,7 @@ CONDA_ENV_NAME=$1
 AUTOML_ENV_FILE=$2
 OPTIONS=$3
 PIP_NO_WARN_SCRIPT_LOCATION=0
+CHECK_CONDA_VERSION_SCRIPT="check_conda_version.py"
 
 if [ "$CONDA_ENV_NAME" == "" ]
 then
@@ -20,6 +21,19 @@ if [ ! -f $AUTOML_ENV_FILE ]; then
     exit 1
 fi
 
+if [ ! -f $CHECK_CONDA_VERSION_SCRIPT ]; then
+    echo "File $CHECK_CONDA_VERSION_SCRIPT not found"
+    exit 1
+fi
+
+python "$CHECK_CONDA_VERSION_SCRIPT"
+if [ $? -ne 0 ]; then
+    exit 1
+fi
+
+sed -i '' 's/AZUREML-SDK-VERSION/latest/' $AUTOML_ENV_FILE
+brew install libomp
+
 if source activate $CONDA_ENV_NAME 2> /dev/null
 then
    echo "Upgrading existing conda environment" $CONDA_ENV_NAME

how-to-use-azureml/automated-machine-learning/check_conda_version.py

@@ -0,0 +1,26 @@
+from distutils.version import LooseVersion
+import platform
+
+try:
+    import conda
+except:
+    print('Failed to import conda.')
+    print('This setup is usually run from the base conda environment.')
+    print('You can activate the base environment using the command "conda activate base"')
+    exit(1)
+
+architecture = platform.architecture()[0]
+
+if architecture != "64bit":
+    print('This setup requires 64bit Anaconda or Miniconda.  Found: ' + architecture)
+    exit(1)
+
+minimumVersion = "4.7.8"
+
+versionInvalid = (LooseVersion(conda.__version__) < LooseVersion(minimumVersion))
+
+if versionInvalid:
+    print('Setup requires conda version ' + minimumVersion + ' or higher.')
+    print('You can use the command "conda update conda" to upgrade conda.')
+
+exit(versionInvalid)

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb

@@ -57,7 +57,7 @@
         "9. Test the ACI service.\n",
         "\n",
         "In addition this notebook showcases the following features\n",
-        "- **Blacklisting** certain pipelines\n",
+        "- **Blocking** certain pipelines\n",
         "- Specifying **target metrics** to indicate stopping criteria\n",
         "- Handling **missing data** in the input"
       ]
@@ -89,7 +89,7 @@
         "from azureml.automl.core.featurization import FeaturizationConfig\n",
         "from azureml.core.dataset import Dataset\n",
         "from azureml.train.automl import AutoMLConfig\n",
-        "from azureml.explain.model._internal.explanation_client import ExplanationClient"
+        "from azureml.interpret import ExplanationClient"
       ]
     },
     {
@@ -105,7 +105,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -165,9 +165,12 @@
       "source": [
         "## Create or Attach existing AmlCompute\n",
         "You will need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "#### Creation of AmlCompute takes approximately 5 minutes. \n",
         "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota."
+        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
     },
     {
@@ -187,7 +190,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=6)\n",
         "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
         "\n",
@@ -314,8 +317,8 @@
         "|**task**|classification or regression or forecasting|\n",
         "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
         "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
-        "|**blacklist_models** | *List* of *strings* indicating machine learning algorithms for AutoML to avoid in this run. <br><br> Allowed values for **Classification**<br><i>LogisticRegression</i><br><i>SGD</i><br><i>MultinomialNaiveBayes</i><br><i>BernoulliNaiveBayes</i><br><i>SVM</i><br><i>LinearSVM</i><br><i>KNN</i><br><i>DecisionTree</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>GradientBoosting</i><br><i>TensorFlowDNN</i><br><i>TensorFlowLinearClassifier</i><br><br>Allowed values for **Regression**<br><i>ElasticNet</i><br><i>GradientBoosting</i><br><i>DecisionTree</i><br><i>KNN</i><br><i>LassoLars</i><br><i>SGD</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>TensorFlowLinearRegressor</i><br><i>TensorFlowDNN</i><br><br>Allowed values for **Forecasting**<br><i>ElasticNet</i><br><i>GradientBoosting</i><br><i>DecisionTree</i><br><i>KNN</i><br><i>LassoLars</i><br><i>SGD</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>TensorFlowLinearRegressor</i><br><i>TensorFlowDNN</i><br><i>Arima</i><br><i>Prophet</i>|\n",
-        "| **whitelist_models** |  *List* of *strings* indicating machine learning algorithms for AutoML to use in this run. Same values listed above for **blacklist_models** allowed for **whitelist_models**.|\n",
+        "|**blocked_models** | *List* of *strings* indicating machine learning algorithms for AutoML to avoid in this run. <br><br> Allowed values for **Classification**<br><i>LogisticRegression</i><br><i>SGD</i><br><i>MultinomialNaiveBayes</i><br><i>BernoulliNaiveBayes</i><br><i>SVM</i><br><i>LinearSVM</i><br><i>KNN</i><br><i>DecisionTree</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>GradientBoosting</i><br><i>TensorFlowDNN</i><br><i>TensorFlowLinearClassifier</i><br><br>Allowed values for **Regression**<br><i>ElasticNet</i><br><i>GradientBoosting</i><br><i>DecisionTree</i><br><i>KNN</i><br><i>LassoLars</i><br><i>SGD</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>TensorFlowLinearRegressor</i><br><i>TensorFlowDNN</i><br><br>Allowed values for **Forecasting**<br><i>ElasticNet</i><br><i>GradientBoosting</i><br><i>DecisionTree</i><br><i>KNN</i><br><i>LassoLars</i><br><i>SGD</i><br><i>RandomForest</i><br><i>ExtremeRandomTrees</i><br><i>LightGBM</i><br><i>TensorFlowLinearRegressor</i><br><i>TensorFlowDNN</i><br><i>Arima</i><br><i>Prophet</i>|\n",
+        "|**allowed_models** |  *List* of *strings* indicating machine learning algorithms for AutoML to use in this run. Same values listed above for **blocked_models** allowed for **allowed_models**.|\n",
         "|**experiment_exit_score**| Value indicating the target for *primary_metric*. <br>Once the target is surpassed the run terminates.|\n",
         "|**experiment_timeout_hours**| Maximum amount of time in hours that all iterations combined can take before the experiment terminates.|\n",
         "|**enable_early_stopping**| Flag to enble early termination if the score is not improving in the short term.|\n",
@@ -349,7 +352,7 @@
         "                             debug_log = 'automl_errors.log',\n",
         "                             compute_target=compute_target,\n",
         "                             experiment_exit_score = 0.9984,\n",
-        "                             blacklist_models = ['KNN','LinearSVM'],\n",
+        "                             blocked_models = ['KNN','LinearSVM'],\n",
         "                             enable_onnx_compatible_models=True,\n",
         "                             training_data = train_data,\n",
         "                             label_column_name = label,\n",
@@ -362,7 +365,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Execution of local runs is synchronous. Depending on the data and the number of iterations this can run for a while."
+        "Call the `submit` method on the experiment object and pass the run configuration. Execution of local runs is synchronous. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -374,15 +377,6 @@
         "remote_run = experiment.submit(automl_config, show_output = False)"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -500,11 +494,10 @@
       "source": [
         "# Wait for the best model explanation run to complete\n",
         "from azureml.core.run import Run\n",
-        "model_explainability_run_id = remote_run.get_properties().get('ModelExplainRunId')\n",
+        "model_explainability_run_id = remote_run.id + \"_\" + \"ModelExplain\"\n",
         "print(model_explainability_run_id)\n",
-        "if model_explainability_run_id is not None:\n",
-        "    model_explainability_run = Run(experiment=experiment, run_id=model_explainability_run_id)\n",
-        "    model_explainability_run.wait_for_completion()\n",
+        "model_explainability_run = Run(experiment=experiment, run_id=model_explainability_run_id)\n",
+        "model_explainability_run.wait_for_completion()\n",
         "\n",
         "# Get the best run object\n",
         "best_run, fitted_model = remote_run.get_output()"
@@ -675,10 +668,8 @@
         "model_name = best_run.properties['model_name']\n",
         "\n",
         "script_file_name = 'inference/score.py'\n",
-        "conda_env_file_name = 'inference/env.yml'\n",
         "\n",
-        "best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'inference/score.py')\n",
-        "best_run.download_file('outputs/conda_env_v_1_0_0.yml', 'inference/env.yml')"
+        "best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'inference/score.py')"
       ]
     },
     {
@@ -721,8 +712,7 @@
         "from azureml.core.model import Model\n",
         "from azureml.core.environment import Environment\n",
         "\n",
-        "myenv = Environment.from_conda_specification(name=\"myenv\", file_path=conda_env_file_name)\n",
-        "inference_config = InferenceConfig(entry_script=script_file_name, environment=myenv)\n",
+        "inference_config = InferenceConfig(entry_script=script_file_name)\n",
         "\n",
         "aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1, \n",
         "                                               memory_gb = 1, \n",
@@ -736,24 +726,6 @@
         "print(aci_service.state)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Delete a Web Service\n",
-        "\n",
-        "Deletes the specified web service."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#aci_service.delete()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -778,7 +750,9 @@
       "source": [
         "## Test\n",
         "\n",
-        "Now that the model is trained, run the test data through the trained model to get the predicted values."
+        "Now that the model is trained, run the test data through the trained model to get the predicted values.  This calls the ACI web service to do the prediction.\n",
+        "\n",
+        "Note that the JSON passed to the ACI web service is an array of rows of data.  Each row should either be an array of values in the same order that was used for training or a dictionary where the keys are the same as the column names used for training.  The example below uses dictionary rows."
       ]
     },
     {
@@ -818,10 +792,27 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "y_pred  = fitted_model.predict(X_test)\n",
+        "import json\n",
+        "import requests\n",
+        "\n",
+        "X_test_json = X_test.to_json(orient='records')\n",
+        "data = \"{\\\"data\\\": \" + X_test_json +\"}\"\n",
+        "headers = {'Content-Type': 'application/json'}\n",
+        "\n",
+        "resp = requests.post(aci_service.scoring_uri, data, headers=headers)\n",
+        "\n",
+        "y_pred = json.loads(json.loads(resp.text))['result']"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
         "actual = array(y_test)\n",
         "actual = actual[:,0]\n",
-        "print(y_pred.shape, \" \", actual.shape)"
+        "print(len(y_pred), \" \", len(actual))"
       ]
     },
     {
@@ -830,8 +821,7 @@
       "source": [
         "### Calculate metrics for the prediction\n",
         "\n",
-        "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
-        "from the trained model that was returned."
+        "Now visualize the data as a confusion matrix that compared the predicted values against the actual values.\n"
       ]
     },
     {
@@ -841,12 +831,45 @@
       "outputs": [],
       "source": [
         "%matplotlib notebook\n",
-        "test_pred = plt.scatter(actual, y_pred, color='b')\n",
-        "test_test = plt.scatter(actual, actual, color='g')\n",
-        "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
+        "from sklearn.metrics import confusion_matrix\n",
+        "import numpy as np\n",
+        "import itertools\n",
+        "\n",
+        "cf =confusion_matrix(actual,y_pred)\n",
+        "plt.imshow(cf,cmap=plt.cm.Blues,interpolation='nearest')\n",
+        "plt.colorbar()\n",
+        "plt.title('Confusion Matrix')\n",
+        "plt.xlabel('Predicted')\n",
+        "plt.ylabel('Actual')\n",
+        "class_labels = ['no','yes']\n",
+        "tick_marks = np.arange(len(class_labels))\n",
+        "plt.xticks(tick_marks,class_labels)\n",
+        "plt.yticks([-0.5,0,1,1.5],['','no','yes',''])\n",
+        "# plotting text value inside cells\n",
+        "thresh = cf.max() / 2.\n",
+        "for i,j in itertools.product(range(cf.shape[0]),range(cf.shape[1])):\n",
+        "    plt.text(j,i,format(cf[i,j],'d'),horizontalalignment='center',color='white' if cf[i,j] >thresh else 'black')\n",
         "plt.show()"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Delete a Web Service\n",
+        "\n",
+        "Deletes the specified web service."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "aci_service.delete()"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -870,7 +893,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "anumamah"
+        "name": "ratanase"
       }
     ],
     "category": "tutorial",

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.yml

@@ -2,7 +2,3 @@ name: auto-ml-classification-bank-marketing-all-features
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib
-  - onnxruntime==1.0.0

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb

@@ -93,7 +93,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -127,6 +127,9 @@
       "source": [
         "## Create or Attach existing AmlCompute\n",
         "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "#### Creation of AmlCompute takes approximately 5 minutes. \n",
         "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
@@ -232,7 +235,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while."
+        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -255,15 +258,6 @@
         "#remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -424,22 +418,33 @@
       "source": [
         "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
         "\n",
+        "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Universit\u00c3\u00a9 Libre de Bruxelles) on big data mining and fraud detection.\n",
+        "More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
         "\n",
-        "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Universit\u00c3\u0192\u00c2\u00a9 Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
-        "Please cite the following works: \n",
-        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tAndrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
-        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
-        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
-        "o\tDal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
-        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
-        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing"
+        "Please cite the following works:\n",
+        "\n",
+        "Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
+        "\n",
+        "Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
+        "\n",
+        "Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
+        "\n",
+        "Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
+        "\n",
+        "Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-A\u00c3\u00abl; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
+        "\n",
+        "Carcillo, Fabrizio; Le Borgne, Yann-A\u00c3\u00abl; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing\n",
+        "\n",
+        "Bertrand Lebichot, Yann-A\u00c3\u00abl Le Borgne, Liyun He, Frederic Obl\u00c3\u00a9, Gianluca Bontempi Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, pp 78-88, 2019\n",
+        "\n",
+        "Fabrizio Carcillo, Yann-A\u00c3\u00abl Le Borgne, Olivier Caelen, Frederic Obl\u00c3\u00a9, Gianluca Bontempi Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection Information Sciences, 2019"
       ]
     }
   ],
   "metadata": {
     "authors": [
       {
-        "name": "tzvikei"
+        "name": "ratanase"
       }
     ],
     "category": "tutorial",

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.yml

@@ -2,6 +2,3 @@ name: auto-ml-classification-credit-card-fraud
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb

@@ -42,9 +42,8 @@
         "\n",
         "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
         "\n",
-        "An Enterprise workspace is required for this notebook. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade).\n",
-        "\n",
         "Notebook synopsis:\n",
+        "\n",
         "1. Creating an Experiment in an existing Workspace\n",
         "2. Configuration and remote run of AutoML for a text dataset (20 Newsgroups dataset from scikit-learn) for classification\n",
         "3. Registering the best model for future use\n",
@@ -97,7 +96,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -139,6 +138,8 @@
         "## Set up a compute cluster\n",
         "This section uses a user-provided compute cluster (named \"dnntext-cluster\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments.\n",
         "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "Whether you provide/select a CPU or GPU cluster, AutoML will choose the appropriate DNN for that setup - BiLSTM or BERT text featurizer will be included in the candidate featurizers on CPU and GPU respectively.  If your goal is to obtain the most accurate model, we recommend you use GPU clusters since BERT featurizers usually outperform BiLSTM featurizers."
       ]
     },
@@ -151,6 +152,8 @@
         "from azureml.core.compute import ComputeTarget, AmlCompute\n",
         "from azureml.core.compute_target import ComputeTargetException\n",
         "\n",
+        "num_nodes = 2\n",
+        "\n",
         "# Choose a name for your cluster.\n",
         "amlcompute_cluster_name = \"dnntext-cluster\"\n",
         "\n",
@@ -159,11 +162,11 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_NC6\", # CPU for BiLSTM, such as \"STANDARD_D2_V2\" \n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_NC6\", # CPU for BiLSTM, such as \"STANDARD_DS12_V2\" \n",
         "                                                           # To use BERT (this is recommended for best performance), select a GPU such as \"STANDARD_NC6\" \n",
         "                                                           # or similar GPU option\n",
         "                                                           # available in your workspace\n",
-        "                                                           max_nodes = 1)\n",
+        "                                                           max_nodes = num_nodes)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
         "compute_target.wait_for_completion(show_output=True)"
@@ -270,7 +273,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade)."
+        "This notebook uses the blocked_models parameter to exclude some models that can take a longer time to train on some text datasets. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
       ]
     },
     {
@@ -280,9 +283,9 @@
       "outputs": [],
       "source": [
         "automl_settings = {\n",
-        "    \"experiment_timeout_minutes\": 20,\n",
+        "    \"experiment_timeout_minutes\": 30,\n",
         "    \"primary_metric\": 'accuracy',\n",
-        "    \"max_concurrent_iterations\": 4, \n",
+        "    \"max_concurrent_iterations\": num_nodes, \n",
         "    \"max_cores_per_iteration\": -1,\n",
         "    \"enable_dnn\": True,\n",
         "    \"enable_early_stopping\": True,\n",
@@ -297,6 +300,7 @@
         "                             compute_target=compute_target,\n",
         "                             training_data=train_dataset,\n",
         "                             label_column_name=target_column_name,\n",
+        "                             blocked_models = ['LightGBM', 'XGBoostClassifier'],\n",
         "                             **automl_settings\n",
         "                            )"
       ]
@@ -317,15 +321,6 @@
         "automl_run = experiment.submit(automl_config, show_output=True)"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "automl_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -492,7 +487,7 @@
       "outputs": [],
       "source": [
         "test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run,\n",
-        "                         train_dataset, test_dataset, target_column_name, model_name)"
+        "                         test_dataset, target_column_name, model_name)"
       ]
     },
     {

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.yml

@@ -2,11 +2,3 @@ name: auto-ml-classification-text-dnn
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib
-  - https://download.pytorch.org/whl/cpu/torch-1.1.0-cp35-cp35m-win_amd64.whl
-  - sentencepiece==0.1.82
-  - pytorch-transformers==1.0
-  - spacy==2.1.8
-  - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz

how-to-use-azureml/automated-machine-learning/classification-text-dnn/helper.py

@@ -1,20 +1,13 @@
 import pandas as pd
 from azureml.core import Environment
-from azureml.core.conda_dependencies import CondaDependencies
 from azureml.train.estimator import Estimator
 from azureml.core.run import Run
 
 
 def run_inference(test_experiment, compute_target, script_folder, train_run,
-                  train_dataset, test_dataset, target_column_name, model_name):
+                  test_dataset, target_column_name, model_name):
 
-    train_run.download_file('outputs/conda_env_v_1_0_0.yml',
-                            'inference/condafile.yml')
-
-    inference_env = Environment("myenv")
-    inference_env.docker.enabled = True
-    inference_env.python.conda_dependencies = CondaDependencies(
-        conda_dependencies_file_path='inference/condafile.yml')
+    inference_env = train_run.get_environment()
 
     est = Estimator(source_directory=script_folder,
                     entry_script='infer.py',
@@ -23,7 +16,6 @@ def run_inference(test_experiment, compute_target, script_folder, train_run,
                         '--model_name': model_name
                     },
                     inputs=[
-                        train_dataset.as_named_input('train_data'),
                         test_dataset.as_named_input('test_data')
                     ],
                     compute_target=compute_target,

how-to-use-azureml/automated-machine-learning/classification-text-dnn/infer.py

@@ -1,5 +1,6 @@
 import argparse
 
+import pandas as pd
 import numpy as np
 
 from sklearn.externals import joblib
@@ -32,22 +33,21 @@ model = joblib.load(model_path)
 run = Run.get_context()
 # get input dataset by name
 test_dataset = run.input_datasets['test_data']
-train_dataset = run.input_datasets['train_data']
 
 X_test_df = test_dataset.drop_columns(columns=[target_column_name]) \
                         .to_pandas_dataframe()
 y_test_df = test_dataset.with_timestamp_columns(None) \
                         .keep_columns(columns=[target_column_name]) \
                         .to_pandas_dataframe()
-y_train_df = test_dataset.with_timestamp_columns(None) \
-                         .keep_columns(columns=[target_column_name]) \
-                         .to_pandas_dataframe()
 
 predicted = model.predict_proba(X_test_df)
 
+if isinstance(predicted, pd.DataFrame):
+    predicted = predicted.values
+
 # Use the AutoML scoring module
-class_labels = np.unique(np.concatenate((y_train_df.values, y_test_df.values)))
 train_labels = model.classes_
+class_labels = np.unique(np.concatenate((y_test_df.values, np.reshape(train_labels, (-1, 1)))))
 classification_metrics = list(constants.CLASSIFICATION_SCALAR_SET)
 scores = scoring.score_classification(y_test_df.values, predicted,
                                       classification_metrics,

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.ipynb

@@ -32,13 +32,6 @@
         "8. [Test Retraining](#Test-Retraining)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "An Enterprise workspace is required for this notebook. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade)"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -88,7 +81,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -148,9 +141,12 @@
         "#### Create or Attach existing AmlCompute\n",
         "\n",
         "You will need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "#### Creation of AmlCompute takes approximately 5 minutes. \n",
         "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota."
+        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
     },
     {
@@ -170,7 +166,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=4)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
@@ -190,7 +186,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.core.runconfig import CondaDependencies, DEFAULT_CPU_IMAGE, RunConfiguration\n",
+        "from azureml.core.runconfig import CondaDependencies, RunConfiguration\n",
         "\n",
         "# create a new RunConfig object\n",
         "conda_run_config = RunConfiguration(framework=\"python\")\n",
@@ -199,12 +195,10 @@
         "conda_run_config.target = compute_target\n",
         "\n",
         "conda_run_config.environment.docker.enabled = True\n",
-        "conda_run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE\n",
         "\n",
-        "cd = CondaDependencies.create(pip_packages=['azureml-sdk[automl]', 'applicationinsights', 'azureml-opendatasets'], \n",
+        "cd = CondaDependencies.create(pip_packages=['azureml-sdk[automl]', 'applicationinsights', 'azureml-opendatasets', 'azureml-defaults'], \n",
         "                              conda_packages=['numpy==1.16.2'], \n",
         "                              pin_sdk_version=False)\n",
-        "#cd.add_pip_package('azureml-explain-model')\n",
         "conda_run_config.environment.python.conda_dependencies = cd\n",
         "\n",
         "print('run config is ready')"

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.yml

@@ -2,7 +2,3 @@ name: auto-ml-continuous-retraining
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib
-  - azureml-pipeline

how-to-use-azureml/automated-machine-learning/continuous-retraining/upload_weather_data.py

@@ -49,22 +49,24 @@ print("Argument 1(ds_name): %s" % args.ds_name)
 
 dstor = ws.get_default_datastore()
 register_dataset = False
+end_time = datetime.utcnow()
+
 try:
     ds = Dataset.get_by_name(ws, args.ds_name)
     end_time_last_slice = ds.data_changed_time.replace(tzinfo=None)
     print("Dataset {0} last updated on {1}".format(args.ds_name,
                                                    end_time_last_slice))
-except Exception as e:
+except Exception:
     print(traceback.format_exc())
     print("Dataset with name {0} not found, registering new dataset.".format(args.ds_name))
     register_dataset = True
-    end_time_last_slice = datetime.today() - relativedelta(weeks=2)
+    end_time = datetime(2021, 5, 1, 0, 0)
+    end_time_last_slice = end_time - relativedelta(weeks=2)
 
-end_time = datetime.utcnow()
 train_df = get_noaa_data(end_time_last_slice, end_time)
 
 if train_df.size > 0:
-    print("Received {0} rows of new data after {0}.".format(
+    print("Received {0} rows of new data after {1}.".format(
         train_df.shape[0], end_time_last_slice))
     folder_name = "{}/{:04d}/{:02d}/{:02d}/{:02d}/{:02d}/{:02d}".format(args.ds_name, end_time.year,
                                                                         end_time.month, end_time.day,

how-to-use-azureml/automated-machine-learning/experimental/README.md

@@ -0,0 +1,92 @@
+# Experimental Notebooks for Automated ML
+Notebooks listed in this folder are leveraging experimental features. Namespaces or function signitures may change in future SDK releases. The notebooks published here will reflect the latest supported APIs. All of these notebooks can run on a client-only installation of the Automated ML SDK.
+The client only installation doesn't contain any of the machine learning libraries, such as scikit-learn, xgboost, or tensorflow, making it much faster to install and is less likely to conflict with any packages in an existing environment. However, since the ML libraries are not available locally, models cannot be downloaded and loaded directly in the client. To replace the functionality of having models locally, these notebooks also demonstrate the ModelProxy feature which will allow you to submit a predict/forecast to the training environment. 
+
+<a name="localconda"></a>
+## Setup using a Local Conda environment
+
+To run these notebook on your own notebook server, use these installation instructions.
+The instructions below will install everything you need and then start a Jupyter notebook.
+If you would like to use a lighter-weight version of the client that does not install all of the machine learning libraries locally, you can leverage the [experimental notebooks.](experimental/README.md)
+
+### 1. Install mini-conda from [here](https://conda.io/miniconda.html), choose 64-bit Python 3.7 or higher.
+- **Note**: if you already have conda installed, you can keep using it but it should be version 4.4.10 or later (as shown by: conda -V).  If you have a previous version installed, you can update it using the command: conda update conda.
+There's no need to install mini-conda specifically.
+
+### 2. Downloading the sample notebooks
+- Download the sample notebooks from [GitHub](https://github.com/Azure/MachineLearningNotebooks) as zip and extract the contents to a local directory.  The automated ML sample notebooks are in the "automated-machine-learning" folder.
+
+### 3. Setup a new conda environment
+The **automl_setup_thin_client** script creates a new conda environment, installs the necessary packages, configures the widget and starts a jupyter notebook. It takes the conda environment name as an optional parameter.  The default conda environment name is azure_automl_experimental.  The exact command depends on the operating system.  See the specific sections below for Windows, Mac and Linux.  It can take about 10 minutes to execute.
+
+Packages installed by the **automl_setup** script:
+    <ul><li>python</li><li>nb_conda</li><li>matplotlib</li><li>numpy</li><li>cython</li><li>urllib3</li><li>pandas</li><li>azureml-sdk</li><li>azureml-widgets</li><li>pandas-ml</li></ul>
+
+For more details refer to the [automl_env_thin_client.yml](./automl_env_thin_client.yml)
+## Windows
+Start an **Anaconda Prompt** window, cd to the **how-to-use-azureml/automated-machine-learning/experimental** folder where the sample notebooks were extracted and then run:
+```
+automl_setup_thin_client
+```
+## Mac
+Install "Command line developer tools" if it is not already installed (you can use the command: `xcode-select --install`).
+
+Start a Terminal windows, cd to the **how-to-use-azureml/automated-machine-learning/experimental** folder where the sample notebooks were extracted and then run:
+
+```
+bash automl_setup_thin_client_mac.sh
+```
+
+## Linux
+cd to the **how-to-use-azureml/automated-machine-learning/experimental** folder where the sample notebooks were extracted and then run:
+
+```
+bash automl_setup_thin_client_linux.sh
+```
+
+### 4. Running configuration.ipynb
+- Before running any samples you next need to run the configuration notebook. Click on [configuration](../../configuration.ipynb) notebook
+- Execute the cells in the notebook to Register Machine Learning Services Resource Provider and create a workspace. (*instructions in notebook*)
+
+### 5. Running Samples
+- Please make sure you use the Python [conda env:azure_automl_experimental] kernel when trying the sample Notebooks.
+- Follow the instructions in the individual notebooks to explore various features in automated ML.
+
+### 6. Starting jupyter notebook manually
+To start your Jupyter notebook manually, use:
+
+```
+conda activate azure_automl
+jupyter notebook
+```
+
+or on Mac or Linux:
+
+```
+source activate azure_automl
+jupyter notebook
+```
+
+
+<a name="samples"></a>
+# Automated ML SDK Sample Notebooks
+
+- [auto-ml-regression-model-proxy.ipynb](regression-model-proxy/auto-ml-regression-model-proxy.ipynb)
+    - Dataset: Hardware Performance Dataset
+    - Simple example of using automated ML for regression
+    - Uses azure compute for training
+    - Uses ModelProxy for submitting prediction to training environment on azure compute
+
+<a name="documentation"></a>
+See [Configure automated machine learning experiments](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-auto-train) to learn how more about the the settings and features available for automated machine learning experiments.
+
+<a name="pythoncommand"></a>
+# Running using python command
+Jupyter notebook provides a File / Download as / Python (.py) option for saving the notebook as a Python file.
+You can then run this file using the python command.
+However, on Windows the file needs to be modified before it can be run.
+The following condition must be added to the main code in the file:
+
+    if __name__ == "__main__":
+
+The main code of the file must be indented so that it is under this condition.

how-to-use-azureml/automated-machine-learning/experimental/automl_setup_thin_client.cmd

@@ -0,0 +1,63 @@
+@echo off
+set conda_env_name=%1
+set automl_env_file=%2
+set options=%3
+set PIP_NO_WARN_SCRIPT_LOCATION=0
+
+IF "%conda_env_name%"=="" SET conda_env_name="azure_automl_experimental"
+IF "%automl_env_file%"=="" SET automl_env_file="automl_thin_client_env.yml"
+
+IF NOT EXIST %automl_env_file% GOTO YmlMissing
+
+IF "%CONDA_EXE%"=="" GOTO CondaMissing
+
+call conda activate %conda_env_name% 2>nul:
+
+if not errorlevel 1 (
+  echo Upgrading existing conda environment %conda_env_name%
+  call pip uninstall azureml-train-automl -y -q
+  call conda env update --name %conda_env_name% --file %automl_env_file%
+  if errorlevel 1 goto ErrorExit
+) else (
+  call conda env create -f %automl_env_file% -n %conda_env_name%
+)
+
+call conda activate %conda_env_name% 2>nul:
+if errorlevel 1 goto ErrorExit
+
+call python -m ipykernel install --user --name %conda_env_name% --display-name "Python (%conda_env_name%)"
+
+REM azureml.widgets is now installed as part of the pip install under the conda env.
+REM Removing the old user install so that the notebooks will use the latest widget.
+call jupyter nbextension uninstall --user --py azureml.widgets
+
+echo.
+echo.
+echo ***************************************
+echo * AutoML setup completed successfully *
+echo ***************************************
+IF NOT "%options%"=="nolaunch" (
+  echo.
+  echo Starting jupyter notebook - please run the configuration notebook 
+  echo.
+  jupyter notebook --log-level=50 --notebook-dir='..\..'
+)
+
+goto End
+
+:CondaMissing
+echo Please run this script from an Anaconda Prompt window.
+echo You can start an Anaconda Prompt window by
+echo typing Anaconda Prompt on the Start menu.
+echo If you don't see the Anaconda Prompt app, install Miniconda.
+echo If you are running an older version of Miniconda or Anaconda,
+echo you can upgrade using the command: conda update conda
+goto End
+
+:YmlMissing
+echo File %automl_env_file% not found.
+
+:ErrorExit
+echo Install failed
+
+:End

how-to-use-azureml/automated-machine-learning/experimental/automl_setup_thin_client_linux.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+CONDA_ENV_NAME=$1
+AUTOML_ENV_FILE=$2
+OPTIONS=$3
+PIP_NO_WARN_SCRIPT_LOCATION=0
+
+if [ "$CONDA_ENV_NAME" == "" ]
+then
+  CONDA_ENV_NAME="azure_automl_experimental"
+fi
+
+if [ "$AUTOML_ENV_FILE" == "" ]
+then
+  AUTOML_ENV_FILE="automl_thin_client_env.yml"
+fi
+
+if [ ! -f $AUTOML_ENV_FILE ]; then
+    echo "File $AUTOML_ENV_FILE not found"
+    exit 1
+fi
+
+if source activate $CONDA_ENV_NAME 2> /dev/null
+then
+   echo "Upgrading existing conda environment" $CONDA_ENV_NAME
+   pip uninstall azureml-train-automl -y -q
+   conda env update --name $CONDA_ENV_NAME --file $AUTOML_ENV_FILE &&
+   jupyter nbextension uninstall --user --py azureml.widgets
+else
+   conda env create -f $AUTOML_ENV_FILE -n $CONDA_ENV_NAME &&
+   source activate $CONDA_ENV_NAME &&
+   python -m ipykernel install --user --name $CONDA_ENV_NAME --display-name "Python ($CONDA_ENV_NAME)" &&
+   jupyter nbextension uninstall --user --py azureml.widgets &&
+   echo "" &&
+   echo "" &&
+   echo "***************************************" &&
+   echo "* AutoML setup completed successfully *" &&
+   echo "***************************************" &&
+   if [ "$OPTIONS" != "nolaunch" ]
+   then
+      echo "" &&
+      echo "Starting jupyter notebook - please run the configuration notebook" &&
+      echo "" &&
+      jupyter notebook --log-level=50 --notebook-dir '../..'
+   fi
+fi
+
+if [ $? -gt 0 ]
+then
+   echo "Installation failed"
+fi
+
+

how-to-use-azureml/automated-machine-learning/experimental/automl_setup_thin_client_mac.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+CONDA_ENV_NAME=$1
+AUTOML_ENV_FILE=$2
+OPTIONS=$3
+PIP_NO_WARN_SCRIPT_LOCATION=0
+
+if [ "$CONDA_ENV_NAME" == "" ]
+then
+  CONDA_ENV_NAME="azure_automl_experimental"
+fi
+
+if [ "$AUTOML_ENV_FILE" == "" ]
+then
+  AUTOML_ENV_FILE="automl_thin_client_env_mac.yml"
+fi
+
+if [ ! -f $AUTOML_ENV_FILE ]; then
+    echo "File $AUTOML_ENV_FILE not found"
+    exit 1
+fi
+
+if source activate $CONDA_ENV_NAME 2> /dev/null
+then
+   echo "Upgrading existing conda environment" $CONDA_ENV_NAME
+   pip uninstall azureml-train-automl -y -q
+   conda env update --name $CONDA_ENV_NAME --file $AUTOML_ENV_FILE &&
+   jupyter nbextension uninstall --user --py azureml.widgets
+else
+   conda env create -f $AUTOML_ENV_FILE -n $CONDA_ENV_NAME &&
+   source activate $CONDA_ENV_NAME &&
+   conda install lightgbm -c conda-forge -y &&
+   python -m ipykernel install --user --name $CONDA_ENV_NAME --display-name "Python ($CONDA_ENV_NAME)" &&
+   jupyter nbextension uninstall --user --py azureml.widgets &&
+   echo "" &&
+   echo "" &&
+   echo "***************************************" &&
+   echo "* AutoML setup completed successfully *" &&
+   echo "***************************************" &&
+   if [ "$OPTIONS" != "nolaunch" ]
+   then
+      echo "" &&
+      echo "Starting jupyter notebook - please run the configuration notebook" &&
+      echo "" &&
+      jupyter notebook --log-level=50 --notebook-dir '../..'
+   fi
+fi
+
+if [ $? -gt 0 ]
+then
+   echo "Installation failed"
+fi
+
+
+

how-to-use-azureml/automated-machine-learning/experimental/automl_thin_client_env.yml

@@ -0,0 +1,18 @@
+name: azure_automl_experimental
+dependencies:
+  # The python interpreter version.
+  # Currently Azure ML only supports 3.5.2 and later.
+- pip<=19.3.1
+- python>=3.5.2,<3.8
+- nb_conda
+- cython
+- urllib3<1.24
+- PyJWT < 2.0.0
+- numpy==1.18.5
+
+- pip:
+  # Required packages for AzureML execution, history, and data preparation.
+  - azureml-defaults
+  - azureml-sdk
+  - azureml-widgets
+  - pandas

how-to-use-azureml/automated-machine-learning/experimental/automl_thin_client_env_mac.yml

@@ -0,0 +1,19 @@
+name: azure_automl_experimental
+dependencies:
+  # The python interpreter version.
+  # Currently Azure ML only supports 3.5.2 and later.
+- pip<=19.3.1
+- nomkl
+- python>=3.5.2,<3.8
+- nb_conda
+- cython
+- urllib3<1.24
+- PyJWT < 2.0.0
+- numpy==1.18.5
+
+- pip:
+  # Required packages for AzureML execution, history, and data preparation.
+  - azureml-defaults
+  - azureml-sdk
+  - azureml-widgets
+  - pandas

how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud-local-managed/auto-ml-classification-credit-card-fraud-local-managed.ipynb

@@ -0,0 +1,420 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Automated Machine Learning\n",
+        "_**Classification of credit card fraudulent transactions on local managed compute **_\n",
+        "\n",
+        "## Contents\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Setup](#Setup)\n",
+        "1. [Train](#Train)\n",
+        "1. [Results](#Results)\n",
+        "1. [Test](#Test)\n",
+        "1. [Acknowledgements](#Acknowledgements)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Introduction\n",
+        "\n",
+        "In this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.\n",
+        "\n",
+        "This notebook is using local managed compute to train the model.\n",
+        "\n",
+        "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
+        "\n",
+        "In this notebook you will learn how to:\n",
+        "1. Create an experiment using an existing workspace.\n",
+        "2. Configure AutoML using `AutoMLConfig`.\n",
+        "3. Train the model using local managed compute.\n",
+        "4. Explore the results.\n",
+        "5. Test the fitted model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Setup\n",
+        "\n",
+        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import logging\n",
+        "\n",
+        "import pandas as pd\n",
+        "\n",
+        "import azureml.core\n",
+        "from azureml.core.compute_target import LocalTarget\n",
+        "from azureml.core.experiment import Experiment\n",
+        "from azureml.core.workspace import Workspace\n",
+        "from azureml.core.dataset import Dataset\n",
+        "from azureml.train.automl import AutoMLConfig"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
+        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "\n",
+        "# choose a name for experiment\n",
+        "experiment_name = 'automl-local-managed'\n",
+        "\n",
+        "experiment=Experiment(ws, experiment_name)\n",
+        "\n",
+        "output = {}\n",
+        "output['Subscription ID'] = ws.subscription_id\n",
+        "output['Workspace'] = ws.name\n",
+        "output['Resource Group'] = ws.resource_group\n",
+        "output['Location'] = ws.location\n",
+        "output['Experiment Name'] = experiment.name\n",
+        "pd.set_option('display.max_colwidth', -1)\n",
+        "outputDf = pd.DataFrame(data = output, index = [''])\n",
+        "outputDf.T"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Determine if local docker is configured for Linux images\n",
+        "\n",
+        "Local managed runs will leverage a Linux docker container to submit the run to. Due to this, the docker needs to be configured to use Linux containers."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Check if Docker is installed and Linux containers are enabled\n",
+        "import subprocess\n",
+        "from subprocess import CalledProcessError\n",
+        "try:\n",
+        "    assert subprocess.run(\"docker -v\", shell=True).returncode == 0, 'Local Managed runs require docker to be installed.'\n",
+        "    out = subprocess.check_output(\"docker system info\", shell=True).decode('ascii')\n",
+        "    assert \"OSType: linux\" in out, 'Docker engine needs to be configured to use Linux containers.' \\\n",
+        "        'https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers'\n",
+        "except CalledProcessError as ex:\n",
+        "    raise Exception('Local Managed runs require docker to be installed.') from ex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Load Data\n",
+        "\n",
+        "Load the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv\"\n",
+        "dataset = Dataset.Tabular.from_delimited_files(data)\n",
+        "training_data, validation_data = dataset.random_split(percentage=0.8, seed=223)\n",
+        "label_column_name = 'Class'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train\n",
+        "\n",
+        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**task**|classification or regression|\n",
+        "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
+        "|**enable_early_stopping**|Stop the run if the metric score is not showing improvement.|\n",
+        "|**n_cross_validations**|Number of cross validation splits.|\n",
+        "|**training_data**|Input dataset, containing both features and label column.|\n",
+        "|**label_column_name**|The name of the label column.|\n",
+        "|**enable_local_managed**|Enable the experimental local-managed scenario.|\n",
+        "\n",
+        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "automl_settings = {\n",
+        "    \"n_cross_validations\": 3,\n",
+        "    \"primary_metric\": 'average_precision_score_weighted',\n",
+        "    \"enable_early_stopping\": True,\n",
+        "    \"experiment_timeout_hours\": 0.3, #for real scenarios we recommend a timeout of at least one hour \n",
+        "    \"verbosity\": logging.INFO,\n",
+        "}\n",
+        "\n",
+        "automl_config = AutoMLConfig(task = 'classification',\n",
+        "                             debug_log = 'automl_errors.log',\n",
+        "                             compute_target = LocalTarget(),\n",
+        "                             enable_local_managed = True,\n",
+        "                             training_data = training_data,\n",
+        "                             label_column_name = label_column_name,\n",
+        "                             **automl_settings\n",
+        "                            )"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "parent_run = experiment.submit(automl_config, show_output = True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# If you need to retrieve a run that already started, use the following code\n",
+        "#from azureml.train.automl.run import AutoMLRun\n",
+        "#parent_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "parent_run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Results"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Explain model\n",
+        "\n",
+        "Automated ML models can be explained and visualized using the SDK Explainability library. "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Analyze results\n",
+        "\n",
+        "### Retrieve the Best Child Run\n",
+        "\n",
+        "Below we select the best pipeline from our iterations. The `get_best_child` method returns the best run. Overloads on `get_best_child` allow you to retrieve the best run for *any* logged metric."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "best_run = parent_run.get_best_child()\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Test the fitted model\n",
+        "\n",
+        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_test_df = validation_data.drop_columns(columns=[label_column_name])\n",
+        "y_test_df = validation_data.keep_columns(columns=[label_column_name], validate=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Creating ModelProxy for submitting prediction runs to the training environment.\n",
+        "We will create a ModelProxy for the best child run, which will allow us to submit a run that does the prediction in the training environment. Unlike the local client, which can have different versions of some libraries, the training environment will have all the compatible libraries for the model already."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.train.automl.model_proxy import ModelProxy\n",
+        "best_model_proxy = ModelProxy(best_run)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# call the predict functions on the model proxy\n",
+        "y_pred = best_model_proxy.predict(X_test_df).to_pandas_dataframe()\n",
+        "y_pred"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Acknowledgements"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
+        "\n",
+        "\n",
+        "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Universit\u00c3\u0192\u00c2\u00a9 Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
+        "Please cite the following works: \n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tAndrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
+        "o\tDal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "sekrupa"
+      }
+    ],
+    "category": "tutorial",
+    "compute": [
+      "AML Compute"
+    ],
+    "datasets": [
+      "Creditcard"
+    ],
+    "deployment": [
+      "None"
+    ],
+    "exclude_from_index": false,
+    "file_extension": ".py",
+    "framework": [
+      "None"
+    ],
+    "friendly_name": "Classification of credit card fraudulent transactions using Automated ML",
+    "index_order": 5,
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.7"
+    },
+    "mimetype": "text/x-python",
+    "name": "python",
+    "nbconvert_exporter": "python",
+    "pygments_lexer": "ipython3",
+    "tags": [
+      "AutomatedML"
+    ],
+    "task": "Classification",
+    "version": "3.6.7"
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud-local-managed/auto-ml-classification-credit-card-fraud-local-managed.yml

@@ -0,0 +1,4 @@
+name: auto-ml-classification-credit-card-fraud-local-managed
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/experimental/regression-model-proxy/auto-ml-regression-model-proxy.ipynb

@@ -0,0 +1,435 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/regression-model-proxy/auto-ml-regression-model-proxy.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Automated Machine Learning\n",
+        "_**Regression with Aml Compute**_\n",
+        "\n",
+        "## Contents\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Setup](#Setup)\n",
+        "1. [Data](#Data)\n",
+        "1. [Train](#Train)\n",
+        "1. [Results](#Results)\n",
+        "1. [Test](#Test)\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Introduction\n",
+        "In this example we use an experimental feature, Model Proxy, to do a predict on the best generated model without downloading the model locally. The prediction will happen on same compute and environment that was used to train the model. This feature is currently in the experimental state, which means that the API is prone to changing, please make sure to run on the latest version of this notebook if you face any issues.\n",
+        "This notebook will also leverage MLFlow for saving models, allowing for more portability of the resulting models. See https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow for more details around MLFlow is AzureML.\n",
+        "\n",
+        "If you are using an Azure Machine Learning Compute Instance, you are all set.  Otherwise, go through the [configuration](../../../../configuration.ipynb)  notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
+        "\n",
+        "In this notebook you will learn how to:\n",
+        "1. Create an `Experiment` in an existing `Workspace`.\n",
+        "2. Configure AutoML using `AutoMLConfig`.\n",
+        "3. Train the model using remote compute.\n",
+        "4. Explore the results.\n",
+        "5. Test the best fitted model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Setup\n",
+        "\n",
+        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import logging\n",
+        "\n",
+        "import json\n",
+        "\n",
+        "\n",
+        "import azureml.core\n",
+        "from azureml.core.experiment import Experiment\n",
+        "from azureml.core.workspace import Workspace\n",
+        "from azureml.core.dataset import Dataset\n",
+        "from azureml.train.automl import AutoMLConfig"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
+        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "\n",
+        "# Choose a name for the experiment.\n",
+        "experiment_name = 'automl-regression-model-proxy'\n",
+        "\n",
+        "experiment = Experiment(ws, experiment_name)\n",
+        "\n",
+        "output = {}\n",
+        "output['Subscription ID'] = ws.subscription_id\n",
+        "output['Workspace'] = ws.name\n",
+        "output['Resource Group'] = ws.resource_group\n",
+        "output['Location'] = ws.location\n",
+        "output['Run History Name'] = experiment_name\n",
+        "output"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Using AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you use `AmlCompute` as your training compute resource."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your CPU cluster\n",
+        "# Try to ensure that the cluster name is unique across the notebooks\n",
+        "cpu_cluster_name = \"reg-model-proxy\"\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
+        "    print('Found existing cluster, use it.')\n",
+        "except ComputeTargetException:\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
+        "                                                           max_nodes=4)\n",
+        "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
+        "\n",
+        "compute_target.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Data\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Load Data\n",
+        "Load the hardware dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/machineData.csv\"\n",
+        "dataset = Dataset.Tabular.from_delimited_files(data)\n",
+        "\n",
+        "# Split the dataset into train and test datasets\n",
+        "train_data, test_data = dataset.random_split(percentage=0.8, seed=223)\n",
+        "\n",
+        "label = \"ERP\"\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train\n",
+        "\n",
+        "Instantiate an `AutoMLConfig` object to specify the settings and data used to run the experiment.\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**task**|classification, regression or forecasting|\n",
+        "|**primary_metric**|This is the metric that you want to optimize. Regression supports the following primary metrics: <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
+        "|**n_cross_validations**|Number of cross validation splits.|\n",
+        "|**training_data**|(sparse) array-like, shape = [n_samples, n_features]|\n",
+        "|**label_column_name**|(sparse) array-like, shape = [n_samples, ], targets values.|\n",
+        "\n",
+        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "tags": [
+          "automlconfig-remarks-sample"
+        ]
+      },
+      "outputs": [],
+      "source": [
+        "automl_settings = {\n",
+        "    \"n_cross_validations\": 3,\n",
+        "    \"primary_metric\": 'r2_score',\n",
+        "    \"enable_early_stopping\": True, \n",
+        "    \"experiment_timeout_hours\": 0.3, #for real scenarios we recommend a timeout of at least one hour \n",
+        "    \"max_concurrent_iterations\": 4,\n",
+        "    \"max_cores_per_iteration\": -1,\n",
+        "    \"verbosity\": logging.INFO,\n",
+        "    \"save_mlflow\": True,\n",
+        "}\n",
+        "\n",
+        "automl_config = AutoMLConfig(task = 'regression',\n",
+        "                             compute_target = compute_target,\n",
+        "                             training_data = train_data,\n",
+        "                             label_column_name = label,\n",
+        "                             **automl_settings\n",
+        "                            )"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Call the `submit` method on the experiment object and pass the run configuration. Execution of remote runs is asynchronous. Depending on the data and the number of iterations this can run for a while.  Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "remote_run = experiment.submit(automl_config, show_output = False)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# If you need to retrieve a run that already started, use the following code\n",
+        "#from azureml.train.automl.run import AutoMLRun\n",
+        "#remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "remote_run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "remote_run.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Retrieve the Best Child Run\n",
+        "\n",
+        "Below we select the best pipeline from our iterations. The `get_best_child` method returns the best run. Overloads on `get_best_child` allow you to retrieve the best run for *any* logged metric."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "best_run = remote_run.get_best_child()\n",
+        "print(best_run)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Show hyperparameters\n",
+        "Show the model pipeline used for the best run with its hyperparameters."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "run_properties = json.loads(best_run.get_details()['properties']['pipeline_script'])\n",
+        "print(json.dumps(run_properties, indent = 1)) "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Best Child Run Based on Any Other Metric\n",
+        "Show the run and the model that has the smallest `root_mean_squared_error` value (which turned out to be the same as the one with largest `spearman_correlation` value):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "lookup_metric = \"root_mean_squared_error\"\n",
+        "best_run = remote_run.get_best_child(metric = lookup_metric)\n",
+        "print(best_run)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "y_test = test_data.keep_columns('ERP')\n",
+        "test_data = test_data.drop_columns('ERP')\n",
+        "\n",
+        "\n",
+        "y_train = train_data.keep_columns('ERP')\n",
+        "train_data = train_data.drop_columns('ERP')\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Creating ModelProxy for submitting prediction runs to the training environment.\n",
+        "We will create a ModelProxy for the best child run, which will allow us to submit a run that does the prediction in the training environment. Unlike the local client, which can have different versions of some libraries, the training environment will have all the compatible libraries for the model already."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.train.automl.model_proxy import ModelProxy\n",
+        "best_model_proxy = ModelProxy(best_run)\n",
+        "y_pred_train = best_model_proxy.predict(train_data)\n",
+        "y_pred_test = best_model_proxy.predict(test_data)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Exploring results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "y_pred_train = y_pred_train.to_pandas_dataframe().values.flatten()\n",
+        "y_train = y_train.to_pandas_dataframe().values.flatten()\n",
+        "y_residual_train = y_train - y_pred_train\n",
+        "\n",
+        "y_pred_test = y_pred_test.to_pandas_dataframe().values.flatten()\n",
+        "y_test = y_test.to_pandas_dataframe().values.flatten()\n",
+        "y_residual_test = y_test - y_pred_test\n",
+        "print(y_residual_train)\n",
+        "print(y_residual_test)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "sekrupa"
+      }
+    ],
+    "categories": [
+      "how-to-use-azureml",
+      "automated-machine-learning"
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.2"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/experimental/regression-model-proxy/auto-ml-regression-model-proxy.yml

@@ -0,0 +1,4 @@
+name: auto-ml-regression-model-proxy
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb

@@ -54,9 +54,8 @@
         "\n",
         "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
         "\n",
-        "An Enterprise workspace is required for this notebook. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade)\n",
-        "\n",
         "Notebook synopsis:\n",
+        "\n",
         "1. Creating an Experiment in an existing Workspace\n",
         "2. Configuration and remote run of AutoML for a time-series model exploring Regression learners, Arima, Prophet and DNNs\n",
         "4. Evaluating the fitted model using a rolling test "
@@ -114,7 +113,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -163,7 +162,9 @@
       },
       "source": [
         "### Using AmlCompute\n",
-        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you use `AmlCompute` as your training compute resource."
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you use `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
       ]
     },
     {
@@ -186,7 +187,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=4)\n",
         "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
         "\n",
@@ -217,7 +218,9 @@
         "\n",
         "**Time column** is the time axis along which to predict.\n",
         "\n",
-        "**Grain** is another word for an individual time series in your dataset. Grains are identified by values of the columns listed `grain_column_names`, for example \"store\" and \"item\" if your data has multiple time series of sales, one series for each combination of store and item sold.\n",
+        "**Time series identifier columns** are identified by values of the columns listed `time_series_id_column_names`, for example \"store\" and \"item\" if your data has multiple time series of sales, one series for each combination of store and item sold.\n",
+        "\n",
+        "**Forecast frequency (freq)** This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
         "\n",
         "This dataset has only one time series. Please see the [orange juice notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales) for an example of a multi-time series dataset."
       ]
@@ -269,7 +272,7 @@
       "source": [
         "target_column_name = 'BeerProduction'\n",
         "time_column_name = 'DATE'\n",
-        "grain_column_names = []\n",
+        "time_series_id_column_names = []\n",
         "freq = 'M' #Monthly data"
       ]
     },
@@ -329,7 +332,7 @@
       },
       "outputs": [],
       "source": [
-        "max_horizon = 12"
+        "forecast_horizon = 12"
       ]
     },
     {
@@ -350,11 +353,7 @@
         "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
         "|**training_data**|Input dataset, containing both features and label column.|\n",
         "|**label_column_name**|The name of the label column.|\n",
-        "|**enable_dnn**|Enable Forecasting DNNs|\n",
-        "\n",
-        "This notebook uses the blacklist_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blacklist_models list but you may need to increase the iteration_timeout_minutes parameter value to get results.\n",
-        "\n",
-        "This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade)."
+        "|**enable_dnn**|Enable Forecasting DNNs|\n"
       ]
     },
     {
@@ -366,11 +365,12 @@
       },
       "outputs": [],
       "source": [
-        "automl_settings = {\n",
-        "    'time_column_name': time_column_name,\n",
-        "    'max_horizon': max_horizon,\n",
-        "    'enable_dnn' : True,\n",
-        "}\n",
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name,\n",
+        "    forecast_horizon=forecast_horizon,\n",
+        "    freq='MS' # Set the forecast frequency to be monthly (start of the month)\n",
+        ")\n",
         "\n",
         "automl_config = AutoMLConfig(task='forecasting',                             \n",
         "                             primary_metric='normalized_root_mean_squared_error',\n",
@@ -382,7 +382,8 @@
         "                             compute_target=compute_target,\n",
         "                             max_concurrent_iterations=4,\n",
         "                             max_cores_per_iteration=-1,\n",
-        "                            **automl_settings)"
+        "                             enable_dnn=True,\n",
+        "                             forecasting_parameters=forecasting_parameters)"
       ]
     },
     {
@@ -392,7 +393,7 @@
         "hidePrompt": false
       },
       "source": [
-        "We will now run the experiment, starting with 10 iterations of model search. The experiment can be continued for more iterations if more accurate results are required."
+        "We will now run the experiment, starting with 10 iterations of model search. The experiment can be continued for more iterations if more accurate results are required. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -404,8 +405,7 @@
       },
       "outputs": [],
       "source": [
-        "remote_run = experiment.submit(automl_config, show_output= False)\n",
-        "remote_run"
+        "remote_run = experiment.submit(automl_config, show_output= True)"
       ]
     },
     {
@@ -422,15 +422,6 @@
         "# remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run.wait_for_completion()"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -583,7 +574,7 @@
       "source": [
         "from helper import run_inference\n",
         "\n",
-        "test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run, test_dataset, valid_dataset, max_horizon,\n",
+        "test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run, test_dataset, valid_dataset, forecast_horizon,\n",
         "                 target_column_name, time_column_name, freq)"
       ]
     },
@@ -605,7 +596,7 @@
         "from helper import run_multiple_inferences\n",
         "\n",
         "summary_df = run_multiple_inferences(summary_df, experiment, test_experiment, compute_target, script_folder, test_dataset, \n",
-        "                  valid_dataset, max_horizon, target_column_name, time_column_name, freq)"
+        "                  valid_dataset, forecast_horizon, target_column_name, time_column_name, freq)"
       ]
     },
     {
@@ -652,7 +643,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "omkarm"
+        "name": "jialiu"
       }
     ],
     "hide_code_all_hidden": false,
@@ -671,7 +662,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.7"
+      "version": "3.6.9"
     }
   },
   "nbformat": 4,

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.yml

@@ -1,11 +1,4 @@
 name: auto-ml-forecasting-beer-remote
 dependencies:
-- py-xgboost<=0.90
 - pip:
   - azureml-sdk
-  - numpy==1.16.2
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib
-  - azureml-train

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/helper.py

@@ -3,11 +3,11 @@ from azureml.core import Environment
 from azureml.core.conda_dependencies import CondaDependencies
 from azureml.train.estimator import Estimator
 from azureml.core.run import Run
+from azureml.automl.core.shared import constants
 
 
 def split_fraction_by_grain(df, fraction, time_column_name,
                             grain_column_names=None):
-
     if not grain_column_names:
         df['tmp_grain_column'] = 'grain'
         grain_column_names = ['tmp_grain_column']
@@ -59,11 +59,13 @@ def get_result_df(remote_run):
                                      'primary_metric', 'Score'])
     goal_minimize = False
     for run in children:
-        if('run_algorithm' in run.properties and 'score' in run.properties):
+        if run.get_status().lower() == constants.RunState.COMPLETE_RUN \
+                and 'run_algorithm' in run.properties and 'score' in run.properties:
+            # We only count in the completed child runs.
             summary_df[run.id] = [run.id, run.properties['run_algorithm'],
                                   run.properties['primary_metric'],
                                   float(run.properties['score'])]
-            if('goal' in run.properties):
+            if ('goal' in run.properties):
                 goal_minimize = run.properties['goal'].split('_')[-1] == 'min'
 
     summary_df = summary_df.T.sort_values(
@@ -118,7 +120,6 @@ def run_multiple_inferences(summary_df, train_experiment, test_experiment,
                             compute_target, script_folder, test_dataset,
                             lookback_dataset, max_horizon, target_column_name,
                             time_column_name, freq):
-
     for run_name, run_summary in summary_df.iterrows():
         print(run_name)
         print(run_summary)

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/infer.py

@@ -1,4 +1,5 @@
 import argparse
+import os
 
 import numpy as np
 import pandas as pd
@@ -10,6 +11,13 @@ from sklearn.metrics import mean_absolute_error, mean_squared_error
 from azureml.automl.runtime.shared.score import scoring, constants
 from azureml.core import Run
 
+try:
+    import torch
+
+    _torch_present = True
+except ImportError:
+    _torch_present = False
+
 
 def align_outputs(y_predicted, X_trans, X_test, y_test,
                   predicted_column_name='predicted',
@@ -48,7 +56,7 @@ def align_outputs(y_predicted, X_trans, X_test, y_test,
     # or at edges of time due to lags/rolling windows
     clean = together[together[[target_column_name,
                                predicted_column_name]].notnull().all(axis=1)]
-    return(clean)
+    return (clean)
 
 
 def do_rolling_forecast_with_lookback(fitted_model, X_test, y_test,
@@ -83,8 +91,7 @@ def do_rolling_forecast_with_lookback(fitted_model, X_test, y_test,
         if origin_time != X[time_column_name].min():
             # Set the context by including actuals up-to the origin time
             test_context_expand_wind = (X[time_column_name] < origin_time)
-            context_expand_wind = (
-                X_test_expand[time_column_name] < origin_time)
+            context_expand_wind = (X_test_expand[time_column_name] < origin_time)
             y_query_expand[context_expand_wind] = y[test_context_expand_wind]
 
         # Print some debug info
@@ -115,8 +122,7 @@ def do_rolling_forecast_with_lookback(fitted_model, X_test, y_test,
         # Align forecast with test set for dates within
         # the current rolling window
         trans_tindex = X_trans.index.get_level_values(time_column_name)
-        trans_roll_wind = (trans_tindex >= origin_time) & (
-            trans_tindex < horizon_time)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
         test_roll_wind = expand_wind & (X[time_column_name] >= origin_time)
         df_list.append(align_outputs(
             y_fcst[trans_roll_wind], X_trans[trans_roll_wind],
@@ -155,8 +161,7 @@ def do_rolling_forecast(fitted_model, X_test, y_test, max_horizon, freq='D'):
         if origin_time != X_test[time_column_name].min():
             # Set the context by including actuals up-to the origin time
             test_context_expand_wind = (X_test[time_column_name] < origin_time)
-            context_expand_wind = (
-                X_test_expand[time_column_name] < origin_time)
+            context_expand_wind = (X_test_expand[time_column_name] < origin_time)
             y_query_expand[context_expand_wind] = y_test[
                 test_context_expand_wind]
 
@@ -186,10 +191,8 @@ def do_rolling_forecast(fitted_model, X_test, y_test, max_horizon, freq='D'):
         # Align forecast with test set for dates within the
         # current rolling window
         trans_tindex = X_trans.index.get_level_values(time_column_name)
-        trans_roll_wind = (trans_tindex >= origin_time) & (
-            trans_tindex < horizon_time)
-        test_roll_wind = expand_wind & (
-            X_test[time_column_name] >= origin_time)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
+        test_roll_wind = expand_wind & (X_test[time_column_name] >= origin_time)
         df_list.append(align_outputs(y_fcst[trans_roll_wind],
                                      X_trans[trans_roll_wind],
                                      X_test[test_roll_wind],
@@ -221,6 +224,10 @@ def MAPE(actual, pred):
     return np.mean(APE(actual_safe, pred_safe))
 
 
+def map_location_cuda(storage, loc):
+    return storage.cuda()
+
+
 parser = argparse.ArgumentParser()
 parser.add_argument(
     '--max_horizon', type=int, dest='max_horizon',
@@ -238,7 +245,6 @@ parser.add_argument(
     '--model_path', type=str, dest='model_path',
     default='model.pkl', help='Filename of model to be loaded')
 
-
 args = parser.parse_args()
 max_horizon = args.max_horizon
 target_column_name = args.target_column_name
@@ -246,7 +252,6 @@ time_column_name = args.time_column_name
 freq = args.freq
 model_path = args.model_path
 
-
 print('args passed are: ')
 print(max_horizon)
 print(target_column_name)
@@ -274,8 +279,19 @@ X_lookback_df = lookback_dataset.drop_columns(columns=[target_column_name])
 y_lookback_df = lookback_dataset.with_timestamp_columns(
     None).keep_columns(columns=[target_column_name])
 
-fitted_model = joblib.load(model_path)
-
+_, ext = os.path.splitext(model_path)
+if ext == '.pt':
+    # Load the fc-tcn torch model.
+    assert _torch_present
+    if torch.cuda.is_available():
+        map_location = map_location_cuda
+    else:
+        map_location = 'cpu'
+    with open(model_path, 'rb') as fh:
+        fitted_model = torch.load(fh, map_location=map_location)
+else:
+    # Load the sklearn pipeline.
+    fitted_model = joblib.load(model_path)
 
 if hasattr(fitted_model, 'get_lookback'):
     lookback = fitted_model.get_lookback()

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.ipynb

@@ -87,7 +87,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -129,9 +129,12 @@
       "source": [
         "## Compute\n",
         "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "#### Creation of AmlCompute takes approximately 5 minutes. \n",
         "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota."
+        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
     },
     {
@@ -151,7 +154,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=4)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
@@ -205,6 +208,10 @@
       "outputs": [],
       "source": [
         "dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'dataset/bike-no.csv')]).with_timestamp_columns(fine_grain_timestamp=time_column_name) \n",
+        "\n",
+        "# Drop the columns 'casual' and 'registered' as these columns are a breakdown of the total and therefore a leak.\n",
+        "dataset = dataset.drop_columns(columns=['casual', 'registered'])\n",
+        "\n",
         "dataset.take(5).to_pandas_dataframe().reset_index(drop=True)"
       ]
     },
@@ -238,6 +245,22 @@
         "test.to_pandas_dataframe().head(5).reset_index(drop=True)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Forecasting Parameters\n",
+        "To define forecasting parameters for your experiment training, you can leverage the ForecastingParameters class. The table below details the forecasting parameter we will be passing into our experiment.\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**time_column_name**|The name of your time column.|\n",
+        "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
+        "|**country_or_region_for_holidays**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
+        "|**target_lags**|The target_lags specifies how far back we will construct the lags of the target variable.|\n",
+        "|**freq**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -250,20 +273,16 @@
         "|-|-|\n",
         "|**task**|forecasting|\n",
         "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>\n",
-        "|**blacklist_models**|Models in blacklist won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
+        "|**blocked_models**|Models in blocked_models won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
         "|**experiment_timeout_hours**|Experimentation timeout in hours.|\n",
         "|**training_data**|Input dataset, containing both features and label column.|\n",
         "|**label_column_name**|The name of the label column.|\n",
         "|**compute_target**|The remote compute for training.|\n",
         "|**n_cross_validations**|Number of cross validation splits.|\n",
         "|**enable_early_stopping**|If early stopping is on, training will stop when the primary metric is no longer improving.|\n",
-        "|**time_column_name**|Name of the datetime column in the input data|\n",
-        "|**max_horizon**|Maximum desired forecast horizon in units of time-series frequency|\n",
-        "|**country_or_region**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
-        "|**target_lags**|The target_lags specifies how far back we will construct the lags of the target variable.|\n",
-        "|**drop_column_names**|Name(s) of columns to drop prior to modeling|\n",
+        "|**forecasting_parameters**|A class that holds all the forecasting related parameters.|\n",
         "\n",
-        "This notebook uses the blacklist_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blacklist_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
+        "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
       ]
     },
     {
@@ -281,7 +300,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "max_horizon = 14"
+        "forecast_horizon = 14"
       ]
     },
     {
@@ -297,17 +316,18 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "time_series_settings = {\n",
-        "    'time_column_name': time_column_name,\n",
-        "    'max_horizon': max_horizon,    \n",
-        "    'country_or_region': 'US', # set country_or_region will trigger holiday featurizer\n",
-        "    'target_lags': 'auto', # use heuristic based lag setting    \n",
-        "    'drop_column_names': ['casual', 'registered'] # these columns are a breakdown of the total and therefore a leak\n",
-        "}\n",
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name,\n",
+        "    forecast_horizon=forecast_horizon,\n",
+        "    country_or_region_for_holidays='US', # set country_or_region will trigger holiday featurizer\n",
+        "    target_lags='auto', # use heuristic based lag setting\n",
+        "    freq='D' # Set the forecast frequency to be daily\n",
+        ")\n",
         "\n",
         "automl_config = AutoMLConfig(task='forecasting',                             \n",
         "                             primary_metric='normalized_root_mean_squared_error',\n",
-        "                             blacklist_models = ['ExtremeRandomTrees'],                             \n",
+        "                             blocked_models = ['ExtremeRandomTrees'],                             \n",
         "                             experiment_timeout_hours=0.3,\n",
         "                             training_data=train,\n",
         "                             label_column_name=target_column_name,\n",
@@ -317,7 +337,7 @@
         "                             max_concurrent_iterations=4,\n",
         "                             max_cores_per_iteration=-1,\n",
         "                             verbosity=logging.INFO,\n",
-        "                            **time_series_settings)"
+        "                             forecasting_parameters=forecasting_parameters)"
       ]
     },
     {
@@ -333,8 +353,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "remote_run = experiment.submit(automl_config, show_output=False)\n",
-        "remote_run"
+        "remote_run = experiment.submit(automl_config, show_output=False)"
       ]
     },
     {
@@ -422,7 +441,7 @@
       "source": [
         "We now use the best fitted model from the AutoML Run to make forecasts for the test set. We will do batch scoring on the test dataset which should have the same schema as training dataset.\n",
         "\n",
-        "The scoring will run on a remote compute. In this example, it will reuse the training compute.|"
+        "The scoring will run on a remote compute. In this example, it will reuse the training compute."
       ]
     },
     {
@@ -439,7 +458,7 @@
       "metadata": {},
       "source": [
         "### Retrieving forecasts from the model\n",
-        "To run the forecast on the remote compute we will use two helper scripts: forecasting_script and forecasting_helper. These scripts contain the utility methods which will be used by the remote estimator. We copy these scripts to the project folder to upload them to remote compute."
+        "To run the forecast on the remote compute we will use a helper script: forecasting_script. This script contains the utility methods which will be used by the remote estimator. We copy the script to the project folder to upload it to remote compute."
       ]
     },
     {
@@ -453,15 +472,14 @@
         "\n",
         "script_folder = os.path.join(os.getcwd(), 'forecast')\n",
         "os.makedirs(script_folder, exist_ok=True)\n",
-        "shutil.copy('forecasting_script.py', script_folder)\n",
-        "shutil.copy('forecasting_helper.py', script_folder)"
+        "shutil.copy('forecasting_script.py', script_folder)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "For brevity we have created the function called run_forecast. It submits the test data to the best model and run the estimation on the selected compute target."
+        "For brevity, we have created a function called run_forecast that submits the test data to the best model determined during the training run and retrieves forecasts. The test set is longer than the forecast horizon specified at train time, so the forecasting script uses a so-called rolling evaluation to generate predictions over the whole test set. A rolling evaluation iterates the forecaster over the test set, using the actuals in the test set to make lag features as needed. "
       ]
     },
     {
@@ -472,8 +490,7 @@
       "source": [
         "from run_forecast import run_rolling_forecast\n",
         "\n",
-        "remote_run = run_rolling_forecast(test_experiment, compute_target, best_run, test, max_horizon,\n",
-        "                 target_column_name, time_column_name)\n",
+        "remote_run = run_rolling_forecast(test_experiment, compute_target, best_run, test, target_column_name)\n",
         "remote_run"
       ]
     },
@@ -537,7 +554,10 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The MAPE seems high; it is being skewed by an actual with a small absolute value. For a more informative evaluation, we can calculate the metrics by forecast horizon:"
+        "For more details on what metrics are included and how they are calculated, please refer to [supported metrics](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#regressionforecasting-metrics). You could also calculate residuals, like described [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#residuals).\n",
+        "\n",
+        "\n",
+        "Since we did a rolling evaluation on the test set, we can analyze the predictions by their forecast horizon relative to the rolling origin. The model was initially trained at a forecast horizon of 14, so each prediction from the model is associated with a horizon value from 1 to 14. The horizon values are in a column named, \"horizon_origin,\" in the prediction set. For example, we can calculate some of the error metrics grouped by the horizon:"
       ]
     },
     {
@@ -557,7 +577,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "It's also interesting to see the distributions of APE (absolute percentage error) by horizon. On a log scale, the outlying APE in the horizon-3 group is clear."
+        "To drill down more, we can look at the distributions of APE (absolute percentage error) by horizon. From the chart, it is clear that the overall MAPE is being skewed by one particular point where the actual value is of small absolute value."
       ]
     },
     {
@@ -567,7 +587,7 @@
       "outputs": [],
       "source": [
         "df_all_APE = df_all.assign(APE=APE(df_all[target_column_name], df_all['predicted']))\n",
-        "APEs = [df_all_APE[df_all['horizon_origin'] == h].APE.values for h in range(1, max_horizon + 1)]\n",
+        "APEs = [df_all_APE[df_all['horizon_origin'] == h].APE.values for h in range(1, forecast_horizon + 1)]\n",
         "\n",
         "%matplotlib inline\n",
         "plt.boxplot(APEs)\n",
@@ -583,7 +603,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "erwright"
+        "name": "jialiu"
       }
     ],
     "category": "tutorial",
@@ -631,5 +651,5 @@
     "version": 3
   },
   "nbformat": 4,
-  "nbformat_minor": 2
+  "nbformat_minor": 4
 }

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.yml

@@ -1,10 +1,4 @@
 name: auto-ml-forecasting-bike-share
 dependencies:
-- py-xgboost<=0.90
 - pip:
   - azureml-sdk
-  - numpy==1.16.2
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/forecasting_helper.py

@@ -1,99 +0,0 @@
-import pandas as pd
-import numpy as np
-from pandas.tseries.frequencies import to_offset
-
-
-def align_outputs(y_predicted, X_trans, X_test, y_test, target_column_name,
-                  predicted_column_name='predicted',
-                  horizon_colname='horizon_origin'):
-    """
-    Demonstrates how to get the output aligned to the inputs
-    using pandas indexes. Helps understand what happened if
-    the output's shape differs from the input shape, or if
-    the data got re-sorted by time and grain during forecasting.
-
-    Typical causes of misalignment are:
-    * we predicted some periods that were missing in actuals -> drop from eval
-    * model was asked to predict past max_horizon -> increase max horizon
-    * data at start of X_test was needed for lags -> provide previous periods
-    """
-
-    if (horizon_colname in X_trans):
-        df_fcst = pd.DataFrame({predicted_column_name: y_predicted,
-                                horizon_colname: X_trans[horizon_colname]})
-    else:
-        df_fcst = pd.DataFrame({predicted_column_name: y_predicted})
-
-    # y and X outputs are aligned by forecast() function contract
-    df_fcst.index = X_trans.index
-
-    # align original X_test to y_test
-    X_test_full = X_test.copy()
-    X_test_full[target_column_name] = y_test
-
-    # X_test_full's index does not include origin, so reset for merge
-    df_fcst.reset_index(inplace=True)
-    X_test_full = X_test_full.reset_index().drop(columns='index')
-    together = df_fcst.merge(X_test_full, how='right')
-
-    # drop rows where prediction or actuals are nan
-    # happens because of missing actuals
-    # or at edges of time due to lags/rolling windows
-    clean = together[together[[target_column_name,
-                               predicted_column_name]].notnull().all(axis=1)]
-    return(clean)
-
-
-def do_rolling_forecast(fitted_model, X_test, y_test, target_column_name,
-                        time_column_name, max_horizon, freq='D'):
-    """
-    Produce forecasts on a rolling origin over the given test set.
-
-    Each iteration makes a forecast for the next 'max_horizon' periods
-    with respect to the current origin, then advances the origin by the
-    horizon time duration. The prediction context for each forecast is set so
-    that the forecaster uses the actual target values prior to the current
-    origin time for constructing lag features.
-
-    This function returns a concatenated DataFrame of rolling forecasts.
-     """
-    df_list = []
-    origin_time = X_test[time_column_name].min()
-    while origin_time <= X_test[time_column_name].max():
-        # Set the horizon time - end date of the forecast
-        horizon_time = origin_time + max_horizon * to_offset(freq)
-
-        # Extract test data from an expanding window up-to the horizon
-        expand_wind = (X_test[time_column_name] < horizon_time)
-        X_test_expand = X_test[expand_wind]
-        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
-        y_query_expand.fill(np.NaN)
-
-        if origin_time != X_test[time_column_name].min():
-            # Set the context by including actuals up-to the origin time
-            test_context_expand_wind = (X_test[time_column_name] < origin_time)
-            context_expand_wind = (
-                X_test_expand[time_column_name] < origin_time)
-            y_query_expand[context_expand_wind] = y_test[
-                test_context_expand_wind]
-
-        # Make a forecast out to the maximum horizon
-        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
-
-        # Align forecast with test set for dates within the
-        # current rolling window
-        trans_tindex = X_trans.index.get_level_values(time_column_name)
-        trans_roll_wind = (trans_tindex >= origin_time) & (
-            trans_tindex < horizon_time)
-        test_roll_wind = expand_wind & (
-            X_test[time_column_name] >= origin_time)
-        df_list.append(align_outputs(y_fcst[trans_roll_wind],
-                                     X_trans[trans_roll_wind],
-                                     X_test[test_roll_wind],
-                                     y_test[test_roll_wind],
-                                     target_column_name))
-
-        # Advance the origin time
-        origin_time = horizon_time
-
-    return pd.concat(df_list, ignore_index=True)

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/forecasting_script.py

@@ -1,52 +1,36 @@
 import argparse
-import azureml.train.automl
-from azureml.automl.runtime.shared import forecasting_models
-from azureml.core import Run
+from azureml.core import Dataset, Run
 from sklearn.externals import joblib
-import forecasting_helper
-
 
 parser = argparse.ArgumentParser()
-parser.add_argument(
-    '--max_horizon', type=int, dest='max_horizon',
-    default=10, help='Max Horizon for forecasting')
 parser.add_argument(
     '--target_column_name', type=str, dest='target_column_name',
     help='Target Column Name')
 parser.add_argument(
-    '--time_column_name', type=str, dest='time_column_name',
-    help='Time Column Name')
-parser.add_argument(
-    '--frequency', type=str, dest='freq',
-    help='Frequency of prediction')
+    '--test_dataset', type=str, dest='test_dataset',
+    help='Test Dataset')
 
 args = parser.parse_args()
-max_horizon = args.max_horizon
 target_column_name = args.target_column_name
-time_column_name = args.time_column_name
-freq = args.freq
+test_dataset_id = args.test_dataset
 
 run = Run.get_context()
-# get input dataset by name
-test_dataset = run.input_datasets['test_data']
+ws = run.experiment.workspace
 
-grain_column_names = []
-
-df = test_dataset.to_pandas_dataframe().reset_index(drop=True)
+# get the input dataset by id
+test_dataset = Dataset.get_by_id(ws, id=test_dataset_id)
 
 X_test_df = test_dataset.drop_columns(columns=[target_column_name]).to_pandas_dataframe().reset_index(drop=True)
 y_test_df = test_dataset.with_timestamp_columns(None).keep_columns(columns=[target_column_name]).to_pandas_dataframe()
 
 fitted_model = joblib.load('model.pkl')
 
-df_all = forecasting_helper.do_rolling_forecast(
-    fitted_model,
-    X_test_df,
-    y_test_df.values.T[0],
-    target_column_name,
-    time_column_name,
-    max_horizon,
-    freq)
+y_pred, X_trans = fitted_model.rolling_evaluation(X_test_df, y_test_df.values)
+
+# Add predictions, actuals, and horizon relative to rolling origin to the test feature data
+assign_dict = {'horizon_origin': X_trans['horizon_origin'].values, 'predicted': y_pred,
+               target_column_name: y_test_df[target_column_name].values}
+df_all = X_test_df.assign(**assign_dict)
 
 file_name = 'outputs/predictions.csv'
 export_csv = df_all.to_csv(file_name, header=True)

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/run_forecast.py

@@ -1,41 +1,32 @@
-from azureml.core import Environment
-from azureml.core.conda_dependencies import CondaDependencies
-from azureml.train.estimator import Estimator
-from azureml.core.run import Run
+from azureml.core import ScriptRunConfig
 
 
-def run_rolling_forecast(test_experiment, compute_target, train_run, test_dataset,
-                         max_horizon, target_column_name, time_column_name,
-                         freq='D', inference_folder='./forecast'):
-    condafile = inference_folder + '/condafile.yml'
+def run_rolling_forecast(test_experiment, compute_target, train_run,
+                         test_dataset, target_column_name,
+                         inference_folder='./forecast'):
     train_run.download_file('outputs/model.pkl',
                             inference_folder + '/model.pkl')
-    train_run.download_file('outputs/conda_env_v_1_0_0.yml', condafile)
 
-    inference_env = Environment("myenv")
-    inference_env.docker.enabled = True
-    inference_env.python.conda_dependencies = CondaDependencies(
-        conda_dependencies_file_path=condafile)
+    inference_env = train_run.get_environment()
 
-    est = Estimator(source_directory=inference_folder,
-                    entry_script='forecasting_script.py',
-                    script_params={
-                        '--max_horizon': max_horizon,
-                        '--target_column_name': target_column_name,
-                        '--time_column_name': time_column_name,
-                        '--frequency': freq
-                    },
-                    inputs=[test_dataset.as_named_input('test_data')],
+    config = ScriptRunConfig(source_directory=inference_folder,
+                             script='forecasting_script.py',
+                             arguments=['--target_column_name',
+                                        target_column_name,
+                                        '--test_dataset',
+                                        test_dataset.as_named_input(test_dataset.name)],
                              compute_target=compute_target,
-                    environment_definition=inference_env)
+                             environment=inference_env)
 
-    run = test_experiment.submit(est,
-                                 tags={
-                                     'training_run_id': train_run.id,
-                                     'run_algorithm': train_run.properties['run_algorithm'],
-                                     'valid_score': train_run.properties['score'],
-                                     'primary_metric': train_run.properties['primary_metric']
-                                 })
+    run = test_experiment.submit(config,
+                                 tags={'training_run_id':
+                                       train_run.id,
+                                       'run_algorithm':
+                                       train_run.properties['run_algorithm'],
+                                       'valid_score':
+                                       train_run.properties['score'],
+                                       'primary_metric':
+                                       train_run.properties['primary_metric']})
 
     run.log("run_algorithm", run.tags['run_algorithm'])
     return run

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb

@@ -24,10 +24,11 @@
         "_**Forecasting using the Energy Demand Dataset**_\n",
         "\n",
         "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Setup](#Setup)\n",
-        "1. [Data and Forecasting Configurations](#Data)\n",
-        "1. [Train](#Train)\n",
+        "1. [Introduction](#introduction)\n",
+        "1. [Setup](#setup)\n",
+        "1. [Data and Forecasting Configurations](#data)\n",
+        "1. [Train](#train)\n",
+        "1. [Generate and Evaluate the Forecast](#forecast)\n",
         "\n",
         "Advanced Forecasting\n",
         "1. [Advanced Training](#advanced_training)\n",
@@ -38,7 +39,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Introduction\n",
+        "# Introduction<a id=\"introduction\"></a>\n",
         "\n",
         "In this example we use the associated New York City energy demand dataset to showcase how you can use AutoML for a simple forecasting problem and explore the results. The goal is predict the energy demand for the next 48 hours based on historic time-series data.\n",
         "\n",
@@ -49,15 +50,16 @@
         "1. Configure AutoML using 'AutoMLConfig'\n",
         "1. Train the model using AmlCompute\n",
         "1. Explore the engineered features and results\n",
+        "1. Generate the forecast and compute the out-of-sample accuracy metrics\n",
         "1. Configuration and remote run of AutoML for a time-series model with lag and rolling window features\n",
-        "1. Run and explore the forecast"
+        "1. Run and explore the forecast with lagging features"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Setup"
+        "# Setup<a id=\"setup\"></a>"
       ]
     },
     {
@@ -97,7 +99,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -177,7 +179,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Data\n",
+        "# Data<a id=\"data\"></a>\n",
         "\n",
         "We will use energy consumption [data from New York City](http://mis.nyiso.com/public/P-58Blist.htm) for model training. The data is stored in a tabular format and includes energy demand and basic weather data at an hourly frequency. \n",
         "\n",
@@ -288,37 +290,50 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "max_horizon = 48"
+        "forecast_horizon = 48"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Train\n",
+        "## Forecasting Parameters\n",
+        "To define forecasting parameters for your experiment training, you can leverage the ForecastingParameters class. The table below details the forecasting parameter we will be passing into our experiment.\n",
         "\n",
-        "Instantiate an AutoMLConfig object. This config defines the settings and data used to run the experiment. We can provide extra configurations within 'automl_settings', for this forecasting task we add the name of the time column and the maximum forecast horizon.\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**time_column_name**|The name of your time column.|\n",
+        "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
+        "|**freq**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Train<a id=\"train\"></a>\n",
+        "\n",
+        "Instantiate an AutoMLConfig object. This config defines the settings and data used to run the experiment. We can provide extra configurations within 'automl_settings', for this forecasting task we add the forecasting parameters to hold all the additional forecasting parameters.\n",
         "\n",
         "|Property|Description|\n",
         "|-|-|\n",
         "|**task**|forecasting|\n",
         "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
-        "|**blacklist_models**|Models in blacklist won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
+        "|**blocked_models**|Models in blocked_models won't be used by AutoML. All supported models can be found at [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py).|\n",
         "|**experiment_timeout_hours**|Maximum amount of time in hours that the experiment take before it terminates.|\n",
         "|**training_data**|The training data to be used within the experiment.|\n",
         "|**label_column_name**|The name of the label column.|\n",
         "|**compute_target**|The remote compute for training.|\n",
         "|**n_cross_validations**|Number of cross validation splits. Rolling Origin Validation is used to split time-series in a temporally consistent way.|\n",
         "|**enable_early_stopping**|Flag to enble early termination if the score is not improving in the short term.|\n",
-        "|**time_column_name**|The name of your time column.|\n",
-        "|**max_horizon**|The number of periods out you would like to predict past your training data. Periods are inferred from your data.|\n"
+        "|**forecasting_parameters**|A class holds all the forecasting related parameters.|\n"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "This notebook uses the blacklist_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blacklist_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
+        "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
       ]
     },
     {
@@ -327,14 +342,16 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "automl_settings = {\n",
-        "    'time_column_name': time_column_name,\n",
-        "    'max_horizon': max_horizon,\n",
-        "}\n",
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name,\n",
+        "    forecast_horizon=forecast_horizon,\n",
+        "    freq='H' # Set the forecast frequency to be hourly\n",
+        ")\n",
         "\n",
         "automl_config = AutoMLConfig(task='forecasting',                             \n",
         "                             primary_metric='normalized_root_mean_squared_error',\n",
-        "                             blacklist_models = ['ExtremeRandomTrees', 'AutoArima', 'Prophet'],                             \n",
+        "                             blocked_models = ['ExtremeRandomTrees', 'AutoArima', 'Prophet'],                             \n",
         "                             experiment_timeout_hours=0.3,\n",
         "                             training_data=train,\n",
         "                             label_column_name=target_column_name,\n",
@@ -342,7 +359,7 @@
         "                             enable_early_stopping=True,\n",
         "                             n_cross_validations=3,                             \n",
         "                             verbosity=logging.INFO,\n",
-        "                            **automl_settings)"
+        "                             forecasting_parameters=forecasting_parameters)"
       ]
     },
     {
@@ -362,15 +379,6 @@
         "remote_run = experiment.submit(automl_config, show_output=False)"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -445,9 +453,11 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Forecasting\n",
+        "# Forecasting<a id=\"forecast\"></a>\n",
         "\n",
-        "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. First, we remove the target values from the test set:"
+        "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. We will do batch scoring on the test dataset which should have the same schema as training dataset.\n",
+        "\n",
+        "The inference will run on a remote compute. In this example, it will re-use the training compute."
       ]
     },
     {
@@ -456,16 +466,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X_test = test.to_pandas_dataframe().reset_index(drop=True)\n",
-        "y_test = X_test.pop(target_column_name).values"
+        "test_experiment = Experiment(ws, experiment_name + \"_inference\")"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "### Forecast Function\n",
-        "For forecasting, we will use the forecast function instead of the predict function. Using the predict method would result in getting predictions for EVERY horizon the forecaster can predict at. This is useful when training and evaluating the performance of the forecaster at various horizons, but the level of detail is excessive for normal use. Forecast function also can handle more complicated scenarios, see the [forecast function notebook](../forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
+        "### Retreiving forecasts from the model\n",
+        "We have created a function called `run_forecast` that submits the test data to the best model determined during the training run and retrieves forecasts. This function uses a helper script `forecasting_script` which is uploaded and expecuted on the remote compute."
       ]
     },
     {
@@ -474,10 +483,16 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# The featurized data, aligned to y, will also be returned.\n",
-        "# This contains the assumptions that were made in the forecast\n",
-        "# and helps align the forecast to the original data\n",
-        "y_predictions, X_trans = fitted_model.forecast(X_test)"
+        "from run_forecast import run_remote_inference\n",
+        "remote_run_infer = run_remote_inference(test_experiment=test_experiment,\n",
+        "                                        compute_target=compute_target,\n",
+        "                                        train_run=best_run,\n",
+        "                                        test_dataset=test,\n",
+        "                                        target_column_name=target_column_name)\n",
+        "remote_run_infer.wait_for_completion(show_output=False)\n",
+        "\n",
+        "# download the inference output file to the local machine\n",
+        "remote_run_infer.download_file('outputs/predictions.csv', 'predictions.csv')"
       ]
     },
     {
@@ -485,9 +500,7 @@
       "metadata": {},
       "source": [
         "### Evaluate\n",
-        "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE).\n",
-        "\n",
-        "It is a good practice to always align the output explicitly to the input, as the count and order of the rows may have changed during transformations that span multiple rows."
+        "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE). For more metrics that can be used for evaluation after training, please see [supported metrics](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#regressionforecasting-metrics), and [how to calculate residuals](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#residuals)."
       ]
     },
     {
@@ -496,9 +509,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from forecasting_helper import align_outputs\n",
-        "\n",
-        "df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)"
+        "# load forecast data frame\n",
+        "fcst_df = pd.read_csv('predictions.csv', parse_dates=[time_column_name])\n",
+        "fcst_df.head()"
       ]
     },
     {
@@ -513,8 +526,8 @@
         "\n",
         "# use automl metrics module\n",
         "scores = scoring.score_regression(\n",
-        "    y_test=df_all[target_column_name],\n",
-        "    y_pred=df_all['predicted'],\n",
+        "    y_test=fcst_df[target_column_name],\n",
+        "    y_pred=fcst_df['predicted'],\n",
         "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
@@ -523,8 +536,8 @@
         "    \n",
         "# Plot outputs\n",
         "%matplotlib inline\n",
-        "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
-        "test_test = plt.scatter(df_all[target_column_name], df_all[target_column_name], color='g')\n",
+        "test_pred = plt.scatter(fcst_df[target_column_name], fcst_df['predicted'], color='b')\n",
+        "test_test = plt.scatter(fcst_df[target_column_name], fcst_df[target_column_name], color='g')\n",
         "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
         "plt.show()"
       ]
@@ -533,24 +546,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Looking at `X_trans` is also useful to see what featurization happened to the data."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "X_trans"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Advanced Training <a id=\"advanced_training\"></a>\n",
-        "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, grain and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
+        "# Advanced Training <a id=\"advanced_training\"></a>\n",
+        "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, time series identifier columns and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
       ]
     },
     {
@@ -558,9 +555,9 @@
       "metadata": {},
       "source": [
         "### Using lags and rolling window features\n",
-        "Now we will configure the target lags, that is the previous values of the target variables, meaning the prediction is no longer horizon-less. We therefore must still specify the `max_horizon` that the model will learn to forecast. The `target_lags` keyword specifies how far back we will construct the lags of the target variable, and the `target_rolling_window_size` specifies the size of the rolling window over which we will generate the `max`, `min` and `sum` features.\n",
+        "Now we will configure the target lags, that is the previous values of the target variables, meaning the prediction is no longer horizon-less. We therefore must still specify the `forecast_horizon` that the model will learn to forecast. The `target_lags` keyword specifies how far back we will construct the lags of the target variable, and the `target_rolling_window_size` specifies the size of the rolling window over which we will generate the `max`, `min` and `sum` features.\n",
         "\n",
-        "This notebook uses the blacklist_models parameter to exclude some models that take a longer time to train on this dataset.  You can choose to remove models from the blacklist_models list but you may need to increase the iteration_timeout_minutes parameter value to get results."
+        "This notebook uses the blocked_models parameter to exclude some models that take a longer time to train on this dataset.  You can choose to remove models from the blocked_models list but you may need to increase the iteration_timeout_minutes parameter value to get results."
       ]
     },
     {
@@ -569,16 +566,14 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "automl_advanced_settings = {\n",
-        "    'time_column_name': time_column_name,\n",
-        "    'max_horizon': max_horizon,\n",
-        "    'target_lags': 12,\n",
-        "    'target_rolling_window_size': 4,\n",
-        "}\n",
+        "advanced_forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name, forecast_horizon=forecast_horizon,\n",
+        "    target_lags=12, target_rolling_window_size=4\n",
+        ")\n",
         "\n",
         "automl_config = AutoMLConfig(task='forecasting',                             \n",
         "                             primary_metric='normalized_root_mean_squared_error',\n",
-        "                             blacklist_models = ['ElasticNet','ExtremeRandomTrees','GradientBoosting','XGBoostRegressor','ExtremeRandomTrees', 'AutoArima', 'Prophet'], #These models are blacklisted for tutorial purposes, remove this for real use cases.                            \n",
+        "                             blocked_models = ['ElasticNet','ExtremeRandomTrees','GradientBoosting','XGBoostRegressor','ExtremeRandomTrees', 'AutoArima', 'Prophet'], #These models are blocked for tutorial purposes, remove this for real use cases.                            \n",
         "                             experiment_timeout_hours=0.3,\n",
         "                             training_data=train,\n",
         "                             label_column_name=target_column_name,\n",
@@ -586,7 +581,7 @@
         "                             enable_early_stopping = True,\n",
         "                             n_cross_validations=3,                             \n",
         "                             verbosity=logging.INFO,\n",
-        "                            **automl_advanced_settings)"
+        "                             forecasting_parameters=advanced_forecasting_parameters)"
       ]
     },
     {
@@ -634,8 +629,8 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Advanced Results<a id=\"advanced_results\"></a>\n",
-        "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, grain and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
+        "# Advanced Results<a id=\"advanced_results\"></a>\n",
+        "We did not use lags in the previous model specification. In effect, the prediction was the result of a simple regression on date, time series identifier columns and any additional features. This is often a very good prediction as common time series patterns like seasonality and trends can be captured in this manner. Such simple regression is horizon-less: it doesn't matter how far into the future we are predicting, because we are not using past data. In the previous example, the horizon was only used to split the data for cross-validation."
       ]
     },
     {
@@ -644,21 +639,27 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# The featurized data, aligned to y, will also be returned.\n",
-        "# This contains the assumptions that were made in the forecast\n",
-        "# and helps align the forecast to the original data\n",
-        "y_predictions, X_trans = fitted_model_lags.forecast(X_test)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from forecasting_helper import align_outputs\n",
+        "test_experiment_advanced = Experiment(ws, experiment_name + \"_inference_advanced\")\n",
+        "advanced_remote_run_infer = run_remote_inference(test_experiment=test_experiment_advanced,\n",
+        "                                                 compute_target=compute_target,\n",
+        "                                                 train_run=best_run_lags,\n",
+        "                                                 test_dataset=test,\n",
+        "                                                 target_column_name=target_column_name,\n",
+        "                                                 inference_folder='./forecast_advanced')\n",
+        "advanced_remote_run_infer.wait_for_completion(show_output=False)\n",
         "\n",
-        "df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)"
+        "# download the inference output file to the local machine\n",
+        "advanced_remote_run_infer.download_file('outputs/predictions.csv', 'predictions_advanced.csv')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "fcst_adv_df = pd.read_csv('predictions_advanced.csv', parse_dates=[time_column_name])\n",
+        "fcst_adv_df.head()"
       ]
     },
     {
@@ -673,8 +674,8 @@
         "\n",
         "# use automl metrics module\n",
         "scores = scoring.score_regression(\n",
-        "    y_test=df_all[target_column_name],\n",
-        "    y_pred=df_all['predicted'],\n",
+        "    y_test=fcst_adv_df[target_column_name],\n",
+        "    y_pred=fcst_adv_df['predicted'],\n",
         "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
@@ -683,8 +684,8 @@
         "    \n",
         "# Plot outputs\n",
         "%matplotlib inline\n",
-        "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
-        "test_test = plt.scatter(df_all[target_column_name], df_all[target_column_name], color='g')\n",
+        "test_pred = plt.scatter(fcst_adv_df[target_column_name], fcst_adv_df['predicted'], color='b')\n",
+        "test_test = plt.scatter(fcst_adv_df[target_column_name], fcst_adv_df[target_column_name], color='g')\n",
         "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
         "plt.show()"
       ]
@@ -693,7 +694,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "erwright"
+        "name": "jialiu"
       }
     ],
     "categories": [
@@ -715,7 +716,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.8"
+      "version": "3.6.9"
     }
   },
   "nbformat": 4,

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.yml

@@ -2,8 +2,3 @@ name: auto-ml-forecasting-energy-demand
 dependencies:
 - pip:
   - azureml-sdk
-  - numpy==1.16.2
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/forecasting_script.py

@@ -1,5 +1,15 @@
+"""
+This is the script that is executed on the compute instance. It relies
+on the model.pkl file which is uploaded along with this script to the
+compute instance.
+"""
+
+import argparse
 import pandas as pd
 import numpy as np
+from azureml.core import Dataset, Run
+from azureml.automl.core.shared.constants import TimeSeriesInternal
+from sklearn.externals import joblib
 from pandas.tseries.frequencies import to_offset
 
 
@@ -42,3 +52,38 @@ def align_outputs(y_predicted, X_trans, X_test, y_test, target_column_name,
     clean = together[together[[target_column_name,
                                predicted_column_name]].notnull().all(axis=1)]
     return(clean)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    '--target_column_name', type=str, dest='target_column_name',
+    help='Target Column Name')
+parser.add_argument(
+    '--test_dataset', type=str, dest='test_dataset',
+    help='Test Dataset')
+
+args = parser.parse_args()
+target_column_name = args.target_column_name
+test_dataset_id = args.test_dataset
+
+run = Run.get_context()
+ws = run.experiment.workspace
+
+# get the input dataset by id
+test_dataset = Dataset.get_by_id(ws, id=test_dataset_id)
+
+X_test = test_dataset.to_pandas_dataframe().reset_index(drop=True)
+y_test = X_test.pop(target_column_name).values
+
+# generate forecast
+fitted_model = joblib.load('model.pkl')
+y_predictions, X_trans = fitted_model.forecast(X_test)
+
+# align output
+df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)
+
+file_name = 'outputs/predictions.csv'
+export_csv = df_all.to_csv(file_name, header=True, index=False)  # added Index
+
+# Upload the predictions into artifacts
+run.upload_file(name=file_name, path_or_stream=file_name)

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/metrics_helper.py

@@ -1,22 +0,0 @@
-import pandas as pd
-import numpy as np
-
-
-def APE(actual, pred):
-    """
-    Calculate absolute percentage error.
-    Returns a vector of APE values with same length as actual/pred.
-    """
-    return 100 * np.abs((actual - pred) / actual)
-
-
-def MAPE(actual, pred):
-    """
-    Calculate mean absolute percentage error.
-    Remove NA and values where actual is close to zero
-    """
-    not_na = ~(np.isnan(actual) | np.isnan(pred))
-    not_zero = ~np.isclose(actual, 0.0)
-    actual_safe = actual[not_na & not_zero]
-    pred_safe = pred[not_na & not_zero]
-    return np.mean(APE(actual_safe, pred_safe))

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/run_forecast.py

@@ -0,0 +1,38 @@
+import os
+import shutil
+from azureml.core import ScriptRunConfig
+
+
+def run_remote_inference(test_experiment, compute_target, train_run,
+                         test_dataset, target_column_name, inference_folder='./forecast'):
+    # Create local directory to copy the model.pkl and forecsting_script.py files into.
+    # These files will be uploaded to and executed on the compute instance.
+    os.makedirs(inference_folder, exist_ok=True)
+    shutil.copy('forecasting_script.py', inference_folder)
+
+    train_run.download_file('outputs/model.pkl',
+                            os.path.join(inference_folder, 'model.pkl'))
+
+    inference_env = train_run.get_environment()
+
+    config = ScriptRunConfig(source_directory=inference_folder,
+                             script='forecasting_script.py',
+                             arguments=['--target_column_name',
+                                        target_column_name,
+                                        '--test_dataset',
+                                        test_dataset.as_named_input(test_dataset.name)],
+                             compute_target=compute_target,
+                             environment=inference_env)
+
+    run = test_experiment.submit(config,
+                                 tags={'training_run_id':
+                                       train_run.id,
+                                       'run_algorithm':
+                                       train_run.properties['run_algorithm'],
+                                       'valid_score':
+                                       train_run.properties['score'],
+                                       'primary_metric':
+                                       train_run.properties['primary_metric']})
+
+    run.log("run_algorithm", run.tags['run_algorithm'])
+    return run

how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.ipynb

@@ -24,7 +24,7 @@
       "metadata": {},
       "source": [
         "## Introduction\n",
-        "This notebook demonstrates the full interface to the `forecast()` function. \n",
+        "This notebook demonstrates the full interface of the `forecast()` function. \n",
         "\n",
         "The best known and most frequent usage of `forecast` enables forecasting on test sets that immediately follows training data. \n",
         "\n",
@@ -94,7 +94,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -142,15 +142,15 @@
       "outputs": [],
       "source": [
         "TIME_COLUMN_NAME = 'date'\n",
-        "GRAIN_COLUMN_NAME = 'grain'\n",
+        "TIME_SERIES_ID_COLUMN_NAME = 'time_series_id'\n",
         "TARGET_COLUMN_NAME = 'y'\n",
         "\n",
         "def get_timeseries(train_len: int,\n",
         "                   test_len: int,\n",
         "                   time_column_name: str,\n",
         "                   target_column_name: str,\n",
-        "                   grain_column_name: str,\n",
-        "                   grains: int = 1,\n",
+        "                   time_series_id_column_name: str,\n",
+        "                   time_series_number: int = 1,\n",
         "                   freq: str = 'H'):\n",
         "    \"\"\"\n",
         "    Return the time series of designed length.\n",
@@ -161,9 +161,8 @@
         "    :type test_len: int\n",
         "    :param time_column_name: The desired name of a time column.\n",
         "    :type time_column_name: str\n",
-        "    :param\n",
-        "    :param grains: The number of grains.\n",
-        "    :type grains: int\n",
+        "    :param time_series_number: The number of time series in the data set.\n",
+        "    :type time_series_number: int\n",
         "    :param freq: The frequency string representing pandas offset.\n",
         "                 see https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html\n",
         "    :type freq: str\n",
@@ -174,14 +173,14 @@
         "    data_train = []  # type: List[pd.DataFrame]\n",
         "    data_test = []  # type: List[pd.DataFrame]\n",
         "    data_length = train_len + test_len\n",
-        "    for i in range(grains):\n",
+        "    for i in range(time_series_number):\n",
         "        X = pd.DataFrame({\n",
         "            time_column_name: pd.date_range(start='2000-01-01',\n",
         "                                            periods=data_length,\n",
         "                                            freq=freq),\n",
         "            target_column_name: np.arange(data_length).astype(float) + np.random.rand(data_length) + i*5,\n",
         "            'ext_predictor': np.asarray(range(42, 42 + data_length)),\n",
-        "            grain_column_name: np.repeat('g{}'.format(i), data_length)\n",
+        "            time_series_id_column_name: np.repeat('ts{}'.format(i), data_length)\n",
         "        })\n",
         "        data_train.append(X[:train_len])\n",
         "        data_test.append(X[train_len:])\n",
@@ -197,8 +196,8 @@
         "                                                  test_len=n_test_periods,\n",
         "                                                  time_column_name=TIME_COLUMN_NAME,\n",
         "                                                  target_column_name=TARGET_COLUMN_NAME,\n",
-        "                                                  grain_column_name=GRAIN_COLUMN_NAME,\n",
-        "                                                  grains=2)"
+        "                                                  time_series_id_column_name=TIME_SERIES_ID_COLUMN_NAME,\n",
+        "                                                  time_series_number=2)"
       ]
     },
     {
@@ -228,7 +227,7 @@
         "whole_data = X_train.copy()\n",
         "target_label = 'y'\n",
         "whole_data[target_label] = y_train\n",
-        "for g in whole_data.groupby('grain'):    \n",
+        "for g in whole_data.groupby('time_series_id'):    \n",
         "    plt.plot(g[1]['date'].values, g[1]['y'].values, label=g[0])\n",
         "plt.legend()\n",
         "plt.show()"
@@ -264,7 +263,9 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource."
+        "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
       ]
     },
     {
@@ -284,7 +285,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=6)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
@@ -297,13 +298,14 @@
       "source": [
         "## Create the configuration and train a forecaster <a id=\"train\"></a>\n",
         "First generate the configuration, in which we:\n",
-        "* Set metadata columns: target, time column and grain column names.\n",
+        "* Set metadata columns: target, time column and time-series id column names.\n",
         "* Validate our data using cross validation with rolling window method.\n",
         "* Set normalized root mean squared error as a metric to select the best model.\n",
         "* Set early termination to True, so the iterations through the models will stop when no improvements in accuracy score will be made.\n",
         "* Set limitations on the length of experiment run to 15 minutes.\n",
         "* Finally, we set the task to be forecasting.\n",
-        "* We apply the lag lead operator to the target value i.e. we use the previous values as a predictor for the future ones."
+        "* We apply the lag lead operator to the target value i.e. we use the previous values as a predictor for the future ones.\n",
+        "* [Optional] Forecast frequency parameter (freq) represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information."
       ]
     },
     {
@@ -312,21 +314,23 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
         "lags = [1,2,3]\n",
-        "max_horizon = n_test_periods\n",
-        "time_series_settings = {    \n",
-        "    'time_column_name': TIME_COLUMN_NAME,\n",
-        "    'grain_column_names': [ GRAIN_COLUMN_NAME ],\n",
-        "    'max_horizon': max_horizon,\n",
-        "    'target_lags': lags\n",
-        "}"
+        "forecast_horizon = n_test_periods\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=TIME_COLUMN_NAME,\n",
+        "    forecast_horizon=forecast_horizon,\n",
+        "    time_series_id_column_names=[ TIME_SERIES_ID_COLUMN_NAME ],\n",
+        "    target_lags=lags,\n",
+        "    freq='H' # Set the forecast frequency to be hourly\n",
+        ")"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Run the model selection and training process."
+        "Run the model selection and training process.  Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -352,7 +356,7 @@
         "                             max_concurrent_iterations=4,\n",
         "                             max_cores_per_iteration=-1,\n",
         "                             label_column_name=target_label,\n",
-        "                             **time_series_settings)\n",
+        "                             forecasting_parameters=forecasting_parameters)\n",
         "\n",
         "remote_run = experiment.submit(automl_config, show_output=False)"
       ]
@@ -482,7 +486,7 @@
         "# use forecast_quantiles function, not the forecast() one\n",
         "y_pred_quantiles =  fitted_model.forecast_quantiles(X_test)\n",
         "\n",
-        "# quantile forecasts returned in a Dataframe along with the time and grain columns \n",
+        "# quantile forecasts returned in a Dataframe along with the time and time series id columns \n",
         "y_pred_quantiles"
       ]
     },
@@ -492,7 +496,7 @@
       "source": [
         "#### Destination-date forecast: \"just do something\"\n",
         "\n",
-        "In some scenarios, the X_test is not known. The forecast is likely to be weak, because it is missing contemporaneous predictors, which we will need to impute. If you still wish to predict forward under the assumption that the last known values will be carried forward, you can forecast out to \"destination date\". The destination date still needs to fit within the maximum horizon from training."
+        "In some scenarios, the X_test is not known. The forecast is likely to be weak, because it is missing contemporaneous predictors, which we will need to impute. If you still wish to predict forward under the assumption that the last known values will be carried forward, you can forecast out to \"destination date\". The destination date still needs to fit within the forecast horizon from training."
       ]
     },
     {
@@ -519,7 +523,7 @@
         "\n",
         "![Forecasting after training](forecast_function_away_from_train.png)\n",
         "\n",
-        "The notion of forecast origin comes into play: the forecast origin is **the last period for which we have seen the target value**. This applies per grain, so each grain can have a different forecast origin. \n",
+        "The notion of forecast origin comes into play: the forecast origin is **the last period for which we have seen the target value**. This applies per time-series, so each time-series can have a different forecast origin. \n",
         "\n",
         "The part of data before the forecast origin is the **prediction context**. To provide the context values the model needs when it looks back, we pass definite values in `y_test` (aligned with corresponding times in `X_test`)."
       ]
@@ -536,13 +540,13 @@
         "                                      test_len=4,\n",
         "                                      time_column_name=TIME_COLUMN_NAME,\n",
         "                                      target_column_name=TARGET_COLUMN_NAME,\n",
-        "                                      grain_column_name=GRAIN_COLUMN_NAME,\n",
-        "                                      grains=2)\n",
+        "                                      time_series_id_column_name=TIME_SERIES_ID_COLUMN_NAME,\n",
+        "                                      time_series_number=2)\n",
         "\n",
         "# end of the data we trained on\n",
-        "print(X_train.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].max())\n",
+        "print(X_train.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].max())\n",
         "# start of the data we want to predict on\n",
-        "print(X_away.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].min())"
+        "print(X_away.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].min())"
       ]
     },
     {
@@ -569,7 +573,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "How should we read that eror message? The forecast origin is at the last time the model saw an actual value of `y` (the target). That was at the end of the training data! The model is attempting to forecast from the end of training data. But the requested forecast periods are past the maximum horizon. We need to provide a define `y` value to establish the forecast origin.\n",
+        "How should we read that eror message? The forecast origin is at the last time the model saw an actual value of `y` (the target). That was at the end of the training data! The model is attempting to forecast from the end of training data. But the requested forecast periods are past the forecast horizon. We need to provide a define `y` value to establish the forecast origin.\n",
         "\n",
         "We will use this helper function to take the required amount of context from the data preceding the testing data. It's definition is intentionally simplified to keep the idea in the clear."
       ]
@@ -584,7 +588,7 @@
         "\n",
         "    \"\"\"\n",
         "    This function will take the full dataset, and create the query\n",
-        "    to predict all values of the grain from the `forecast_origin`\n",
+        "    to predict all values of the time series from the `forecast_origin`\n",
         "    forward for the next `horizon` horizons. Context from previous\n",
         "    `lookback` periods will be included.\n",
         "\n",
@@ -654,8 +658,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(X_context.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].agg(['min','max','count']))\n",
-        "print(X_away.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].agg(['min','max','count']))\n",
+        "print(X_context.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].agg(['min','max','count']))\n",
+        "print(X_away.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].agg(['min','max','count']))\n",
         "X_context.tail(5)"
       ]
     },
@@ -685,7 +689,7 @@
         "n_lookback_periods = max(lags)\n",
         "lookback = pd.DateOffset(hours=n_lookback_periods)\n",
         "\n",
-        "horizon = pd.DateOffset(hours=max_horizon)\n",
+        "horizon = pd.DateOffset(hours=forecast_horizon)\n",
         "\n",
         "# now make the forecast query from context (refer to figure)\n",
         "X_pred, y_pred = make_forecasting_query(fulldata, TIME_COLUMN_NAME, TARGET_COLUMN_NAME,\n",
@@ -701,7 +705,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Note that the forecast origin is at 17:00 for both grains, and periods from 18:00 are to be forecast."
+        "Note that the forecast origin is at 17:00 for both time-series, and periods from 18:00 are to be forecast."
       ]
     },
     {
@@ -716,7 +720,7 @@
         "# show the forecast aligned\n",
         "X_show = xy_away.reset_index()\n",
         "# without the generated features\n",
-        "X_show[['date', 'grain', 'ext_predictor', '_automl_target_col']]\n",
+        "X_show[['date', 'time_series_id', 'ext_predictor', '_automl_target_col']]\n",
         "# prediction is in _automl_target_col"
       ]
     },
@@ -724,14 +728,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Forecasting farther than the maximum horizon <a id=\"recursive forecasting\"></a>\n",
-        "When the forecast destination, or the latest date in the prediction data frame, is farther into the future than the specified maximum horizon, the `forecast()` function will still make point predictions out to the later date using a recursive operation mode. Internally, the method recursively applies the regular forecaster to generate context so that we can forecast further into the future. \n",
+        "## Forecasting farther than the forecast horizon <a id=\"recursive forecasting\"></a>\n",
+        "When the forecast destination, or the latest date in the prediction data frame, is farther into the future than the specified forecast horizon, the `forecast()` function will still make point predictions out to the later date using a recursive operation mode. Internally, the method recursively applies the regular forecaster to generate context so that we can forecast further into the future. \n",
         "\n",
-        "To illustrate the use-case and operation of recursive forecasting, we'll consider an example with a single time-series where the forecasting period directly follows the training period and is twice as long as the maximum horizon given at training time.\n",
+        "To illustrate the use-case and operation of recursive forecasting, we'll consider an example with a single time-series where the forecasting period directly follows the training period and is twice as long as the forecasting horizon given at training time.\n",
         "\n",
         "![Recursive_forecast_overview](recursive_forecast_overview_small.png)\n",
         "\n",
-        "Internally, we apply the forecaster in an iterative manner and finish the forecast task in two interations. In the first iteration, we apply the forecaster and get the prediction for the first max-horizon periods (y_pred1). In the second iteraction, y_pred1 is used as the context to produce the prediction for the next max-horizon periods (y_pred2). The combination of (y_pred1 and y_pred2) gives the results for the total forecast periods. \n",
+        "Internally, we apply the forecaster in an iterative manner and finish the forecast task in two interations. In the first iteration, we apply the forecaster and get the prediction for the first forecast-horizon periods (y_pred1). In the second iteraction, y_pred1 is used as the context to produce the prediction for the next forecast-horizon periods (y_pred2). The combination of (y_pred1 and y_pred2) gives the results for the total forecast periods. \n",
         "\n",
         "A caveat: forecast accuracy will likely be worse the farther we predict into the future since errors are compounded with recursive application of the forecaster.\n",
         "\n",
@@ -745,16 +749,17 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# generate the same kind of test data we trained on, but with a single grain/time-series and test period twice as long as the max_horizon\n",
+        "# generate the same kind of test data we trained on, but with a single time-series and test period twice as long\n",
+        "# as the forecast_horizon.\n",
         "_, _, X_test_long, y_test_long = get_timeseries(train_len=n_train_periods,\n",
-        "                                                  test_len=max_horizon*2,\n",
+        "                                                  test_len=forecast_horizon*2,\n",
         "                                                  time_column_name=TIME_COLUMN_NAME,\n",
         "                                                  target_column_name=TARGET_COLUMN_NAME,\n",
-        "                                                  grain_column_name=GRAIN_COLUMN_NAME,\n",
-        "                                                  grains=1)\n",
+        "                                                  time_series_id_column_name=TIME_SERIES_ID_COLUMN_NAME,\n",
+        "                                                  time_series_number=1)\n",
         "\n",
-        "print(X_test_long.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].min())\n",
-        "print(X_test_long.groupby(GRAIN_COLUMN_NAME)[TIME_COLUMN_NAME].max())"
+        "print(X_test_long.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].min())\n",
+        "print(X_test_long.groupby(TIME_SERIES_ID_COLUMN_NAME)[TIME_COLUMN_NAME].max())"
       ]
     },
     {
@@ -775,8 +780,8 @@
       "outputs": [],
       "source": [
         "# What forecast() function does in this case is equivalent to iterating it twice over the test set as the following. \n",
-        "y_pred1, _ = fitted_model.forecast(X_test_long[:max_horizon])\n",
-        "y_pred_all, _ = fitted_model.forecast(X_test_long, np.concatenate((y_pred1, np.full(max_horizon, np.nan))))\n",
+        "y_pred1, _ = fitted_model.forecast(X_test_long[:forecast_horizon])\n",
+        "y_pred_all, _ = fitted_model.forecast(X_test_long, np.concatenate((y_pred1, np.full(forecast_horizon, np.nan))))\n",
         "np.array_equal(y_pred_all, y_pred_long)"
       ]
     },
@@ -785,7 +790,7 @@
       "metadata": {},
       "source": [
         "#### Confidence interval and distributional forecasts\n",
-        "AutoML cannot currently estimate forecast errors beyond the maximum horizon set during training, so the `forecast_quantiles()` function will return missing values for quantiles not equal to 0.5 beyond the maximum horizon. "
+        "AutoML cannot currently estimate forecast errors beyond the forecast horizon set during training, so the `forecast_quantiles()` function will return missing values for quantiles not equal to 0.5 beyond the forecast horizon. "
       ]
     },
     {
@@ -801,14 +806,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Similarly with the simple senarios illustrated above, forecasting farther than the max horizon in other senarios like 'multiple grain', 'Destination-date forecast', and 'forecast away from the training data' are also automatically handled by the `forecast()` function. "
+        "Similarly with the simple senarios illustrated above, forecasting farther than the forecast horizon in other senarios like 'multiple time-series', 'Destination-date forecast', and 'forecast away from the training data' are also automatically handled by the `forecast()` function. "
       ]
     }
   ],
   "metadata": {
     "authors": [
       {
-        "name": "erwright"
+        "name": "jialiu"
       }
     ],
     "category": "tutorial",

how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.yml

@@ -1,10 +1,4 @@
 name: auto-ml-forecasting-function
 dependencies:
-- py-xgboost<=0.90
 - pip:
   - azureml-sdk
-  - numpy==1.16.2
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb

@@ -24,20 +24,20 @@
         "_**Orange Juice Sales Forecasting**_\n",
         "\n",
         "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Setup](#Setup)\n",
-        "1. [Compute](#Compute)\n",
-        "1. [Data](#Data)\n",
-        "1. [Train](#Train)\n",
-        "1. [Predict](#Predict)\n",
-        "1. [Operationalize](#Operationalize)"
+        "1. [Introduction](#introduction)\n",
+        "1. [Setup](#setup)\n",
+        "1. [Compute](#compute)\n",
+        "1. [Data](#data)\n",
+        "1. [Train](#train)\n",
+        "1. [Forecast](#forecast)\n",
+        "1. [Operationalize](#operationalize)"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Introduction\n",
+        "## Introduction<a id=\"introduction\"></a>\n",
         "In this example, we use AutoML to train, select, and operationalize a time-series forecasting model for multiple time-series.\n",
         "\n",
         "Make sure you have executed the [configuration notebook](../../../configuration.ipynb) before running this notebook.\n",
@@ -49,7 +49,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Setup"
+        "## Setup<a id=\"setup\"></a>"
       ]
     },
     {
@@ -82,7 +82,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -122,11 +122,14 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Compute\n",
+        "## Compute<a id=\"compute\"></a>\n",
         "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "#### Creation of AmlCompute takes approximately 5 minutes. \n",
         "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota."
+        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
       ]
     },
     {
@@ -146,7 +149,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D12_V2',\n",
         "                                                           max_nodes=6)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
@@ -157,7 +160,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Data\n",
+        "## Data<a id=\"data\"></a>\n",
         "You are now ready to load the historical orange juice sales data. We will load the CSV file into a plain pandas DataFrame; the time column in the CSV is called _WeekStarting_, so it will be specially parsed into the datetime type."
       ]
     },
@@ -169,6 +172,10 @@
       "source": [
         "time_column_name = 'WeekStarting'\n",
         "data = pd.read_csv(\"dominicks_OJ.csv\", parse_dates=[time_column_name])\n",
+        "\n",
+        "# Drop the columns 'logQuantity' as it is a leaky feature.\n",
+        "data.drop('logQuantity', axis=1, inplace=True)\n",
+        "\n",
         "data.head()"
       ]
     },
@@ -178,7 +185,7 @@
       "source": [
         "Each row in the DataFrame holds a quantity of weekly sales for an OJ brand at a single store. The data also includes the sales price, a flag indicating if the OJ brand was advertised in the store that week, and some customer demographic information based on the store location. For historical reasons, the data also include the logarithm of the sales quantity. The Dominick's grocery data is commonly used to illustrate econometric modeling techniques where logarithms of quantities are generally preferred.    \n",
         "\n",
-        "The task is now to build a time-series model for the _Quantity_ column. It is important to note that this dataset is comprised of many individual time-series - one for each unique combination of _Store_ and _Brand_. To distinguish the individual time-series, we thus define the **grain** - the columns whose values determine the boundaries between time-series: "
+        "The task is now to build a time-series model for the _Quantity_ column. It is important to note that this dataset is comprised of many individual time-series - one for each unique combination of _Store_ and _Brand_. To distinguish the individual time-series, we define the **time_series_id_column_names** - the columns whose values determine the boundaries between time-series: "
       ]
     },
     {
@@ -187,8 +194,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "grain_column_names = ['Store', 'Brand']\n",
-        "nseries = data.groupby(grain_column_names).ngroups\n",
+        "time_series_id_column_names = ['Store', 'Brand']\n",
+        "nseries = data.groupby(time_series_id_column_names).ngroups\n",
         "print('Data contains {0} individual time-series.'.format(nseries))"
       ]
     },
@@ -207,7 +214,7 @@
       "source": [
         "use_stores = [2, 5, 8]\n",
         "data_subset = data[data.Store.isin(use_stores)]\n",
-        "nseries = data_subset.groupby(grain_column_names).ngroups\n",
+        "nseries = data_subset.groupby(time_series_id_column_names).ngroups\n",
         "print('Data subset contains {0} individual time-series.'.format(nseries))"
       ]
     },
@@ -216,7 +223,7 @@
       "metadata": {},
       "source": [
         "### Data Splitting\n",
-        "We now split the data into a training and a testing set for later forecast evaluation. The test set will contain the final 20 weeks of observed sales for each time-series. The splits should be stratified by series, so we use a group-by statement on the grain columns."
+        "We now split the data into a training and a testing set for later forecast evaluation. The test set will contain the final 20 weeks of observed sales for each time-series. The splits should be stratified by series, so we use a group-by statement on the time series identifier columns."
       ]
     },
     {
@@ -227,15 +234,15 @@
       "source": [
         "n_test_periods = 20\n",
         "\n",
-        "def split_last_n_by_grain(df, n):\n",
-        "    \"\"\"Group df by grain and split on last n rows for each group.\"\"\"\n",
+        "def split_last_n_by_series_id(df, n):\n",
+        "    \"\"\"Group df by series identifiers and split on last n rows for each group.\"\"\"\n",
         "    df_grouped = (df.sort_values(time_column_name) # Sort by ascending time\n",
-        "                  .groupby(grain_column_names, group_keys=False))\n",
+        "                  .groupby(time_series_id_column_names, group_keys=False))\n",
         "    df_head = df_grouped.apply(lambda dfg: dfg.iloc[:-n])\n",
         "    df_tail = df_grouped.apply(lambda dfg: dfg.iloc[-n:])\n",
         "    return df_head, df_tail\n",
         "\n",
-        "train, test = split_last_n_by_grain(data_subset, n_test_periods)"
+        "train, test = split_last_n_by_series_id(data_subset, n_test_periods)"
       ]
     },
     {
@@ -280,7 +287,8 @@
       "outputs": [],
       "source": [
         "from azureml.core.dataset import Dataset\n",
-        "train_dataset = Dataset.Tabular.from_delimited_files(path=datastore.path('dataset/dominicks_OJ_train.csv'))"
+        "train_dataset = Dataset.Tabular.from_delimited_files(path=datastore.path('dataset/dominicks_OJ_train.csv'))\n",
+        "test_dataset = Dataset.Tabular.from_delimited_files(path=datastore.path('dataset/dominicks_OJ_test.csv'))"
       ]
     },
     {
@@ -301,11 +309,11 @@
         "For forecasting tasks, AutoML uses pre-processing and estimation steps that are specific to time-series. AutoML will undertake the following pre-processing steps:\n",
         "* Detect time-series sample frequency (e.g. hourly, daily, weekly) and create new records for absent time points to make the series regular. A regular time series has a well-defined frequency and has a value at every sample point in a contiguous time span \n",
         "* Impute missing values in the target (via forward-fill) and feature columns (using median column values) \n",
-        "* Create grain-based features to enable fixed effects across different series\n",
+        "* Create features based on time series identifiers to enable fixed effects across different series\n",
         "* Create time-based features to assist in learning seasonal patterns\n",
         "* Encode categorical variables to numeric quantities\n",
         "\n",
-        "In this notebook, AutoML will train a single, regression-type model across **all** time-series in a given training set. This allows the model to generalize across related series. If you're looking for training multiple models for different time-series, please check out the forecasting grouping notebook. \n",
+        "In this notebook, AutoML will train a single, regression-type model across **all** time-series in a given training set. This allows the model to generalize across related series. If you're looking for training multiple models for different time-series, please see the many-models notebook.\n",
         "\n",
         "You are almost ready to start an AutoML training job. First, we need to separate the target column from the rest of the DataFrame: "
       ]
@@ -325,12 +333,11 @@
       "source": [
         "## Customization\n",
         "\n",
-        "The featurization customization in forecasting is an advanced feature in AutoML which allows our customers to change the default forecasting featurization behaviors and column types through `FeaturizationConfig`. The supported scenarios include,\n",
-        "1. Column purposes update: Override feature type for the specified column. Currently supports DateTime, Categorical and Numeric. This customization can be used in the scenario that the type of the column cannot correctly reflect its purpose. Some numerical columns, for instance, can be treated as Categorical columns which need to be converted to categorical while some can be treated as epoch timestamp which need to be converted to datetime. To tell our SDK to correctly preprocess these columns, a configuration need to be add with the columns and their desired types.\n",
-        "2. Transformer parameters update: Currently supports parameter change for Imputer only. User can customize imputation methods, the supported methods are constant for target data and mean, median, most frequent and constant for training data. This customization can be used for the scenario that our customers know which imputation methods fit best to the input data. For instance, some datasets use NaN to represent 0 which the correct behavior should impute all the missing value with 0. To achieve this behavior, these columns need to be configured as constant imputation with `fill_value` 0.\n",
-        "3. Drop columns: Columns to drop from being featurized. These usually are the columns which are leaky or the columns contain no useful data.\n",
+        "The featurization customization in forecasting is an advanced feature in AutoML which allows our customers to change the default forecasting featurization behaviors and column types through `FeaturizationConfig`. The supported scenarios include:\n",
         "\n",
-        "This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade)"
+        "1. Column purposes update: Override feature type for the specified column. Currently supports DateTime, Categorical and Numeric. This customization can be used in the scenario that the type of the column cannot correctly reflect its purpose. Some numerical columns, for instance, can be treated as Categorical columns which need to be converted to categorical while some can be treated as epoch timestamp which need to be converted to datetime. To tell our SDK to correctly preprocess these columns, a configuration need to be add with the columns and their desired types.\n",
+        "2. Transformer parameters update: Currently supports parameter change for Imputer only. User can customize imputation methods. The supported imputing methods for target column are constant and ffill (forward fill). The supported imputing methods for feature columns are mean, median, most frequent, constant and ffill (forward fill). This customization can be used for the scenario that our customers know which imputation methods fit best to the input data. For instance, some datasets use NaN to represent 0 which the correct behavior should impute all the missing value with 0. To achieve this behavior, these columns need to be configured as constant imputation with `fill_value` 0.\n",
+        "3. Drop columns: Columns to drop from being featurized. These usually are the columns which are leaky or the columns contain no useful data."
       ]
     },
     {
@@ -344,29 +351,46 @@
       "outputs": [],
       "source": [
         "featurization_config = FeaturizationConfig()\n",
-        "featurization_config.drop_columns = ['logQuantity']  # 'logQuantity' is a leaky feature, so we remove it.\n",
         "# Force the CPWVOL5 feature to be numeric type.\n",
         "featurization_config.add_column_purpose('CPWVOL5', 'Numeric')\n",
         "# Fill missing values in the target column, Quantity, with zeros.\n",
         "featurization_config.add_transformer_params('Imputer', ['Quantity'], {\"strategy\": \"constant\", \"fill_value\": 0})\n",
         "# Fill missing values in the INCOME column with median value.\n",
-        "featurization_config.add_transformer_params('Imputer', ['INCOME'], {\"strategy\": \"median\"})"
+        "featurization_config.add_transformer_params('Imputer', ['INCOME'], {\"strategy\": \"median\"})\n",
+        "# Fill missing values in the Price column with forward fill (last value carried forward).\n",
+        "featurization_config.add_transformer_params('Imputer', ['Price'], {\"strategy\": \"ffill\"})"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Train\n",
+        "## Forecasting Parameters\n",
+        "To define forecasting parameters for your experiment training, you can leverage the ForecastingParameters class. The table below details the forecasting parameter we will be passing into our experiment.\n",
+        "\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**time_column_name**|The name of your time column.|\n",
+        "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
+        "|**time_series_id_column_names**|The column names used to uniquely identify the time series in data that has multiple rows with the same timestamp. If the time series identifiers are not defined, the data set is assumed to be one time series.|\n",
+        "|**freq**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train<a id=\"train\"></a>\n",
         "\n",
         "The [AutoMLConfig](https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig?view=azure-ml-py) object defines the settings and data for an AutoML training job. Here, we set necessary inputs like the task type, the number of AutoML iterations to try, the training data, and cross-validation parameters.\n",
         "\n",
-        "For forecasting tasks, there are some additional parameters that can be set: the name of the column holding the date/time, the grain column names, and the maximum forecast horizon. A time column is required for forecasting, while the grain is optional. If grain columns are not given, AutoML assumes that the whole dataset is a single time-series. We also pass a list of columns to drop prior to modeling. The _logQuantity_ column is completely correlated with the target quantity, so it must be removed to prevent a target leak.\n",
+        "For forecasting tasks, there are some additional parameters that can be set in the `ForecastingParameters` class: the name of the column holding the date/time, the timeseries id column names, and the maximum forecast horizon. A time column is required for forecasting, while the time_series_id is optional. If time_series_id columns are not given, AutoML assumes that the whole dataset is a single time-series. We also pass a list of columns to drop prior to modeling. The _logQuantity_ column is completely correlated with the target quantity, so it must be removed to prevent a target leak.\n",
         "\n",
-        "The forecast horizon is given in units of the time-series frequency; for instance, the OJ series frequency is weekly, so a horizon of 20 means that a trained model will estimate sales up to 20 weeks beyond the latest date in the training data for each series. In this example, we set the maximum horizon to the number of samples per series in the test set (n_test_periods). Generally, the value of this parameter will be dictated by business needs. For example, a demand planning application that estimates the next month of sales should set the horizon according to suitable planning time-scales. Please see the [energy_demand notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand) for more discussion of forecast horizon.\n",
+        "The forecast horizon is given in units of the time-series frequency; for instance, the OJ series frequency is weekly, so a horizon of 20 means that a trained model will estimate sales up to 20 weeks beyond the latest date in the training data for each series. In this example, we set the forecast horizon to the number of samples per series in the test set (n_test_periods). Generally, the value of this parameter will be dictated by business needs. For example, a demand planning application that estimates the next month of sales should set the horizon according to suitable planning time-scales. Please see the [energy_demand notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand) for more discussion of forecast horizon.\n",
         "\n",
         "We note here that AutoML can sweep over two types of time-series models:\n",
-        "* Models that are trained for each series such as ARIMA and Facebook's Prophet. Note that these models are only available for [Enterprise Edition Workspaces](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-workspace#upgrade).\n",
+        "* Models that are trained for each series such as ARIMA and Facebook's Prophet.\n",
         "* Models trained across multiple time-series using a regression approach.\n",
         "\n",
         "In the first case, AutoML loops over all time-series in your dataset and trains one model (e.g. AutoArima or Prophet, as the case may be) for each series. This can result in long runtimes to train these models if there are a lot of series in the data. One way to mitigate this problem is to fit models for different series in parallel if you have multiple compute cores available. To enable this behavior, set the `max_cores_per_iteration` parameter in your AutoMLConfig as shown in the example in the next cell. \n",
@@ -389,11 +413,8 @@
         "|**enable_voting_ensemble**|Allow AutoML to create a Voting ensemble of the best performing models|\n",
         "|**enable_stack_ensemble**|Allow AutoML to create a Stack ensemble of the best performing models|\n",
         "|**debug_log**|Log file path for writing debugging information|\n",
-        "|**time_column_name**|Name of the datetime column in the input data|\n",
-        "|**grain_column_names**|Name(s) of the columns defining individual series in the input data|\n",
-        "|**max_horizon**|Maximum desired forecast horizon in units of time-series frequency|\n",
         "|**featurization**| 'auto' / 'off' / FeaturizationConfig Indicator for whether featurization step should be done automatically or not, or whether customized featurization should be used. Setting this enables AutoML to perform featurization on the input to handle *missing data*, and to perform some common *feature extraction*.|\n",
-        "|**max_cores_per_iteration**|Maximum number of cores to utilize per iteration. A value of -1 indicates all available cores should be used.|"
+        "|**max_cores_per_iteration**|Maximum number of cores to utilize per iteration. A value of -1 indicates all available cores should be used"
       ]
     },
     {
@@ -402,11 +423,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "time_series_settings = {\n",
-        "    'time_column_name': time_column_name,\n",
-        "    'grain_column_names': grain_column_names,\n",
-        "    'max_horizon': n_test_periods\n",
-        "}\n",
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name,\n",
+        "    forecast_horizon=n_test_periods,\n",
+        "    time_series_id_column_names=time_series_id_column_names,\n",
+        "    freq='W-THU' # Set the forecast frequency to be weekly (start on each Thursday)\n",
+        ")\n",
         "\n",
         "automl_config = AutoMLConfig(task='forecasting',\n",
         "                             debug_log='automl_oj_sales_errors.log',\n",
@@ -420,7 +443,7 @@
         "                             n_cross_validations=3,\n",
         "                             verbosity=logging.INFO,\n",
         "                             max_cores_per_iteration=-1,\n",
-        "                             **time_series_settings)"
+        "                             forecasting_parameters=forecasting_parameters)"
       ]
     },
     {
@@ -428,7 +451,7 @@
       "metadata": {},
       "source": [
         "You can now submit a new training run. Depending on the data and number of iterations this operation may take several minutes.\n",
-        "Information from each iteration will be printed to the console."
+        "Information from each iteration will be printed to the console.  Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -437,8 +460,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "remote_run = experiment.submit(automl_config, show_output=False)\n",
-        "remote_run"
+        "remote_run = experiment.submit(automl_config, show_output=False)"
       ]
     },
     {
@@ -500,9 +522,11 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Forecasting\n",
+        "# Forecast<a id=\"forecast\"></a>\n",
         "\n",
-        "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. First, we remove the target values from the test set:"
+        "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. We will do batch scoring on the test dataset which should have the same schema as training dataset.\n",
+        "\n",
+        "The inference will run on a remote compute. In this example, it will re-use the training compute."
       ]
     },
     {
@@ -511,17 +535,15 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X_test = test\n",
-        "y_test = X_test.pop(target_column_name).values"
+        "test_experiment = Experiment(ws, experiment_name + \"_inference\")"
       ]
     },
     {
-      "cell_type": "code",
-      "execution_count": null,
+      "cell_type": "markdown",
       "metadata": {},
-      "outputs": [],
       "source": [
-        "X_test.head()"
+        "### Retreiving forecasts from the model\n",
+        "We have created a function called `run_forecast` that submits the test data to the best model determined during the training run and retrieves forecasts. This function uses a helper script `forecasting_script` which is uploaded and expecuted on the remote compute."
       ]
     },
     {
@@ -537,19 +559,16 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# The featurized data, aligned to y, will also be returned.\n",
-        "# This contains the assumptions that were made in the forecast\n",
-        "# and helps align the forecast to the original data\n",
-        "y_predictions, X_trans = fitted_model.forecast(X_test)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If you are used to scikit pipelines, perhaps you expected `predict(X_test)`. However, forecasting requires a more general interface that also supplies the past target `y` values. Please use `forecast(X,y)` as `predict(X)` is reserved for internal purposes on forecasting models.\n",
+        "from run_forecast import run_remote_inference\n",
+        "remote_run_infer = run_remote_inference(test_experiment=test_experiment, \n",
+        "                                        compute_target=compute_target,\n",
+        "                                        train_run=best_run,\n",
+        "                                        test_dataset=test_dataset,\n",
+        "                                        target_column_name=target_column_name)\n",
+        "remote_run_infer.wait_for_completion(show_output=False)\n",
         "\n",
-        "The [forecast function notebook](../forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
+        "# download the forecast file to the local machine\n",
+        "remote_run_infer.download_file('outputs/predictions.csv', 'predictions.csv')"
       ]
     },
     {
@@ -558,9 +577,9 @@
       "source": [
         "# Evaluate\n",
         "\n",
-        "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE). \n",
+        "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE). For more metrics that can be used for evaluation after training, please see [supported metrics](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#regressionforecasting-metrics), and [how to calculate residuals](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#residuals).\n",
         "\n",
-        "It is a good practice to always align the output explicitly to the input, as the count and order of the rows may have changed during transformations that span multiple rows."
+        "We'll add predictions and actuals into a single dataframe for convenience in calculating the metrics."
       ]
     },
     {
@@ -569,9 +588,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from forecasting_helper import align_outputs\n",
-        "\n",
-        "df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)"
+        "# load forecast data frame\n",
+        "fcst_df = pd.read_csv('predictions.csv', parse_dates=[time_column_name])\n",
+        "fcst_df.head()"
       ]
     },
     {
@@ -586,8 +605,8 @@
         "\n",
         "# use automl scoring module\n",
         "scores = scoring.score_regression(\n",
-        "    y_test=df_all[target_column_name],\n",
-        "    y_pred=df_all['predicted'],\n",
+        "    y_test=fcst_df[target_column_name],\n",
+        "    y_pred=fcst_df['predicted'],\n",
         "    metrics=list(constants.Metric.SCALAR_REGRESSION_SET))\n",
         "\n",
         "print(\"[Test data scores]\\n\")\n",
@@ -596,8 +615,8 @@
         "    \n",
         "# Plot outputs\n",
         "%matplotlib inline\n",
-        "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
-        "test_test = plt.scatter(df_all[target_column_name], df_all[target_column_name], color='g')\n",
+        "test_pred = plt.scatter(fcst_df[target_column_name], fcst_df['predicted'], color='b')\n",
+        "test_test = plt.scatter(fcst_df[target_column_name], fcst_df[target_column_name], color='g')\n",
         "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
         "plt.show()"
       ]
@@ -606,7 +625,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Operationalize"
+        "# Operationalize<a id=\"operationalize\"></a>"
       ]
     },
     {
@@ -704,12 +723,13 @@
       "outputs": [],
       "source": [
         "import json\n",
-        "X_query = X_test.copy()\n",
+        "X_query = test.copy()\n",
+        "X_query.pop(target_column_name)\n",
         "# We have to convert datetime to string, because Timestamps cannot be serialized to JSON.\n",
         "X_query[time_column_name] = X_query[time_column_name].astype(str)\n",
         "# The Service object accept the complex dictionary, which is internally converted to JSON string.\n",
         "# The section 'data' contains the data frame in the form of dictionary.\n",
-        "test_sample = json.dumps({'data': X_query.to_dict(orient='records')})\n",
+        "test_sample = json.dumps({\"data\": json.loads(X_query.to_json(orient=\"records\"))})\n",
         "response = aci_service.run(input_data = test_sample)\n",
         "# translate from networkese to datascientese\n",
         "try: \n",
@@ -751,7 +771,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "erwright"
+        "name": "jialiu"
       }
     ],
     "category": "tutorial",
@@ -786,7 +806,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.8"
+      "version": "3.6.9"
     },
     "tags": [
       "None"
@@ -794,5 +814,5 @@
     "task": "Forecasting"
   },
   "nbformat": 4,
-  "nbformat_minor": 2
+  "nbformat_minor": 4
 }

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.yml

@@ -1,10 +1,4 @@
 name: auto-ml-forecasting-orange-juice-sales
 dependencies:
-- py-xgboost<=0.90
 - pip:
   - azureml-sdk
-  - numpy==1.16.2
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/forecasting_helper.py

@@ -1,98 +0,0 @@
-import pandas as pd
-import numpy as np
-from pandas.tseries.frequencies import to_offset
-
-
-def align_outputs(y_predicted, X_trans, X_test, y_test, target_column_name,
-                  predicted_column_name='predicted',
-                  horizon_colname='horizon_origin'):
-    """
-    Demonstrates how to get the output aligned to the inputs
-    using pandas indexes. Helps understand what happened if
-    the output's shape differs from the input shape, or if
-    the data got re-sorted by time and grain during forecasting.
-
-    Typical causes of misalignment are:
-    * we predicted some periods that were missing in actuals -> drop from eval
-    * model was asked to predict past max_horizon -> increase max horizon
-    * data at start of X_test was needed for lags -> provide previous periods
-    """
-
-    if (horizon_colname in X_trans):
-        df_fcst = pd.DataFrame({predicted_column_name: y_predicted,
-                                horizon_colname: X_trans[horizon_colname]})
-    else:
-        df_fcst = pd.DataFrame({predicted_column_name: y_predicted})
-
-    # y and X outputs are aligned by forecast() function contract
-    df_fcst.index = X_trans.index
-
-    # align original X_test to y_test
-    X_test_full = X_test.copy()
-    X_test_full[target_column_name] = y_test
-
-    # X_test_full's index does not include origin, so reset for merge
-    df_fcst.reset_index(inplace=True)
-    X_test_full = X_test_full.reset_index().drop(columns='index')
-    together = df_fcst.merge(X_test_full, how='right')
-
-    # drop rows where prediction or actuals are nan
-    # happens because of missing actuals
-    # or at edges of time due to lags/rolling windows
-    clean = together[together[[target_column_name,
-                               predicted_column_name]].notnull().all(axis=1)]
-    return(clean)
-
-
-def do_rolling_forecast(fitted_model, X_test, y_test, target_column_name, time_column_name, max_horizon, freq='D'):
-    """
-    Produce forecasts on a rolling origin over the given test set.
-
-    Each iteration makes a forecast for the next 'max_horizon' periods
-    with respect to the current origin, then advances the origin by the
-    horizon time duration. The prediction context for each forecast is set so
-    that the forecaster uses the actual target values prior to the current
-    origin time for constructing lag features.
-
-    This function returns a concatenated DataFrame of rolling forecasts.
-     """
-    df_list = []
-    origin_time = X_test[time_column_name].min()
-    while origin_time <= X_test[time_column_name].max():
-        # Set the horizon time - end date of the forecast
-        horizon_time = origin_time + max_horizon * to_offset(freq)
-
-        # Extract test data from an expanding window up-to the horizon
-        expand_wind = (X_test[time_column_name] < horizon_time)
-        X_test_expand = X_test[expand_wind]
-        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
-        y_query_expand.fill(np.NaN)
-
-        if origin_time != X_test[time_column_name].min():
-            # Set the context by including actuals up-to the origin time
-            test_context_expand_wind = (X_test[time_column_name] < origin_time)
-            context_expand_wind = (
-                X_test_expand[time_column_name] < origin_time)
-            y_query_expand[context_expand_wind] = y_test[
-                test_context_expand_wind]
-
-        # Make a forecast out to the maximum horizon
-        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
-
-        # Align forecast with test set for dates within the
-        # current rolling window
-        trans_tindex = X_trans.index.get_level_values(time_column_name)
-        trans_roll_wind = (trans_tindex >= origin_time) & (
-            trans_tindex < horizon_time)
-        test_roll_wind = expand_wind & (
-            X_test[time_column_name] >= origin_time)
-        df_list.append(align_outputs(y_fcst[trans_roll_wind],
-                                     X_trans[trans_roll_wind],
-                                     X_test[test_roll_wind],
-                                     y_test[test_roll_wind],
-                                     target_column_name))
-
-        # Advance the origin time
-        origin_time = horizon_time
-
-    return pd.concat(df_list, ignore_index=True)

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/forecasting_script.py

@@ -0,0 +1,89 @@
+"""
+This is the script that is executed on the compute instance. It relies
+on the model.pkl file which is uploaded along with this script to the
+compute instance.
+"""
+
+import argparse
+import pandas as pd
+import numpy as np
+from azureml.core import Dataset, Run
+from azureml.automl.core.shared.constants import TimeSeriesInternal
+from sklearn.externals import joblib
+from pandas.tseries.frequencies import to_offset
+
+
+def align_outputs(y_predicted, X_trans, X_test, y_test, target_column_name,
+                  predicted_column_name='predicted',
+                  horizon_colname='horizon_origin'):
+    """
+    Demonstrates how to get the output aligned to the inputs
+    using pandas indexes. Helps understand what happened if
+    the output's shape differs from the input shape, or if
+    the data got re-sorted by time and grain during forecasting.
+
+    Typical causes of misalignment are:
+    * we predicted some periods that were missing in actuals -> drop from eval
+    * model was asked to predict past max_horizon -> increase max horizon
+    * data at start of X_test was needed for lags -> provide previous periods
+    """
+
+    if (horizon_colname in X_trans):
+        df_fcst = pd.DataFrame({predicted_column_name: y_predicted,
+                                horizon_colname: X_trans[horizon_colname]})
+    else:
+        df_fcst = pd.DataFrame({predicted_column_name: y_predicted})
+
+    # y and X outputs are aligned by forecast() function contract
+    df_fcst.index = X_trans.index
+
+    # align original X_test to y_test
+    X_test_full = X_test.copy()
+    X_test_full[target_column_name] = y_test
+
+    # X_test_full's index does not include origin, so reset for merge
+    df_fcst.reset_index(inplace=True)
+    X_test_full = X_test_full.reset_index().drop(columns='index')
+    together = df_fcst.merge(X_test_full, how='right')
+
+    # drop rows where prediction or actuals are nan
+    # happens because of missing actuals
+    # or at edges of time due to lags/rolling windows
+    clean = together[together[[target_column_name,
+                               predicted_column_name]].notnull().all(axis=1)]
+    return(clean)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    '--target_column_name', type=str, dest='target_column_name',
+    help='Target Column Name')
+parser.add_argument(
+    '--test_dataset', type=str, dest='test_dataset',
+    help='Test Dataset')
+
+args = parser.parse_args()
+target_column_name = args.target_column_name
+test_dataset_id = args.test_dataset
+
+run = Run.get_context()
+ws = run.experiment.workspace
+
+# get the input dataset by id
+test_dataset = Dataset.get_by_id(ws, id=test_dataset_id)
+
+X_test = test_dataset.to_pandas_dataframe().reset_index(drop=True)
+y_test = X_test.pop(target_column_name).values
+
+# generate forecast
+fitted_model = joblib.load('model.pkl')
+y_predictions, X_trans = fitted_model.forecast(X_test)
+
+# align output
+df_all = align_outputs(y_predictions, X_trans, X_test, y_test, target_column_name)
+
+file_name = 'outputs/predictions.csv'
+export_csv = df_all.to_csv(file_name, header=True, index=False)  # added Index
+
+# Upload the predictions into artifacts
+run.upload_file(name=file_name, path_or_stream=file_name)

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/metrics_helper.py

@@ -1,22 +0,0 @@
-import pandas as pd
-import numpy as np
-
-
-def APE(actual, pred):
-    """
-    Calculate absolute percentage error.
-    Returns a vector of APE values with same length as actual/pred.
-    """
-    return 100 * np.abs((actual - pred) / actual)
-
-
-def MAPE(actual, pred):
-    """
-    Calculate mean absolute percentage error.
-    Remove NA and values where actual is close to zero
-    """
-    not_na = ~(np.isnan(actual) | np.isnan(pred))
-    not_zero = ~np.isclose(actual, 0.0)
-    actual_safe = actual[not_na & not_zero]
-    pred_safe = pred[not_na & not_zero]
-    return np.mean(APE(actual_safe, pred_safe))

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/run_forecast.py

@@ -0,0 +1,38 @@
+import os
+import shutil
+from azureml.core import ScriptRunConfig
+
+
+def run_remote_inference(test_experiment, compute_target, train_run,
+                         test_dataset, target_column_name, inference_folder='./forecast'):
+    # Create local directory to copy the model.pkl and forecsting_script.py files into.
+    # These files will be uploaded to and executed on the compute instance.
+    os.makedirs(inference_folder, exist_ok=True)
+    shutil.copy('forecasting_script.py', inference_folder)
+
+    train_run.download_file('outputs/model.pkl',
+                            os.path.join(inference_folder, 'model.pkl'))
+
+    inference_env = train_run.get_environment()
+
+    config = ScriptRunConfig(source_directory=inference_folder,
+                             script='forecasting_script.py',
+                             arguments=['--target_column_name',
+                                        target_column_name,
+                                        '--test_dataset',
+                                        test_dataset.as_named_input(test_dataset.name)],
+                             compute_target=compute_target,
+                             environment=inference_env)
+
+    run = test_experiment.submit(config,
+                                 tags={'training_run_id':
+                                       train_run.id,
+                                       'run_algorithm':
+                                       train_run.properties['run_algorithm'],
+                                       'valid_score':
+                                       train_run.properties['score'],
+                                       'primary_metric':
+                                       train_run.properties['primary_metric']})
+
+    run.log("run_algorithm", run.tags['run_algorithm'])
+    return run

how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb

@@ -80,7 +80,7 @@
         "from azureml.core.workspace import Workspace\n",
         "from azureml.core.dataset import Dataset\n",
         "from azureml.train.automl import AutoMLConfig\n",
-        "from azureml.explain.model._internal.explanation_client import ExplanationClient"
+        "from azureml.interpret import ExplanationClient"
       ]
     },
     {
@@ -96,7 +96,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -215,15 +215,6 @@
         "#local_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "local_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -354,12 +345,12 @@
       "metadata": {},
       "source": [
         "## Explanation\n",
-        "In this section, we will show how to compute model explanations and visualize the explanations using azureml-explain-model package. We will also show how to run the automl model and the explainer model through deploying an AKS web service.\n",
+        "In this section, we will show how to compute model explanations and visualize the explanations using azureml-interpret package. We will also show how to run the automl model and the explainer model through deploying an AKS web service.\n",
         "\n",
         "Besides retrieving an existing model explanation for an AutoML model, you can also explain your AutoML model with different test data. The following steps will allow you to compute and visualize engineered feature importance based on your test data.\n",
         "\n",
         "### Run the explanation\n",
-        "#### Download engineered feature importance from artifact store\n",
+        "#### Download the engineered feature importance from artifact store\n",
         "You can use ExplanationClient to download the engineered feature explanations from the artifact store of the best_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the engineered features."
       ]
     },
@@ -375,6 +366,25 @@
         "print(\"You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + best_run.get_portal_url())"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Download the raw feature importance from artifact store\n",
+        "You can use ExplanationClient to download the raw feature explanations from the artifact store of the best_run. You can also use azure portal url to view the dash board visualization of the feature importance values of the raw features."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "raw_explanations = client.download_model_explanation(raw=True)\n",
+        "print(raw_explanations.get_feature_importance_dict())\n",
+        "print(\"You can visualize the raw explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + best_run.get_portal_url())"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -426,7 +436,8 @@
         "\n",
         "automl_explainer_setup_obj = automl_setup_model_explanations(fitted_model, X=X_train, \n",
         "                                                             X_test=X_test, y=y_train, \n",
-        "                                                             task='classification')"
+        "                                                             task='classification',\n",
+        "                                                             automl_run=automl_run)"
       ]
     },
     {
@@ -434,7 +445,7 @@
       "metadata": {},
       "source": [
         "#### Initialize the Mimic Explainer for feature importance\n",
-        "For explaining the AutoML models, use the MimicWrapper from azureml.explain.model package. The MimicWrapper can be initialized with fields in automl_explainer_setup_obj, your workspace and a surrogate model to explain the AutoML model (fitted_model here). The MimicWrapper also takes the automl_run object where engineered explanations will be uploaded."
+        "For explaining the AutoML models, use the MimicWrapper from azureml-interpret package. The MimicWrapper can be initialized with fields in automl_explainer_setup_obj, your workspace and a surrogate model to explain the AutoML model (fitted_model here). The MimicWrapper also takes the automl_run object where engineered explanations will be uploaded."
       ]
     },
     {
@@ -443,10 +454,10 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.explain.model.mimic_wrapper import MimicWrapper\n",
+        "from azureml.interpret.mimic_wrapper import MimicWrapper\n",
         "explainer = MimicWrapper(ws, automl_explainer_setup_obj.automl_estimator,\n",
         "                         explainable_model=automl_explainer_setup_obj.surrogate_model, \n",
-        "                         init_dataset=automl_explainer_setup_obj.X_transform, run=automl_run,\n",
+        "                         init_dataset=automl_explainer_setup_obj.X_transform, run=automl_explainer_setup_obj.automl_run,\n",
         "                         features=automl_explainer_setup_obj.engineered_feature_names, \n",
         "                         feature_maps=[automl_explainer_setup_obj.feature_map],\n",
         "                         classes=automl_explainer_setup_obj.classes,\n",
@@ -473,6 +484,29 @@
         "print(\"You can visualize the engineered explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + automl_run.get_portal_url())"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Use Mimic Explainer for computing and visualizing raw feature importance\n",
+        "The explain() method in MimicWrapper can be called with the transformed test samples to get the feature importance for the original features in your data. You can also use azure portal url to view the dash board visualization of the feature importance values of the original/raw features."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Compute the raw explanations\n",
+        "raw_explanations = explainer.explain(['local', 'global'], get_raw=True,\n",
+        "                                     raw_feature_names=automl_explainer_setup_obj.raw_feature_names,\n",
+        "                                     eval_dataset=automl_explainer_setup_obj.X_test_transform,\n",
+        "                                     raw_eval_dataset=automl_explainer_setup_obj.X_test_raw)\n",
+        "print(raw_explanations.get_feature_importance_dict())\n",
+        "print(\"You can visualize the raw explanations under the 'Explanations (preview)' tab in the AutoML run at:-\\n\" + automl_run.get_portal_url())"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -486,7 +520,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.explain.model.scoring.scoring_explainer import TreeScoringExplainer\n",
+        "from azureml.interpret.scoring.scoring_explainer import TreeScoringExplainer\n",
         "import joblib\n",
         "\n",
         "# Initialize the ScoringExplainer\n",
@@ -507,7 +541,7 @@
       "source": [
         "### Deploying the scoring and explainer models to a web service to Azure Kubernetes Service (AKS)\n",
         "\n",
-        "We use the TreeScoringExplainer from azureml.explain.model package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service."
+        "We use the TreeScoringExplainer from azureml.interpret package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service."
       ]
     },
     {
@@ -529,7 +563,7 @@
       "source": [
         "#### Create the conda dependencies for setting up the service\n",
         "\n",
-        "We need to create the conda dependencies comprising of the azureml-explain-model, azureml-train-automl and azureml-defaults packages."
+        "We need to download the conda dependencies using the automl_run object."
       ]
     },
     {
@@ -561,16 +595,10 @@
       "outputs": [],
       "source": [
         "%%writefile score.py\n",
-        "import numpy as np\n",
-        "import pandas as pd\n",
-        "import os\n",
-        "import pickle\n",
-        "import azureml.train.automl\n",
-        "import azureml.explain.model\n",
-        "from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \\\n",
-        "    automl_setup_model_explanations\n",
         "import joblib\n",
+        "import pandas as pd\n",
         "from azureml.core.model import Model\n",
+        "from azureml.train.automl.runtime.automl_explain_utilities import automl_setup_model_explanations\n",
         "\n",
         "\n",
         "def init():\n",
@@ -594,10 +622,13 @@
         "    automl_explainer_setup_obj = automl_setup_model_explanations(automl_model,\n",
         "                                                                 X_test=data, task='classification')\n",
         "    # Retrieve model explanations for engineered explanations\n",
-        "    engineered_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform)    \n",
+        "    engineered_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform)\n",
+        "    # Retrieve model explanations for raw explanations\n",
+        "    raw_local_importance_values = scoring_explainer.explain(automl_explainer_setup_obj.X_test_transform, get_raw=True)\n",
         "    # You can return any data type as long as it is JSON-serializable\n",
         "    return {'predictions': predictions.tolist(),\n",
-        "            'engineered_local_importance_values': engineered_local_importance_values}\n"
+        "            'engineered_local_importance_values': engineered_local_importance_values,\n",
+        "            'raw_local_importance_values': raw_local_importance_values}\n"
       ]
     },
     {
@@ -730,7 +761,9 @@
         "# Print the predicted value\n",
         "print('predictions:\\n{}\\n'.format(output['predictions']))\n",
         "# Print the engineered feature importances for the predicted value\n",
-        "print('engineered_local_importance_values:\\n{}\\n'.format(output['engineered_local_importance_values']))"
+        "print('engineered_local_importance_values:\\n{}\\n'.format(output['engineered_local_importance_values']))\n",
+        "# Print the raw feature importances for the predicted value\n",
+        "print('raw_local_importance_values:\\n{}\\n'.format(output['raw_local_importance_values']))\n"
       ]
     },
     {
@@ -778,7 +811,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "anumamah"
+        "name": "ratanase"
       }
     ],
     "category": "tutorial",

how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.yml

@@ -2,6 +2,3 @@ name: auto-ml-classification-credit-card-fraud-local
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/auto-ml-regression-explanation-featurization.ipynb

@@ -42,8 +42,6 @@
         "\n",
         "If you are using an Azure Machine Learning Compute Instance, you are all set.  Otherwise, go through the [configuration](../../../configuration.ipynb)  notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
         "\n",
-        "An Enterprise workspace is required for this notebook. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade) \n",
-        "\n",
         "In this notebook you will learn how to:\n",
         "1. Create an `Experiment` in an existing `Workspace`.\n",
         "2. Instantiating AutoMLConfig with FeaturizationConfig for customization\n",
@@ -98,7 +96,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -132,6 +130,8 @@
         "### Create or Attach existing AmlCompute\n",
         "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you create `AmlCompute` as your training compute resource.\n",
         "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+        "\n",
         "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
         "\n",
         "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
@@ -154,7 +154,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=4)\n",
         "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
         "\n",
@@ -223,9 +223,8 @@
       "source": [
         "## Customization\n",
         "\n",
-        "This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade). \n",
-        "\n",
         "Supported customization includes:\n",
+        "\n",
         "1. Column purpose update: Override feature type for the specified column.\n",
         "2. Transformer parameter update: Update parameters for the specified transformer. Currently supports Imputer and HashOneHotEncoder.\n",
         "3. Drop columns: Columns to drop from being featurized.\n",
@@ -308,15 +307,6 @@
         "remote_run = experiment.submit(automl_config, show_output = False)"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -447,12 +437,11 @@
       "metadata": {},
       "source": [
         "## Explanations\n",
-        "This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspace#upgrade). \n",
         "This section will walk you through the workflow to compute model explanations for an AutoML model on your remote compute.\n",
         "\n",
         "### Retrieve any AutoML Model for explanations\n",
         "\n",
-        "Below we select the some AutoML pipeline from our iterations. The `get_output` method returns the a AutoML run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
+        "Below we select an AutoML pipeline from our iterations. The `get_output` method returns the a AutoML run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for any logged `metric` or for a particular `iteration`."
       ]
     },
     {
@@ -461,7 +450,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "automl_run, fitted_model = remote_run.get_output(metric='r2_score')"
+        "#automl_run, fitted_model = remote_run.get_output(metric='r2_score')\n",
+        "automl_run, fitted_model = remote_run.get_output(iteration=2)"
       ]
     },
     {
@@ -625,7 +615,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "from azureml.explain.model._internal.explanation_client import ExplanationClient\n",
+        "from azureml.interpret import ExplanationClient\n",
         "client = ExplanationClient.from_run(automl_run)\n",
         "engineered_explanations = client.download_model_explanation(raw=False, comment='engineered explanations')\n",
         "print(engineered_explanations.get_feature_importance_dict())\n",
@@ -655,11 +645,11 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Operationailze\n",
+        "## Operationalize\n",
         "In this section we will show how you can operationalize an AutoML model and the explainer which was used to compute the explanations in the previous section.\n",
         "\n",
         "### Register the AutoML model and the scoring explainer\n",
-        "We use the *TreeScoringExplainer* from *azureml.explain.model* package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. \n",
+        "We use the *TreeScoringExplainer* from *azureml-interpret* package to create the scoring explainer which will be used to compute the raw and engineered feature importances at the inference time. \n",
         "In the cell below, we register the AutoML model and the scoring explainer with the Model Management Service."
       ]
     },
@@ -681,7 +671,7 @@
       "metadata": {},
       "source": [
         "### Create the conda dependencies for setting up the service\n",
-        "We need to create the conda dependencies comprising of the *azureml-explain-model*, *azureml-train-automl* and *azureml-defaults* packages. "
+        "We need to create the conda dependencies comprising of the *azureml* packages using the training environment from the *automl_run*."
       ]
     },
     {
@@ -905,7 +895,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "anumamah"
+        "name": "anshirga"
       }
     ],
     "categories": [

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/auto-ml-regression-explanation-featurization.yml

@@ -2,6 +2,3 @@ name: auto-ml-regression-explanation-featurization
 dependencies:
 - pip:
   - azureml-sdk
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/score_explain.py

@@ -1,14 +1,7 @@
-import json
-import numpy as np
 import pandas as pd
-import os
-import pickle
-import azureml.train.automl
-import azureml.explain.model
-from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \
-    automl_setup_model_explanations
 import joblib
 from azureml.core.model import Model
+from azureml.train.automl.runtime.automl_explain_utilities import automl_setup_model_explanations
 
 
 def init():

how-to-use-azureml/automated-machine-learning/regression-explanation-featurization/train_explainer.py

@@ -1,17 +1,18 @@
 # Copyright (c) Microsoft. All rights reserved.
 # Licensed under the MIT license.
 import os
+import joblib
 
-from azureml.core.run import Run
+from interpret.ext.glassbox import LGBMExplainableModel
+from azureml.automl.core.shared.constants import MODEL_PATH
 from azureml.core.experiment import Experiment
 from azureml.core.dataset import Dataset
-from azureml.train.automl.runtime.automl_explain_utilities import AutoMLExplainerSetupClass, \
-    automl_setup_model_explanations, automl_check_model_if_explainable
-from azureml.explain.model.mimic.models.lightgbm_model import LGBMExplainableModel
-from azureml.explain.model.mimic_wrapper import MimicWrapper
-from azureml.automl.core.shared.constants import MODEL_PATH
-from azureml.explain.model.scoring.scoring_explainer import TreeScoringExplainer
-import joblib
+from azureml.core.run import Run
+from azureml.interpret.mimic_wrapper import MimicWrapper
+from azureml.interpret.scoring.scoring_explainer import TreeScoringExplainer
+from azureml.train.automl.runtime.automl_explain_utilities import automl_setup_model_explanations, \
+    automl_check_model_if_explainable
+
 
 OUTPUT_DIR = './outputs/'
 os.makedirs(OUTPUT_DIR, exist_ok=True)
@@ -26,7 +27,7 @@ automl_run = Run(experiment=experiment, run_id='<<run_id>>')
 
 # Check if this AutoML model is explainable
 if not automl_check_model_if_explainable(automl_run):
-    raise Exception("Model explanations is currently not supported for " + automl_run.get_properties().get(
+    raise Exception("Model explanations are currently not supported for " + automl_run.get_properties().get(
         'run_algorithm'))
 
 # Download the best model from the artifact store
@@ -37,23 +38,25 @@ fitted_model = joblib.load('model.pkl')
 
 # Get the train dataset from the workspace
 train_dataset = Dataset.get_by_name(workspace=ws, name='<<train_dataset_name>>')
-# Drop the lablled column to get the training set.
+# Drop the labeled column to get the training set.
 X_train = train_dataset.drop_columns(columns=['<<target_column_name>>'])
 y_train = train_dataset.keep_columns(columns=['<<target_column_name>>'], validate=True)
 
-# Get the train dataset from the workspace
+# Get the test dataset from the workspace
 test_dataset = Dataset.get_by_name(workspace=ws, name='<<test_dataset_name>>')
-# Drop the lablled column to get the testing set.
+# Drop the labeled column to get the testing set.
 X_test = test_dataset.drop_columns(columns=['<<target_column_name>>'])
 
-# Setup the class for explaining the AtuoML models
+# Setup the class for explaining the AutoML models
 automl_explainer_setup_obj = automl_setup_model_explanations(fitted_model, '<<task>>',
                                                              X=X_train, X_test=X_test,
-                                                             y=y_train)
+                                                             y=y_train,
+                                                             automl_run=automl_run)
 
 # Initialize the Mimic Explainer
 explainer = MimicWrapper(ws, automl_explainer_setup_obj.automl_estimator, LGBMExplainableModel,
-                         init_dataset=automl_explainer_setup_obj.X_transform, run=automl_run,
+                         init_dataset=automl_explainer_setup_obj.X_transform,
+                         run=automl_explainer_setup_obj.automl_run,
                          features=automl_explainer_setup_obj.engineered_feature_names,
                          feature_maps=[automl_explainer_setup_obj.feature_map],
                          classes=automl_explainer_setup_obj.classes)
@@ -65,7 +68,8 @@ engineered_explanations = explainer.explain(['local', 'global'], tag='engineered
 # Compute the raw explanations
 raw_explanations = explainer.explain(['local', 'global'], get_raw=True, tag='raw explanations',
                                      raw_feature_names=automl_explainer_setup_obj.raw_feature_names,
-                                     eval_dataset=automl_explainer_setup_obj.X_test_transform)
+                                     eval_dataset=automl_explainer_setup_obj.X_test_transform,
+                                     raw_eval_dataset=automl_explainer_setup_obj.X_test_raw)
 
 print("Engineered and raw explanations computed successfully")
 

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.ipynb

@@ -92,7 +92,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.8.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.32.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -145,7 +145,7 @@
         "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
         "    print('Found existing cluster, use it.')\n",
         "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
         "                                                           max_nodes=4)\n",
         "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
         "\n",
@@ -233,7 +233,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Execution of remote runs is asynchronous. Depending on the data and the number of iterations this can run for a while."
+        "Call the `submit` method on the experiment object and pass the run configuration. Execution of remote runs is asynchronous. Depending on the data and the number of iterations this can run for a while.  Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
       ]
     },
     {
@@ -256,15 +256,6 @@
         "#remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -375,18 +366,12 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# preview the first 3 rows of the dataset\n",
-        "\n",
-        "test_data = test_data.to_pandas_dataframe()\n",
-        "y_test = test_data['ERP'].fillna(0)\n",
-        "test_data = test_data.drop('ERP', 1)\n",
-        "test_data = test_data.fillna(0)\n",
+        "y_test = test_data.keep_columns('ERP').to_pandas_dataframe()\n",
+        "test_data = test_data.drop_columns('ERP').to_pandas_dataframe()\n",
         "\n",
         "\n",
-        "train_data = train_data.to_pandas_dataframe()\n",
-        "y_train = train_data['ERP'].fillna(0)\n",
-        "train_data = train_data.drop('ERP', 1)\n",
-        "train_data = train_data.fillna(0)\n"
+        "y_train = train_data.keep_columns('ERP').to_pandas_dataframe()\n",
+        "train_data = train_data.drop_columns('ERP').to_pandas_dataframe()\n"
       ]
     },
     {
@@ -396,10 +381,10 @@
       "outputs": [],
       "source": [
         "y_pred_train = fitted_model.predict(train_data)\n",
-        "y_residual_train = y_train - y_pred_train\n",
+        "y_residual_train = y_train.values - y_pred_train\n",
         "\n",
         "y_pred_test = fitted_model.predict(test_data)\n",
-        "y_residual_test = y_test - y_pred_test"
+        "y_residual_test = y_test.values - y_pred_test"
       ]
     },
     {
@@ -462,7 +447,7 @@
   "metadata": {
     "authors": [
       {
-        "name": "rakellam"
+        "name": "ratanase"
       }
     ],
     "categories": [

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.yml

@@ -2,7 +2,3 @@ name: auto-ml-regression
 dependencies:
 - pip:
   - azureml-sdk
-  - pandas==0.23.4
-  - azureml-train-automl
-  - azureml-widgets
-  - matplotlib

how-to-use-azureml/azure-databricks/README.md

@@ -1,33 +0,0 @@
-Azure Databricks is a managed Spark offering on Azure and customers already use it for advanced analytics. It provides a collaborative Notebook based environment with CPU or GPU based compute cluster. 
-
-In this section, you will find sample notebooks on how to use Azure Machine Learning SDK with Azure Databricks. You can train a model using Spark MLlib and then deploy the model to ACI/AKS from within Azure Databricks. You can also use Automated ML capability (**public preview**) of Azure ML SDK with Azure Databricks. 
-
-- Customers who use Azure Databricks for advanced analytics can now use the same cluster to run experiments with or without automated machine learning. 
-- You can keep the data within the same cluster. 
-- You can leverage the local worker nodes with autoscale and auto termination capabilities. 
-- You can use multiple cores of your Azure Databricks cluster to perform simultenous training. 
-- You can further tune the model generated by automated machine learning if you chose to. 
-- Every run (including the best run) is available as a pipeline, which you can tune further if needed. 
-- The model trained using Azure Databricks can be registered in Azure ML SDK workspace and then deployed to Azure managed compute (ACI or AKS) using the Azure Machine learning SDK.
-
-Please follow our [Azure doc](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-environment#azure-databricks) to install the sdk in your Azure Databricks cluster before trying any of the sample notebooks.
-
-**Single file** - 
-The following archive contains all the sample notebooks. You can the run notebooks after importing [DBC](Databricks_AMLSDK_1-4_6.dbc) in your Databricks workspace instead of downloading individually.
-
-Notebooks 1-4 have to be run sequentially & are related to Income prediction experiment based on this [dataset](https://archive.ics.uci.edu/ml/datasets/adult) and demonstrate how to data prep, train and operationalize a Spark ML model with Azure ML Python SDK from within Azure Databricks. 
-
-Notebook 6 is an Automated ML sample notebook for Classification.
-
-Learn more about [how to use Azure Databricks as a development environment](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment#azure-databricks) for Azure Machine Learning service.
-
-**Databricks as a Compute Target from AML Pipelines**
-You can use Azure Databricks as a compute target from [Azure Machine Learning Pipelines](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-ml-pipelines). Take a look at this notebook for details: [aml-pipelines-use-databricks-as-compute-target.ipynb](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.ipynb). 
-
-For more on SDK concepts, please refer to [notebooks](https://github.com/Azure/MachineLearningNotebooks).
-
-**Please let us know your feedback.**
-
- 
-
-![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/README.png) 

how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.ipynb

@@ -1,373 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Azure ML & Azure Databricks notebooks by Parashar Shah.\n",
-        "\n",
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Model Building"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "import pprint\n",
-        "import numpy as np\n",
-        "\n",
-        "from pyspark.ml import Pipeline, PipelineModel\n",
-        "from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler\n",
-        "from pyspark.ml.classification import LogisticRegression\n",
-        "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
-        "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import azureml.core\n",
-        "\n",
-        "# Check core SDK version number\n",
-        "print(\"SDK version:\", azureml.core.VERSION)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Set auth to be used by workspace related APIs.\n",
-        "# For automation or CI/CD ServicePrincipalAuthentication can be used.\n",
-        "# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.authentication.serviceprincipalauthentication?view=azure-ml-py\n",
-        "auth = None"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# import the Workspace class and check the azureml SDK version\n",
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.from_config(auth = auth)\n",
-        "print('Workspace name: ' + ws.name, \n",
-        "      'Azure region: ' + ws.location, \n",
-        "      'Subscription id: ' + ws.subscription_id, \n",
-        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#get the train and test datasets\n",
-        "train_data_path = \"AdultCensusIncomeTrain\"\n",
-        "test_data_path = \"AdultCensusIncomeTest\"\n",
-        "\n",
-        "train = spark.read.parquet(train_data_path)\n",
-        "test = spark.read.parquet(test_data_path)\n",
-        "\n",
-        "print(\"train: ({}, {})\".format(train.count(), len(train.columns)))\n",
-        "print(\"test: ({}, {})\".format(test.count(), len(test.columns)))\n",
-        "\n",
-        "train.printSchema()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Define Model"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "label = \"income\"\n",
-        "dtypes = dict(train.dtypes)\n",
-        "dtypes.pop(label)\n",
-        "\n",
-        "si_xvars = []\n",
-        "ohe_xvars = []\n",
-        "featureCols = []\n",
-        "for idx,key in enumerate(dtypes):\n",
-        "    if dtypes[key] == \"string\":\n",
-        "        featureCol = \"-\".join([key, \"encoded\"])\n",
-        "        featureCols.append(featureCol)\n",
-        "        \n",
-        "        tmpCol = \"-\".join([key, \"tmp\"])\n",
-        "        # string-index and one-hot encode the string column\n",
-        "        #https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/ml/feature/StringIndexer.html\n",
-        "        #handleInvalid: Param for how to handle invalid data (unseen labels or NULL values). \n",
-        "        #Options are 'skip' (filter out rows with invalid data), 'error' (throw an error), \n",
-        "        #or 'keep' (put invalid data in a special additional bucket, at index numLabels). Default: \"error\"\n",
-        "        si_xvars.append(StringIndexer(inputCol=key, outputCol=tmpCol, handleInvalid=\"skip\"))\n",
-        "        ohe_xvars.append(OneHotEncoder(inputCol=tmpCol, outputCol=featureCol))\n",
-        "    else:\n",
-        "        featureCols.append(key)\n",
-        "\n",
-        "# string-index the label column into a column named \"label\"\n",
-        "si_label = StringIndexer(inputCol=label, outputCol='label')\n",
-        "\n",
-        "# assemble the encoded feature columns in to a column named \"features\"\n",
-        "assembler = VectorAssembler(inputCols=featureCols, outputCol=\"features\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.run import Run\n",
-        "from azureml.core.experiment import Experiment\n",
-        "import numpy as np\n",
-        "import os\n",
-        "import shutil\n",
-        "\n",
-        "model_name = \"AdultCensus_runHistory.mml\"\n",
-        "model_dbfs = os.path.join(\"/dbfs\", model_name)\n",
-        "run_history_name = 'spark-ml-notebook'\n",
-        "\n",
-        "# start a training run by defining an experiment\n",
-        "myexperiment = Experiment(ws, \"Ignite_AI_Talk\")\n",
-        "root_run = myexperiment.start_logging()\n",
-        "\n",
-        "# Regularization Rates - \n",
-        "regs = [0.0001, 0.001, 0.01, 0.1]\n",
-        " \n",
-        "# try a bunch of regularization rate in a Logistic Regression model\n",
-        "for reg in regs:\n",
-        "    print(\"Regularization rate: {}\".format(reg))\n",
-        "    # create a bunch of child runs\n",
-        "    with root_run.child_run(\"reg-\" + str(reg)) as run:\n",
-        "        # create a new Logistic Regression model.\n",
-        "        lr = LogisticRegression(regParam=reg)\n",
-        "        \n",
-        "        # put together the pipeline\n",
-        "        pipe = Pipeline(stages=[*si_xvars, *ohe_xvars, si_label, assembler, lr])\n",
-        "\n",
-        "        # train the model\n",
-        "        model_p = pipe.fit(train)\n",
-        "        \n",
-        "        # make prediction\n",
-        "        pred = model_p.transform(test)\n",
-        "        \n",
-        "        # evaluate. note only 2 metrics are supported out of the box by Spark ML.\n",
-        "        bce = BinaryClassificationEvaluator(rawPredictionCol='rawPrediction')\n",
-        "        au_roc = bce.setMetricName('areaUnderROC').evaluate(pred)\n",
-        "        au_prc = bce.setMetricName('areaUnderPR').evaluate(pred)\n",
-        "\n",
-        "        print(\"Area under ROC: {}\".format(au_roc))\n",
-        "        print(\"Area Under PR: {}\".format(au_prc))\n",
-        "      \n",
-        "        # log reg, au_roc, au_prc and feature names in run history\n",
-        "        run.log(\"reg\", reg)\n",
-        "        run.log(\"au_roc\", au_roc)\n",
-        "        run.log(\"au_prc\", au_prc)\n",
-        "        run.log_list(\"columns\", train.columns)\n",
-        "\n",
-        "        # save model\n",
-        "        model_p.write().overwrite().save(model_name)\n",
-        "        \n",
-        "        # upload the serialized model into run history record\n",
-        "        mdl, ext = model_name.split(\".\")\n",
-        "        model_zip = mdl + \".zip\"\n",
-        "        shutil.make_archive(mdl, 'zip', model_dbfs)\n",
-        "        run.upload_file(\"outputs/\" + model_name, model_zip)        \n",
-        "        #run.upload_file(\"outputs/\" + model_name, path_or_stream = model_dbfs) #cannot deal with folders\n",
-        "\n",
-        "        # now delete the serialized model from local folder since it is already uploaded to run history \n",
-        "        shutil.rmtree(model_dbfs)\n",
-        "        os.remove(model_zip)\n",
-        "        \n",
-        "# Declare run completed\n",
-        "root_run.complete()\n",
-        "root_run_id = root_run.id\n",
-        "print (\"run id:\", root_run.id)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "metrics = root_run.get_metrics(recursive=True)\n",
-        "best_run_id = max(metrics, key = lambda k: metrics[k]['au_roc'])\n",
-        "print(best_run_id, metrics[best_run_id]['au_roc'], metrics[best_run_id]['reg'])"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#Get the best run\n",
-        "child_runs = {}\n",
-        "\n",
-        "for r in root_run.get_children():\n",
-        "    child_runs[r.id] = r\n",
-        "   \n",
-        "best_run = child_runs[best_run_id]"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#Download the model from the best run to a local folder\n",
-        "best_model_file_name = \"best_model.zip\"\n",
-        "best_run.download_file(name = 'outputs/' + model_name, output_file_path = best_model_file_name)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Model Evaluation"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "##unzip the model to dbfs (as load() seems to require that) and load it.\n",
-        "if os.path.isfile(model_dbfs) or os.path.isdir(model_dbfs):\n",
-        "    shutil.rmtree(model_dbfs)\n",
-        "shutil.unpack_archive(best_model_file_name, model_dbfs)\n",
-        "\n",
-        "model_p_best = PipelineModel.load(model_name)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# make prediction\n",
-        "pred = model_p_best.transform(test)\n",
-        "output = pred[['hours_per_week','age','workclass','marital_status','income','prediction']]\n",
-        "display(output.limit(5))"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# evaluate. note only 2 metrics are supported out of the box by Spark ML.\n",
-        "bce = BinaryClassificationEvaluator(rawPredictionCol='rawPrediction')\n",
-        "au_roc = bce.setMetricName('areaUnderROC').evaluate(pred)\n",
-        "au_prc = bce.setMetricName('areaUnderPR').evaluate(pred)\n",
-        "\n",
-        "print(\"Area under ROC: {}\".format(au_roc))\n",
-        "print(\"Area Under PR: {}\".format(au_prc))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Model Persistence"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "##NOTE: by default the model is saved to and loaded from /dbfs/ instead of cwd!\n",
-        "model_p_best.write().overwrite().save(model_name)\n",
-        "print(\"saved model to {}\".format(model_dbfs))"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "%sh\n",
-        "\n",
-        "ls -la /dbfs/AdultCensus_runHistory.mml/*"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "dbutils.notebook.exit(\"success\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/build-model-run-history-03.png)"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "pasha"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.6"
-    },
-    "name": "build-model-run-history-03",
-    "notebookId": 3836944406456339
-  },
-  "nbformat": 4,
-  "nbformat_minor": 1
-}

how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.ipynb

@@ -1,320 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Azure ML & Azure Databricks notebooks by Parashar Shah.\n",
-        "\n",
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Register Azure Databricks trained model and deploy it to ACI\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Please ensure you have run all previous notebooks in sequence before running this.\n",
-        "\n",
-        "Please Register Azure Container Instance(ACI) using Azure Portal: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-supported-services#portal in your subscription before using the SDK to deploy your ML model to ACI."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import azureml.core\n",
-        "\n",
-        "# Check core SDK version number\n",
-        "print(\"SDK version:\", azureml.core.VERSION)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Set auth to be used by workspace related APIs.\n",
-        "# For automation or CI/CD ServicePrincipalAuthentication can be used.\n",
-        "# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.authentication.serviceprincipalauthentication?view=azure-ml-py\n",
-        "auth = None"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.from_config(auth = auth)\n",
-        "print('Workspace name: ' + ws.name, \n",
-        "      'Azure region: ' + ws.location, \n",
-        "      'Subscription id: ' + ws.subscription_id, \n",
-        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "##NOTE: service deployment always gets the model from the current working dir.\n",
-        "import os\n",
-        "\n",
-        "model_name = \"AdultCensus_runHistory.mml\" # \n",
-        "model_name_dbfs = os.path.join(\"/dbfs\", model_name)\n",
-        "\n",
-        "print(\"copy model from dbfs to local\")\n",
-        "model_local = \"file:\" + os.getcwd() + \"/\" + model_name\n",
-        "dbutils.fs.cp(model_name, model_local, True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#Register the model\n",
-        "from azureml.core.model import Model\n",
-        "mymodel = Model.register(model_path = model_name, # this points to a local file\n",
-        "                       model_name = model_name, # this is the name the model is registered as, am using same name for both path and name.                 \n",
-        "                       description = \"ADB trained model by Parashar\",\n",
-        "                       workspace = ws)\n",
-        "\n",
-        "print(mymodel.name, mymodel.description, mymodel.version)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#%%writefile score_sparkml.py\n",
-        "score_sparkml = \"\"\"\n",
-        " \n",
-        "import json\n",
-        " \n",
-        "def init():\n",
-        "    # One-time initialization of PySpark and predictive model\n",
-        "    import pyspark\n",
-        "    import os\n",
-        "    from azureml.core.model import Model\n",
-        "    from pyspark.ml import PipelineModel\n",
-        " \n",
-        "    global trainedModel\n",
-        "    global spark\n",
-        " \n",
-        "    spark = pyspark.sql.SparkSession.builder.appName(\"ADB and AML notebook by Parashar\").getOrCreate()\n",
-        "    model_name = \"{model_name}\" #interpolated\n",
-        "    # AZUREML_MODEL_DIR is an environment variable created during deployment.\n",
-        "    # It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)\n",
-        "    # For multiple models, it points to the folder containing all deployed models (./azureml-models)\n",
-        "    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), model_name)\n",
-        "    trainedModel = PipelineModel.load(model_path)\n",
-        "    \n",
-        "def run(input_json):\n",
-        "    if isinstance(trainedModel, Exception):\n",
-        "        return json.dumps({{\"trainedModel\":str(trainedModel)}})\n",
-        "      \n",
-        "    try:\n",
-        "        sc = spark.sparkContext\n",
-        "        input_list = json.loads(input_json)\n",
-        "        input_rdd = sc.parallelize(input_list)\n",
-        "        input_df = spark.read.json(input_rdd)\n",
-        "    \n",
-        "        # Compute prediction\n",
-        "        prediction = trainedModel.transform(input_df)\n",
-        "        #result = prediction.first().prediction\n",
-        "        predictions = prediction.collect()\n",
-        " \n",
-        "        #Get each scored result\n",
-        "        preds = [str(x['prediction']) for x in predictions]\n",
-        "        result = \",\".join(preds)\n",
-        "        # you can return any data type as long as it is JSON-serializable\n",
-        "        return result.tolist()\n",
-        "    except Exception as e:\n",
-        "        result = str(e)\n",
-        "        return result\n",
-        "    \n",
-        "\"\"\".format(model_name=model_name)\n",
-        " \n",
-        "exec(score_sparkml)\n",
-        " \n",
-        "with open(\"score_sparkml.py\", \"w\") as file:\n",
-        "    file.write(score_sparkml)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.conda_dependencies import CondaDependencies \n",
-        "\n",
-        "myacienv = CondaDependencies.create(conda_packages=['scikit-learn','numpy','pandas']) # showing how to add libs as an eg. - not needed for this model.\n",
-        "\n",
-        "with open(\"myenv.yml\",\"w\") as f:\n",
-        "    f.write(myacienv.serialize_to_string())"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#deploy to ACI\n",
-        "from azureml.core.webservice import AciWebservice, Webservice\n",
-        "from azureml.exceptions import WebserviceException\n",
-        "from azureml.core.model import InferenceConfig\n",
-        "from azureml.core.environment import Environment\n",
-        "from azureml.core.conda_dependencies import CondaDependencies\n",
-        "\n",
-        "\n",
-        "myaci_config = AciWebservice.deploy_configuration(cpu_cores = 2, \n",
-        "                                                  memory_gb = 2, \n",
-        "                                                  tags = {'name':'Databricks Azure ML ACI'}, \n",
-        "                                                  description = 'This is for ADB and AML example.')\n",
-        "\n",
-        "service_name = 'aciws'\n",
-        "\n",
-        "# Remove any existing service under the same name.\n",
-        "try:\n",
-        "    Webservice(ws, service_name).delete()\n",
-        "except WebserviceException:\n",
-        "    pass\n",
-        "\n",
-        "myenv = Environment.get(ws, name='AzureML-PySpark-MmlSpark-0.15')\n",
-        "# we need to add extra packages to procured environment\n",
-        "# in order to deploy amended environment we need to rename it\n",
-        "myenv.name = 'myenv'\n",
-        "model_dependencies = CondaDependencies('myenv.yml')\n",
-        "for pip_dep in model_dependencies.pip_packages:\n",
-        "    myenv.python.conda_dependencies.add_pip_package(pip_dep)\n",
-        "for conda_dep in model_dependencies.conda_packages:\n",
-        "    myenv.python.conda_dependencies.add_conda_package(conda_dep)\n",
-        "inference_config = InferenceConfig(entry_script='score_sparkml.py', environment=myenv)\n",
-        "\n",
-        "myservice = Model.deploy(ws, service_name, [mymodel], inference_config, myaci_config)\n",
-        "myservice.wait_for_deployment(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "help(Webservice)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#for using the Web HTTP API \n",
-        "print(myservice.scoring_uri)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import json\n",
-        "\n",
-        "#get the some sample data\n",
-        "test_data_path = \"AdultCensusIncomeTest\"\n",
-        "test = spark.read.parquet(test_data_path).limit(5)\n",
-        "\n",
-        "test_json = json.dumps(test.toJSON().collect())\n",
-        "\n",
-        "print(test_json)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#using data defined above predict if income is >50K (1) or <=50K (0)\n",
-        "myservice.run(input_data=test_json)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#comment to not delete the web service\n",
-        "myservice.delete()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Deploying to other types of computes\n",
-        "\n",
-        "In order to learn how to deploy to other types of compute targets, such as AKS, please take a look at the set of notebooks in the [deployment](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/deployment) folder."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/deploy-to-aci-04.png)"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "pasha"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.8"
-    },
-    "name": "deploy-to-aci-04",
-    "notebookId": 3836944406456376
-  },
-  "nbformat": 4,
-  "nbformat_minor": 1
-}

how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.ipynb

@@ -1,179 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Azure ML & Azure Databricks notebooks by Parashar Shah.\n",
-        "\n",
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Data Ingestion"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "import urllib"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Download AdultCensusIncome.csv from Azure CDN. This file has 32,561 rows.\n",
-        "dataurl = \"https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv\"\n",
-        "datafile = \"AdultCensusIncome.csv\"\n",
-        "datafile_dbfs = os.path.join(\"/dbfs\", datafile)\n",
-        "\n",
-        "if os.path.isfile(datafile_dbfs):\n",
-        "    print(\"found {} at {}\".format(datafile, datafile_dbfs))\n",
-        "else:\n",
-        "    print(\"downloading {} to {}\".format(datafile, datafile_dbfs))\n",
-        "    urllib.request.urlretrieve(dataurl, datafile_dbfs)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Create a Spark dataframe out of the csv file.\n",
-        "data_all = sqlContext.read.format('csv').options(header='true', inferSchema='true', ignoreLeadingWhiteSpace='true', ignoreTrailingWhiteSpace='true').load(datafile)\n",
-        "print(\"({}, {})\".format(data_all.count(), len(data_all.columns)))\n",
-        "data_all.printSchema()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#renaming columns\n",
-        "columns_new = [col.replace(\"-\", \"_\") for col in data_all.columns]\n",
-        "data_all = data_all.toDF(*columns_new)\n",
-        "data_all.printSchema()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "display(data_all.limit(5))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Data Preparation"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Choose feature columns and the label column.\n",
-        "label = \"income\"\n",
-        "xvars = set(data_all.columns) - {label}\n",
-        "\n",
-        "print(\"label = {}\".format(label))\n",
-        "print(\"features = {}\".format(xvars))\n",
-        "\n",
-        "data = data_all.select([*xvars, label])\n",
-        "\n",
-        "# Split data into train and test.\n",
-        "train, test = data.randomSplit([0.75, 0.25], seed=123)\n",
-        "\n",
-        "print(\"train ({}, {})\".format(train.count(), len(train.columns)))\n",
-        "print(\"test ({}, {})\".format(test.count(), len(test.columns)))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#Data Persistence"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Write the train and test data sets to intermediate storage\n",
-        "train_data_path = \"AdultCensusIncomeTrain\"\n",
-        "test_data_path = \"AdultCensusIncomeTest\"\n",
-        "\n",
-        "train_data_path_dbfs = os.path.join(\"/dbfs\", \"AdultCensusIncomeTrain\")\n",
-        "test_data_path_dbfs = os.path.join(\"/dbfs\", \"AdultCensusIncomeTest\")\n",
-        "\n",
-        "train.write.mode('overwrite').parquet(train_data_path)\n",
-        "test.write.mode('overwrite').parquet(test_data_path)\n",
-        "print(\"train and test datasets saved to {} and {}\".format(train_data_path_dbfs, test_data_path_dbfs))"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": []
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/ingest-data-02.png)"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "pasha"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.6"
-    },
-    "name": "ingest-data-02",
-    "notebookId": 3836944406456362
-  },
-  "nbformat": 4,
-  "nbformat_minor": 1
-}

how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.ipynb

@@ -1,183 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Azure ML & Azure Databricks notebooks by Parashar Shah.\n",
-        "\n",
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "We support installing AML SDK as library from GUI. When attaching a library follow this https://docs.databricks.com/user-guide/libraries.html and add the below string as your PyPi package. You can select the option to attach the library to all clusters or just one cluster.\n",
-        "\n",
-        "**install azureml-sdk**\n",
-        "* Source: Upload Python Egg or PyPi\n",
-        "* PyPi Name: `azureml-sdk[databricks]`\n",
-        "* Select Install Library"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import azureml.core\n",
-        "\n",
-        "# Check core SDK version number - based on build number of preview/master.\n",
-        "print(\"SDK version:\", azureml.core.VERSION)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Please specify the Azure subscription Id, resource group name, workspace name, and the region in which you want to create the Azure Machine Learning Workspace.\n",
-        "\n",
-        "You can get the value of your Azure subscription ID from the Azure Portal, and then selecting Subscriptions from the menu on the left.\n",
-        "\n",
-        "For the resource_group, use the name of the resource group that contains your Azure Databricks Workspace.\n",
-        "\n",
-        "NOTE: If you provide a resource group name that does not exist, the resource group will be automatically created. This may or may not succeed in your environment, depending on the permissions you have on your Azure Subscription."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# subscription_id = \"<your-subscription-id>\"\n",
-        "# resource_group = \"<your-existing-resource-group>\"\n",
-        "# workspace_name = \"<a-new-or-existing-workspace; it is unrelated to Databricks workspace>\"\n",
-        "# workspace_region = \"<your-resource group-region>\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Set auth to be used by workspace related APIs.\n",
-        "# For automation or CI/CD ServicePrincipalAuthentication can be used.\n",
-        "# https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.authentication.serviceprincipalauthentication?view=azure-ml-py\n",
-        "auth = None"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# import the Workspace class and check the azureml SDK version\n",
-        "# exist_ok checks if workspace exists or not.\n",
-        "\n",
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.create(name = workspace_name,\n",
-        "                      subscription_id = subscription_id,\n",
-        "                      resource_group = resource_group, \n",
-        "                      location = workspace_region,\n",
-        "                      auth = auth,\n",
-        "                      exist_ok=True)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "#get workspace details\n",
-        "ws.get_details()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace(workspace_name = workspace_name,\n",
-        "               subscription_id = subscription_id,\n",
-        "               resource_group = resource_group,\n",
-        "               auth = auth)\n",
-        "\n",
-        "# persist the subscription id, resource group name, and workspace name in aml_config/config.json.\n",
-        "ws.write_config()\n",
-        "#if you need to give a different path/filename please use this\n",
-        "#write_config(path=\"/databricks/driver/aml_config/\",file_name=<alias_conf.cfg>)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "help(Workspace)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# import the Workspace class and check the azureml SDK version\n",
-        "from azureml.core import Workspace\n",
-        "\n",
-        "ws = Workspace.from_config(auth = auth)\n",
-        "#ws = Workspace.from_config(<full path>)\n",
-        "print('Workspace name: ' + ws.name, \n",
-        "      'Azure region: ' + ws.location, \n",
-        "      'Subscription id: ' + ws.subscription_id, \n",
-        "      'Resource group: ' + ws.resource_group, sep = '\\n')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/amlsdk/installation-and-configuration-01.png)"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "pasha"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.6"
-    },
-    "name": "installation-and-configuration-01",
-    "notebookId": 3688394266452835
-  },
-  "nbformat": 4,
-  "nbformat_minor": 1
-}

how-to-use-azureml/azure-databricks/automl/README.md

@@ -0,0 +1,70 @@
+# Automated ML introduction
+Automated machine learning (automated ML) builds high quality machine learning models for you by automating model and hyperparameter selection. Bring a labelled dataset that you want to build a model for, automated ML will give you a high quality machine learning model that you can use for predictions.
+
+
+If you are new to Data Science, automated ML will help you get jumpstarted by simplifying machine learning model building. It abstracts you from needing to perform model selection, hyperparameter selection and in one step creates a high quality trained model for you to use.
+
+If you are an experienced data scientist, automated ML will help increase your productivity by intelligently performing the model and hyperparameter selection for your training and generates high quality models much quicker than manually specifying several combinations of the parameters and running training jobs. Automated ML provides visibility and access to all the training jobs and the performance characteristics of the models to help you further tune the pipeline if you desire.
+
+# Install Instructions using Azure Databricks :
+
+#### For Databricks non ML runtime 7.1(scala 2.21, spark 3.0.0) and up, Install Automated Machine Learning sdk by adding and running the following command as the first cell of your notebook. This will install AutoML dependencies specific for your notebook.
+
+%pip install --upgrade --force-reinstall -r https://aka.ms/automl_linux_requirements.txt
+
+
+#### For Databricks non ML runtime 7.0 and lower, Install Automated Machine Learning sdk using init script as shown below before running the notebook.**
+
+**Create the Azure Databricks cluster-scoped init script 'azureml-cluster-init.sh' as below
+
+1. Create the base directory you want to store the init script in if it does not exist.
+    ```
+    dbutils.fs.mkdirs("dbfs:/databricks/init/")
+    ```
+
+2. Create the script azureml-cluster-init.sh
+    ```
+    dbutils.fs.put("/databricks/init/azureml-cluster-init.sh","""
+    #!/bin/bash
+	set -ex
+	/databricks/python/bin/pip install --upgrade --force-reinstall -r https://aka.ms/automl_linux_requirements.txt
+    """, True)
+    ```
+
+3. Check that the script exists.
+    ```
+    display(dbutils.fs.ls("dbfs:/databricks/init/azureml-cluster-init.sh"))
+    ```
+
+**Install libraries to cluster using init script 'azureml-cluster-init.sh' created in previous step
+
+1. Configure the cluster to run the script.
+    * Using the cluster configuration page
+        1. On the cluster configuration page, click the Advanced Options toggle.
+        1. At the bottom of the page, click the Init Scripts tab.
+        1. In the Destination drop-down, select a destination type. Example: 'DBFS'
+        1. Specify a path to the init script.
+            ```
+            dbfs:/databricks/init/azureml-cluster-init.sh
+            ```
+        1. Click Add
+
+    * Using the API.
+        ```
+        curl -n -X POST -H 'Content-Type: application/json' -d '{
+        "cluster_id": "<cluster_id>",
+        "num_workers": <num_workers>,
+        "spark_version": "<spark_version>",
+        "node_type_id": "<node_type_id>",
+        "cluster_log_conf": {
+            "dbfs" : {
+            "destination": "dbfs:/cluster-logs"
+            }
+        },
+        "init_scripts": [ {
+            "dbfs": {
+            "destination": "dbfs:/databricks/init/azureml-cluster-init.sh"
+            }
+        } ]
+        }' https://<databricks-instance>/api/2.0/clusters/edit
+        ```

how-to-use-azureml/azure-databricks/automl/automl-databricks-local-01.ipynb

@@ -13,32 +13,46 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "# Automated ML on Azure Databricks\n",
+        "## AutoML Installation\n",
         "\n",
-        "In this example we use the scikit-learn's <a href=\"http://scikit-learn.org/stable/datasets/index.html#optical-recognition-of-handwritten-digits-dataset\" target=\"_blank\">digit dataset</a> to showcase how you can use AutoML for a simple classification problem.\n",
+        "**For Databricks non ML runtime 7.1(scala 2.21, spark 3.0.0) and up, Install AML sdk by running the following command in the first cell of the notebook.**\n",
         "\n",
-        "In this notebook you will learn how to:\n",
-        "1. Create Azure Machine Learning Workspace object and initialize your notebook directory to easily reload this object from a configuration file.\n",
-        "2. Create an `Experiment` in an existing `Workspace`.\n",
-        "3. Configure Automated ML using `AutoMLConfig`.\n",
-        "4. Train the model using Azure Databricks.\n",
-        "5. Explore the results.\n",
-        "6. Viewing the engineered names for featurized data and featurization summary for all raw features.\n",
-        "7. Test the best fitted model.\n",
+        "%pip install --upgrade --force-reinstall -r https://aka.ms/automl_linux_requirements.txt\n",
         "\n",
-        "Before running this notebook, please follow the <a href=\"https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/azure-databricks\" target=\"_blank\">readme for using Automated ML on Azure Databricks</a> for installing necessary libraries to your cluster."
+        "**For Databricks non ML runtime 7.0 and lower, Install AML sdk using init script as shown in [readme](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/azure-databricks/automl/README.md) before running this notebook.**\n"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "We support installing AML SDK with Automated ML as library from GUI. When attaching a library follow <a href=\"https://docs.databricks.com/user-guide/libraries.html\" target=\"_blank\">this link</a> and add the below string as your PyPi package. You can select the option to attach the library to all clusters or just one cluster.\n",
+        "# AutoML : Classification with Local Compute on Azure DataBricks\n",
         "\n",
-        "**azureml-sdk with automated ml**\n",
-        "* Source: Upload Python Egg or PyPi\n",
-        "* PyPi Name: `azureml-sdk[automl]`\n",
-        "* Select Install Library"
+        "In this example we use the scikit-learn's to showcase how you can use AutoML for a simple classification problem.\n",
+        "\n",
+        "In this notebook you will learn how to:\n",
+        "1. Create Azure Machine Learning Workspace object and initialize your notebook directory to easily reload this object from a configuration file.\n",
+        "2. Create an `Experiment` in an existing `Workspace`.\n",
+        "3. Configure AutoML using `AutoMLConfig`.\n",
+        "4. Train the model using AzureDataBricks.\n",
+        "5. Explore the results.\n",
+        "6. Test the best fitted model.\n",
+        "\n",
+        "Prerequisites:\n",
+        "Before running this notebook, please follow the readme for installing necessary libraries to your cluster."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Register Machine Learning Services Resource Provider\n",
+        "Microsoft.MachineLearningServices only needs to be registed once in the subscription. To register it:\n",
+        "Start the Azure portal.\n",
+        "Select your All services and then Subscription.\n",
+        "Select the subscription that you want to use.\n",
+        "Click on Resource providers\n",
+        "Click the Register link next to Microsoft.MachineLearningServices"
       ]
     },
     {
@@ -145,31 +159,8 @@
         "               resource_group = resource_group)\n",
         "\n",
         "# Persist the subscription id, resource group name, and workspace name in aml_config/config.json.\n",
-        "ws.write_config()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Create a Folder to Host Sample Projects\n",
-        "Finally, create a folder where all the sample projects will be hosted."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "sample_projects_folder = './sample_projects'\n",
-        "\n",
-        "if not os.path.isdir(sample_projects_folder):\n",
-        "    os.mkdir(sample_projects_folder)\n",
-        "    \n",
-        "print('Sample projects will be created in {}.'.format(sample_projects_folder))"
+        "ws.write_config()\n",
+        "write_config(path=\"/databricks/driver/aml_config/\",file_name=<alias_conf.cfg>)"
       ]
     },
     {
@@ -178,7 +169,7 @@
       "source": [
         "## Create an Experiment\n",
         "\n",
-        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
+        "As part of the setup you have already created an Azure ML `Workspace` object. For AutoML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
       ]
     },
     {
@@ -191,6 +182,7 @@
         "import os\n",
         "import random\n",
         "import time\n",
+        "import json\n",
         "\n",
         "from matplotlib import pyplot as plt\n",
         "from matplotlib.pyplot import imshow\n",
@@ -212,7 +204,6 @@
       "source": [
         "# Choose a name for the experiment and specify the project folder.\n",
         "experiment_name = 'automl-local-classification'\n",
-        "project_folder = './sample_projects/automl-local-classification'\n",
         "\n",
         "experiment = Experiment(ws, experiment_name)\n",
         "\n",
@@ -222,94 +213,11 @@
         "output['Workspace Name'] = ws.name\n",
         "output['Resource Group'] = ws.resource_group\n",
         "output['Location'] = ws.location\n",
-        "output['Project Directory'] = project_folder\n",
         "output['Experiment Name'] = experiment.name\n",
         "pd.set_option('display.max_colwidth', -1)\n",
         "pd.DataFrame(data = output, index = ['']).T"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Diagnostics\n",
-        "\n",
-        "Opt-in diagnostics for better experience, quality, and security of future releases."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.telemetry import set_diagnostics_collection\n",
-        "set_diagnostics_collection(send_diagnostics = True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Registering Datastore"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Datastore is the way to save connection information to a storage service (e.g. Azure Blob, Azure Data Lake, Azure SQL) information to your workspace so you can access them without exposing credentials in your code. The first thing you will need to do is register a datastore, you can refer to our [python SDK documentation](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore.datastore?view=azure-ml-py) on how to register datastores. __Note: for best security practices, please do not check in code that contains registering datastores with secrets into your source control__\n",
-        "\n",
-        "The code below registers a datastore pointing to a publicly readable blob container."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Datastore\n",
-        "\n",
-        "datastore_name = 'demo_training'\n",
-        "container_name = 'digits' \n",
-        "account_name = 'automlpublicdatasets'\n",
-        "Datastore.register_azure_blob_container(\n",
-        "    workspace = ws, \n",
-        "    datastore_name = datastore_name, \n",
-        "    container_name = container_name, \n",
-        "    account_name = account_name,\n",
-        "    overwrite = True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Below is an example on how to register a private blob container\n",
-        "```python\n",
-        "datastore = Datastore.register_azure_blob_container(\n",
-        "    workspace = ws, \n",
-        "    datastore_name = 'example_datastore', \n",
-        "    container_name = 'example-container', \n",
-        "    account_name = 'storageaccount',\n",
-        "    account_key = 'accountkey'\n",
-        ")\n",
-        "```\n",
-        "The example below shows how  to register an Azure Data Lake store. Please make sure you have granted the necessary permissions for the service principal to access the data lake.\n",
-        "```python\n",
-        "datastore = Datastore.register_azure_data_lake(\n",
-        "    workspace = ws,\n",
-        "    datastore_name = 'example_datastore',\n",
-        "    store_name = 'adlsstore',\n",
-        "    tenant_id = 'tenant-id-of-service-principal',\n",
-        "    client_id = 'client-id-of-service-principal',\n",
-        "    client_secret = 'client-secret-of-service-principal'\n",
-        ")\n",
-        "```"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -323,9 +231,7 @@
       "source": [
         "Automated ML takes a `TabularDataset` as input.\n",
         "\n",
-        "You are free to use the data preparation libraries/tools of your choice to do the require preparation and once you are done, you can write it to a datastore and create a TabularDataset from it.\n",
-        "\n",
-        "You will get the datastore you registered previously and pass it to Dataset for reading. The data comes from the digits dataset: `sklearn.datasets.load_digits()`. `DataPath` points to a specific location within a datastore. "
+        "You are free to use the data preparation libraries/tools of your choice to do the require preparation and once you are done, you can write it to a datastore and create a TabularDataset from it."
       ]
     },
     {
@@ -334,13 +240,12 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "# The data referenced here was a 1MB simple random sample of the Chicago Crime data into a local temporary directory.\n",
         "from azureml.core.dataset import Dataset\n",
-        "from azureml.data.datapath import DataPath\n",
         "\n",
-        "datastore = Datastore.get(workspace = ws, datastore_name = datastore_name)\n",
-        "\n",
-        "X_train = Dataset.Tabular.from_delimited_files(datastore.path('X.csv'))\n",
-        "y_train = Dataset.Tabular.from_delimited_files(datastore.path('y.csv'))"
+        "example_data = 'https://dprepdata.blob.core.windows.net/demo/crime0-random.csv'\n",
+        "dataset = Dataset.Tabular.from_delimited_files(example_data)\n",
+        "dataset.take(5).to_pandas_dataframe()"
       ]
     },
     {
@@ -357,16 +262,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X_train.take(5).to_pandas_dataframe()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "y_train.take(5).to_pandas_dataframe()"
+        "training_data = dataset.drop_columns(columns=['FBI Code'])\n",
+        "label = 'Primary Type'"
       ]
     },
     {
@@ -384,14 +281,11 @@
         "|**primary_metric**|This is the metric that you want to optimize. Regression supports the following primary metrics: <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
         "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
         "|**iterations**|Number of iterations. In each iteration AutoML trains a specific pipeline with the data.|\n",
-        "|**n_cross_validations**|Number of cross validation splits.|\n",
         "|**spark_context**|Spark Context object. for Databricks, use spark_context=sc|\n",
         "|**max_concurrent_iterations**|Maximum number of iterations to execute in parallel. This should be <= number of worker nodes in your Azure Databricks cluster.|\n",
-        "|**X**|(sparse) array-like, shape = [n_samples, n_features]|\n",
-        "|**y**|(sparse) array-like, shape = [n_samples, ], [n_samples, n_classes]<br>Multi-class targets. An indicator matrix turns on multilabel classification. This should be an array of integers.|\n",
-        "|**path**|Relative path to the project folder. AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder.|\n",
-        "|**preprocess**|set this to True to enable pre-processing of data eg. string to numeric using one-hot encoding|\n",
-        "|**exit_score**|Target score for experiment. It is associated with the metric. eg. exit_score=0.995 will exit experiment after that|"
+        "|**n_cross_validations**|Number of cross validation splits.|\n",
+        "|**training_data**|Input dataset, containing both features and label column.|\n",
+        "|**label_column_name**|The name of the label column.|"
       ]
     },
     {
@@ -404,15 +298,13 @@
         "                             debug_log = 'automl_errors.log',\n",
         "                             primary_metric = 'AUC_weighted',\n",
         "                             iteration_timeout_minutes = 10,\n",
-        "                             iterations = 3,\n",
-        "                             preprocess = True,\n",
+        "                             iterations = 5,\n",
         "                             n_cross_validations = 10,\n",
         "                             max_concurrent_iterations = 2, #change it based on number of worker nodes\n",
         "                             verbosity = logging.INFO,\n",
         "                             spark_context=sc, #databricks/spark related\n",
-        "                             X = X_train, \n",
-        "                             y = y_train,\n",
-        "                             path = project_folder)"
+        "                             training_data=training_data,\n",
+        "                             label_column_name=label)"
       ]
     },
     {
@@ -433,26 +325,6 @@
         "local_run = experiment.submit(automl_config, show_output = True)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Continue experiment"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "local_run.continue_experiment(iterations=2,\n",
-        "                              X=X_train, \n",
-        "                              y=y_train,\n",
-        "                              spark_context=sc,\n",
-        "                              show_output=True)"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -475,48 +347,18 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "displayHTML(\"<a href={} target='_blank'>Your experiment in Azure Portal: {}</a>\".format(local_run.get_portal_url(), local_run.id))"
+        "displayHTML(\"<a href={} target='_blank'>Azure Portal: {}</a>\".format(local_run.get_portal_url(), local_run.id))"
       ]
     },
     {
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "The following will show the child runs and waits for the parent run to complete."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Retrieve All Child Runs after the experiment is completed (in portal)\n",
-        "You can also use SDK methods to fetch all the child runs and see individual metrics that we log."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "children = list(local_run.get_children())\n",
-        "metricslist = {}\n",
-        "for run in children:\n",
-        "    properties = run.get_properties()\n",
-        "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    \n",
-        "    metricslist[int(properties['iteration'])] = metrics\n",
+        "## Deploy\n",
         "\n",
-        "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
-        "rundata"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Retrieve the Best Model after the above run is complete \n",
+        "### Retrieve the Best Model\n",
         "\n",
-        "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model. The Model includes the pipeline and any pre-processing.  Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
+        "Below we select the best pipeline from our iterations. The `get_output` method on `automl_classifier` returns the best run and the fitted model for the last invocation. Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
       ]
     },
     {
@@ -525,71 +367,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "best_run, fitted_model = local_run.get_output()\n",
-        "print(best_run)\n",
-        "print(fitted_model)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Best Model Based on Any Other Metric after the above run is complete based on the child run\n",
-        "Show the run and the model that has the smallest `log_loss` value:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "lookup_metric = \"log_loss\"\n",
-        "best_run, fitted_model = local_run.get_output(metric = lookup_metric)\n",
-        "print(best_run)\n",
-        "print(fitted_model)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View the engineered names for featurized data\n",
-        "Below we display the engineered feature names generated for the featurized data using the preprocessing featurization."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "fitted_model.named_steps['datatransformer'].get_engineered_feature_names()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View the featurization summary\n",
-        "Below we display the featurization that was performed on different raw features in the user data. For each raw feature in the user data, the following information is displayed:-\n",
-        "- Raw feature name\n",
-        "- Number of engineered features formed out of this raw feature\n",
-        "- Type detected\n",
-        "- If feature was dropped\n",
-        "- List of feature transformations for the raw feature"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Get the featurization summary as a list of JSON\n",
-        "featurization_summary = fitted_model.named_steps['datatransformer'].get_featurization_summary()\n",
-        "# View the featurization summary as a pandas dataframe\n",
-        "pd.DataFrame.from_records(featurization_summary)"
+        "best_run, fitted_model = local_run.get_output()"
       ]
     },
     {
@@ -607,11 +385,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "blob_location = \"https://{}.blob.core.windows.net/{}\".format(account_name, container_name)\n",
-        "X_test = pd.read_csv(\"{}./X_valid.csv\".format(blob_location), header=0)\n",
-        "y_test = pd.read_csv(\"{}/y_valid.csv\".format(blob_location), header=0)\n",
-        "images  = pd.read_csv(\"{}/images.csv\".format(blob_location), header=None)\n",
-        "images = np.reshape(images.values, (100,8,8))"
+        "dataset_test = Dataset.Tabular.from_delimited_files(path='https://dprepdata.blob.core.windows.net/demo/crime0-test.csv')\n",
+        "\n",
+        "df_test = dataset_test.to_pandas_dataframe()\n",
+        "df_test = df_test[pd.notnull(df_test['Primary Type'])]\n",
+        "\n",
+        "y_test = df_test[['Primary Type']]\n",
+        "X_test = df_test.drop(['Primary Type', 'FBI Code'], axis=1)"
       ]
     },
     {
@@ -628,35 +408,9 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "# Randomly select digits and test.\n",
-        "for index in np.random.choice(len(y_test), 2, replace = False):\n",
-        "    print(index)\n",
-        "    predicted = fitted_model.predict(X_test[index:index + 1])[0]\n",
-        "    label = y_test.values[index]\n",
-        "    title = \"Label value = %d  Predicted value = %d \" % (label, predicted)\n",
-        "    fig = plt.figure(3, figsize = (5,5))\n",
-        "    ax1 = fig.add_axes((0,0,.8,.8))\n",
-        "    ax1.set_title(title)\n",
-        "    plt.imshow(images[index], cmap = plt.cm.gray_r, interpolation = 'nearest')\n",
-        "    display(fig)"
+        "fitted_model.predict(X_test)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "When deploying an automated ML trained model, please specify _pippackages=['azureml-sdk[automl]']_ in your CondaDependencies.\n",
-        "\n",
-        "Please refer to only the **Deploy** section in this notebook - <a href=\"https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/classification-with-deployment\" target=\"_blank\">Deployment of Automated ML trained model</a>"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": []
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -689,10 +443,10 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.5"
+      "version": "3.6.8"
     },
     "name": "auto-ml-classification-local-adb",
-    "notebookId": 587284549713154
+    "notebookId": 1275190406842063
   },
   "nbformat": 4,
   "nbformat_minor": 1

how-to-use-azureml/azure-databricks/automl/automl-databricks-local-with-deployment.ipynb

@@ -13,12 +13,13 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "We support installing AML SDK as library from GUI. When attaching a library follow this https://docs.databricks.com/user-guide/libraries.html and add the below string as your PyPi package. You can select the option to attach the library to all clusters or just one cluster.\n",
+        "## AutoML Installation\n",
         "\n",
-        "**install azureml-sdk with Automated ML**\n",
-        "* Source: Upload Python Egg or PyPi\n",
-        "* PyPi Name: `azureml-sdk[automl]`\n",
-        "* Select Install Library"
+        "**For Databricks non ML runtime 7.1(scala 2.21, spark 3.0.0) and up, Install AML sdk by running the following command in the first cell of the notebook.**\n",
+        "\n",
+        "%pip install --upgrade --force-reinstall -r https://aka.ms/automl_linux_requirements.txt\n",
+        "\n",
+        "**For Databricks non ML runtime 7.0 and lower, Install AML sdk using init script as shown in [readme](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/azure-databricks/automl/README.md) before running this notebook.**"
       ]
     },
     {
@@ -27,7 +28,7 @@
       "source": [
         "# AutoML : Classification with Local Compute on Azure DataBricks with deployment to ACI\n",
         "\n",
-        "In this example we use the scikit-learn's [digit dataset](http://scikit-learn.org/stable/datasets/index.html#optical-recognition-of-handwritten-digits-dataset) to showcase how you can use AutoML for a simple classification problem.\n",
+        "In this example we use the scikit-learn's to showcase how you can use AutoML for a simple classification problem.\n",
         "\n",
         "In this notebook you will learn how to:\n",
         "1. Create Azure Machine Learning Workspace object and initialize your notebook directory to easily reload this object from a configuration file.\n",
@@ -164,30 +165,6 @@
         "write_config(path=\"/databricks/driver/aml_config/\",file_name=<alias_conf.cfg>)"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Create a Folder to Host Sample Projects\n",
-        "Finally, create a folder where all the sample projects will be hosted."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "\n",
-        "sample_projects_folder = './sample_projects'\n",
-        "\n",
-        "if not os.path.isdir(sample_projects_folder):\n",
-        "    os.mkdir(sample_projects_folder)\n",
-        "    \n",
-        "print('Sample projects will be created in {}.'.format(sample_projects_folder))"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -229,7 +206,6 @@
       "source": [
         "# Choose a name for the experiment and specify the project folder.\n",
         "experiment_name = 'automl-local-classification'\n",
-        "project_folder = './sample_projects/automl-local-classification'\n",
         "\n",
         "experiment = Experiment(ws, experiment_name)\n",
         "\n",
@@ -239,94 +215,11 @@
         "output['Workspace Name'] = ws.name\n",
         "output['Resource Group'] = ws.resource_group\n",
         "output['Location'] = ws.location\n",
-        "output['Project Directory'] = project_folder\n",
         "output['Experiment Name'] = experiment.name\n",
         "pd.set_option('display.max_colwidth', -1)\n",
         "pd.DataFrame(data = output, index = ['']).T"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Diagnostics\n",
-        "\n",
-        "Opt-in diagnostics for better experience, quality, and security of future releases."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.telemetry import set_diagnostics_collection\n",
-        "set_diagnostics_collection(send_diagnostics = True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Registering Datastore"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Datastore is the way to save connection information to a storage service (e.g. Azure Blob, Azure Data Lake, Azure SQL) information to your workspace so you can access them without exposing credentials in your code. The first thing you will need to do is register a datastore, you can refer to our [python SDK documentation](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore.datastore?view=azure-ml-py) on how to register datastores. __Note: for best security practices, please do not check in code that contains registering datastores with secrets into your source control__\n",
-        "\n",
-        "The code below registers a datastore pointing to a publicly readable blob container."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Datastore\n",
-        "\n",
-        "datastore_name = 'demo_training'\n",
-        "container_name = 'digits' \n",
-        "account_name = 'automlpublicdatasets'\n",
-        "Datastore.register_azure_blob_container(\n",
-        "    workspace = ws, \n",
-        "    datastore_name = datastore_name, \n",
-        "    container_name = container_name, \n",
-        "    account_name = account_name,\n",
-        "     overwrite = True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Below is an example on how to register a private blob container\n",
-        "```python\n",
-        "datastore = Datastore.register_azure_blob_container(\n",
-        "    workspace = ws, \n",
-        "    datastore_name = 'example_datastore', \n",
-        "    container_name = 'example-container', \n",
-        "    account_name = 'storageaccount',\n",
-        "    account_key = 'accountkey'\n",
-        ")\n",
-        "```\n",
-        "The example below shows how  to register an Azure Data Lake store. Please make sure you have granted the necessary permissions for the service principal to access the data lake.\n",
-        "```python\n",
-        "datastore = Datastore.register_azure_data_lake(\n",
-        "    workspace = ws,\n",
-        "    datastore_name = 'example_datastore',\n",
-        "    store_name = 'adlsstore',\n",
-        "    tenant_id = 'tenant-id-of-service-principal',\n",
-        "    client_id = 'client-id-of-service-principal',\n",
-        "    client_secret = 'client-secret-of-service-principal'\n",
-        ")\n",
-        "```"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -340,9 +233,7 @@
       "source": [
         "Automated ML takes a `TabularDataset` as input.\n",
         "\n",
-        "You are free to use the data preparation libraries/tools of your choice to do the require preparation and once you are done, you can write it to a datastore and create a TabularDataset from it.\n",
-        "\n",
-        "You will get the datastore you registered previously and pass it to Dataset for reading. The data comes from the digits dataset: `sklearn.datasets.load_digits()`. `DataPath` points to a specific location within a datastore. "
+        "You are free to use the data preparation libraries/tools of your choice to do the require preparation and once you are done, you can write it to a datastore and create a TabularDataset from it."
       ]
     },
     {
@@ -351,13 +242,12 @@
       "metadata": {},
       "outputs": [],
       "source": [
+        "# The data referenced here was a 1MB simple random sample of the Chicago Crime data into a local temporary directory.\n",
         "from azureml.core.dataset import Dataset\n",
-        "from azureml.data.datapath import DataPath\n",
         "\n",
-        "datastore = Datastore.get(workspace = ws, datastore_name = datastore_name)\n",
-        "\n",
-        "X_train = Dataset.Tabular.from_delimited_files(datastore.path('X.csv'))\n",
-        "y_train = Dataset.Tabular.from_delimited_files(datastore.path('y.csv'))"
+        "example_data = 'https://dprepdata.blob.core.windows.net/demo/crime0-random.csv'\n",
+        "dataset = Dataset.Tabular.from_delimited_files(example_data)\n",
+        "dataset.take(5).to_pandas_dataframe()"
       ]
     },
     {
@@ -374,16 +264,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "X_train.take(5).to_pandas_dataframe()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "y_train.take(5).to_pandas_dataframe()"
+        "training_data = dataset.drop_columns(columns=['FBI Code'])\n",
+        "label = 'Primary Type'"
       ]
     },
     {
@@ -401,14 +283,11 @@
         "|**primary_metric**|This is the metric that you want to optimize. Regression supports the following primary metrics: <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
         "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
         "|**iterations**|Number of iterations. In each iteration AutoML trains a specific pipeline with the data.|\n",
-        "|**n_cross_validations**|Number of cross validation splits.|\n",
         "|**spark_context**|Spark Context object. for Databricks, use spark_context=sc|\n",
         "|**max_concurrent_iterations**|Maximum number of iterations to execute in parallel. This should be <= number of worker nodes in your Azure Databricks cluster.|\n",
-        "|**X**|(sparse) array-like, shape = [n_samples, n_features]|\n",
-        "|**y**|(sparse) array-like, shape = [n_samples, ], [n_samples, n_classes]<br>Multi-class targets. An indicator matrix turns on multilabel classification. This should be an array of integers.|\n",
-        "|**path**|Relative path to the project folder. AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder.|\n",
-        "|**preprocess**|set this to True to enable pre-processing of data eg. string to numeric using one-hot encoding|\n",
-        "|**exit_score**|Target score for experiment. It is associated with the metric. eg. exit_score=0.995 will exit experiment after that|"
+        "|**n_cross_validations**|Number of cross validation splits.|\n",
+        "|**training_data**|Input dataset, containing both features and label column.|\n",
+        "|**label_column_name**|The name of the label column.|"
       ]
     },
     {
@@ -422,14 +301,12 @@
         "                             primary_metric = 'AUC_weighted',\n",
         "                             iteration_timeout_minutes = 10,\n",
         "                             iterations = 5,\n",
-        "                             preprocess = True,\n",
         "                             n_cross_validations = 10,\n",
         "                             max_concurrent_iterations = 2, #change it based on number of worker nodes\n",
         "                             verbosity = logging.INFO,\n",
         "                             spark_context=sc, #databricks/spark related\n",
-        "                             X = X_train, \n",
-        "                             y = y_train,\n",
-        "                             path = project_folder)"
+        "                             training_data=training_data,\n",
+        "                             label_column_name=label)"
       ]
     },
     {
@@ -475,39 +352,6 @@
         "displayHTML(\"<a href={} target='_blank'>Azure Portal: {}</a>\".format(local_run.get_portal_url(), local_run.id))"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "The following will show the child runs and waits for the parent run to complete."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Retrieve All Child Runs after the experiment is completed (in portal)\n",
-        "You can also use SDK methods to fetch all the child runs and see individual metrics that we log."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "children = list(local_run.get_children())\n",
-        "metricslist = {}\n",
-        "for run in children:\n",
-        "    properties = run.get_properties()\n",
-        "    #print(properties)\n",
-        "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}    \n",
-        "    metricslist[int(properties['iteration'])] = metrics\n",
-        "\n",
-        "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
-        "rundata"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -651,11 +495,13 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "blob_location = \"https://{}.blob.core.windows.net/{}\".format(account_name, container_name)\n",
-        "X_test = pd.read_csv(\"{}./X_valid.csv\".format(blob_location), header=0)\n",
-        "y_test = pd.read_csv(\"{}/y_valid.csv\".format(blob_location), header=0)\n",
-        "images  = pd.read_csv(\"{}/images.csv\".format(blob_location), header=None)\n",
-        "images = np.reshape(images.values, (100,8,8))"
+        "dataset_test = Dataset.Tabular.from_delimited_files(path='https://dprepdata.blob.core.windows.net/demo/crime0-test.csv')\n",
+        "\n",
+        "df_test = dataset_test.to_pandas_dataframe()\n",
+        "df_test = df_test[pd.notnull(df_test['Primary Type'])]\n",
+        "\n",
+        "y_test = df_test[['Primary Type']]\n",
+        "X_test = df_test.drop(['Primary Type', 'FBI Code'], axis=1)"
       ]
     },
     {
@@ -672,20 +518,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "import json\n",
-        "# Randomly select digits and test.\n",
-        "for index in np.random.choice(len(y_test), 2, replace = False):\n",
-        "    print(index)\n",
-        "    test_sample = json.dumps({'data':X_test[index:index + 1].values.tolist()})\n",
-        "    predicted = aci_service.run(input_data = test_sample)\n",
-        "    label = y_test.values[index]\n",
-        "    predictedDict = json.loads(predicted)\n",
-        "    title = \"Label value = %d  Predicted value = %s \" % ( label,predictedDict['result'][0])    \n",
-        "    fig = plt.figure(3, figsize = (5,5))\n",
-        "    ax1 = fig.add_axes((0,0,.8,.8))\n",
-        "    ax1.set_title(title)\n",
-        "    plt.imshow(images[index], cmap = plt.cm.gray_r, interpolation = 'nearest')\n",
-        "    display(fig)"
+        "fitted_model.predict(X_test)"
       ]
     },
     {
@@ -703,7 +536,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "myservice.delete()"
+        "aci_service.delete()"
       ]
     },
     {
@@ -741,7 +574,7 @@
       "version": "3.6.8"
     },
     "name": "auto-ml-classification-local-adb",
-    "notebookId": 2733885892129020
+    "notebookId": 3772036807853791
   },
   "nbformat": 4,
   "nbformat_minor": 1

how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.ipynb

@@ -1,719 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Using Databricks as a Compute Target from Azure Machine Learning Pipeline\n",
-        "To use Databricks as a compute target from [Azure Machine Learning Pipeline](https://aka.ms/pl-concept), a [DatabricksStep](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py) is used. This notebook demonstrates the use of DatabricksStep in Azure Machine Learning Pipeline.\n",
-        "\n",
-        "The notebook will show:\n",
-        "1. Running an arbitrary Databricks notebook that the customer has in Databricks workspace\n",
-        "2. Running an arbitrary Python script that the customer has in DBFS\n",
-        "3. Running an arbitrary Python script that is available on local computer (will upload to DBFS, and then run in Databricks) \n",
-        "4. Running a JAR job that the customer has in DBFS.\n",
-        "\n",
-        "## Before you begin:\n",
-        "\n",
-        "1. **Create an Azure Databricks workspace** in the same subscription where you have your Azure Machine Learning workspace. You will need details of this workspace later on to define DatabricksStep. [Click here](https://ms.portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.Databricks%2Fworkspaces) for more information.\n",
-        "2. **Create PAT (access token)**: Manually create a Databricks access token at the Azure Databricks portal. See [this](https://docs.databricks.com/api/latest/authentication.html#generate-a-token) for more information.\n",
-        "3. **Add demo notebook to ADB**: This notebook has a sample you can use as is. Launch Azure Databricks attached to your Azure Machine Learning workspace and add a new notebook. \n",
-        "4. **Create/attach a Blob storage** for use from ADB"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Add demo notebook to ADB Workspace\n",
-        "Copy and paste the below code to create a new notebook in your ADB workspace."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "```python\n",
-        "# direct access\n",
-        "dbutils.widgets.get(\"myparam\")\n",
-        "p = getArgument(\"myparam\")\n",
-        "print (\"Param -\\'myparam':\")\n",
-        "print (p)\n",
-        "\n",
-        "dbutils.widgets.get(\"input\")\n",
-        "i = getArgument(\"input\")\n",
-        "print (\"Param -\\'input':\")\n",
-        "print (i)\n",
-        "\n",
-        "dbutils.widgets.get(\"output\")\n",
-        "o = getArgument(\"output\")\n",
-        "print (\"Param -\\'output':\")\n",
-        "print (o)\n",
-        "\n",
-        "n = i + \"/testdata.txt\"\n",
-        "df = spark.read.csv(n)\n",
-        "\n",
-        "display (df)\n",
-        "\n",
-        "data = [('value1', 'value2')]\n",
-        "df2 = spark.createDataFrame(data)\n",
-        "\n",
-        "z = o + \"/output.txt\"\n",
-        "df2.write.csv(z)\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Azure Machine Learning and Pipeline SDK-specific imports"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import os\n",
-        "import azureml.core\n",
-        "from azureml.core.runconfig import JarLibrary\n",
-        "from azureml.core.compute import ComputeTarget, DatabricksCompute\n",
-        "from azureml.exceptions import ComputeTargetException\n",
-        "from azureml.core import Workspace, Experiment\n",
-        "from azureml.pipeline.core import Pipeline, PipelineData\n",
-        "from azureml.pipeline.steps import DatabricksStep\n",
-        "from azureml.core.datastore import Datastore\n",
-        "from azureml.data.data_reference import DataReference\n",
-        "\n",
-        "# Check core SDK version number\n",
-        "print(\"SDK version:\", azureml.core.VERSION)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Initialize Workspace\n",
-        "\n",
-        "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Attach Databricks compute target\n",
-        "Next, you need to add your Databricks workspace to Azure Machine Learning as a compute target and give it a name. You will use this name to refer to your Databricks workspace compute target inside Azure Machine Learning.\n",
-        "\n",
-        "- **Resource Group** - The resource group name of your Azure Machine Learning workspace\n",
-        "- **Databricks Workspace Name** - The workspace name of your Azure Databricks workspace\n",
-        "- **Databricks Access Token** - The access token you created in ADB\n",
-        "\n",
-        "**The Databricks workspace need to be present in the same subscription as your AML workspace**"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Replace with your account info before running.\n",
-        " \n",
-        "db_compute_name=os.getenv(\"DATABRICKS_COMPUTE_NAME\", \"<my-databricks-compute-name>\") # Databricks compute name\n",
-        "db_resource_group=os.getenv(\"DATABRICKS_RESOURCE_GROUP\", \"<my-db-resource-group>\") # Databricks resource group\n",
-        "db_workspace_name=os.getenv(\"DATABRICKS_WORKSPACE_NAME\", \"<my-db-workspace-name>\") # Databricks workspace name\n",
-        "db_access_token=os.getenv(\"DATABRICKS_ACCESS_TOKEN\", \"<my-access-token>\") # Databricks access token\n",
-        " \n",
-        "try:\n",
-        "    databricks_compute = DatabricksCompute(workspace=ws, name=db_compute_name)\n",
-        "    print('Compute target {} already exists'.format(db_compute_name))\n",
-        "except ComputeTargetException:\n",
-        "    print('Compute not found, will use below parameters to attach new one')\n",
-        "    print('db_compute_name {}'.format(db_compute_name))\n",
-        "    print('db_resource_group {}'.format(db_resource_group))\n",
-        "    print('db_workspace_name {}'.format(db_workspace_name))\n",
-        "    print('db_access_token {}'.format(db_access_token))\n",
-        " \n",
-        "    config = DatabricksCompute.attach_configuration(\n",
-        "        resource_group = db_resource_group,\n",
-        "        workspace_name = db_workspace_name,\n",
-        "        access_token= db_access_token)\n",
-        "    databricks_compute=ComputeTarget.attach(ws, db_compute_name, config)\n",
-        "    databricks_compute.wait_for_completion(True)\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Data Connections with Inputs and Outputs\n",
-        "The DatabricksStep supports Azure Bloband ADLS for inputs and outputs. You also will need to define a [Secrets](https://docs.azuredatabricks.net/user-guide/secrets/index.html) scope to enable authentication to external data sources such as Blob and ADLS from Databricks.\n",
-        "\n",
-        "- Databricks documentation on [Azure Blob](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-storage.html)\n",
-        "- Databricks documentation on [ADLS](https://docs.databricks.com/spark/latest/data-sources/azure/azure-datalake.html)\n",
-        "\n",
-        "### Type of Data Access\n",
-        "Databricks allows to interact with Azure Blob and ADLS in two ways.\n",
-        "- **Direct Access**: Databricks allows you to interact with Azure Blob or ADLS URIs directly. The input or output URIs will be mapped to a Databricks widget param in the Databricks notebook.\n",
-        "- **Mounting**: You will be supplied with additional parameters and secrets that will enable you to mount your ADLS or Azure Blob input or output location in your Databricks notebook."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Direct Access: Python sample code\n",
-        "If you have a data reference named \"input\" it will represent the URI of the input and you can access it directly in the Databricks python notebook like so:"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "```python\n",
-        "dbutils.widgets.get(\"input\")\n",
-        "y = getArgument(\"input\")\n",
-        "df = spark.read.csv(y)\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Mounting: Python sample code for Azure Blob\n",
-        "Given an Azure Blob data reference named \"input\" the following widget params will be made available in the Databricks notebook:"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "```python\n",
-        "# This contains the input URI\n",
-        "dbutils.widgets.get(\"input\")\n",
-        "myinput_uri = getArgument(\"input\")\n",
-        "\n",
-        "# How to get the input datastore name inside ADB notebook\n",
-        "# This contains the name of a Databricks secret (in the predefined \"amlscope\" secret scope) \n",
-        "# that contians an access key or sas for the Azure Blob input (this name is obtained by appending \n",
-        "# the name of the input with \"_blob_secretname\". \n",
-        "dbutils.widgets.get(\"input_blob_secretname\") \n",
-        "myinput_blob_secretname = getArgument(\"input_blob_secretname\")\n",
-        "\n",
-        "# This contains the required configuration for mounting\n",
-        "dbutils.widgets.get(\"input_blob_config\")\n",
-        "myinput_blob_config = getArgument(\"input_blob_config\")\n",
-        "\n",
-        "# Usage\n",
-        "dbutils.fs.mount(\n",
-        "  source = myinput_uri,\n",
-        "  mount_point = \"/mnt/input\",\n",
-        "  extra_configs = {myinput_blob_config:dbutils.secrets.get(scope = \"amlscope\", key = myinput_blob_secretname)})\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Mounting: Python sample code for ADLS\n",
-        "Given an ADLS data reference named \"input\" the following widget params will be made available in the Databricks notebook:"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "```python\n",
-        "# This contains the input URI\n",
-        "dbutils.widgets.get(\"input\") \n",
-        "myinput_uri = getArgument(\"input\")\n",
-        "\n",
-        "# This contains the client id for the service principal \n",
-        "# that has access to the adls input\n",
-        "dbutils.widgets.get(\"input_adls_clientid\") \n",
-        "myinput_adls_clientid = getArgument(\"input_adls_clientid\")\n",
-        "\n",
-        "# This contains the name of a Databricks secret (in the predefined \"amlscope\" secret scope) \n",
-        "# that contains the secret for the above mentioned service principal\n",
-        "dbutils.widgets.get(\"input_adls_secretname\") \n",
-        "myinput_adls_secretname = getArgument(\"input_adls_secretname\")\n",
-        "\n",
-        "# This contains the refresh url for the mounting configs\n",
-        "dbutils.widgets.get(\"input_adls_refresh_url\") \n",
-        "myinput_adls_refresh_url = getArgument(\"input_adls_refresh_url\")\n",
-        "\n",
-        "# Usage \n",
-        "configs = {\"dfs.adls.oauth2.access.token.provider.type\": \"ClientCredential\",\n",
-        "           \"dfs.adls.oauth2.client.id\": myinput_adls_clientid,\n",
-        "           \"dfs.adls.oauth2.credential\": dbutils.secrets.get(scope = \"amlscope\", key =myinput_adls_secretname),\n",
-        "           \"dfs.adls.oauth2.refresh.url\": myinput_adls_refresh_url}\n",
-        "\n",
-        "dbutils.fs.mount(\n",
-        "  source = myinput_uri,\n",
-        "  mount_point = \"/mnt/output\",\n",
-        "  extra_configs = configs)\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Use Databricks from Azure Machine Learning Pipeline\n",
-        "To use Databricks as a compute target from Azure Machine Learning Pipeline, a DatabricksStep is used. Let's define a datasource (via DataReference) and intermediate data (via PipelineData) to be used in DatabricksStep."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Use the default blob storage\n",
-        "def_blob_store = Datastore(ws, \"workspaceblobstore\")\n",
-        "print('Datastore {} will be used'.format(def_blob_store.name))\n",
-        "\n",
-        "# We are uploading a sample file in the local directory to be used as a datasource\n",
-        "def_blob_store.upload_files(files=[\"./testdata.txt\"], target_path=\"dbtest\", overwrite=False)\n",
-        "\n",
-        "step_1_input = DataReference(datastore=def_blob_store, path_on_datastore=\"dbtest\",\n",
-        "                                     data_reference_name=\"input\")\n",
-        "\n",
-        "step_1_output = PipelineData(\"output\", datastore=def_blob_store)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Add a DatabricksStep\n",
-        "Adds a Databricks notebook as a step in a Pipeline.\n",
-        "- ***name:** Name of the Module\n",
-        "- **inputs:** List of input connections for data consumed by this step. Fetch this inside the notebook using dbutils.widgets.get(\"input\")\n",
-        "- **outputs:** List of output port definitions for outputs produced by this step. Fetch this inside the notebook using dbutils.widgets.get(\"output\")\n",
-        "- **existing_cluster_id:** Cluster ID of an existing Interactive cluster on the Databricks workspace. If you are providing this, do not provide any of the parameters below that are used to create a new cluster such as spark_version, node_type, etc.\n",
-        "- **spark_version:** Version of spark for the databricks run cluster. default value: 4.0.x-scala2.11\n",
-        "- **node_type:** Azure vm node types for the databricks run cluster. default value: Standard_D3_v2\n",
-        "- **num_workers:** Specifies a static number of workers for the databricks run cluster\n",
-        "- **min_workers:** Specifies a min number of workers to use for auto-scaling the databricks run cluster\n",
-        "- **max_workers:** Specifies a max number of workers to use for auto-scaling the databricks run cluster\n",
-        "- **spark_env_variables:** Spark environment variables for the databricks run cluster (dictionary of {str:str}). default value: {'PYSPARK_PYTHON': '/databricks/python3/bin/python3'}\n",
-        "- **notebook_path:** Path to the notebook in the databricks instance. If you are providing this, do not provide python script related paramaters or JAR related parameters.\n",
-        "- **notebook_params:** Parameters  for the databricks notebook (dictionary of {str:str}). Fetch this inside the notebook using dbutils.widgets.get(\"myparam\")\n",
-        "- **python_script_path:** The path to the python script in the DBFS or S3. If you are providing this, do not provide python_script_name which is used for uploading script from local machine.\n",
-        "- **python_script_params:** Parameters for the python script (list of str)\n",
-        "- **main_class_name:** The name of the entry point in a JAR module. If you are providing this, do not provide any python script or notebook related parameters.\n",
-        "- **jar_params:** Parameters for the JAR module (list of str)\n",
-        "- **python_script_name:** name of a python script on your local machine (relative to source_directory). If you are providing this do not provide python_script_path which is used to execute a remote python script; or any of the JAR or notebook related parameters.\n",
-        "- **source_directory:** folder that contains the script and other files\n",
-        "- **hash_paths:** list of paths to hash to detect a change in source_directory (script file is always hashed)\n",
-        "- **run_name:** Name in databricks for this run\n",
-        "- **timeout_seconds:** Timeout for the databricks run\n",
-        "- **runconfig:** Runconfig to use. Either pass runconfig or each library type as a separate parameter but do not mix the two\n",
-        "- **maven_libraries:** maven libraries for the databricks run\n",
-        "- **pypi_libraries:** pypi libraries for the databricks run\n",
-        "- **egg_libraries:** egg libraries for the databricks run\n",
-        "- **jar_libraries:** jar libraries for the databricks run\n",
-        "- **rcran_libraries:** rcran libraries for the databricks run\n",
-        "- **compute_target:** Azure Databricks compute\n",
-        "- **allow_reuse:** Whether the step should reuse previous results when run with the same settings/inputs\n",
-        "- **version:** Optional version tag to denote a change in functionality for the step\n",
-        "\n",
-        "\\* *denotes required fields*  \n",
-        "*You must provide exactly one of num_workers or min_workers and max_workers paramaters*  \n",
-        "*You must provide exactly one of databricks_compute or databricks_compute_name parameters*\n",
-        "\n",
-        "## Use runconfig to specify library dependencies\n",
-        "You can use a runconfig to specify the library dependencies for your cluster in Databricks. The runconfig will contain a databricks section as follows:\n",
-        "\n",
-        "```yaml\n",
-        "environment:\n",
-        "# Databricks details\n",
-        "  databricks:\n",
-        "# List of maven libraries.\n",
-        "    mavenLibraries:\n",
-        "    - coordinates: org.jsoup:jsoup:1.7.1\n",
-        "      repo: ''\n",
-        "      exclusions:\n",
-        "      - slf4j:slf4j\n",
-        "      - '*:hadoop-client'\n",
-        "# List of PyPi libraries\n",
-        "    pypiLibraries:\n",
-        "    - package: beautifulsoup4\n",
-        "      repo: ''\n",
-        "# List of RCran libraries\n",
-        "    rcranLibraries:\n",
-        "    -\n",
-        "# Coordinates.\n",
-        "      package: ada\n",
-        "# Repo\n",
-        "      repo: http://cran.us.r-project.org\n",
-        "# List of JAR libraries\n",
-        "    jarLibraries:\n",
-        "    -\n",
-        "# Coordinates.\n",
-        "      library: dbfs:/mnt/libraries/library.jar\n",
-        "# List of Egg libraries\n",
-        "    eggLibraries:\n",
-        "    -\n",
-        "# Coordinates.\n",
-        "      library: dbfs:/mnt/libraries/library.egg\n",
-        "```\n",
-        "\n",
-        "You can then create a RunConfiguration object using this file and pass it as the runconfig parameter to DatabricksStep.\n",
-        "```python\n",
-        "from azureml.core.runconfig import RunConfiguration\n",
-        "\n",
-        "runconfig = RunConfiguration()\n",
-        "runconfig.load(path='<directory_where_runconfig_is_stored>', name='<runconfig_file_name>')\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### 1. Running the demo notebook already added to the Databricks workspace\n",
-        "Create a notebook in the Azure Databricks workspace, and provide the path to that notebook as the value associated with the environment variable \"DATABRICKS_NOTEBOOK_PATH\". This will then set the variable\u00c2\u00a0notebook_path\u00c2\u00a0when you run the code cell below:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "databricksstep-remarks-sample"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "notebook_path=os.getenv(\"DATABRICKS_NOTEBOOK_PATH\", \"<my-databricks-notebook-path>\") # Databricks notebook path\n",
-        "\n",
-        "dbNbStep = DatabricksStep(\n",
-        "    name=\"DBNotebookInWS\",\n",
-        "    inputs=[step_1_input],\n",
-        "    outputs=[step_1_output],\n",
-        "    num_workers=1,\n",
-        "    notebook_path=notebook_path,\n",
-        "    notebook_params={'myparam': 'testparam'},\n",
-        "    run_name='DB_Notebook_demo',\n",
-        "    compute_target=databricks_compute,\n",
-        "    allow_reuse=True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Build and submit the Experiment"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "steps = [dbNbStep]\n",
-        "pipeline = Pipeline(workspace=ws, steps=steps)\n",
-        "pipeline_run = Experiment(ws, 'DB_Notebook_demo').submit(pipeline)\n",
-        "pipeline_run.wait_for_completion()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View Run Details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(pipeline_run).show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### 2. Running a Python script from DBFS\n",
-        "This shows how to run a Python script in DBFS. \n",
-        "\n",
-        "To complete this, you will need to first upload the Python script in your local machine to DBFS using the [CLI](https://docs.azuredatabricks.net/user-guide/dbfs-databricks-file-system.html). The CLI command is given below:\n",
-        "\n",
-        "```\n",
-        "dbfs cp ./train-db-dbfs.py dbfs:/train-db-dbfs.py\n",
-        "```\n",
-        "\n",
-        "The code in the below cell assumes that you have completed the previous step of uploading the script `train-db-dbfs.py` to the root folder in DBFS."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "python_script_path = os.getenv(\"DATABRICKS_PYTHON_SCRIPT_PATH\", \"<my-databricks-python-script-path>\") # Databricks python script path\n",
-        "\n",
-        "dbPythonInDbfsStep = DatabricksStep(\n",
-        "    name=\"DBPythonInDBFS\",\n",
-        "    inputs=[step_1_input],\n",
-        "    num_workers=1,\n",
-        "    python_script_path=python_script_path,\n",
-        "    python_script_params={'--input_data'},\n",
-        "    run_name='DB_Python_demo',\n",
-        "    compute_target=databricks_compute,\n",
-        "    allow_reuse=True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Build and submit the Experiment"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "steps = [dbPythonInDbfsStep]\n",
-        "pipeline = Pipeline(workspace=ws, steps=steps)\n",
-        "pipeline_run = Experiment(ws, 'DB_Python_demo').submit(pipeline)\n",
-        "pipeline_run.wait_for_completion()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View Run Details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(pipeline_run).show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### 3. Running a Python script in Databricks that currenlty is in local computer\n",
-        "To run a Python script that is currently in your local computer, follow the instructions below. \n",
-        "\n",
-        "The commented out code below code assumes that you have `train-db-local.py` in the `scripts` subdirectory under the current working directory.\n",
-        "\n",
-        "In this case, the Python script will be uploaded first to DBFS, and then the script will be run in Databricks."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "python_script_name = \"train-db-local.py\"\n",
-        "source_directory = \".\"\n",
-        "\n",
-        "dbPythonInLocalMachineStep = DatabricksStep(\n",
-        "    name=\"DBPythonInLocalMachine\",\n",
-        "    inputs=[step_1_input],\n",
-        "    num_workers=1,\n",
-        "    python_script_name=python_script_name,\n",
-        "    source_directory=source_directory,\n",
-        "    run_name='DB_Python_Local_demo',\n",
-        "    compute_target=databricks_compute,\n",
-        "    allow_reuse=True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Build and submit the Experiment"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "steps = [dbPythonInLocalMachineStep]\n",
-        "pipeline = Pipeline(workspace=ws, steps=steps)\n",
-        "pipeline_run = Experiment(ws, 'DB_Python_Local_demo').submit(pipeline)\n",
-        "pipeline_run.wait_for_completion()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View Run Details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(pipeline_run).show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### 4. Running a JAR job that is alreay added in DBFS\n",
-        "To run a JAR job that is already uploaded to DBFS, follow the instructions below. You will first upload the JAR file to DBFS using the [CLI](https://docs.azuredatabricks.net/user-guide/dbfs-databricks-file-system.html).\n",
-        "\n",
-        "The commented out code in the below cell assumes that you have uploaded `train-db-dbfs.jar` to the root folder in DBFS. You can upload `train-db-dbfs.jar` to the root folder in DBFS using this commandline so you can use `jar_library_dbfs_path = \"dbfs:/train-db-dbfs.jar\"`:\n",
-        "\n",
-        "```\n",
-        "dbfs cp ./train-db-dbfs.jar dbfs:/train-db-dbfs.jar\n",
-        "```"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "main_jar_class_name = \"com.microsoft.aeva.Main\"\n",
-        "jar_library_dbfs_path = os.getenv(\"DATABRICKS_JAR_LIB_PATH\", \"<my-databricks-jar-lib-path>\") # Databricks jar library path\n",
-        "\n",
-        "dbJarInDbfsStep = DatabricksStep(\n",
-        "    name=\"DBJarInDBFS\",\n",
-        "    inputs=[step_1_input],\n",
-        "    num_workers=1,\n",
-        "    main_class_name=main_jar_class_name,\n",
-        "    jar_params={'arg1', 'arg2'},\n",
-        "    run_name='DB_JAR_demo',\n",
-        "    jar_libraries=[JarLibrary(jar_library_dbfs_path)],\n",
-        "    compute_target=databricks_compute,\n",
-        "    allow_reuse=True\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Build and submit the Experiment"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "steps = [dbJarInDbfsStep]\n",
-        "pipeline = Pipeline(workspace=ws, steps=steps)\n",
-        "pipeline_run = Experiment(ws, 'DB_JAR_demo').submit(pipeline)\n",
-        "pipeline_run.wait_for_completion()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### View Run Details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(pipeline_run).show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Next: ADLA as a Compute Target\n",
-        "To use ADLA as a compute target from Azure Machine Learning Pipeline, a AdlaStep is used. This [notebook](https://aka.ms/pl-adla) demonstrates the use of AdlaStep in Azure Machine Learning Pipeline."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/aml-pipelines-use-databricks-as-compute-target.png)"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "diray"
-      }
-    ],
-    "kernelspec": {
-      "display_name": "Python 3.6",
-      "language": "python",
-      "name": "python36"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.2"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}

how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/testdata.txt

@@ -1 +0,0 @@
-Test1

how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/train-db-dbfs.py

@@ -1,5 +0,0 @@
-# Copyright (c) Microsoft. All rights reserved.
-# Licensed under the MIT license.
-
-print("In train.py")
-print("As a data scientist, this is where I use my training code.")

how-to-use-azureml/azure-databricks/databricks-as-remote-compute-target/train-db-local.py

@@ -1,5 +0,0 @@
-# Copyright (c) Microsoft. All rights reserved.
-# Licensed under the MIT license.
-
-print("In train.py")
-print("As a data scientist, this is where I use my training code.")

how-to-use-azureml/azure-synapse/README.md

@@ -0,0 +1,84 @@
+Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. A core offering within Azure Synapse Analytics are serverless Apache Spark pools enhanced for big data workloads.  
+
+Synapse in Aml integration is for customers who want to use Apache Spark in Azure Synapse Analytics to prepare data at scale in Azure ML before training their ML model. This will allow customers to work on their end-to-end ML lifecycle including large-scale data preparation, model training and deployment within Azure ML workspace without having to use suboptimal tools for machine learning or switch between multiple tools for data preparation and model training. The ability to perform all ML tasks within Azure ML will reduce time required for customers to iterate on a machine learning project which typically includes multiple rounds of data preparation and training.
+
+In the public preview, the capabilities are provided:
+
+- Link Azure Synapse Analytics workspace to Azure Machine Learning workspace (via ARM, UI or SDK) 
+- Attach Apache Spark pools powered by Azure Synapse Analytics as Azure Machine Learning compute targets (via ARM, UI or SDK) 
+- Launch Apache Spark sessions in notebooks and perform interactive data exploration and preparation.  This interactive experience leverages Apache Spark magic and customers will have session-level Conda support to install packages. 
+- Productionize ML pipelines by leveraging Apache Spark pools to pre-process big data 
+
+# Using Synapse in Azure machine learning
+
+## Create synapse resources
+
+Follow up the documents to create Synapse workspace and resource-setup.sh is available for you to create the resources.
+
+- Create from [Portal](https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace)
+- Create from [Cli](https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace-cli)
+
+Follow up the documents to create Synapse spark pool
+
+- Create from [Portal](https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-apache-spark-pool-portal)
+- Create from [Cli](https://docs.microsoft.com/en-us/cli/azure/ext/synapse/synapse/spark/pool?view=azure-cli-latest)
+
+## Link Synapse Workspace
+
+Make sure you are the owner of synapse workspace so that you can link synapse workspace into AML.
+You can run resource-setup.py to link the synapse workspace and attach compute
+
+```python
+from azureml.core import Workspace
+ws = Workspace.from_config()
+
+from azureml.core import LinkedService, SynapseWorkspaceLinkedServiceConfiguration
+synapse_link_config = SynapseWorkspaceLinkedServiceConfiguration(
+    subscription_id="<subscription id>",
+    resource_group="<resource group",
+    name="<synapse workspace name>"
+)
+
+linked_service = LinkedService.register(
+    workspace=ws,
+    name='<link name>',
+    linked_service_config=synapse_link_config)
+
+```
+
+## Attach synapse spark pool as AzureML compute
+
+```python
+
+from azureml.core.compute import SynapseCompute, ComputeTarget
+spark_pool_name = "<spark pool name>"
+attached_synapse_name = "<attached compute name>"
+
+attach_config = SynapseCompute.attach_configuration(
+        linked_service,
+        type="SynapseSpark",
+        pool_name=spark_pool_name)
+
+synapse_compute=ComputeTarget.attach(
+        workspace=ws,
+        name=attached_synapse_name,
+        attach_configuration=attach_config)
+
+synapse_compute.wait_for_completion()
+```
+
+## Set up permission
+
+Grant Spark admin role to system assigned identity of the linked service so that the user can submit experiment run or pipeline run from AML workspace to synapse spark pool.
+
+Grant Spark admin role to the specific user so that the user can start spark session to synapse spark pool.
+
+You can get the system assigned identity information by running
+
+```python
+print(linked_service.system_assigned_identity_principal_id)
+```
+
+- Launch synapse studio of the synapse workspace and grant linked service MSI "Synapse Apache Spark administrator" role.
+
+- In azure portal grant linked service MSI "Storage Blob Data Contributor" role of the primary adlsgen2 account of synapse workspace to use the library management feature.

how-to-use-azureml/azure-synapse/Synapse_Job_Scala_Support.ipynb

@@ -0,0 +1,186 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved. \n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-arcadia/Synapse_Job_Scala_Support.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Get AML workspace which has synapse spark pool attached"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace, Experiment, Dataset, Environment\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Leverage ScriptRunConfig to submit scala job to an attached synapse spark cluster"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Prepare data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.datastore import Datastore\n",
+        "# Use the default blob storage\n",
+        "def_blob_store = Datastore(ws, \"workspaceblobstore\")\n",
+        "\n",
+        "# We are uploading a sample file in the local directory to be used as a datasource\n",
+        "file_name = \"shakespeare.txt\"\n",
+        "def_blob_store.upload_files(files=[\"./{}\".format(file_name)], overwrite=False)\n",
+        "\n",
+        "# Create file dataset\n",
+        "file_dataset = Dataset.File.from_files(path=[(def_blob_store, file_name)])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.runconfig import RunConfiguration\n",
+        "from azureml.data import HDFSOutputDatasetConfig\n",
+        "import uuid\n",
+        "\n",
+        "run_config = RunConfiguration(framework=\"pyspark\")\n",
+        "run_config.target = \"link-pool\"\n",
+        "run_config.spark.configuration[\"spark.driver.memory\"] = \"2g\"\n",
+        "run_config.spark.configuration[\"spark.driver.cores\"] = 2\n",
+        "run_config.spark.configuration[\"spark.executor.memory\"] = \"2g\"\n",
+        "run_config.spark.configuration[\"spark.executor.cores\"] = 1\n",
+        "run_config.spark.configuration[\"spark.executor.instances\"] = 1\n",
+        "# This can be removed if you are using local jars in source folder\n",
+        "run_config.spark.configuration[\"spark.yarn.dist.jars\"]=\"wasbs://synapse@azuremlexamples.blob.core.windows.net/shared/wordcount.jar\"\n",
+        "\n",
+        "dir_name = \"wordcount-{}\".format(str(uuid.uuid4()))\n",
+        "input = file_dataset.as_named_input(\"input\").as_hdfs()\n",
+        "output = HDFSOutputDatasetConfig(destination=(ws.get_default_datastore(), \"{}/result\".format(dir_name)))\n",
+        "\n",
+        "from azureml.core import ScriptRunConfig\n",
+        "args = ['--input', input, '--output', output]\n",
+        "script_run_config = ScriptRunConfig(source_directory = '.',\n",
+        "                                    script= 'start_script.py',\n",
+        "                                    arguments= args,\n",
+        "                                    run_config = run_config)\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Experiment\n",
+        "exp = Experiment(workspace=ws, name='synapse-spark')\n",
+        "run = exp.submit(config=script_run_config)\n",
+        "run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Leverage SynapseSparkStep in an AML pipeline to add dataprep step on synapse spark cluster"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.pipeline.core import Pipeline\n",
+        "from azureml.pipeline.steps import SynapseSparkStep\n",
+        "\n",
+        "configs = {}\n",
+        "#configs[\"spark.yarn.dist.jars\"] = \"wasbs://synapse@azuremlexamples.blob.core.windows.net/shared/wordcount.jar\"\n",
+        "step_1 = SynapseSparkStep(name = 'synapse-spark',\n",
+        "                          file = 'start_script.py',\n",
+        "                          jars = \"wasbs://synapse@azuremlexamples.blob.core.windows.net/shared/wordcount.jar\",\n",
+        "                          source_directory=\".\",\n",
+        "                          arguments = args,\n",
+        "                          compute_target = 'link-pool',\n",
+        "                          driver_memory = \"2g\",\n",
+        "                          driver_cores = 2,\n",
+        "                          executor_memory = \"2g\",\n",
+        "                          executor_cores = 1,\n",
+        "                          num_executors = 1,\n",
+        "                          conf = configs)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "pipeline = Pipeline(workspace=ws, steps=[step_1])\n",
+        "pipeline_run = pipeline.submit('synapse-pipeline', regenerate_outputs=True)"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "feli1"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.5"
+    },
+    "nteract": {
+      "version": "0.28.0"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/azure-synapse/Synapse_Session_Scala_Support.ipynb

@@ -0,0 +1,240 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved. \n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-arcadia/Synapse_Session_Scala_Support.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Interactive Spark Session on Synapse Spark Pool"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install -U \"azureml-synapse\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "For JupyterLab, please additionally run:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!jupyter lab build --minimize=False"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## PLEASE restart kernel and then refresh web page before starting spark session."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 0. Magic Usage"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "execution": {
+          "iopub.execute_input": "2020-06-05T03:22:14.965395Z",
+          "iopub.status.busy": "2020-06-05T03:22:14.965395Z",
+          "iopub.status.idle": "2020-06-05T03:22:14.970398Z",
+          "shell.execute_reply": "2020-06-05T03:22:14.969397Z",
+          "shell.execute_reply.started": "2020-06-05T03:22:14.965395Z"
+        },
+        "gather": {
+          "logged": 1615594584642
+        }
+      },
+      "outputs": [],
+      "source": [
+        "# show help\n",
+        "%synapse ?"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# 1. Start Synapse Session"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "gather": {
+          "logged": 1615577715289
+        }
+      },
+      "outputs": [],
+      "source": [
+        "%synapse start -c linktestpool --start-timeout 1000"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "nteract": {
+          "transient": {
+            "deleting": false
+          }
+        }
+      },
+      "source": [
+        "# 2. Use Scala"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "nteract": {
+          "transient": {
+            "deleting": false
+          }
+        }
+      },
+      "source": [
+        "## (1) Read Data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "jupyter": {
+          "outputs_hidden": false,
+          "source_hidden": false
+        },
+        "nteract": {
+          "transient": {
+            "deleting": false
+          }
+        }
+      },
+      "outputs": [],
+      "source": [
+        "%%synapse scala\n",
+        "\n",
+        "var df = spark.read.option(\"header\", \"true\").csv(\"wasbs://demo@dprepdata.blob.core.windows.net/Titanic.csv\")\n",
+        "df.show(5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## (2) Use Scala Sql"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "jupyter": {
+          "outputs_hidden": false,
+          "source_hidden": false
+        },
+        "nteract": {
+          "transient": {
+            "deleting": false
+          }
+        }
+      },
+      "outputs": [],
+      "source": [
+        "%%synapse scala\n",
+        "\n",
+        "df.createOrReplaceTempView(\"titanic\")\n",
+        "var sqlDF = spark.sql(\"SELECT Name, Fare from titanic\")\n",
+        "sqlDF.show(5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Stop Session"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "jupyter": {
+          "outputs_hidden": false,
+          "source_hidden": false
+        },
+        "nteract": {
+          "transient": {
+            "deleting": false
+          }
+        }
+      },
+      "outputs": [],
+      "source": [
+        "%synapse stop"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "feli1"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.5"
+    },
+    "nteract": {
+      "version": "0.28.0"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}

how-to-use-azureml/azure-synapse/Titanic.csv

@@ -0,0 +1,892 @@
+PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked
+1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S
+2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C
+3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S
+4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S
+5,0,3,"Allen, Mr. William Henry",male,35,0,0,373450,8.05,,S
+6,0,3,"Moran, Mr. James",male,,0,0,330877,8.4583,,Q
+7,0,1,"McCarthy, Mr. Timothy J",male,54,0,0,17463,51.8625,E46,S
+8,0,3,"Palsson, Master. Gosta Leonard",male,2,3,1,349909,21.075,,S
+9,1,3,"Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)",female,27,0,2,347742,11.1333,,S
+10,1,2,"Nasser, Mrs. Nicholas (Adele Achem)",female,14,1,0,237736,30.0708,,C
+11,1,3,"Sandstrom, Miss. Marguerite Rut",female,4,1,1,PP 9549,16.7,G6,S
+12,1,1,"Bonnell, Miss. Elizabeth",female,58,0,0,113783,26.55,C103,S
+13,0,3,"Saundercock, Mr. William Henry",male,20,0,0,A/5. 2151,8.05,,S
+14,0,3,"Andersson, Mr. Anders Johan",male,39,1,5,347082,31.275,,S
+15,0,3,"Vestrom, Miss. Hulda Amanda Adolfina",female,14,0,0,350406,7.8542,,S
+16,1,2,"Hewlett, Mrs. (Mary D Kingcome) ",female,55,0,0,248706,16,,S
+17,0,3,"Rice, Master. Eugene",male,2,4,1,382652,29.125,,Q
+18,1,2,"Williams, Mr. Charles Eugene",male,,0,0,244373,13,,S
+19,0,3,"Vander Planke, Mrs. Julius (Emelia Maria Vandemoortele)",female,31,1,0,345763,18,,S
+20,1,3,"Masselmani, Mrs. Fatima",female,,0,0,2649,7.225,,C
+21,0,2,"Fynney, Mr. Joseph J",male,35,0,0,239865,26,,S
+22,1,2,"Beesley, Mr. Lawrence",male,34,0,0,248698,13,D56,S
+23,1,3,"McGowan, Miss. Anna ""Annie""",female,15,0,0,330923,8.0292,,Q
+24,1,1,"Sloper, Mr. William Thompson",male,28,0,0,113788,35.5,A6,S
+25,0,3,"Palsson, Miss. Torborg Danira",female,8,3,1,349909,21.075,,S
+26,1,3,"Asplund, Mrs. Carl Oscar (Selma Augusta Emilia Johansson)",female,38,1,5,347077,31.3875,,S
+27,0,3,"Emir, Mr. Farred Chehab",male,,0,0,2631,7.225,,C
+28,0,1,"Fortune, Mr. Charles Alexander",male,19,3,2,19950,263,C23 C25 C27,S
+29,1,3,"O'Dwyer, Miss. Ellen ""Nellie""",female,,0,0,330959,7.8792,,Q
+30,0,3,"Todoroff, Mr. Lalio",male,,0,0,349216,7.8958,,S
+31,0,1,"Uruchurtu, Don. Manuel E",male,40,0,0,PC 17601,27.7208,,C
+32,1,1,"Spencer, Mrs. William Augustus (Marie Eugenie)",female,,1,0,PC 17569,146.5208,B78,C
+33,1,3,"Glynn, Miss. Mary Agatha",female,,0,0,335677,7.75,,Q
+34,0,2,"Wheadon, Mr. Edward H",male,66,0,0,C.A. 24579,10.5,,S
+35,0,1,"Meyer, Mr. Edgar Joseph",male,28,1,0,PC 17604,82.1708,,C
+36,0,1,"Holverson, Mr. Alexander Oskar",male,42,1,0,113789,52,,S
+37,1,3,"Mamee, Mr. Hanna",male,,0,0,2677,7.2292,,C
+38,0,3,"Cann, Mr. Ernest Charles",male,21,0,0,A./5. 2152,8.05,,S
+39,0,3,"Vander Planke, Miss. Augusta Maria",female,18,2,0,345764,18,,S
+40,1,3,"Nicola-Yarred, Miss. Jamila",female,14,1,0,2651,11.2417,,C
+41,0,3,"Ahlin, Mrs. Johan (Johanna Persdotter Larsson)",female,40,1,0,7546,9.475,,S
+42,0,2,"Turpin, Mrs. William John Robert (Dorothy Ann Wonnacott)",female,27,1,0,11668,21,,S
+43,0,3,"Kraeff, Mr. Theodor",male,,0,0,349253,7.8958,,C
+44,1,2,"Laroche, Miss. Simonne Marie Anne Andree",female,3,1,2,SC/Paris 2123,41.5792,,C
+45,1,3,"Devaney, Miss. Margaret Delia",female,19,0,0,330958,7.8792,,Q
+46,0,3,"Rogers, Mr. William John",male,,0,0,S.C./A.4. 23567,8.05,,S
+47,0,3,"Lennon, Mr. Denis",male,,1,0,370371,15.5,,Q
+48,1,3,"O'Driscoll, Miss. Bridget",female,,0,0,14311,7.75,,Q
+49,0,3,"Samaan, Mr. Youssef",male,,2,0,2662,21.6792,,C
+50,0,3,"Arnold-Franchi, Mrs. Josef (Josefine Franchi)",female,18,1,0,349237,17.8,,S
+51,0,3,"Panula, Master. Juha Niilo",male,7,4,1,3101295,39.6875,,S
+52,0,3,"Nosworthy, Mr. Richard Cater",male,21,0,0,A/4. 39886,7.8,,S
+53,1,1,"Harper, Mrs. Henry Sleeper (Myna Haxtun)",female,49,1,0,PC 17572,76.7292,D33,C
+54,1,2,"Faunthorpe, Mrs. Lizzie (Elizabeth Anne Wilkinson)",female,29,1,0,2926,26,,S
+55,0,1,"Ostby, Mr. Engelhart Cornelius",male,65,0,1,113509,61.9792,B30,C
+56,1,1,"Woolner, Mr. Hugh",male,,0,0,19947,35.5,C52,S
+57,1,2,"Rugg, Miss. Emily",female,21,0,0,C.A. 31026,10.5,,S
+58,0,3,"Novel, Mr. Mansouer",male,28.5,0,0,2697,7.2292,,C
+59,1,2,"West, Miss. Constance Mirium",female,5,1,2,C.A. 34651,27.75,,S
+60,0,3,"Goodwin, Master. William Frederick",male,11,5,2,CA 2144,46.9,,S
+61,0,3,"Sirayanian, Mr. Orsen",male,22,0,0,2669,7.2292,,C
+62,1,1,"Icard, Miss. Amelie",female,38,0,0,113572,80,B28,
+63,0,1,"Harris, Mr. Henry Birkhardt",male,45,1,0,36973,83.475,C83,S
+64,0,3,"Skoog, Master. Harald",male,4,3,2,347088,27.9,,S
+65,0,1,"Stewart, Mr. Albert A",male,,0,0,PC 17605,27.7208,,C
+66,1,3,"Moubarek, Master. Gerios",male,,1,1,2661,15.2458,,C
+67,1,2,"Nye, Mrs. (Elizabeth Ramell)",female,29,0,0,C.A. 29395,10.5,F33,S
+68,0,3,"Crease, Mr. Ernest James",male,19,0,0,S.P. 3464,8.1583,,S
+69,1,3,"Andersson, Miss. Erna Alexandra",female,17,4,2,3101281,7.925,,S
+70,0,3,"Kink, Mr. Vincenz",male,26,2,0,315151,8.6625,,S
+71,0,2,"Jenkin, Mr. Stephen Curnow",male,32,0,0,C.A. 33111,10.5,,S
+72,0,3,"Goodwin, Miss. Lillian Amy",female,16,5,2,CA 2144,46.9,,S
+73,0,2,"Hood, Mr. Ambrose Jr",male,21,0,0,S.O.C. 14879,73.5,,S
+74,0,3,"Chronopoulos, Mr. Apostolos",male,26,1,0,2680,14.4542,,C
+75,1,3,"Bing, Mr. Lee",male,32,0,0,1601,56.4958,,S
+76,0,3,"Moen, Mr. Sigurd Hansen",male,25,0,0,348123,7.65,F G73,S
+77,0,3,"Staneff, Mr. Ivan",male,,0,0,349208,7.8958,,S
+78,0,3,"Moutal, Mr. Rahamin Haim",male,,0,0,374746,8.05,,S
+79,1,2,"Caldwell, Master. Alden Gates",male,0.83,0,2,248738,29,,S
+80,1,3,"Dowdell, Miss. Elizabeth",female,30,0,0,364516,12.475,,S
+81,0,3,"Waelens, Mr. Achille",male,22,0,0,345767,9,,S
+82,1,3,"Sheerlinck, Mr. Jan Baptist",male,29,0,0,345779,9.5,,S
+83,1,3,"McDermott, Miss. Brigdet Delia",female,,0,0,330932,7.7875,,Q
+84,0,1,"Carrau, Mr. Francisco M",male,28,0,0,113059,47.1,,S
+85,1,2,"Ilett, Miss. Bertha",female,17,0,0,SO/C 14885,10.5,,S
+86,1,3,"Backstrom, Mrs. Karl Alfred (Maria Mathilda Gustafsson)",female,33,3,0,3101278,15.85,,S
+87,0,3,"Ford, Mr. William Neal",male,16,1,3,W./C. 6608,34.375,,S
+88,0,3,"Slocovski, Mr. Selman Francis",male,,0,0,SOTON/OQ 392086,8.05,,S
+89,1,1,"Fortune, Miss. Mabel Helen",female,23,3,2,19950,263,C23 C25 C27,S
+90,0,3,"Celotti, Mr. Francesco",male,24,0,0,343275,8.05,,S
+91,0,3,"Christmann, Mr. Emil",male,29,0,0,343276,8.05,,S
+92,0,3,"Andreasson, Mr. Paul Edvin",male,20,0,0,347466,7.8542,,S
+93,0,1,"Chaffee, Mr. Herbert Fuller",male,46,1,0,W.E.P. 5734,61.175,E31,S
+94,0,3,"Dean, Mr. Bertram Frank",male,26,1,2,C.A. 2315,20.575,,S
+95,0,3,"Coxon, Mr. Daniel",male,59,0,0,364500,7.25,,S
+96,0,3,"Shorney, Mr. Charles Joseph",male,,0,0,374910,8.05,,S
+97,0,1,"Goldschmidt, Mr. George B",male,71,0,0,PC 17754,34.6542,A5,C
+98,1,1,"Greenfield, Mr. William Bertram",male,23,0,1,PC 17759,63.3583,D10 D12,C
+99,1,2,"Doling, Mrs. John T (Ada Julia Bone)",female,34,0,1,231919,23,,S
+100,0,2,"Kantor, Mr. Sinai",male,34,1,0,244367,26,,S
+101,0,3,"Petranec, Miss. Matilda",female,28,0,0,349245,7.8958,,S
+102,0,3,"Petroff, Mr. Pastcho (""Pentcho"")",male,,0,0,349215,7.8958,,S
+103,0,1,"White, Mr. Richard Frasar",male,21,0,1,35281,77.2875,D26,S
+104,0,3,"Johansson, Mr. Gustaf Joel",male,33,0,0,7540,8.6542,,S
+105,0,3,"Gustafsson, Mr. Anders Vilhelm",male,37,2,0,3101276,7.925,,S
+106,0,3,"Mionoff, Mr. Stoytcho",male,28,0,0,349207,7.8958,,S
+107,1,3,"Salkjelsvik, Miss. Anna Kristine",female,21,0,0,343120,7.65,,S
+108,1,3,"Moss, Mr. Albert Johan",male,,0,0,312991,7.775,,S
+109,0,3,"Rekic, Mr. Tido",male,38,0,0,349249,7.8958,,S
+110,1,3,"Moran, Miss. Bertha",female,,1,0,371110,24.15,,Q
+111,0,1,"Porter, Mr. Walter Chamberlain",male,47,0,0,110465,52,C110,S
+112,0,3,"Zabour, Miss. Hileni",female,14.5,1,0,2665,14.4542,,C
+113,0,3,"Barton, Mr. David John",male,22,0,0,324669,8.05,,S
+114,0,3,"Jussila, Miss. Katriina",female,20,1,0,4136,9.825,,S
+115,0,3,"Attalah, Miss. Malake",female,17,0,0,2627,14.4583,,C
+116,0,3,"Pekoniemi, Mr. Edvard",male,21,0,0,STON/O 2. 3101294,7.925,,S
+117,0,3,"Connors, Mr. Patrick",male,70.5,0,0,370369,7.75,,Q
+118,0,2,"Turpin, Mr. William John Robert",male,29,1,0,11668,21,,S
+119,0,1,"Baxter, Mr. Quigg Edmond",male,24,0,1,PC 17558,247.5208,B58 B60,C
+120,0,3,"Andersson, Miss. Ellis Anna Maria",female,2,4,2,347082,31.275,,S
+121,0,2,"Hickman, Mr. Stanley George",male,21,2,0,S.O.C. 14879,73.5,,S
+122,0,3,"Moore, Mr. Leonard Charles",male,,0,0,A4. 54510,8.05,,S
+123,0,2,"Nasser, Mr. Nicholas",male,32.5,1,0,237736,30.0708,,C
+124,1,2,"Webber, Miss. Susan",female,32.5,0,0,27267,13,E101,S
+125,0,1,"White, Mr. Percival Wayland",male,54,0,1,35281,77.2875,D26,S
+126,1,3,"Nicola-Yarred, Master. Elias",male,12,1,0,2651,11.2417,,C
+127,0,3,"McMahon, Mr. Martin",male,,0,0,370372,7.75,,Q
+128,1,3,"Madsen, Mr. Fridtjof Arne",male,24,0,0,C 17369,7.1417,,S
+129,1,3,"Peter, Miss. Anna",female,,1,1,2668,22.3583,F E69,C
+130,0,3,"Ekstrom, Mr. Johan",male,45,0,0,347061,6.975,,S
+131,0,3,"Drazenoic, Mr. Jozef",male,33,0,0,349241,7.8958,,C
+132,0,3,"Coelho, Mr. Domingos Fernandeo",male,20,0,0,SOTON/O.Q. 3101307,7.05,,S
+133,0,3,"Robins, Mrs. Alexander A (Grace Charity Laury)",female,47,1,0,A/5. 3337,14.5,,S
+134,1,2,"Weisz, Mrs. Leopold (Mathilde Francoise Pede)",female,29,1,0,228414,26,,S
+135,0,2,"Sobey, Mr. Samuel James Hayden",male,25,0,0,C.A. 29178,13,,S
+136,0,2,"Richard, Mr. Emile",male,23,0,0,SC/PARIS 2133,15.0458,,C
+137,1,1,"Newsom, Miss. Helen Monypeny",female,19,0,2,11752,26.2833,D47,S
+138,0,1,"Futrelle, Mr. Jacques Heath",male,37,1,0,113803,53.1,C123,S
+139,0,3,"Osen, Mr. Olaf Elon",male,16,0,0,7534,9.2167,,S
+140,0,1,"Giglio, Mr. Victor",male,24,0,0,PC 17593,79.2,B86,C
+141,0,3,"Boulos, Mrs. Joseph (Sultana)",female,,0,2,2678,15.2458,,C
+142,1,3,"Nysten, Miss. Anna Sofia",female,22,0,0,347081,7.75,,S
+143,1,3,"Hakkarainen, Mrs. Pekka Pietari (Elin Matilda Dolck)",female,24,1,0,STON/O2. 3101279,15.85,,S
+144,0,3,"Burke, Mr. Jeremiah",male,19,0,0,365222,6.75,,Q
+145,0,2,"Andrew, Mr. Edgardo Samuel",male,18,0,0,231945,11.5,,S
+146,0,2,"Nicholls, Mr. Joseph Charles",male,19,1,1,C.A. 33112,36.75,,S
+147,1,3,"Andersson, Mr. August Edvard (""Wennerstrom"")",male,27,0,0,350043,7.7958,,S
+148,0,3,"Ford, Miss. Robina Maggie ""Ruby""",female,9,2,2,W./C. 6608,34.375,,S
+149,0,2,"Navratil, Mr. Michel (""Louis M Hoffman"")",male,36.5,0,2,230080,26,F2,S
+150,0,2,"Byles, Rev. Thomas Roussel Davids",male,42,0,0,244310,13,,S
+151,0,2,"Bateman, Rev. Robert James",male,51,0,0,S.O.P. 1166,12.525,,S
+152,1,1,"Pears, Mrs. Thomas (Edith Wearne)",female,22,1,0,113776,66.6,C2,S
+153,0,3,"Meo, Mr. Alfonzo",male,55.5,0,0,A.5. 11206,8.05,,S
+154,0,3,"van Billiard, Mr. Austin Blyler",male,40.5,0,2,A/5. 851,14.5,,S
+155,0,3,"Olsen, Mr. Ole Martin",male,,0,0,Fa 265302,7.3125,,S
+156,0,1,"Williams, Mr. Charles Duane",male,51,0,1,PC 17597,61.3792,,C
+157,1,3,"Gilnagh, Miss. Katherine ""Katie""",female,16,0,0,35851,7.7333,,Q
+158,0,3,"Corn, Mr. Harry",male,30,0,0,SOTON/OQ 392090,8.05,,S
+159,0,3,"Smiljanic, Mr. Mile",male,,0,0,315037,8.6625,,S
+160,0,3,"Sage, Master. Thomas Henry",male,,8,2,CA. 2343,69.55,,S
+161,0,3,"Cribb, Mr. John Hatfield",male,44,0,1,371362,16.1,,S
+162,1,2,"Watt, Mrs. James (Elizabeth ""Bessie"" Inglis Milne)",female,40,0,0,C.A. 33595,15.75,,S
+163,0,3,"Bengtsson, Mr. John Viktor",male,26,0,0,347068,7.775,,S
+164,0,3,"Calic, Mr. Jovo",male,17,0,0,315093,8.6625,,S
+165,0,3,"Panula, Master. Eino Viljami",male,1,4,1,3101295,39.6875,,S
+166,1,3,"Goldsmith, Master. Frank John William ""Frankie""",male,9,0,2,363291,20.525,,S
+167,1,1,"Chibnall, Mrs. (Edith Martha Bowerman)",female,,0,1,113505,55,E33,S
+168,0,3,"Skoog, Mrs. William (Anna Bernhardina Karlsson)",female,45,1,4,347088,27.9,,S
+169,0,1,"Baumann, Mr. John D",male,,0,0,PC 17318,25.925,,S
+170,0,3,"Ling, Mr. Lee",male,28,0,0,1601,56.4958,,S
+171,0,1,"Van der hoef, Mr. Wyckoff",male,61,0,0,111240,33.5,B19,S
+172,0,3,"Rice, Master. Arthur",male,4,4,1,382652,29.125,,Q
+173,1,3,"Johnson, Miss. Eleanor Ileen",female,1,1,1,347742,11.1333,,S
+174,0,3,"Sivola, Mr. Antti Wilhelm",male,21,0,0,STON/O 2. 3101280,7.925,,S
+175,0,1,"Smith, Mr. James Clinch",male,56,0,0,17764,30.6958,A7,C
+176,0,3,"Klasen, Mr. Klas Albin",male,18,1,1,350404,7.8542,,S
+177,0,3,"Lefebre, Master. Henry Forbes",male,,3,1,4133,25.4667,,S
+178,0,1,"Isham, Miss. Ann Elizabeth",female,50,0,0,PC 17595,28.7125,C49,C
+179,0,2,"Hale, Mr. Reginald",male,30,0,0,250653,13,,S
+180,0,3,"Leonard, Mr. Lionel",male,36,0,0,LINE,0,,S
+181,0,3,"Sage, Miss. Constance Gladys",female,,8,2,CA. 2343,69.55,,S
+182,0,2,"Pernot, Mr. Rene",male,,0,0,SC/PARIS 2131,15.05,,C
+183,0,3,"Asplund, Master. Clarence Gustaf Hugo",male,9,4,2,347077,31.3875,,S
+184,1,2,"Becker, Master. Richard F",male,1,2,1,230136,39,F4,S
+185,1,3,"Kink-Heilmann, Miss. Luise Gretchen",female,4,0,2,315153,22.025,,S
+186,0,1,"Rood, Mr. Hugh Roscoe",male,,0,0,113767,50,A32,S
+187,1,3,"O'Brien, Mrs. Thomas (Johanna ""Hannah"" Godfrey)",female,,1,0,370365,15.5,,Q
+188,1,1,"Romaine, Mr. Charles Hallace (""Mr C Rolmane"")",male,45,0,0,111428,26.55,,S
+189,0,3,"Bourke, Mr. John",male,40,1,1,364849,15.5,,Q
+190,0,3,"Turcin, Mr. Stjepan",male,36,0,0,349247,7.8958,,S
+191,1,2,"Pinsky, Mrs. (Rosa)",female,32,0,0,234604,13,,S
+192,0,2,"Carbines, Mr. William",male,19,0,0,28424,13,,S
+193,1,3,"Andersen-Jensen, Miss. Carla Christine Nielsine",female,19,1,0,350046,7.8542,,S
+194,1,2,"Navratil, Master. Michel M",male,3,1,1,230080,26,F2,S
+195,1,1,"Brown, Mrs. James Joseph (Margaret Tobin)",female,44,0,0,PC 17610,27.7208,B4,C
+196,1,1,"Lurette, Miss. Elise",female,58,0,0,PC 17569,146.5208,B80,C
+197,0,3,"Mernagh, Mr. Robert",male,,0,0,368703,7.75,,Q
+198,0,3,"Olsen, Mr. Karl Siegwart Andreas",male,42,0,1,4579,8.4042,,S
+199,1,3,"Madigan, Miss. Margaret ""Maggie""",female,,0,0,370370,7.75,,Q
+200,0,2,"Yrois, Miss. Henriette (""Mrs Harbeck"")",female,24,0,0,248747,13,,S
+201,0,3,"Vande Walle, Mr. Nestor Cyriel",male,28,0,0,345770,9.5,,S
+202,0,3,"Sage, Mr. Frederick",male,,8,2,CA. 2343,69.55,,S
+203,0,3,"Johanson, Mr. Jakob Alfred",male,34,0,0,3101264,6.4958,,S
+204,0,3,"Youseff, Mr. Gerious",male,45.5,0,0,2628,7.225,,C
+205,1,3,"Cohen, Mr. Gurshon ""Gus""",male,18,0,0,A/5 3540,8.05,,S
+206,0,3,"Strom, Miss. Telma Matilda",female,2,0,1,347054,10.4625,G6,S
+207,0,3,"Backstrom, Mr. Karl Alfred",male,32,1,0,3101278,15.85,,S
+208,1,3,"Albimona, Mr. Nassef Cassem",male,26,0,0,2699,18.7875,,C
+209,1,3,"Carr, Miss. Helen ""Ellen""",female,16,0,0,367231,7.75,,Q
+210,1,1,"Blank, Mr. Henry",male,40,0,0,112277,31,A31,C
+211,0,3,"Ali, Mr. Ahmed",male,24,0,0,SOTON/O.Q. 3101311,7.05,,S
+212,1,2,"Cameron, Miss. Clear Annie",female,35,0,0,F.C.C. 13528,21,,S
+213,0,3,"Perkin, Mr. John Henry",male,22,0,0,A/5 21174,7.25,,S
+214,0,2,"Givard, Mr. Hans Kristensen",male,30,0,0,250646,13,,S
+215,0,3,"Kiernan, Mr. Philip",male,,1,0,367229,7.75,,Q
+216,1,1,"Newell, Miss. Madeleine",female,31,1,0,35273,113.275,D36,C
+217,1,3,"Honkanen, Miss. Eliina",female,27,0,0,STON/O2. 3101283,7.925,,S
+218,0,2,"Jacobsohn, Mr. Sidney Samuel",male,42,1,0,243847,27,,S
+219,1,1,"Bazzani, Miss. Albina",female,32,0,0,11813,76.2917,D15,C
+220,0,2,"Harris, Mr. Walter",male,30,0,0,W/C 14208,10.5,,S
+221,1,3,"Sunderland, Mr. Victor Francis",male,16,0,0,SOTON/OQ 392089,8.05,,S
+222,0,2,"Bracken, Mr. James H",male,27,0,0,220367,13,,S
+223,0,3,"Green, Mr. George Henry",male,51,0,0,21440,8.05,,S
+224,0,3,"Nenkoff, Mr. Christo",male,,0,0,349234,7.8958,,S
+225,1,1,"Hoyt, Mr. Frederick Maxfield",male,38,1,0,19943,90,C93,S
+226,0,3,"Berglund, Mr. Karl Ivar Sven",male,22,0,0,PP 4348,9.35,,S
+227,1,2,"Mellors, Mr. William John",male,19,0,0,SW/PP 751,10.5,,S
+228,0,3,"Lovell, Mr. John Hall (""Henry"")",male,20.5,0,0,A/5 21173,7.25,,S
+229,0,2,"Fahlstrom, Mr. Arne Jonas",male,18,0,0,236171,13,,S
+230,0,3,"Lefebre, Miss. Mathilde",female,,3,1,4133,25.4667,,S
+231,1,1,"Harris, Mrs. Henry Birkhardt (Irene Wallach)",female,35,1,0,36973,83.475,C83,S
+232,0,3,"Larsson, Mr. Bengt Edvin",male,29,0,0,347067,7.775,,S
+233,0,2,"Sjostedt, Mr. Ernst Adolf",male,59,0,0,237442,13.5,,S
+234,1,3,"Asplund, Miss. Lillian Gertrud",female,5,4,2,347077,31.3875,,S
+235,0,2,"Leyson, Mr. Robert William Norman",male,24,0,0,C.A. 29566,10.5,,S
+236,0,3,"Harknett, Miss. Alice Phoebe",female,,0,0,W./C. 6609,7.55,,S
+237,0,2,"Hold, Mr. Stephen",male,44,1,0,26707,26,,S
+238,1,2,"Collyer, Miss. Marjorie ""Lottie""",female,8,0,2,C.A. 31921,26.25,,S
+239,0,2,"Pengelly, Mr. Frederick William",male,19,0,0,28665,10.5,,S
+240,0,2,"Hunt, Mr. George Henry",male,33,0,0,SCO/W 1585,12.275,,S
+241,0,3,"Zabour, Miss. Thamine",female,,1,0,2665,14.4542,,C
+242,1,3,"Murphy, Miss. Katherine ""Kate""",female,,1,0,367230,15.5,,Q
+243,0,2,"Coleridge, Mr. Reginald Charles",male,29,0,0,W./C. 14263,10.5,,S
+244,0,3,"Maenpaa, Mr. Matti Alexanteri",male,22,0,0,STON/O 2. 3101275,7.125,,S
+245,0,3,"Attalah, Mr. Sleiman",male,30,0,0,2694,7.225,,C
+246,0,1,"Minahan, Dr. William Edward",male,44,2,0,19928,90,C78,Q
+247,0,3,"Lindahl, Miss. Agda Thorilda Viktoria",female,25,0,0,347071,7.775,,S
+248,1,2,"Hamalainen, Mrs. William (Anna)",female,24,0,2,250649,14.5,,S
+249,1,1,"Beckwith, Mr. Richard Leonard",male,37,1,1,11751,52.5542,D35,S
+250,0,2,"Carter, Rev. Ernest Courtenay",male,54,1,0,244252,26,,S
+251,0,3,"Reed, Mr. James George",male,,0,0,362316,7.25,,S
+252,0,3,"Strom, Mrs. Wilhelm (Elna Matilda Persson)",female,29,1,1,347054,10.4625,G6,S
+253,0,1,"Stead, Mr. William Thomas",male,62,0,0,113514,26.55,C87,S
+254,0,3,"Lobb, Mr. William Arthur",male,30,1,0,A/5. 3336,16.1,,S
+255,0,3,"Rosblom, Mrs. Viktor (Helena Wilhelmina)",female,41,0,2,370129,20.2125,,S
+256,1,3,"Touma, Mrs. Darwis (Hanne Youssef Razi)",female,29,0,2,2650,15.2458,,C
+257,1,1,"Thorne, Mrs. Gertrude Maybelle",female,,0,0,PC 17585,79.2,,C
+258,1,1,"Cherry, Miss. Gladys",female,30,0,0,110152,86.5,B77,S
+259,1,1,"Ward, Miss. Anna",female,35,0,0,PC 17755,512.3292,,C
+260,1,2,"Parrish, Mrs. (Lutie Davis)",female,50,0,1,230433,26,,S
+261,0,3,"Smith, Mr. Thomas",male,,0,0,384461,7.75,,Q
+262,1,3,"Asplund, Master. Edvin Rojj Felix",male,3,4,2,347077,31.3875,,S
+263,0,1,"Taussig, Mr. Emil",male,52,1,1,110413,79.65,E67,S
+264,0,1,"Harrison, Mr. William",male,40,0,0,112059,0,B94,S
+265,0,3,"Henry, Miss. Delia",female,,0,0,382649,7.75,,Q
+266,0,2,"Reeves, Mr. David",male,36,0,0,C.A. 17248,10.5,,S
+267,0,3,"Panula, Mr. Ernesti Arvid",male,16,4,1,3101295,39.6875,,S
+268,1,3,"Persson, Mr. Ernst Ulrik",male,25,1,0,347083,7.775,,S
+269,1,1,"Graham, Mrs. William Thompson (Edith Junkins)",female,58,0,1,PC 17582,153.4625,C125,S
+270,1,1,"Bissette, Miss. Amelia",female,35,0,0,PC 17760,135.6333,C99,S
+271,0,1,"Cairns, Mr. Alexander",male,,0,0,113798,31,,S
+272,1,3,"Tornquist, Mr. William Henry",male,25,0,0,LINE,0,,S
+273,1,2,"Mellinger, Mrs. (Elizabeth Anne Maidment)",female,41,0,1,250644,19.5,,S
+274,0,1,"Natsch, Mr. Charles H",male,37,0,1,PC 17596,29.7,C118,C
+275,1,3,"Healy, Miss. Hanora ""Nora""",female,,0,0,370375,7.75,,Q
+276,1,1,"Andrews, Miss. Kornelia Theodosia",female,63,1,0,13502,77.9583,D7,S
+277,0,3,"Lindblom, Miss. Augusta Charlotta",female,45,0,0,347073,7.75,,S
+278,0,2,"Parkes, Mr. Francis ""Frank""",male,,0,0,239853,0,,S
+279,0,3,"Rice, Master. Eric",male,7,4,1,382652,29.125,,Q
+280,1,3,"Abbott, Mrs. Stanton (Rosa Hunt)",female,35,1,1,C.A. 2673,20.25,,S
+281,0,3,"Duane, Mr. Frank",male,65,0,0,336439,7.75,,Q
+282,0,3,"Olsson, Mr. Nils Johan Goransson",male,28,0,0,347464,7.8542,,S
+283,0,3,"de Pelsmaeker, Mr. Alfons",male,16,0,0,345778,9.5,,S
+284,1,3,"Dorking, Mr. Edward Arthur",male,19,0,0,A/5. 10482,8.05,,S
+285,0,1,"Smith, Mr. Richard William",male,,0,0,113056,26,A19,S
+286,0,3,"Stankovic, Mr. Ivan",male,33,0,0,349239,8.6625,,C
+287,1,3,"de Mulder, Mr. Theodore",male,30,0,0,345774,9.5,,S
+288,0,3,"Naidenoff, Mr. Penko",male,22,0,0,349206,7.8958,,S
+289,1,2,"Hosono, Mr. Masabumi",male,42,0,0,237798,13,,S
+290,1,3,"Connolly, Miss. Kate",female,22,0,0,370373,7.75,,Q
+291,1,1,"Barber, Miss. Ellen ""Nellie""",female,26,0,0,19877,78.85,,S
+292,1,1,"Bishop, Mrs. Dickinson H (Helen Walton)",female,19,1,0,11967,91.0792,B49,C
+293,0,2,"Levy, Mr. Rene Jacques",male,36,0,0,SC/Paris 2163,12.875,D,C
+294,0,3,"Haas, Miss. Aloisia",female,24,0,0,349236,8.85,,S
+295,0,3,"Mineff, Mr. Ivan",male,24,0,0,349233,7.8958,,S
+296,0,1,"Lewy, Mr. Ervin G",male,,0,0,PC 17612,27.7208,,C
+297,0,3,"Hanna, Mr. Mansour",male,23.5,0,0,2693,7.2292,,C
+298,0,1,"Allison, Miss. Helen Loraine",female,2,1,2,113781,151.55,C22 C26,S
+299,1,1,"Saalfeld, Mr. Adolphe",male,,0,0,19988,30.5,C106,S
+300,1,1,"Baxter, Mrs. James (Helene DeLaudeniere Chaput)",female,50,0,1,PC 17558,247.5208,B58 B60,C
+301,1,3,"Kelly, Miss. Anna Katherine ""Annie Kate""",female,,0,0,9234,7.75,,Q
+302,1,3,"McCoy, Mr. Bernard",male,,2,0,367226,23.25,,Q
+303,0,3,"Johnson, Mr. William Cahoone Jr",male,19,0,0,LINE,0,,S
+304,1,2,"Keane, Miss. Nora A",female,,0,0,226593,12.35,E101,Q
+305,0,3,"Williams, Mr. Howard Hugh ""Harry""",male,,0,0,A/5 2466,8.05,,S
+306,1,1,"Allison, Master. Hudson Trevor",male,0.92,1,2,113781,151.55,C22 C26,S
+307,1,1,"Fleming, Miss. Margaret",female,,0,0,17421,110.8833,,C
+308,1,1,"Penasco y Castellana, Mrs. Victor de Satode (Maria Josefa Perez de Soto y Vallejo)",female,17,1,0,PC 17758,108.9,C65,C
+309,0,2,"Abelson, Mr. Samuel",male,30,1,0,P/PP 3381,24,,C
+310,1,1,"Francatelli, Miss. Laura Mabel",female,30,0,0,PC 17485,56.9292,E36,C
+311,1,1,"Hays, Miss. Margaret Bechstein",female,24,0,0,11767,83.1583,C54,C
+312,1,1,"Ryerson, Miss. Emily Borie",female,18,2,2,PC 17608,262.375,B57 B59 B63 B66,C
+313,0,2,"Lahtinen, Mrs. William (Anna Sylfven)",female,26,1,1,250651,26,,S
+314,0,3,"Hendekovic, Mr. Ignjac",male,28,0,0,349243,7.8958,,S
+315,0,2,"Hart, Mr. Benjamin",male,43,1,1,F.C.C. 13529,26.25,,S
+316,1,3,"Nilsson, Miss. Helmina Josefina",female,26,0,0,347470,7.8542,,S
+317,1,2,"Kantor, Mrs. Sinai (Miriam Sternin)",female,24,1,0,244367,26,,S
+318,0,2,"Moraweck, Dr. Ernest",male,54,0,0,29011,14,,S
+319,1,1,"Wick, Miss. Mary Natalie",female,31,0,2,36928,164.8667,C7,S
+320,1,1,"Spedden, Mrs. Frederic Oakley (Margaretta Corning Stone)",female,40,1,1,16966,134.5,E34,C
+321,0,3,"Dennis, Mr. Samuel",male,22,0,0,A/5 21172,7.25,,S
+322,0,3,"Danoff, Mr. Yoto",male,27,0,0,349219,7.8958,,S
+323,1,2,"Slayter, Miss. Hilda Mary",female,30,0,0,234818,12.35,,Q
+324,1,2,"Caldwell, Mrs. Albert Francis (Sylvia Mae Harbaugh)",female,22,1,1,248738,29,,S
+325,0,3,"Sage, Mr. George John Jr",male,,8,2,CA. 2343,69.55,,S
+326,1,1,"Young, Miss. Marie Grice",female,36,0,0,PC 17760,135.6333,C32,C
+327,0,3,"Nysveen, Mr. Johan Hansen",male,61,0,0,345364,6.2375,,S
+328,1,2,"Ball, Mrs. (Ada E Hall)",female,36,0,0,28551,13,D,S
+329,1,3,"Goldsmith, Mrs. Frank John (Emily Alice Brown)",female,31,1,1,363291,20.525,,S
+330,1,1,"Hippach, Miss. Jean Gertrude",female,16,0,1,111361,57.9792,B18,C
+331,1,3,"McCoy, Miss. Agnes",female,,2,0,367226,23.25,,Q
+332,0,1,"Partner, Mr. Austen",male,45.5,0,0,113043,28.5,C124,S
+333,0,1,"Graham, Mr. George Edward",male,38,0,1,PC 17582,153.4625,C91,S
+334,0,3,"Vander Planke, Mr. Leo Edmondus",male,16,2,0,345764,18,,S
+335,1,1,"Frauenthal, Mrs. Henry William (Clara Heinsheimer)",female,,1,0,PC 17611,133.65,,S
+336,0,3,"Denkoff, Mr. Mitto",male,,0,0,349225,7.8958,,S
+337,0,1,"Pears, Mr. Thomas Clinton",male,29,1,0,113776,66.6,C2,S
+338,1,1,"Burns, Miss. Elizabeth Margaret",female,41,0,0,16966,134.5,E40,C
+339,1,3,"Dahl, Mr. Karl Edwart",male,45,0,0,7598,8.05,,S
+340,0,1,"Blackwell, Mr. Stephen Weart",male,45,0,0,113784,35.5,T,S
+341,1,2,"Navratil, Master. Edmond Roger",male,2,1,1,230080,26,F2,S
+342,1,1,"Fortune, Miss. Alice Elizabeth",female,24,3,2,19950,263,C23 C25 C27,S
+343,0,2,"Collander, Mr. Erik Gustaf",male,28,0,0,248740,13,,S
+344,0,2,"Sedgwick, Mr. Charles Frederick Waddington",male,25,0,0,244361,13,,S
+345,0,2,"Fox, Mr. Stanley Hubert",male,36,0,0,229236,13,,S
+346,1,2,"Brown, Miss. Amelia ""Mildred""",female,24,0,0,248733,13,F33,S
+347,1,2,"Smith, Miss. Marion Elsie",female,40,0,0,31418,13,,S
+348,1,3,"Davison, Mrs. Thomas Henry (Mary E Finck)",female,,1,0,386525,16.1,,S
+349,1,3,"Coutts, Master. William Loch ""William""",male,3,1,1,C.A. 37671,15.9,,S
+350,0,3,"Dimic, Mr. Jovan",male,42,0,0,315088,8.6625,,S
+351,0,3,"Odahl, Mr. Nils Martin",male,23,0,0,7267,9.225,,S
+352,0,1,"Williams-Lambert, Mr. Fletcher Fellows",male,,0,0,113510,35,C128,S
+353,0,3,"Elias, Mr. Tannous",male,15,1,1,2695,7.2292,,C
+354,0,3,"Arnold-Franchi, Mr. Josef",male,25,1,0,349237,17.8,,S
+355,0,3,"Yousif, Mr. Wazli",male,,0,0,2647,7.225,,C
+356,0,3,"Vanden Steen, Mr. Leo Peter",male,28,0,0,345783,9.5,,S
+357,1,1,"Bowerman, Miss. Elsie Edith",female,22,0,1,113505,55,E33,S
+358,0,2,"Funk, Miss. Annie Clemmer",female,38,0,0,237671,13,,S
+359,1,3,"McGovern, Miss. Mary",female,,0,0,330931,7.8792,,Q
+360,1,3,"Mockler, Miss. Helen Mary ""Ellie""",female,,0,0,330980,7.8792,,Q
+361,0,3,"Skoog, Mr. Wilhelm",male,40,1,4,347088,27.9,,S
+362,0,2,"del Carlo, Mr. Sebastiano",male,29,1,0,SC/PARIS 2167,27.7208,,C
+363,0,3,"Barbara, Mrs. (Catherine David)",female,45,0,1,2691,14.4542,,C
+364,0,3,"Asim, Mr. Adola",male,35,0,0,SOTON/O.Q. 3101310,7.05,,S
+365,0,3,"O'Brien, Mr. Thomas",male,,1,0,370365,15.5,,Q
+366,0,3,"Adahl, Mr. Mauritz Nils Martin",male,30,0,0,C 7076,7.25,,S
+367,1,1,"Warren, Mrs. Frank Manley (Anna Sophia Atkinson)",female,60,1,0,110813,75.25,D37,C
+368,1,3,"Moussa, Mrs. (Mantoura Boulos)",female,,0,0,2626,7.2292,,C
+369,1,3,"Jermyn, Miss. Annie",female,,0,0,14313,7.75,,Q
+370,1,1,"Aubart, Mme. Leontine Pauline",female,24,0,0,PC 17477,69.3,B35,C
+371,1,1,"Harder, Mr. George Achilles",male,25,1,0,11765,55.4417,E50,C
+372,0,3,"Wiklund, Mr. Jakob Alfred",male,18,1,0,3101267,6.4958,,S
+373,0,3,"Beavan, Mr. William Thomas",male,19,0,0,323951,8.05,,S
+374,0,1,"Ringhini, Mr. Sante",male,22,0,0,PC 17760,135.6333,,C
+375,0,3,"Palsson, Miss. Stina Viola",female,3,3,1,349909,21.075,,S
+376,1,1,"Meyer, Mrs. Edgar Joseph (Leila Saks)",female,,1,0,PC 17604,82.1708,,C
+377,1,3,"Landergren, Miss. Aurora Adelia",female,22,0,0,C 7077,7.25,,S
+378,0,1,"Widener, Mr. Harry Elkins",male,27,0,2,113503,211.5,C82,C
+379,0,3,"Betros, Mr. Tannous",male,20,0,0,2648,4.0125,,C
+380,0,3,"Gustafsson, Mr. Karl Gideon",male,19,0,0,347069,7.775,,S
+381,1,1,"Bidois, Miss. Rosalie",female,42,0,0,PC 17757,227.525,,C
+382,1,3,"Nakid, Miss. Maria (""Mary"")",female,1,0,2,2653,15.7417,,C
+383,0,3,"Tikkanen, Mr. Juho",male,32,0,0,STON/O 2. 3101293,7.925,,S
+384,1,1,"Holverson, Mrs. Alexander Oskar (Mary Aline Towner)",female,35,1,0,113789,52,,S
+385,0,3,"Plotcharsky, Mr. Vasil",male,,0,0,349227,7.8958,,S
+386,0,2,"Davies, Mr. Charles Henry",male,18,0,0,S.O.C. 14879,73.5,,S
+387,0,3,"Goodwin, Master. Sidney Leonard",male,1,5,2,CA 2144,46.9,,S
+388,1,2,"Buss, Miss. Kate",female,36,0,0,27849,13,,S
+389,0,3,"Sadlier, Mr. Matthew",male,,0,0,367655,7.7292,,Q
+390,1,2,"Lehmann, Miss. Bertha",female,17,0,0,SC 1748,12,,C
+391,1,1,"Carter, Mr. William Ernest",male,36,1,2,113760,120,B96 B98,S
+392,1,3,"Jansson, Mr. Carl Olof",male,21,0,0,350034,7.7958,,S
+393,0,3,"Gustafsson, Mr. Johan Birger",male,28,2,0,3101277,7.925,,S
+394,1,1,"Newell, Miss. Marjorie",female,23,1,0,35273,113.275,D36,C
+395,1,3,"Sandstrom, Mrs. Hjalmar (Agnes Charlotta Bengtsson)",female,24,0,2,PP 9549,16.7,G6,S
+396,0,3,"Johansson, Mr. Erik",male,22,0,0,350052,7.7958,,S
+397,0,3,"Olsson, Miss. Elina",female,31,0,0,350407,7.8542,,S
+398,0,2,"McKane, Mr. Peter David",male,46,0,0,28403,26,,S
+399,0,2,"Pain, Dr. Alfred",male,23,0,0,244278,10.5,,S
+400,1,2,"Trout, Mrs. William H (Jessie L)",female,28,0,0,240929,12.65,,S
+401,1,3,"Niskanen, Mr. Juha",male,39,0,0,STON/O 2. 3101289,7.925,,S
+402,0,3,"Adams, Mr. John",male,26,0,0,341826,8.05,,S
+403,0,3,"Jussila, Miss. Mari Aina",female,21,1,0,4137,9.825,,S
+404,0,3,"Hakkarainen, Mr. Pekka Pietari",male,28,1,0,STON/O2. 3101279,15.85,,S
+405,0,3,"Oreskovic, Miss. Marija",female,20,0,0,315096,8.6625,,S
+406,0,2,"Gale, Mr. Shadrach",male,34,1,0,28664,21,,S
+407,0,3,"Widegren, Mr. Carl/Charles Peter",male,51,0,0,347064,7.75,,S
+408,1,2,"Richards, Master. William Rowe",male,3,1,1,29106,18.75,,S
+409,0,3,"Birkeland, Mr. Hans Martin Monsen",male,21,0,0,312992,7.775,,S
+410,0,3,"Lefebre, Miss. Ida",female,,3,1,4133,25.4667,,S
+411,0,3,"Sdycoff, Mr. Todor",male,,0,0,349222,7.8958,,S
+412,0,3,"Hart, Mr. Henry",male,,0,0,394140,6.8583,,Q
+413,1,1,"Minahan, Miss. Daisy E",female,33,1,0,19928,90,C78,Q
+414,0,2,"Cunningham, Mr. Alfred Fleming",male,,0,0,239853,0,,S
+415,1,3,"Sundman, Mr. Johan Julian",male,44,0,0,STON/O 2. 3101269,7.925,,S
+416,0,3,"Meek, Mrs. Thomas (Annie Louise Rowley)",female,,0,0,343095,8.05,,S
+417,1,2,"Drew, Mrs. James Vivian (Lulu Thorne Christian)",female,34,1,1,28220,32.5,,S
+418,1,2,"Silven, Miss. Lyyli Karoliina",female,18,0,2,250652,13,,S
+419,0,2,"Matthews, Mr. William John",male,30,0,0,28228,13,,S
+420,0,3,"Van Impe, Miss. Catharina",female,10,0,2,345773,24.15,,S
+421,0,3,"Gheorgheff, Mr. Stanio",male,,0,0,349254,7.8958,,C
+422,0,3,"Charters, Mr. David",male,21,0,0,A/5. 13032,7.7333,,Q
+423,0,3,"Zimmerman, Mr. Leo",male,29,0,0,315082,7.875,,S
+424,0,3,"Danbom, Mrs. Ernst Gilbert (Anna Sigrid Maria Brogren)",female,28,1,1,347080,14.4,,S
+425,0,3,"Rosblom, Mr. Viktor Richard",male,18,1,1,370129,20.2125,,S
+426,0,3,"Wiseman, Mr. Phillippe",male,,0,0,A/4. 34244,7.25,,S
+427,1,2,"Clarke, Mrs. Charles V (Ada Maria Winfield)",female,28,1,0,2003,26,,S
+428,1,2,"Phillips, Miss. Kate Florence (""Mrs Kate Louise Phillips Marshall"")",female,19,0,0,250655,26,,S
+429,0,3,"Flynn, Mr. James",male,,0,0,364851,7.75,,Q
+430,1,3,"Pickard, Mr. Berk (Berk Trembisky)",male,32,0,0,SOTON/O.Q. 392078,8.05,E10,S
+431,1,1,"Bjornstrom-Steffansson, Mr. Mauritz Hakan",male,28,0,0,110564,26.55,C52,S
+432,1,3,"Thorneycroft, Mrs. Percival (Florence Kate White)",female,,1,0,376564,16.1,,S
+433,1,2,"Louch, Mrs. Charles Alexander (Alice Adelaide Slow)",female,42,1,0,SC/AH 3085,26,,S
+434,0,3,"Kallio, Mr. Nikolai Erland",male,17,0,0,STON/O 2. 3101274,7.125,,S
+435,0,1,"Silvey, Mr. William Baird",male,50,1,0,13507,55.9,E44,S
+436,1,1,"Carter, Miss. Lucile Polk",female,14,1,2,113760,120,B96 B98,S
+437,0,3,"Ford, Miss. Doolina Margaret ""Daisy""",female,21,2,2,W./C. 6608,34.375,,S
+438,1,2,"Richards, Mrs. Sidney (Emily Hocking)",female,24,2,3,29106,18.75,,S
+439,0,1,"Fortune, Mr. Mark",male,64,1,4,19950,263,C23 C25 C27,S
+440,0,2,"Kvillner, Mr. Johan Henrik Johannesson",male,31,0,0,C.A. 18723,10.5,,S
+441,1,2,"Hart, Mrs. Benjamin (Esther Ada Bloomfield)",female,45,1,1,F.C.C. 13529,26.25,,S
+442,0,3,"Hampe, Mr. Leon",male,20,0,0,345769,9.5,,S
+443,0,3,"Petterson, Mr. Johan Emil",male,25,1,0,347076,7.775,,S
+444,1,2,"Reynaldo, Ms. Encarnacion",female,28,0,0,230434,13,,S
+445,1,3,"Johannesen-Bratthammer, Mr. Bernt",male,,0,0,65306,8.1125,,S
+446,1,1,"Dodge, Master. Washington",male,4,0,2,33638,81.8583,A34,S
+447,1,2,"Mellinger, Miss. Madeleine Violet",female,13,0,1,250644,19.5,,S
+448,1,1,"Seward, Mr. Frederic Kimber",male,34,0,0,113794,26.55,,S
+449,1,3,"Baclini, Miss. Marie Catherine",female,5,2,1,2666,19.2583,,C
+450,1,1,"Peuchen, Major. Arthur Godfrey",male,52,0,0,113786,30.5,C104,S
+451,0,2,"West, Mr. Edwy Arthur",male,36,1,2,C.A. 34651,27.75,,S
+452,0,3,"Hagland, Mr. Ingvald Olai Olsen",male,,1,0,65303,19.9667,,S
+453,0,1,"Foreman, Mr. Benjamin Laventall",male,30,0,0,113051,27.75,C111,C
+454,1,1,"Goldenberg, Mr. Samuel L",male,49,1,0,17453,89.1042,C92,C
+455,0,3,"Peduzzi, Mr. Joseph",male,,0,0,A/5 2817,8.05,,S
+456,1,3,"Jalsevac, Mr. Ivan",male,29,0,0,349240,7.8958,,C
+457,0,1,"Millet, Mr. Francis Davis",male,65,0,0,13509,26.55,E38,S
+458,1,1,"Kenyon, Mrs. Frederick R (Marion)",female,,1,0,17464,51.8625,D21,S
+459,1,2,"Toomey, Miss. Ellen",female,50,0,0,F.C.C. 13531,10.5,,S
+460,0,3,"O'Connor, Mr. Maurice",male,,0,0,371060,7.75,,Q
+461,1,1,"Anderson, Mr. Harry",male,48,0,0,19952,26.55,E12,S
+462,0,3,"Morley, Mr. William",male,34,0,0,364506,8.05,,S
+463,0,1,"Gee, Mr. Arthur H",male,47,0,0,111320,38.5,E63,S
+464,0,2,"Milling, Mr. Jacob Christian",male,48,0,0,234360,13,,S
+465,0,3,"Maisner, Mr. Simon",male,,0,0,A/S 2816,8.05,,S
+466,0,3,"Goncalves, Mr. Manuel Estanslas",male,38,0,0,SOTON/O.Q. 3101306,7.05,,S
+467,0,2,"Campbell, Mr. William",male,,0,0,239853,0,,S
+468,0,1,"Smart, Mr. John Montgomery",male,56,0,0,113792,26.55,,S
+469,0,3,"Scanlan, Mr. James",male,,0,0,36209,7.725,,Q
+470,1,3,"Baclini, Miss. Helene Barbara",female,0.75,2,1,2666,19.2583,,C
+471,0,3,"Keefe, Mr. Arthur",male,,0,0,323592,7.25,,S
+472,0,3,"Cacic, Mr. Luka",male,38,0,0,315089,8.6625,,S
+473,1,2,"West, Mrs. Edwy Arthur (Ada Mary Worth)",female,33,1,2,C.A. 34651,27.75,,S
+474,1,2,"Jerwan, Mrs. Amin S (Marie Marthe Thuillard)",female,23,0,0,SC/AH Basle 541,13.7917,D,C
+475,0,3,"Strandberg, Miss. Ida Sofia",female,22,0,0,7553,9.8375,,S
+476,0,1,"Clifford, Mr. George Quincy",male,,0,0,110465,52,A14,S
+477,0,2,"Renouf, Mr. Peter Henry",male,34,1,0,31027,21,,S
+478,0,3,"Braund, Mr. Lewis Richard",male,29,1,0,3460,7.0458,,S
+479,0,3,"Karlsson, Mr. Nils August",male,22,0,0,350060,7.5208,,S
+480,1,3,"Hirvonen, Miss. Hildur E",female,2,0,1,3101298,12.2875,,S
+481,0,3,"Goodwin, Master. Harold Victor",male,9,5,2,CA 2144,46.9,,S
+482,0,2,"Frost, Mr. Anthony Wood ""Archie""",male,,0,0,239854,0,,S
+483,0,3,"Rouse, Mr. Richard Henry",male,50,0,0,A/5 3594,8.05,,S
+484,1,3,"Turkula, Mrs. (Hedwig)",female,63,0,0,4134,9.5875,,S
+485,1,1,"Bishop, Mr. Dickinson H",male,25,1,0,11967,91.0792,B49,C
+486,0,3,"Lefebre, Miss. Jeannie",female,,3,1,4133,25.4667,,S
+487,1,1,"Hoyt, Mrs. Frederick Maxfield (Jane Anne Forby)",female,35,1,0,19943,90,C93,S
+488,0,1,"Kent, Mr. Edward Austin",male,58,0,0,11771,29.7,B37,C
+489,0,3,"Somerton, Mr. Francis William",male,30,0,0,A.5. 18509,8.05,,S
+490,1,3,"Coutts, Master. Eden Leslie ""Neville""",male,9,1,1,C.A. 37671,15.9,,S
+491,0,3,"Hagland, Mr. Konrad Mathias Reiersen",male,,1,0,65304,19.9667,,S
+492,0,3,"Windelov, Mr. Einar",male,21,0,0,SOTON/OQ 3101317,7.25,,S
+493,0,1,"Molson, Mr. Harry Markland",male,55,0,0,113787,30.5,C30,S
+494,0,1,"Artagaveytia, Mr. Ramon",male,71,0,0,PC 17609,49.5042,,C
+495,0,3,"Stanley, Mr. Edward Roland",male,21,0,0,A/4 45380,8.05,,S
+496,0,3,"Yousseff, Mr. Gerious",male,,0,0,2627,14.4583,,C
+497,1,1,"Eustis, Miss. Elizabeth Mussey",female,54,1,0,36947,78.2667,D20,C
+498,0,3,"Shellard, Mr. Frederick William",male,,0,0,C.A. 6212,15.1,,S
+499,0,1,"Allison, Mrs. Hudson J C (Bessie Waldo Daniels)",female,25,1,2,113781,151.55,C22 C26,S
+500,0,3,"Svensson, Mr. Olof",male,24,0,0,350035,7.7958,,S
+501,0,3,"Calic, Mr. Petar",male,17,0,0,315086,8.6625,,S
+502,0,3,"Canavan, Miss. Mary",female,21,0,0,364846,7.75,,Q
+503,0,3,"O'Sullivan, Miss. Bridget Mary",female,,0,0,330909,7.6292,,Q
+504,0,3,"Laitinen, Miss. Kristina Sofia",female,37,0,0,4135,9.5875,,S
+505,1,1,"Maioni, Miss. Roberta",female,16,0,0,110152,86.5,B79,S
+506,0,1,"Penasco y Castellana, Mr. Victor de Satode",male,18,1,0,PC 17758,108.9,C65,C
+507,1,2,"Quick, Mrs. Frederick Charles (Jane Richards)",female,33,0,2,26360,26,,S
+508,1,1,"Bradley, Mr. George (""George Arthur Brayton"")",male,,0,0,111427,26.55,,S
+509,0,3,"Olsen, Mr. Henry Margido",male,28,0,0,C 4001,22.525,,S
+510,1,3,"Lang, Mr. Fang",male,26,0,0,1601,56.4958,,S
+511,1,3,"Daly, Mr. Eugene Patrick",male,29,0,0,382651,7.75,,Q
+512,0,3,"Webber, Mr. James",male,,0,0,SOTON/OQ 3101316,8.05,,S
+513,1,1,"McGough, Mr. James Robert",male,36,0,0,PC 17473,26.2875,E25,S
+514,1,1,"Rothschild, Mrs. Martin (Elizabeth L. Barrett)",female,54,1,0,PC 17603,59.4,,C
+515,0,3,"Coleff, Mr. Satio",male,24,0,0,349209,7.4958,,S
+516,0,1,"Walker, Mr. William Anderson",male,47,0,0,36967,34.0208,D46,S
+517,1,2,"Lemore, Mrs. (Amelia Milley)",female,34,0,0,C.A. 34260,10.5,F33,S
+518,0,3,"Ryan, Mr. Patrick",male,,0,0,371110,24.15,,Q
+519,1,2,"Angle, Mrs. William A (Florence ""Mary"" Agnes Hughes)",female,36,1,0,226875,26,,S
+520,0,3,"Pavlovic, Mr. Stefo",male,32,0,0,349242,7.8958,,S
+521,1,1,"Perreault, Miss. Anne",female,30,0,0,12749,93.5,B73,S
+522,0,3,"Vovk, Mr. Janko",male,22,0,0,349252,7.8958,,S
+523,0,3,"Lahoud, Mr. Sarkis",male,,0,0,2624,7.225,,C
+524,1,1,"Hippach, Mrs. Louis Albert (Ida Sophia Fischer)",female,44,0,1,111361,57.9792,B18,C
+525,0,3,"Kassem, Mr. Fared",male,,0,0,2700,7.2292,,C
+526,0,3,"Farrell, Mr. James",male,40.5,0,0,367232,7.75,,Q
+527,1,2,"Ridsdale, Miss. Lucy",female,50,0,0,W./C. 14258,10.5,,S
+528,0,1,"Farthing, Mr. John",male,,0,0,PC 17483,221.7792,C95,S
+529,0,3,"Salonen, Mr. Johan Werner",male,39,0,0,3101296,7.925,,S
+530,0,2,"Hocking, Mr. Richard George",male,23,2,1,29104,11.5,,S
+531,1,2,"Quick, Miss. Phyllis May",female,2,1,1,26360,26,,S
+532,0,3,"Toufik, Mr. Nakli",male,,0,0,2641,7.2292,,C
+533,0,3,"Elias, Mr. Joseph Jr",male,17,1,1,2690,7.2292,,C
+534,1,3,"Peter, Mrs. Catherine (Catherine Rizk)",female,,0,2,2668,22.3583,,C
+535,0,3,"Cacic, Miss. Marija",female,30,0,0,315084,8.6625,,S
+536,1,2,"Hart, Miss. Eva Miriam",female,7,0,2,F.C.C. 13529,26.25,,S
+537,0,1,"Butt, Major. Archibald Willingham",male,45,0,0,113050,26.55,B38,S
+538,1,1,"LeRoy, Miss. Bertha",female,30,0,0,PC 17761,106.425,,C
+539,0,3,"Risien, Mr. Samuel Beard",male,,0,0,364498,14.5,,S
+540,1,1,"Frolicher, Miss. Hedwig Margaritha",female,22,0,2,13568,49.5,B39,C
+541,1,1,"Crosby, Miss. Harriet R",female,36,0,2,WE/P 5735,71,B22,S
+542,0,3,"Andersson, Miss. Ingeborg Constanzia",female,9,4,2,347082,31.275,,S
+543,0,3,"Andersson, Miss. Sigrid Elisabeth",female,11,4,2,347082,31.275,,S
+544,1,2,"Beane, Mr. Edward",male,32,1,0,2908,26,,S
+545,0,1,"Douglas, Mr. Walter Donald",male,50,1,0,PC 17761,106.425,C86,C
+546,0,1,"Nicholson, Mr. Arthur Ernest",male,64,0,0,693,26,,S
+547,1,2,"Beane, Mrs. Edward (Ethel Clarke)",female,19,1,0,2908,26,,S
+548,1,2,"Padro y Manent, Mr. Julian",male,,0,0,SC/PARIS 2146,13.8625,,C
+549,0,3,"Goldsmith, Mr. Frank John",male,33,1,1,363291,20.525,,S
+550,1,2,"Davies, Master. John Morgan Jr",male,8,1,1,C.A. 33112,36.75,,S
+551,1,1,"Thayer, Mr. John Borland Jr",male,17,0,2,17421,110.8833,C70,C
+552,0,2,"Sharp, Mr. Percival James R",male,27,0,0,244358,26,,S
+553,0,3,"O'Brien, Mr. Timothy",male,,0,0,330979,7.8292,,Q
+554,1,3,"Leeni, Mr. Fahim (""Philip Zenni"")",male,22,0,0,2620,7.225,,C
+555,1,3,"Ohman, Miss. Velin",female,22,0,0,347085,7.775,,S
+556,0,1,"Wright, Mr. George",male,62,0,0,113807,26.55,,S
+557,1,1,"Duff Gordon, Lady. (Lucille Christiana Sutherland) (""Mrs Morgan"")",female,48,1,0,11755,39.6,A16,C
+558,0,1,"Robbins, Mr. Victor",male,,0,0,PC 17757,227.525,,C
+559,1,1,"Taussig, Mrs. Emil (Tillie Mandelbaum)",female,39,1,1,110413,79.65,E67,S
+560,1,3,"de Messemaeker, Mrs. Guillaume Joseph (Emma)",female,36,1,0,345572,17.4,,S
+561,0,3,"Morrow, Mr. Thomas Rowan",male,,0,0,372622,7.75,,Q
+562,0,3,"Sivic, Mr. Husein",male,40,0,0,349251,7.8958,,S
+563,0,2,"Norman, Mr. Robert Douglas",male,28,0,0,218629,13.5,,S
+564,0,3,"Simmons, Mr. John",male,,0,0,SOTON/OQ 392082,8.05,,S
+565,0,3,"Meanwell, Miss. (Marion Ogden)",female,,0,0,SOTON/O.Q. 392087,8.05,,S
+566,0,3,"Davies, Mr. Alfred J",male,24,2,0,A/4 48871,24.15,,S
+567,0,3,"Stoytcheff, Mr. Ilia",male,19,0,0,349205,7.8958,,S
+568,0,3,"Palsson, Mrs. Nils (Alma Cornelia Berglund)",female,29,0,4,349909,21.075,,S
+569,0,3,"Doharr, Mr. Tannous",male,,0,0,2686,7.2292,,C
+570,1,3,"Jonsson, Mr. Carl",male,32,0,0,350417,7.8542,,S
+571,1,2,"Harris, Mr. George",male,62,0,0,S.W./PP 752,10.5,,S
+572,1,1,"Appleton, Mrs. Edward Dale (Charlotte Lamson)",female,53,2,0,11769,51.4792,C101,S
+573,1,1,"Flynn, Mr. John Irwin (""Irving"")",male,36,0,0,PC 17474,26.3875,E25,S
+574,1,3,"Kelly, Miss. Mary",female,,0,0,14312,7.75,,Q
+575,0,3,"Rush, Mr. Alfred George John",male,16,0,0,A/4. 20589,8.05,,S
+576,0,3,"Patchett, Mr. George",male,19,0,0,358585,14.5,,S
+577,1,2,"Garside, Miss. Ethel",female,34,0,0,243880,13,,S
+578,1,1,"Silvey, Mrs. William Baird (Alice Munger)",female,39,1,0,13507,55.9,E44,S
+579,0,3,"Caram, Mrs. Joseph (Maria Elias)",female,,1,0,2689,14.4583,,C
+580,1,3,"Jussila, Mr. Eiriik",male,32,0,0,STON/O 2. 3101286,7.925,,S
+581,1,2,"Christy, Miss. Julie Rachel",female,25,1,1,237789,30,,S
+582,1,1,"Thayer, Mrs. John Borland (Marian Longstreth Morris)",female,39,1,1,17421,110.8833,C68,C
+583,0,2,"Downton, Mr. William James",male,54,0,0,28403,26,,S
+584,0,1,"Ross, Mr. John Hugo",male,36,0,0,13049,40.125,A10,C
+585,0,3,"Paulner, Mr. Uscher",male,,0,0,3411,8.7125,,C
+586,1,1,"Taussig, Miss. Ruth",female,18,0,2,110413,79.65,E68,S
+587,0,2,"Jarvis, Mr. John Denzil",male,47,0,0,237565,15,,S
+588,1,1,"Frolicher-Stehli, Mr. Maxmillian",male,60,1,1,13567,79.2,B41,C
+589,0,3,"Gilinski, Mr. Eliezer",male,22,0,0,14973,8.05,,S
+590,0,3,"Murdlin, Mr. Joseph",male,,0,0,A./5. 3235,8.05,,S
+591,0,3,"Rintamaki, Mr. Matti",male,35,0,0,STON/O 2. 3101273,7.125,,S
+592,1,1,"Stephenson, Mrs. Walter Bertram (Martha Eustis)",female,52,1,0,36947,78.2667,D20,C
+593,0,3,"Elsbury, Mr. William James",male,47,0,0,A/5 3902,7.25,,S
+594,0,3,"Bourke, Miss. Mary",female,,0,2,364848,7.75,,Q
+595,0,2,"Chapman, Mr. John Henry",male,37,1,0,SC/AH 29037,26,,S
+596,0,3,"Van Impe, Mr. Jean Baptiste",male,36,1,1,345773,24.15,,S
+597,1,2,"Leitch, Miss. Jessie Wills",female,,0,0,248727,33,,S
+598,0,3,"Johnson, Mr. Alfred",male,49,0,0,LINE,0,,S
+599,0,3,"Boulos, Mr. Hanna",male,,0,0,2664,7.225,,C
+600,1,1,"Duff Gordon, Sir. Cosmo Edmund (""Mr Morgan"")",male,49,1,0,PC 17485,56.9292,A20,C
+601,1,2,"Jacobsohn, Mrs. Sidney Samuel (Amy Frances Christy)",female,24,2,1,243847,27,,S
+602,0,3,"Slabenoff, Mr. Petco",male,,0,0,349214,7.8958,,S
+603,0,1,"Harrington, Mr. Charles H",male,,0,0,113796,42.4,,S
+604,0,3,"Torber, Mr. Ernst William",male,44,0,0,364511,8.05,,S
+605,1,1,"Homer, Mr. Harry (""Mr E Haven"")",male,35,0,0,111426,26.55,,C
+606,0,3,"Lindell, Mr. Edvard Bengtsson",male,36,1,0,349910,15.55,,S
+607,0,3,"Karaic, Mr. Milan",male,30,0,0,349246,7.8958,,S
+608,1,1,"Daniel, Mr. Robert Williams",male,27,0,0,113804,30.5,,S
+609,1,2,"Laroche, Mrs. Joseph (Juliette Marie Louise Lafargue)",female,22,1,2,SC/Paris 2123,41.5792,,C
+610,1,1,"Shutes, Miss. Elizabeth W",female,40,0,0,PC 17582,153.4625,C125,S
+611,0,3,"Andersson, Mrs. Anders Johan (Alfrida Konstantia Brogren)",female,39,1,5,347082,31.275,,S
+612,0,3,"Jardin, Mr. Jose Neto",male,,0,0,SOTON/O.Q. 3101305,7.05,,S
+613,1,3,"Murphy, Miss. Margaret Jane",female,,1,0,367230,15.5,,Q
+614,0,3,"Horgan, Mr. John",male,,0,0,370377,7.75,,Q
+615,0,3,"Brocklebank, Mr. William Alfred",male,35,0,0,364512,8.05,,S
+616,1,2,"Herman, Miss. Alice",female,24,1,2,220845,65,,S
+617,0,3,"Danbom, Mr. Ernst Gilbert",male,34,1,1,347080,14.4,,S
+618,0,3,"Lobb, Mrs. William Arthur (Cordelia K Stanlick)",female,26,1,0,A/5. 3336,16.1,,S
+619,1,2,"Becker, Miss. Marion Louise",female,4,2,1,230136,39,F4,S
+620,0,2,"Gavey, Mr. Lawrence",male,26,0,0,31028,10.5,,S
+621,0,3,"Yasbeck, Mr. Antoni",male,27,1,0,2659,14.4542,,C
+622,1,1,"Kimball, Mr. Edwin Nelson Jr",male,42,1,0,11753,52.5542,D19,S
+623,1,3,"Nakid, Mr. Sahid",male,20,1,1,2653,15.7417,,C
+624,0,3,"Hansen, Mr. Henry Damsgaard",male,21,0,0,350029,7.8542,,S
+625,0,3,"Bowen, Mr. David John ""Dai""",male,21,0,0,54636,16.1,,S
+626,0,1,"Sutton, Mr. Frederick",male,61,0,0,36963,32.3208,D50,S
+627,0,2,"Kirkland, Rev. Charles Leonard",male,57,0,0,219533,12.35,,Q
+628,1,1,"Longley, Miss. Gretchen Fiske",female,21,0,0,13502,77.9583,D9,S
+629,0,3,"Bostandyeff, Mr. Guentcho",male,26,0,0,349224,7.8958,,S
+630,0,3,"O'Connell, Mr. Patrick D",male,,0,0,334912,7.7333,,Q
+631,1,1,"Barkworth, Mr. Algernon Henry Wilson",male,80,0,0,27042,30,A23,S
+632,0,3,"Lundahl, Mr. Johan Svensson",male,51,0,0,347743,7.0542,,S
+633,1,1,"Stahelin-Maeglin, Dr. Max",male,32,0,0,13214,30.5,B50,C
+634,0,1,"Parr, Mr. William Henry Marsh",male,,0,0,112052,0,,S
+635,0,3,"Skoog, Miss. Mabel",female,9,3,2,347088,27.9,,S
+636,1,2,"Davis, Miss. Mary",female,28,0,0,237668,13,,S
+637,0,3,"Leinonen, Mr. Antti Gustaf",male,32,0,0,STON/O 2. 3101292,7.925,,S
+638,0,2,"Collyer, Mr. Harvey",male,31,1,1,C.A. 31921,26.25,,S
+639,0,3,"Panula, Mrs. Juha (Maria Emilia Ojala)",female,41,0,5,3101295,39.6875,,S
+640,0,3,"Thorneycroft, Mr. Percival",male,,1,0,376564,16.1,,S
+641,0,3,"Jensen, Mr. Hans Peder",male,20,0,0,350050,7.8542,,S
+642,1,1,"Sagesser, Mlle. Emma",female,24,0,0,PC 17477,69.3,B35,C
+643,0,3,"Skoog, Miss. Margit Elizabeth",female,2,3,2,347088,27.9,,S
+644,1,3,"Foo, Mr. Choong",male,,0,0,1601,56.4958,,S
+645,1,3,"Baclini, Miss. Eugenie",female,0.75,2,1,2666,19.2583,,C
+646,1,1,"Harper, Mr. Henry Sleeper",male,48,1,0,PC 17572,76.7292,D33,C
+647,0,3,"Cor, Mr. Liudevit",male,19,0,0,349231,7.8958,,S
+648,1,1,"Simonius-Blumer, Col. Oberst Alfons",male,56,0,0,13213,35.5,A26,C
+649,0,3,"Willey, Mr. Edward",male,,0,0,S.O./P.P. 751,7.55,,S
+650,1,3,"Stanley, Miss. Amy Zillah Elsie",female,23,0,0,CA. 2314,7.55,,S
+651,0,3,"Mitkoff, Mr. Mito",male,,0,0,349221,7.8958,,S
+652,1,2,"Doling, Miss. Elsie",female,18,0,1,231919,23,,S
+653,0,3,"Kalvik, Mr. Johannes Halvorsen",male,21,0,0,8475,8.4333,,S
+654,1,3,"O'Leary, Miss. Hanora ""Norah""",female,,0,0,330919,7.8292,,Q
+655,0,3,"Hegarty, Miss. Hanora ""Nora""",female,18,0,0,365226,6.75,,Q
+656,0,2,"Hickman, Mr. Leonard Mark",male,24,2,0,S.O.C. 14879,73.5,,S
+657,0,3,"Radeff, Mr. Alexander",male,,0,0,349223,7.8958,,S
+658,0,3,"Bourke, Mrs. John (Catherine)",female,32,1,1,364849,15.5,,Q
+659,0,2,"Eitemiller, Mr. George Floyd",male,23,0,0,29751,13,,S
+660,0,1,"Newell, Mr. Arthur Webster",male,58,0,2,35273,113.275,D48,C
+661,1,1,"Frauenthal, Dr. Henry William",male,50,2,0,PC 17611,133.65,,S
+662,0,3,"Badt, Mr. Mohamed",male,40,0,0,2623,7.225,,C
+663,0,1,"Colley, Mr. Edward Pomeroy",male,47,0,0,5727,25.5875,E58,S
+664,0,3,"Coleff, Mr. Peju",male,36,0,0,349210,7.4958,,S
+665,1,3,"Lindqvist, Mr. Eino William",male,20,1,0,STON/O 2. 3101285,7.925,,S
+666,0,2,"Hickman, Mr. Lewis",male,32,2,0,S.O.C. 14879,73.5,,S
+667,0,2,"Butler, Mr. Reginald Fenton",male,25,0,0,234686,13,,S
+668,0,3,"Rommetvedt, Mr. Knud Paust",male,,0,0,312993,7.775,,S
+669,0,3,"Cook, Mr. Jacob",male,43,0,0,A/5 3536,8.05,,S
+670,1,1,"Taylor, Mrs. Elmer Zebley (Juliet Cummins Wright)",female,,1,0,19996,52,C126,S
+671,1,2,"Brown, Mrs. Thomas William Solomon (Elizabeth Catherine Ford)",female,40,1,1,29750,39,,S
+672,0,1,"Davidson, Mr. Thornton",male,31,1,0,F.C. 12750,52,B71,S
+673,0,2,"Mitchell, Mr. Henry Michael",male,70,0,0,C.A. 24580,10.5,,S
+674,1,2,"Wilhelms, Mr. Charles",male,31,0,0,244270,13,,S
+675,0,2,"Watson, Mr. Ennis Hastings",male,,0,0,239856,0,,S
+676,0,3,"Edvardsson, Mr. Gustaf Hjalmar",male,18,0,0,349912,7.775,,S
+677,0,3,"Sawyer, Mr. Frederick Charles",male,24.5,0,0,342826,8.05,,S
+678,1,3,"Turja, Miss. Anna Sofia",female,18,0,0,4138,9.8417,,S
+679,0,3,"Goodwin, Mrs. Frederick (Augusta Tyler)",female,43,1,6,CA 2144,46.9,,S
+680,1,1,"Cardeza, Mr. Thomas Drake Martinez",male,36,0,1,PC 17755,512.3292,B51 B53 B55,C
+681,0,3,"Peters, Miss. Katie",female,,0,0,330935,8.1375,,Q
+682,1,1,"Hassab, Mr. Hammad",male,27,0,0,PC 17572,76.7292,D49,C
+683,0,3,"Olsvigen, Mr. Thor Anderson",male,20,0,0,6563,9.225,,S
+684,0,3,"Goodwin, Mr. Charles Edward",male,14,5,2,CA 2144,46.9,,S
+685,0,2,"Brown, Mr. Thomas William Solomon",male,60,1,1,29750,39,,S
+686,0,2,"Laroche, Mr. Joseph Philippe Lemercier",male,25,1,2,SC/Paris 2123,41.5792,,C
+687,0,3,"Panula, Mr. Jaako Arnold",male,14,4,1,3101295,39.6875,,S
+688,0,3,"Dakic, Mr. Branko",male,19,0,0,349228,10.1708,,S
+689,0,3,"Fischer, Mr. Eberhard Thelander",male,18,0,0,350036,7.7958,,S
+690,1,1,"Madill, Miss. Georgette Alexandra",female,15,0,1,24160,211.3375,B5,S
+691,1,1,"Dick, Mr. Albert Adrian",male,31,1,0,17474,57,B20,S
+692,1,3,"Karun, Miss. Manca",female,4,0,1,349256,13.4167,,C
+693,1,3,"Lam, Mr. Ali",male,,0,0,1601,56.4958,,S
+694,0,3,"Saad, Mr. Khalil",male,25,0,0,2672,7.225,,C
+695,0,1,"Weir, Col. John",male,60,0,0,113800,26.55,,S
+696,0,2,"Chapman, Mr. Charles Henry",male,52,0,0,248731,13.5,,S
+697,0,3,"Kelly, Mr. James",male,44,0,0,363592,8.05,,S
+698,1,3,"Mullens, Miss. Katherine ""Katie""",female,,0,0,35852,7.7333,,Q
+699,0,1,"Thayer, Mr. John Borland",male,49,1,1,17421,110.8833,C68,C
+700,0,3,"Humblen, Mr. Adolf Mathias Nicolai Olsen",male,42,0,0,348121,7.65,F G63,S
+701,1,1,"Astor, Mrs. John Jacob (Madeleine Talmadge Force)",female,18,1,0,PC 17757,227.525,C62 C64,C
+702,1,1,"Silverthorne, Mr. Spencer Victor",male,35,0,0,PC 17475,26.2875,E24,S
+703,0,3,"Barbara, Miss. Saiide",female,18,0,1,2691,14.4542,,C
+704,0,3,"Gallagher, Mr. Martin",male,25,0,0,36864,7.7417,,Q
+705,0,3,"Hansen, Mr. Henrik Juul",male,26,1,0,350025,7.8542,,S
+706,0,2,"Morley, Mr. Henry Samuel (""Mr Henry Marshall"")",male,39,0,0,250655,26,,S
+707,1,2,"Kelly, Mrs. Florence ""Fannie""",female,45,0,0,223596,13.5,,S
+708,1,1,"Calderhead, Mr. Edward Pennington",male,42,0,0,PC 17476,26.2875,E24,S
+709,1,1,"Cleaver, Miss. Alice",female,22,0,0,113781,151.55,,S
+710,1,3,"Moubarek, Master. Halim Gonios (""William George"")",male,,1,1,2661,15.2458,,C
+711,1,1,"Mayne, Mlle. Berthe Antonine (""Mrs de Villiers"")",female,24,0,0,PC 17482,49.5042,C90,C
+712,0,1,"Klaber, Mr. Herman",male,,0,0,113028,26.55,C124,S
+713,1,1,"Taylor, Mr. Elmer Zebley",male,48,1,0,19996,52,C126,S
+714,0,3,"Larsson, Mr. August Viktor",male,29,0,0,7545,9.4833,,S
+715,0,2,"Greenberg, Mr. Samuel",male,52,0,0,250647,13,,S
+716,0,3,"Soholt, Mr. Peter Andreas Lauritz Andersen",male,19,0,0,348124,7.65,F G73,S
+717,1,1,"Endres, Miss. Caroline Louise",female,38,0,0,PC 17757,227.525,C45,C
+718,1,2,"Troutt, Miss. Edwina Celia ""Winnie""",female,27,0,0,34218,10.5,E101,S
+719,0,3,"McEvoy, Mr. Michael",male,,0,0,36568,15.5,,Q
+720,0,3,"Johnson, Mr. Malkolm Joackim",male,33,0,0,347062,7.775,,S
+721,1,2,"Harper, Miss. Annie Jessie ""Nina""",female,6,0,1,248727,33,,S
+722,0,3,"Jensen, Mr. Svend Lauritz",male,17,1,0,350048,7.0542,,S
+723,0,2,"Gillespie, Mr. William Henry",male,34,0,0,12233,13,,S
+724,0,2,"Hodges, Mr. Henry Price",male,50,0,0,250643,13,,S
+725,1,1,"Chambers, Mr. Norman Campbell",male,27,1,0,113806,53.1,E8,S
+726,0,3,"Oreskovic, Mr. Luka",male,20,0,0,315094,8.6625,,S
+727,1,2,"Renouf, Mrs. Peter Henry (Lillian Jefferys)",female,30,3,0,31027,21,,S
+728,1,3,"Mannion, Miss. Margareth",female,,0,0,36866,7.7375,,Q
+729,0,2,"Bryhl, Mr. Kurt Arnold Gottfrid",male,25,1,0,236853,26,,S
+730,0,3,"Ilmakangas, Miss. Pieta Sofia",female,25,1,0,STON/O2. 3101271,7.925,,S
+731,1,1,"Allen, Miss. Elisabeth Walton",female,29,0,0,24160,211.3375,B5,S
+732,0,3,"Hassan, Mr. Houssein G N",male,11,0,0,2699,18.7875,,C
+733,0,2,"Knight, Mr. Robert J",male,,0,0,239855,0,,S
+734,0,2,"Berriman, Mr. William John",male,23,0,0,28425,13,,S
+735,0,2,"Troupiansky, Mr. Moses Aaron",male,23,0,0,233639,13,,S
+736,0,3,"Williams, Mr. Leslie",male,28.5,0,0,54636,16.1,,S
+737,0,3,"Ford, Mrs. Edward (Margaret Ann Watson)",female,48,1,3,W./C. 6608,34.375,,S
+738,1,1,"Lesurer, Mr. Gustave J",male,35,0,0,PC 17755,512.3292,B101,C
+739,0,3,"Ivanoff, Mr. Kanio",male,,0,0,349201,7.8958,,S
+740,0,3,"Nankoff, Mr. Minko",male,,0,0,349218,7.8958,,S
+741,1,1,"Hawksford, Mr. Walter James",male,,0,0,16988,30,D45,S
+742,0,1,"Cavendish, Mr. Tyrell William",male,36,1,0,19877,78.85,C46,S
+743,1,1,"Ryerson, Miss. Susan Parker ""Suzette""",female,21,2,2,PC 17608,262.375,B57 B59 B63 B66,C
+744,0,3,"McNamee, Mr. Neal",male,24,1,0,376566,16.1,,S
+745,1,3,"Stranden, Mr. Juho",male,31,0,0,STON/O 2. 3101288,7.925,,S
+746,0,1,"Crosby, Capt. Edward Gifford",male,70,1,1,WE/P 5735,71,B22,S
+747,0,3,"Abbott, Mr. Rossmore Edward",male,16,1,1,C.A. 2673,20.25,,S
+748,1,2,"Sinkkonen, Miss. Anna",female,30,0,0,250648,13,,S
+749,0,1,"Marvin, Mr. Daniel Warner",male,19,1,0,113773,53.1,D30,S
+750,0,3,"Connaghton, Mr. Michael",male,31,0,0,335097,7.75,,Q
+751,1,2,"Wells, Miss. Joan",female,4,1,1,29103,23,,S
+752,1,3,"Moor, Master. Meier",male,6,0,1,392096,12.475,E121,S
+753,0,3,"Vande Velde, Mr. Johannes Joseph",male,33,0,0,345780,9.5,,S
+754,0,3,"Jonkoff, Mr. Lalio",male,23,0,0,349204,7.8958,,S
+755,1,2,"Herman, Mrs. Samuel (Jane Laver)",female,48,1,2,220845,65,,S
+756,1,2,"Hamalainen, Master. Viljo",male,0.67,1,1,250649,14.5,,S
+757,0,3,"Carlsson, Mr. August Sigfrid",male,28,0,0,350042,7.7958,,S
+758,0,2,"Bailey, Mr. Percy Andrew",male,18,0,0,29108,11.5,,S
+759,0,3,"Theobald, Mr. Thomas Leonard",male,34,0,0,363294,8.05,,S
+760,1,1,"Rothes, the Countess. of (Lucy Noel Martha Dyer-Edwards)",female,33,0,0,110152,86.5,B77,S
+761,0,3,"Garfirth, Mr. John",male,,0,0,358585,14.5,,S
+762,0,3,"Nirva, Mr. Iisakki Antino Aijo",male,41,0,0,SOTON/O2 3101272,7.125,,S
+763,1,3,"Barah, Mr. Hanna Assi",male,20,0,0,2663,7.2292,,C
+764,1,1,"Carter, Mrs. William Ernest (Lucile Polk)",female,36,1,2,113760,120,B96 B98,S
+765,0,3,"Eklund, Mr. Hans Linus",male,16,0,0,347074,7.775,,S
+766,1,1,"Hogeboom, Mrs. John C (Anna Andrews)",female,51,1,0,13502,77.9583,D11,S
+767,0,1,"Brewe, Dr. Arthur Jackson",male,,0,0,112379,39.6,,C
+768,0,3,"Mangan, Miss. Mary",female,30.5,0,0,364850,7.75,,Q
+769,0,3,"Moran, Mr. Daniel J",male,,1,0,371110,24.15,,Q
+770,0,3,"Gronnestad, Mr. Daniel Danielsen",male,32,0,0,8471,8.3625,,S
+771,0,3,"Lievens, Mr. Rene Aime",male,24,0,0,345781,9.5,,S
+772,0,3,"Jensen, Mr. Niels Peder",male,48,0,0,350047,7.8542,,S
+773,0,2,"Mack, Mrs. (Mary)",female,57,0,0,S.O./P.P. 3,10.5,E77,S
+774,0,3,"Elias, Mr. Dibo",male,,0,0,2674,7.225,,C
+775,1,2,"Hocking, Mrs. Elizabeth (Eliza Needs)",female,54,1,3,29105,23,,S
+776,0,3,"Myhrman, Mr. Pehr Fabian Oliver Malkolm",male,18,0,0,347078,7.75,,S
+777,0,3,"Tobin, Mr. Roger",male,,0,0,383121,7.75,F38,Q
+778,1,3,"Emanuel, Miss. Virginia Ethel",female,5,0,0,364516,12.475,,S
+779,0,3,"Kilgannon, Mr. Thomas J",male,,0,0,36865,7.7375,,Q
+780,1,1,"Robert, Mrs. Edward Scott (Elisabeth Walton McMillan)",female,43,0,1,24160,211.3375,B3,S
+781,1,3,"Ayoub, Miss. Banoura",female,13,0,0,2687,7.2292,,C
+782,1,1,"Dick, Mrs. Albert Adrian (Vera Gillespie)",female,17,1,0,17474,57,B20,S
+783,0,1,"Long, Mr. Milton Clyde",male,29,0,0,113501,30,D6,S
+784,0,3,"Johnston, Mr. Andrew G",male,,1,2,W./C. 6607,23.45,,S
+785,0,3,"Ali, Mr. William",male,25,0,0,SOTON/O.Q. 3101312,7.05,,S
+786,0,3,"Harmer, Mr. Abraham (David Lishin)",male,25,0,0,374887,7.25,,S
+787,1,3,"Sjoblom, Miss. Anna Sofia",female,18,0,0,3101265,7.4958,,S
+788,0,3,"Rice, Master. George Hugh",male,8,4,1,382652,29.125,,Q
+789,1,3,"Dean, Master. Bertram Vere",male,1,1,2,C.A. 2315,20.575,,S
+790,0,1,"Guggenheim, Mr. Benjamin",male,46,0,0,PC 17593,79.2,B82 B84,C
+791,0,3,"Keane, Mr. Andrew ""Andy""",male,,0,0,12460,7.75,,Q
+792,0,2,"Gaskell, Mr. Alfred",male,16,0,0,239865,26,,S
+793,0,3,"Sage, Miss. Stella Anna",female,,8,2,CA. 2343,69.55,,S
+794,0,1,"Hoyt, Mr. William Fisher",male,,0,0,PC 17600,30.6958,,C
+795,0,3,"Dantcheff, Mr. Ristiu",male,25,0,0,349203,7.8958,,S
+796,0,2,"Otter, Mr. Richard",male,39,0,0,28213,13,,S
+797,1,1,"Leader, Dr. Alice (Farnham)",female,49,0,0,17465,25.9292,D17,S
+798,1,3,"Osman, Mrs. Mara",female,31,0,0,349244,8.6833,,S
+799,0,3,"Ibrahim Shawah, Mr. Yousseff",male,30,0,0,2685,7.2292,,C
+800,0,3,"Van Impe, Mrs. Jean Baptiste (Rosalie Paula Govaert)",female,30,1,1,345773,24.15,,S
+801,0,2,"Ponesell, Mr. Martin",male,34,0,0,250647,13,,S
+802,1,2,"Collyer, Mrs. Harvey (Charlotte Annie Tate)",female,31,1,1,C.A. 31921,26.25,,S
+803,1,1,"Carter, Master. William Thornton II",male,11,1,2,113760,120,B96 B98,S
+804,1,3,"Thomas, Master. Assad Alexander",male,0.42,0,1,2625,8.5167,,C
+805,1,3,"Hedman, Mr. Oskar Arvid",male,27,0,0,347089,6.975,,S
+806,0,3,"Johansson, Mr. Karl Johan",male,31,0,0,347063,7.775,,S
+807,0,1,"Andrews, Mr. Thomas Jr",male,39,0,0,112050,0,A36,S
+808,0,3,"Pettersson, Miss. Ellen Natalia",female,18,0,0,347087,7.775,,S
+809,0,2,"Meyer, Mr. August",male,39,0,0,248723,13,,S
+810,1,1,"Chambers, Mrs. Norman Campbell (Bertha Griggs)",female,33,1,0,113806,53.1,E8,S
+811,0,3,"Alexander, Mr. William",male,26,0,0,3474,7.8875,,S
+812,0,3,"Lester, Mr. James",male,39,0,0,A/4 48871,24.15,,S
+813,0,2,"Slemen, Mr. Richard James",male,35,0,0,28206,10.5,,S
+814,0,3,"Andersson, Miss. Ebba Iris Alfrida",female,6,4,2,347082,31.275,,S
+815,0,3,"Tomlin, Mr. Ernest Portage",male,30.5,0,0,364499,8.05,,S
+816,0,1,"Fry, Mr. Richard",male,,0,0,112058,0,B102,S
+817,0,3,"Heininen, Miss. Wendla Maria",female,23,0,0,STON/O2. 3101290,7.925,,S
+818,0,2,"Mallet, Mr. Albert",male,31,1,1,S.C./PARIS 2079,37.0042,,C
+819,0,3,"Holm, Mr. John Fredrik Alexander",male,43,0,0,C 7075,6.45,,S
+820,0,3,"Skoog, Master. Karl Thorsten",male,10,3,2,347088,27.9,,S
+821,1,1,"Hays, Mrs. Charles Melville (Clara Jennings Gregg)",female,52,1,1,12749,93.5,B69,S
+822,1,3,"Lulic, Mr. Nikola",male,27,0,0,315098,8.6625,,S
+823,0,1,"Reuchlin, Jonkheer. John George",male,38,0,0,19972,0,,S
+824,1,3,"Moor, Mrs. (Beila)",female,27,0,1,392096,12.475,E121,S
+825,0,3,"Panula, Master. Urho Abraham",male,2,4,1,3101295,39.6875,,S
+826,0,3,"Flynn, Mr. John",male,,0,0,368323,6.95,,Q
+827,0,3,"Lam, Mr. Len",male,,0,0,1601,56.4958,,S
+828,1,2,"Mallet, Master. Andre",male,1,0,2,S.C./PARIS 2079,37.0042,,C
+829,1,3,"McCormack, Mr. Thomas Joseph",male,,0,0,367228,7.75,,Q
+830,1,1,"Stone, Mrs. George Nelson (Martha Evelyn)",female,62,0,0,113572,80,B28,
+831,1,3,"Yasbeck, Mrs. Antoni (Selini Alexander)",female,15,1,0,2659,14.4542,,C
+832,1,2,"Richards, Master. George Sibley",male,0.83,1,1,29106,18.75,,S
+833,0,3,"Saad, Mr. Amin",male,,0,0,2671,7.2292,,C
+834,0,3,"Augustsson, Mr. Albert",male,23,0,0,347468,7.8542,,S
+835,0,3,"Allum, Mr. Owen George",male,18,0,0,2223,8.3,,S
+836,1,1,"Compton, Miss. Sara Rebecca",female,39,1,1,PC 17756,83.1583,E49,C
+837,0,3,"Pasic, Mr. Jakob",male,21,0,0,315097,8.6625,,S
+838,0,3,"Sirota, Mr. Maurice",male,,0,0,392092,8.05,,S
+839,1,3,"Chip, Mr. Chang",male,32,0,0,1601,56.4958,,S
+840,1,1,"Marechal, Mr. Pierre",male,,0,0,11774,29.7,C47,C
+841,0,3,"Alhomaki, Mr. Ilmari Rudolf",male,20,0,0,SOTON/O2 3101287,7.925,,S
+842,0,2,"Mudd, Mr. Thomas Charles",male,16,0,0,S.O./P.P. 3,10.5,,S
+843,1,1,"Serepeca, Miss. Augusta",female,30,0,0,113798,31,,C
+844,0,3,"Lemberopolous, Mr. Peter L",male,34.5,0,0,2683,6.4375,,C
+845,0,3,"Culumovic, Mr. Jeso",male,17,0,0,315090,8.6625,,S
+846,0,3,"Abbing, Mr. Anthony",male,42,0,0,C.A. 5547,7.55,,S
+847,0,3,"Sage, Mr. Douglas Bullen",male,,8,2,CA. 2343,69.55,,S
+848,0,3,"Markoff, Mr. Marin",male,35,0,0,349213,7.8958,,C
+849,0,2,"Harper, Rev. John",male,28,0,1,248727,33,,S
+850,1,1,"Goldenberg, Mrs. Samuel L (Edwiga Grabowska)",female,,1,0,17453,89.1042,C92,C
+851,0,3,"Andersson, Master. Sigvard Harald Elias",male,4,4,2,347082,31.275,,S
+852,0,3,"Svensson, Mr. Johan",male,74,0,0,347060,7.775,,S
+853,0,3,"Boulos, Miss. Nourelain",female,9,1,1,2678,15.2458,,C
+854,1,1,"Lines, Miss. Mary Conover",female,16,0,1,PC 17592,39.4,D28,S
+855,0,2,"Carter, Mrs. Ernest Courtenay (Lilian Hughes)",female,44,1,0,244252,26,,S
+856,1,3,"Aks, Mrs. Sam (Leah Rosen)",female,18,0,1,392091,9.35,,S
+857,1,1,"Wick, Mrs. George Dennick (Mary Hitchcock)",female,45,1,1,36928,164.8667,,S
+858,1,1,"Daly, Mr. Peter Denis ",male,51,0,0,113055,26.55,E17,S
+859,1,3,"Baclini, Mrs. Solomon (Latifa Qurban)",female,24,0,3,2666,19.2583,,C
+860,0,3,"Razi, Mr. Raihed",male,,0,0,2629,7.2292,,C
+861,0,3,"Hansen, Mr. Claus Peter",male,41,2,0,350026,14.1083,,S
+862,0,2,"Giles, Mr. Frederick Edward",male,21,1,0,28134,11.5,,S
+863,1,1,"Swift, Mrs. Frederick Joel (Margaret Welles Barron)",female,48,0,0,17466,25.9292,D17,S
+864,0,3,"Sage, Miss. Dorothy Edith ""Dolly""",female,,8,2,CA. 2343,69.55,,S
+865,0,2,"Gill, Mr. John William",male,24,0,0,233866,13,,S
+866,1,2,"Bystrom, Mrs. (Karolina)",female,42,0,0,236852,13,,S
+867,1,2,"Duran y More, Miss. Asuncion",female,27,1,0,SC/PARIS 2149,13.8583,,C
+868,0,1,"Roebling, Mr. Washington Augustus II",male,31,0,0,PC 17590,50.4958,A24,S
+869,0,3,"van Melkebeke, Mr. Philemon",male,,0,0,345777,9.5,,S
+870,1,3,"Johnson, Master. Harold Theodor",male,4,1,1,347742,11.1333,,S
+871,0,3,"Balkic, Mr. Cerin",male,26,0,0,349248,7.8958,,S
+872,1,1,"Beckwith, Mrs. Richard Leonard (Sallie Monypeny)",female,47,1,1,11751,52.5542,D35,S
+873,0,1,"Carlsson, Mr. Frans Olof",male,33,0,0,695,5,B51 B53 B55,S
+874,0,3,"Vander Cruyssen, Mr. Victor",male,47,0,0,345765,9,,S
+875,1,2,"Abelson, Mrs. Samuel (Hannah Wizosky)",female,28,1,0,P/PP 3381,24,,C
+876,1,3,"Najib, Miss. Adele Kiamie ""Jane""",female,15,0,0,2667,7.225,,C
+877,0,3,"Gustafsson, Mr. Alfred Ossian",male,20,0,0,7534,9.8458,,S
+878,0,3,"Petroff, Mr. Nedelio",male,19,0,0,349212,7.8958,,S
+879,0,3,"Laleff, Mr. Kristo",male,,0,0,349217,7.8958,,S
+880,1,1,"Potter, Mrs. Thomas Jr (Lily Alexenia Wilson)",female,56,0,1,11767,83.1583,C50,C
+881,1,2,"Shelley, Mrs. William (Imanita Parrish Hall)",female,25,0,1,230433,26,,S
+882,0,3,"Markun, Mr. Johann",male,33,0,0,349257,7.8958,,S
+883,0,3,"Dahlberg, Miss. Gerda Ulrika",female,22,0,0,7552,10.5167,,S
+884,0,2,"Banfield, Mr. Frederick James",male,28,0,0,C.A./SOTON 34068,10.5,,S
+885,0,3,"Sutehall, Mr. Henry Jr",male,25,0,0,SOTON/OQ 392076,7.05,,S
+886,0,3,"Rice, Mrs. William (Margaret Norton)",female,39,0,5,382652,29.125,,Q
+887,0,2,"Montvila, Rev. Juozas",male,27,0,0,211536,13,,S
+888,1,1,"Graham, Miss. Margaret Edith",female,19,0,0,112053,30,B42,S
+889,0,3,"Johnston, Miss. Catherine Helen ""Carrie""",female,,1,2,W./C. 6607,23.45,,S
+890,1,1,"Behr, Mr. Karl Howell",male,26,0,0,111369,30,C148,C
+891,0,3,"Dooley, Mr. Patrick",male,32,0,0,370376,7.75,,Q

how-to-use-azureml/azure-synapse/shakespeare.txt

@@ -0,0 +1,270 @@
+This is the 100th Etext file presented by Project Gutenberg, and
+is presented in cooperation with World Library, Inc., from their
+Library of the Future and Shakespeare CDROMS.  Project Gutenberg
+often releases Etexts that are NOT placed in the Public Domain!!
+
+Shakespeare
+
+*This Etext has certain copyright implications you should read!*
+
+<<THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM
+SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS
+PROVIDED BY PROJECT GUTENBERG ETEXT OF ILLINOIS BENEDICTINE COLLEGE
+WITH PERMISSION.  ELECTRONIC AND MACHINE READABLE COPIES MAY BE
+DISTRIBUTED SO LONG AS SUCH COPIES (1) ARE FOR YOUR OR OTHERS
+PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED
+COMMERCIALLY.  PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY
+SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>>
+
+*Project Gutenberg is proud to cooperate with The World Library*
+in the presentation of The Complete Works of William Shakespeare
+for your reading for education and entertainment.  HOWEVER, THIS
+IS NEITHER SHAREWARE NOR PUBLIC DOMAIN. . .AND UNDER THE LIBRARY
+OF THE FUTURE CONDITIONS OF THIS PRESENTATION. . .NO CHARGES MAY
+BE MADE FOR *ANY* ACCESS TO THIS MATERIAL.  YOU ARE ENCOURAGED!!
+TO GIVE IT AWAY TO ANYONE YOU LIKE, BUT NO CHARGES ARE ALLOWED!!
+
+
+**Welcome To The World of Free Plain Vanilla Electronic Texts**
+
+**Etexts Readable By Both Humans and By Computers, Since 1971**
+
+*These Etexts Prepared By Hundreds of Volunteers and Donations*
+
+Information on contacting Project Gutenberg to get Etexts, and
+further information is included below.  We need your donations.
+
+
+The Complete Works of William Shakespeare 
+
+January, 1994  [Etext #100]
+
+
+The Library of the Future Complete Works of William Shakespeare 
+Library of the Future is a TradeMark (TM) of World Library Inc.
+******This file should be named shaks12.txt or shaks12.zip*****
+
+Corrected EDITIONS of our etexts get a new NUMBER, shaks13.txt
+VERSIONS based on separate sources get new LETTER, shaks10a.txt
+
+If you would like further information about World Library, Inc.
+Please call them at 1-800-443-0238 or email julianc@netcom.com
+Please give them our thanks for their Shakespeare cooperation!
+
+
+The official release date of all Project Gutenberg Etexts is at
+Midnight, Central Time, of the last day of the stated month.  A
+preliminary version may often be posted for suggestion, comment
+and editing by those who wish to do so.  To be sure you have an
+up to date first edition [xxxxx10x.xxx] please check file sizes
+in the first week of the next month.  Since our ftp program has
+a bug in it that scrambles the date [tried to fix and failed] a
+look at the file size will have to do, but we will try to see a
+new copy has at least one byte more or less.
+
+
+Information about Project Gutenberg (one page)
+
+We produce about two million dollars for each hour we work.  The
+fifty hours is one conservative estimate for how long it we take
+to get any etext selected, entered, proofread, edited, copyright
+searched and analyzed, the copyright letters written, etc.  This
+projected audience is one hundred million readers.  If our value
+per text is nominally estimated at one dollar, then we produce 2
+million dollars per hour this year we, will have to do four text
+files per month:  thus upping our productivity from one million.
+The Goal of Project Gutenberg is to Give Away One Trillion Etext
+Files by the December 31, 2001.  [10,000 x 100,000,000=Trillion]
+This is ten thousand titles each to one hundred million readers,
+which is 10% of the expected number of computer users by the end
+of the year 2001.
+
+We need your donations more than ever!
+
+All donations should be made to "Project Gutenberg/IBC", and are
+tax deductible to the extent allowable by law ("IBC" is Illinois
+Benedictine College).  (Subscriptions to our paper newsletter go
+to IBC, too)
+
+For these and other matters, please mail to:
+
+Project Gutenberg
+P. O. Box  2782
+Champaign, IL 61825
+
+When all other email fails try our Michael S. Hart, Executive Director:
+hart@vmd.cso.uiuc.edu (internet)   hart@uiucvmd   (bitnet)
+
+We would prefer to send you this information by email
+(Internet, Bitnet, Compuserve, ATTMAIL or MCImail).
+
+******
+If you have an FTP program (or emulator), please
+FTP directly to the Project Gutenberg archives:
+[Mac users, do NOT point and click. . .type]
+
+ftp mrcnext.cso.uiuc.edu
+login:  anonymous
+password:  your@login
+cd etext/etext91
+or cd etext92
+or cd etext93 [for new books]  [now also in cd etext/etext93]
+or cd etext/articles [get suggest gut for more information]
+dir [to see files]
+get or mget [to get files. . .set bin for zip files]
+GET 0INDEX.GUT
+for a list of books
+and
+GET NEW GUT for general information
+and
+MGET GUT* for newsletters.
+
+**Information prepared by the Project Gutenberg legal advisor**
+
+
+***** SMALL PRINT! for COMPLETE SHAKESPEARE *****
+
+THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM
+SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC.,
+AND IS PROVIDED BY PROJECT GUTENBERG ETEXT OF
+ILLINOIS BENEDICTINE COLLEGE WITH PERMISSION.
+
+Since unlike many other Project Gutenberg-tm etexts, this etext
+is copyright protected, and since the materials and methods you
+use will effect the Project's reputation, your right to copy and
+distribute it is limited by the copyright and other laws, and by
+the conditions of this "Small Print!" statement.
+
+1.  LICENSE
+
+  A) YOU MAY (AND ARE ENCOURAGED) TO DISTRIBUTE ELECTRONIC AND
+MACHINE READABLE COPIES OF THIS ETEXT, SO LONG AS SUCH COPIES
+(1) ARE FOR YOUR OR OTHERS PERSONAL USE ONLY, AND (2) ARE NOT
+DISTRIBUTED OR USED COMMERCIALLY.  PROHIBITED COMMERCIAL
+DISTRIBUTION INCLUDES BY ANY SERVICE THAT CHARGES FOR DOWNLOAD
+TIME OR FOR MEMBERSHIP.
+
+  B) This license is subject to the conditions that you honor
+the refund and replacement provisions of this "small print!"
+statement; and that you distribute exact copies of this etext,
+including this Small Print statement.  Such copies can be
+compressed or any proprietary form (including any form resulting
+from word processing or hypertext software), so long as
+*EITHER*:
+
+    (1) The etext, when displayed, is clearly readable, and does
+  *not* contain characters other than those intended by the
+  author of the work, although tilde (~), asterisk (*) and
+  underline (_) characters may be used to convey punctuation
+  intended by the author, and additional characters may be used
+  to indicate hypertext links; OR
+
+    (2) The etext is readily convertible by the reader at no
+  expense into plain ASCII, EBCDIC or equivalent form by the
+  program that displays the etext (as is the case, for instance,
+  with most word processors); OR
+
+    (3) You provide or agree to provide on request at no
+  additional cost, fee or expense, a copy of the etext in plain
+  ASCII.
+
+2.  LIMITED WARRANTY; DISCLAIMER OF DAMAGES
+
+This etext may contain a "Defect" in the form of incomplete,
+inaccurate or corrupt data, transcription errors, a copyright or
+other infringement, a defective or damaged disk, computer virus,
+or codes that damage or cannot be read by your equipment.  But
+for the "Right of Replacement or Refund" described below, the
+Project (and any other party you may receive this etext from as
+a PROJECT GUTENBERG-tm etext) disclaims all liability to you for
+damages, costs and expenses, including legal fees, and YOU HAVE
+NO REMEDIES FOR NEGLIGENCE OR UNDER STRICT LIABILITY, OR FOR
+BREACH OF WARRANTY OR CONTRACT, INCLUDING BUT NOT LIMITED TO
+INDIRECT, CONSEQUENTIAL, PUNITIVE OR INCIDENTAL DAMAGES, EVEN IF
+YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH DAMAGES.
+
+If you discover a Defect in this etext within 90 days of receiv-
+ing it, you can receive a refund of the money (if any) you paid
+for it by sending an explanatory note within that time to the
+person you received it from.  If you received it on a physical
+medium, you must return it with your note, and such person may
+choose to alternatively give you a replacement copy.  If you
+received it electronically, such person may choose to
+alternatively give you a second opportunity to receive it
+electronically.
+
+THIS ETEXT IS OTHERWISE PROVIDED TO YOU "AS-IS".  NO OTHER
+WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, ARE MADE TO YOU AS
+TO THE ETEXT OR ANY MEDIUM IT MAY BE ON, INCLUDING BUT NOT
+LIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
+PARTICULAR PURPOSE.  Some states do not allow disclaimers of
+implied warranties or the exclusion or limitation of consequen-
+tial damages, so the above disclaimers and exclusions may not
+apply to you, and you may have other legal rights.
+
+3.  INDEMNITY: You will indemnify and hold the Project, its
+directors, officers, members and agents harmless from all lia-
+bility, cost and expense, including legal fees, that arise
+directly or indirectly from any of the following that you do or
+cause: [A] distribution of this etext, [B] alteration,
+modification, or addition to the etext, or [C] any Defect.
+
+4.  WHAT IF YOU *WANT* TO SEND MONEY EVEN IF YOU DON'T HAVE TO?
+Project Gutenberg is dedicated to increasing the number of
+public domain and licensed works that can be freely distributed
+in machine readable form.  The Project gratefully accepts
+contributions in money, time, scanning machines, OCR software,
+public domain etexts, royalty free copyright licenses, and
+whatever else you can think of.  Money should be paid to "Pro-
+ject Gutenberg Association / Illinois Benedictine College".
+
+WRITE TO US! We can be reached at:
+     Internet: hart@vmd.cso.uiuc.edu
+       Bitnet: hart@uiucvmd
+   CompuServe: >internet:hart@.vmd.cso.uiuc.edu
+      Attmail: internet!vmd.cso.uiuc.edu!Hart
+        Mail:  Prof. Michael Hart
+               P.O. Box 2782
+               Champaign, IL 61825
+
+This "Small Print!" by Charles B. Kramer, Attorney
+Internet (72600.2026@compuserve.com); TEL: (212-254-5093)
+****   SMALL PRINT! FOR __ COMPLETE SHAKESPEARE ****
+["Small Print" V.12.08.93]
+
+<<THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM         
+SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS  
+PROVIDED BY PROJECT GUTENBERG ETEXT OF ILLINOIS BENEDICTINE COLLEGE
+WITH PERMISSION.  ELECTRONIC AND MACHINE READABLE COPIES MAY BE    
+DISTRIBUTED SO LONG AS SUCH COPIES (1) ARE FOR YOUR OR OTHERS      
+PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED             
+COMMERCIALLY.  PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY  
+SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>> 
+
+
+1609
+
+THE SONNETS
+
+by William Shakespeare
+
+
+THE END
+
+
+
+<<THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM
+SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS
+PROVIDED BY PROJECT GUTENBERG ETEXT OF ILLINOIS BENEDICTINE COLLEGE
+WITH PERMISSION.  ELECTRONIC AND MACHINE READABLE COPIES MAY BE
+DISTRIBUTED SO LONG AS SUCH COPIES (1) ARE FOR YOUR OR OTHERS
+PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED
+COMMERCIALLY.  PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY
+SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>>
+
+
+
+End of this Etext of The Complete Works of William Shakespeare
+
+
+

how-to-use-azureml/azure-synapse/spark_job_on_synapse_spark_pool.ipynb

@@ -0,0 +1,507 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved. \n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-arcadia/spark_job_on_synapse_spark_pool.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Using Synapse Spark Pool as a Compute Target from Azure Machine Learning Remote Run\n",
+        "1. To use Synapse Spark Pool as a compute target from Experiment Run, [ScriptRunConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.script_run_config.scriptrunconfig?view=azure-ml-py) is used, the same as other Experiment Runs. This notebook demonstrates how to leverage ScriptRunConfig to submit an experiment run to an attached Synapse Spark cluster.\n",
+        "2. To use Synapse Spark Pool as a compute target from [Azure Machine Learning Pipeline](https://aka.ms/pl-concept), a [SynapseSparkStep](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.synapse_spark_step.synapsesparkstep?view=azure-ml-py) is used. This notebook demonstrates how to leverage SynapseSparkStep in Azure Machine Learning Pipeline.\n",
+        "\n",
+        "## Before you begin:\n",
+        "1. **Create an Azure Synapse workspace**, check [this] (https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace) for more information.\n",
+        "2. **Create Spark Pool in Synapse workspace**: check [this] (https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-apache-spark-pool-portal) for more information."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Azure Machine Learning and Pipeline SDK-specific imports"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import azureml.core\n",
+        "from azureml.core import Workspace, Experiment\n",
+        "from azureml.core import LinkedService, SynapseWorkspaceLinkedServiceConfiguration\n",
+        "from azureml.core.compute import ComputeTarget, AmlCompute, SynapseCompute\n",
+        "from azureml.exceptions import ComputeTargetException\n",
+        "from azureml.data import HDFSOutputDatasetConfig\n",
+        "from azureml.core.datastore import Datastore\n",
+        "from azureml.core.runconfig import RunConfiguration\n",
+        "from azureml.core.conda_dependencies import CondaDependencies\n",
+        "from azureml.pipeline.core import Pipeline\n",
+        "from azureml.pipeline.steps import PythonScriptStep, SynapseSparkStep\n",
+        "\n",
+        "# Check core SDK version number\n",
+        "print(\"SDK version:\", azureml.core.VERSION)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Link Synapse workspace to AML \n",
+        "You have to be an \"Owner\" of Synapse workspace resource to perform linking. You can check your role in the Azure resource management portal, if you don't have an \"Owner\" role, you can contact an \"Owner\" to link the workspaces for you."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "\n",
+        "# Replace with your resource info before running.\n",
+        "\n",
+        "synapse_subscription_id=os.getenv(\"SYNAPSE_SUBSCRIPTION_ID\", \"<my-synapse-subscription-id>\")\n",
+        "synapse_resource_group=os.getenv(\"SYNAPSE_RESOURCE_GROUP\", \"<my-synapse-resource-group>\")\n",
+        "synapse_workspace_name=os.getenv(\"SYNAPSE_WORKSPACE_NAME\", \"<my-synapse-workspace-name>\")\n",
+        "synapse_linked_service_name=os.getenv(\"SYNAPSE_LINKED_SERVICE_NAME\", \"<my-synapse-linked-service-name>\")\n",
+        "\n",
+        "synapse_link_config = SynapseWorkspaceLinkedServiceConfiguration(\n",
+        "    subscription_id=synapse_subscription_id,\n",
+        "    resource_group=synapse_resource_group,\n",
+        "    name=synapse_workspace_name\n",
+        ")\n",
+        "\n",
+        "linked_service = LinkedService.register(\n",
+        "    workspace=ws,\n",
+        "    name=synapse_linked_service_name,\n",
+        "    linked_service_config=synapse_link_config)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Linked service property\n",
+        "\n",
+        "A MSI (system_assigned_identity_principal_id) will be generated for each linked service, for example:\n",
+        "\n",
+        "name=synapselink,</p>\n",
+        "type=Synapse, </p>\n",
+        "linked_service_resource_id=/subscriptions/4faaaf21-663f-4391-96fd-47197c630979/resourceGroups/static_resources_synapse_test/providers/Microsoft.Synapse/workspaces/synapsetest2, </p>\n",
+        "system_assigned_identity_principal_id=eb355d52-3806-4c5a-aec9-91447e8cfc2e </p>\n",
+        "\n",
+        "#### Make sure you grant \"Synapse Apache Spark Administrator\" role of the synapse workspace to the generated workspace linking MSI in Synapse studio portal before you submit job."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "linked_service"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "LinkedService.list(ws)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Attach Synapse spark pool as AML compute target"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "synapse_spark_pool_name=os.getenv(\"SYNAPSE_SPARK_POOL_NAME\", \"<my-synapse-spark-pool-name>\")\n",
+        "synapse_compute_name=os.getenv(\"SYNAPSE_COMPUTE_NAME\", \"<my-synapse-compute-name>\")\n",
+        "\n",
+        "attach_config = SynapseCompute.attach_configuration(\n",
+        "        linked_service,\n",
+        "        type=\"SynapseSpark\",\n",
+        "        pool_name=synapse_spark_pool_name)\n",
+        "\n",
+        "synapse_compute=ComputeTarget.attach(\n",
+        "        workspace=ws,\n",
+        "        name=synapse_compute_name,\n",
+        "        attach_configuration=attach_config)\n",
+        "\n",
+        "synapse_compute.wait_for_completion()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Start an experiment run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Prepare data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Use the default blob storage\n",
+        "def_blob_store = Datastore(ws, \"workspaceblobstore\")\n",
+        "print('Datastore {} will be used'.format(def_blob_store.name))\n",
+        "\n",
+        "# We are uploading a sample file in the local directory to be used as a datasource\n",
+        "file_name = \"Titanic.csv\"\n",
+        "def_blob_store.upload_files(files=[\"./{}\".format(file_name)], overwrite=False)\n",
+        " "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Tabular dataset as input"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Dataset\n",
+        "titanic_tabular_dataset = Dataset.Tabular.from_delimited_files(path=[(def_blob_store, file_name)])\n",
+        "input1 = titanic_tabular_dataset.as_named_input(\"tabular_input\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## File dataset as input"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Dataset\n",
+        "titanic_file_dataset = Dataset.File.from_files(path=[(def_blob_store, file_name)])\n",
+        "input2 = titanic_file_dataset.as_named_input(\"file_input\").as_hdfs()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Output config: the output will be registered as a File dataset\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.data import HDFSOutputDatasetConfig\n",
+        "output = HDFSOutputDatasetConfig(destination=(def_blob_store,\"test\")).register_on_complete(name=\"registered_dataset\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Dataprep script"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "os.makedirs(\"code\", exist_ok=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile code/dataprep.py\n",
+        "import os\n",
+        "import sys\n",
+        "import azureml.core\n",
+        "from pyspark.sql import SparkSession\n",
+        "from azureml.core import Run, Dataset\n",
+        "\n",
+        "print(azureml.core.VERSION)\n",
+        "print(os.environ)\n",
+        "\n",
+        "import argparse\n",
+        "parser = argparse.ArgumentParser()\n",
+        "parser.add_argument(\"--tabular_input\")\n",
+        "parser.add_argument(\"--file_input\")\n",
+        "parser.add_argument(\"--output_dir\")\n",
+        "args = parser.parse_args()\n",
+        "\n",
+        "# use dataset sdk to read tabular dataset\n",
+        "run_context = Run.get_context()\n",
+        "dataset = Dataset.get_by_id(run_context.experiment.workspace,id=args.tabular_input)\n",
+        "sdf = dataset.to_spark_dataframe()\n",
+        "sdf.show()\n",
+        "\n",
+        "# use hdfs path to read file dataset\n",
+        "spark= SparkSession.builder.getOrCreate()\n",
+        "sdf = spark.read.option(\"header\", \"true\").csv(args.file_input)\n",
+        "sdf.show()\n",
+        "\n",
+        "sdf.coalesce(1).write\\\n",
+        ".option(\"header\", \"true\")\\\n",
+        ".mode(\"append\")\\\n",
+        ".csv(args.output_dir)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Set up Conda dependency for the following Script Run"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core.environment import CondaDependencies\n",
+        "conda_dep = CondaDependencies()\n",
+        "conda_dep.add_pip_package(\"azureml-core==1.20.0\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## How to leverage ScriptRunConfig to submit an experiment run to an attached Synapse Spark cluster"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import RunConfiguration\n",
+        "from azureml.core import ScriptRunConfig \n",
+        "from azureml.core import Experiment\n",
+        "\n",
+        "run_config = RunConfiguration(framework=\"pyspark\")\n",
+        "run_config.target = synapse_compute_name\n",
+        "\n",
+        "run_config.spark.configuration[\"spark.driver.memory\"] = \"1g\" \n",
+        "run_config.spark.configuration[\"spark.driver.cores\"] = 2 \n",
+        "run_config.spark.configuration[\"spark.executor.memory\"] = \"1g\" \n",
+        "run_config.spark.configuration[\"spark.executor.cores\"] = 1 \n",
+        "run_config.spark.configuration[\"spark.executor.instances\"] = 1 \n",
+        "\n",
+        "run_config.environment.python.conda_dependencies = conda_dep\n",
+        "\n",
+        "script_run_config = ScriptRunConfig(source_directory = './code',\n",
+        "                                    script= 'dataprep.py',\n",
+        "                                    arguments = [\"--tabular_input\", input1, \n",
+        "                                                 \"--file_input\", input2,\n",
+        "                                                 \"--output_dir\", output],\n",
+        "                                    run_config = run_config) "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Experiment \n",
+        "exp = Experiment(workspace=ws, name=\"synapse-spark\") \n",
+        "run = exp.submit(config=script_run_config) \n",
+        "run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## How to leverage SynapseSparkStep in an AML pipeline to orchestrate data prep step on Synapse Spark and training step on AzureML compute."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Choose a name for your CPU cluster\n",
+        "cpu_cluster_name = \"cpucluster\"\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
+        "    print('Found existing cluster, use it.')\n",
+        "except ComputeTargetException:\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
+        "                                                           max_nodes=1)\n",
+        "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
+        "\n",
+        "cpu_cluster.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%writefile code/train.py\n",
+        "import glob\n",
+        "import os\n",
+        "import sys\n",
+        "from os import listdir\n",
+        "from os.path import isfile, join\n",
+        "\n",
+        "mypath = os.environ[\"step2_input\"]\n",
+        "files = [f for f in listdir(mypath) if isfile(join(mypath, f))]\n",
+        "for file in files:\n",
+        "    with open(join(mypath,file)) as f:\n",
+        "        print(f.read())"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "titanic_tabular_dataset = Dataset.Tabular.from_delimited_files(path=[(def_blob_store, file_name)])\n",
+        "titanic_file_dataset = Dataset.File.from_files(path=[(def_blob_store, file_name)])\n",
+        "\n",
+        "step1_input1 = titanic_tabular_dataset.as_named_input(\"tabular_input\")\n",
+        "step1_input2 = titanic_file_dataset.as_named_input(\"file_input\").as_hdfs()\n",
+        "step1_output = HDFSOutputDatasetConfig(destination=(def_blob_store,\"test\")).register_on_complete(name=\"registered_dataset\")\n",
+        "\n",
+        "step2_input = step1_output.as_input(\"step2_input\").as_download()\n",
+        "\n",
+        "\n",
+        "from azureml.core.environment import Environment\n",
+        "env = Environment(name=\"myenv\")\n",
+        "env.python.conda_dependencies.add_pip_package(\"azureml-core==1.20.0\")\n",
+        "\n",
+        "step_1 = SynapseSparkStep(name = 'synapse-spark',\n",
+        "                          file = 'dataprep.py',\n",
+        "                          source_directory=\"./code\", \n",
+        "                          inputs=[step1_input1, step1_input2],\n",
+        "                          outputs=[step1_output],\n",
+        "                          arguments = [\"--tabular_input\", step1_input1, \n",
+        "                                       \"--file_input\", step1_input2,\n",
+        "                                       \"--output_dir\", step1_output],\n",
+        "                          compute_target = synapse_compute_name,\n",
+        "                          driver_memory = \"7g\",\n",
+        "                          driver_cores = 4,\n",
+        "                          executor_memory = \"7g\",\n",
+        "                          executor_cores = 2,\n",
+        "                          num_executors = 1,\n",
+        "                          environment = env)\n",
+        "\n",
+        "step_2 = PythonScriptStep(script_name=\"train.py\",\n",
+        "                          arguments=[step2_input],\n",
+        "                          inputs=[step2_input],\n",
+        "                          compute_target=cpu_cluster_name,\n",
+        "                          source_directory=\"./code\",\n",
+        "                          allow_reuse=False)\n",
+        "\n",
+        "pipeline = Pipeline(workspace=ws, steps=[step_1, step_2])\n",
+        "pipeline_run = pipeline.submit('synapse-pipeline', regenerate_outputs=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "yunzhan"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.7"
+    },
+    "nteract": {
+      "version": "0.28.0"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/azure-synapse/spark_session_on_synapse_spark_pool.ipynb

@@ -0,0 +1,327 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/azure-arcadia/spark_session_on_synapse_spark_pool.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Interactive Spark Session on Synapse Spark Pool"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Install package"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install -U \"azureml-synapse\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "For JupyterLab, please additionally run:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!jupyter lab build --minimize=False"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## PLEASE restart kernel and then refresh web page before starting spark session."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 0. How to leverage Spark Magic for interactive Spark experience"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "execution": {
+          "iopub.execute_input": "2020-06-05T03:22:14.965395Z",
+          "iopub.status.busy": "2020-06-05T03:22:14.965395Z",
+          "iopub.status.idle": "2020-06-05T03:22:14.970398Z",
+          "shell.execute_reply": "2020-06-05T03:22:14.969397Z",
+          "shell.execute_reply.started": "2020-06-05T03:22:14.965395Z"
+        }
+      },
+      "outputs": [],
+      "source": [
+        "# show help\n",
+        "%synapse ?"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 1. Start Synapse Session"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "synapse_compute_name=os.getenv(\"SYNAPSE_COMPUTE_NAME\", \"<my-synapse-compute-name>\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# use Synapse compute linked to the Compute Instance's workspace with an aml envrionment.\n",
+        "# conda dependencies specified in the environment will be installed before the spark session started.\n",
+        "\n",
+        "%synapse start -c $synapse_compute_name -e AzureML-Minimal"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# use Synapse compute from anther workspace via its config file\n",
+        "\n",
+        "# %synapse start -c <compute-name> -f config.json"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# use Synapse compute from anther workspace via subscription_id, resource_group and workspace_name\n",
+        "\n",
+        "# %synapse start -c <compute-name> -s <subscription-id> -r <resource group> -w <workspace-name>"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# start a spark session with an AML environment, \n",
+        "# %synapse start -c <compute-name> -s <subscription-id> -r <resource group> -w <workspace-name> -e AzureML-Minimal"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 2. Data prepration\n",
+        "\n",
+        "Three types of datastore are supported in synapse spark, and you have two ways to load the data.\n",
+        "\n",
+        "\n",
+        "| Datastore Type     | Data Acess                    |\n",
+        "|--------------------|-------------------------------|\n",
+        "| Blob               | Credential                    |\n",
+        "| Adlsgen1           | Credential & Credential-less  |\n",
+        "| Adlsgen2           | Credential & Credential-less  |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Example 1: Data loading by HDFS path"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "**Read data from Blob**\n",
+        "\n",
+        "```python\n",
+        "# setup access key or sas token\n",
+        "\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.key.<storage account name>.blob.core.windows.net\", \"<acess key>\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.sas.<container name>.<storage account name>.blob.core.windows.net\", \"sas token\")\n",
+        "\n",
+        "df = spark.read.parquet(\"wasbs://<container name>@<storage account name>.blob.core.windows.net/<path>\")\n",
+        "```\n",
+        "\n",
+        "**Read data from Adlsgen1**\n",
+        "\n",
+        "```python\n",
+        "# setup service pricinpal which has access of the data\n",
+        "# If no data Credential is setup, the user identity will be used to do access control\n",
+        "\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.adl.account.<storage account name>.oauth2.access.token.provider.type\",\"ClientCredential\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.adl.account.<storage account name>.oauth2.client.id\", \"<client id>\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.adl.account.<storage account name>.oauth2.credential\", \"<client secret>\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.adl.account.<storage account name>.oauth2.refresh.url\", \"https://login.microsoftonline.com/<tenant id>/oauth2/token\")\n",
+        "\n",
+        "df = spark.read.csv(\"adl://<storage account name>.azuredatalakestore.net/<path>\")\n",
+        "```\n",
+        "\n",
+        "**Read data from Adlsgen2**\n",
+        "\n",
+        "```python\n",
+        "# setup service pricinpal which has access of the data\n",
+        "# If no data Credential is setup, the user identity will be used to do access control\n",
+        "\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.auth.type.<storage account name>.dfs.core.windows.net\",\"OAuth\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.oauth.provider.type.<storage account name>.dfs.core.windows.net\", \"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.oauth2.client.id.<storage account name>.dfs.core.windows.net\", \"<client id>\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.oauth2.client.secret.<storage account name>.dfs.core.windows.net\", \"<client secret>\")\n",
+        "sc._jsc.hadoopConfiguration().set(\"fs.azure.account.oauth2.client.endpoint.<storage account name>.dfs.core.windows.net\", \"https://login.microsoftonline.com/<tenant id>/oauth2/token\")\n",
+        "\n",
+        "df = spark.read.csv(\"abfss://<container name>@<storage account>.dfs.core.windows.net/<path>\")\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "execution": {
+          "iopub.execute_input": "2020-06-04T08:11:18.812276Z",
+          "iopub.status.busy": "2020-06-04T08:11:18.812276Z",
+          "iopub.status.idle": "2020-06-04T08:11:23.854526Z",
+          "shell.execute_reply": "2020-06-04T08:11:23.853525Z",
+          "shell.execute_reply.started": "2020-06-04T08:11:18.812276Z"
+        }
+      },
+      "outputs": [],
+      "source": [
+        "%%synapse\n",
+        "\n",
+        "from pyspark.sql.functions import col, desc\n",
+        "\n",
+        "df = spark.read.option(\"header\", \"true\").csv(\"wasbs://demo@dprepdata.blob.core.windows.net/Titanic.csv\")\n",
+        "df.filter(col('Survived') == 1).groupBy('Age').count().orderBy(desc('count')).show(10)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Example 2: Data loading by AML Dataset\n",
+        "\n",
+        "You can create tabular data by following the [guidance](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-register-datasets) and use to_spark_dataframe() to load the data.\n",
+        "\n",
+        "```text\n",
+        "%%synapse\n",
+        "\n",
+        "import azureml.core\n",
+        "print(azureml.core.VERSION)\n",
+        "\n",
+        "from azureml.core import Workspace, Dataset\n",
+        "ws = Workspace.get(name='<workspace name>', subscription_id='<subscription id>', resource_group='<resource group>')\n",
+        "ds = Dataset.get_by_name(ws, \"<tabular dataset name>\")\n",
+        "df = ds.to_spark_dataframe()\n",
+        "\n",
+        "# You can do more data transformation on spark dataframe\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 3. Session Metadata\n",
+        "After session started, you can check the session's metadata, find the links to Synapse portal."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%synapse meta"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 4. Stop Session\n",
+        "When current session reach the status timeout, dead or any failure, you must explicitly stop it before start new one. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%synapse stop"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "yunzhan"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.7"
+    },
+    "nteract": {
+      "version": "0.28.0"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}