update samples from Release-114 as a part of 1.38.0 SDK stable release

README.md

@@ -1,6 +1,6 @@
 # Azure Machine Learning Python SDK notebooks
 
-### **With the introduction of AzureML SDK v2, this samples repository for the v1 SDK is now deprecated and will not be monitored or updated. Users are encouraged to visit the [v2 SDK samples repository](https://github.com/Azure/azureml-examples) instead for up-to-date and enhanced examples of how to build, train, and deploy machine learning models with AzureML's newest features.**
+> a community-driven repository of examples using mlflow for tracking can be found at https://github.com/Azure/azureml-examples
 
 Welcome to the Azure Machine Learning Python SDK notebooks repository!
 

SECURITY.md

@@ -1,41 +0,0 @@
-<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->
-
-## Security
-
-Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
-
-If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.
-
-## Reporting Security Issues
-
-**Please do not report security vulnerabilities through public GitHub issues.**
-
-Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).
-
-If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
-
-You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 
-
-Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
-
-  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
-  * Full paths of source file(s) related to the manifestation of the issue
-  * The location of the affected source code (tag/branch/commit or direct URL)
-  * Any special configuration required to reproduce the issue
-  * Step-by-step instructions to reproduce the issue
-  * Proof-of-concept or exploit code (if possible)
-  * Impact of the issue, including how an attacker might exploit the issue
-
-This information will help us triage your report more quickly.
-
-If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
-
-## Preferred Languages
-
-We prefer all communications to be in English.
-
-## Policy
-
-Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
-
-<!-- END MICROSOFT SECURITY.MD BLOCK -->

configuration.ipynb

@@ -103,7 +103,7 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"This notebook was created using version 1.59.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.38.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -329,7 +329,7 @@
         "    print(\"Creating new gpu-cluster\")\n",
         "    \n",
         "    # Specify the configuration for the new cluster\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"Standard_NC6s_v3\",\n",
+        "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n",
         "                                                           min_nodes=0,\n",
         "                                                           max_nodes=4)\n",
         "    # Create the cluster with the specified name and configuration\n",
@@ -367,9 +367,9 @@
       }
     ],
     "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
+      "display_name": "Python 3.6",
       "language": "python",
-      "name": "python38-azureml"
+      "name": "python36"
     },
     "language_info": {
       "codemirror_mode": {

contrib/RAPIDS/azure-ml-with-nvidia-rapids.ipynb

@@ -174,7 +174,7 @@
         "else:\n",
         "    print(\"creating new cluster\")\n",
         "    # vm_size parameter below could be modified to one of the RAPIDS-supported VM types\n",
-        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"Standard_NC6s_v3\", min_nodes=1, max_nodes = 1)\n",
+        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"Standard_NC6s_v2\", min_nodes=1, max_nodes = 1)\n",
         "\n",
         "    # create the cluster\n",
         "    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, provisioning_config)\n",
@@ -188,6 +188,13 @@
         "### Script to process data and train model"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The _process_data.py_ script used in the step below is a slightly modified implementation of [RAPIDS Mortgage E2E example](https://github.com/rapidsai/notebooks-contrib/blob/master/intermediate_notebooks/E2E/mortgage/mortgage_e2e.ipynb)."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -366,7 +373,7 @@
         "run_config.target = gpu_cluster_name\n",
         "run_config.environment.docker.enabled = True\n",
         "run_config.environment.docker.gpu_support = True\n",
-        "run_config.environment.docker.base_image = \"mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.1-cudnn8-ubuntu20.04\"\n",
+        "run_config.environment.docker.base_image = \"mcr.microsoft.com/azureml/base-gpu:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04\"\n",
         "run_config.environment.spark.precache_packages = False\n",
         "run_config.data_references={'data':data_ref.to_config()}"
       ]
@@ -398,7 +405,7 @@
         "# run_config.target = gpu_cluster_name\n",
         "# run_config.environment.docker.enabled = True\n",
         "# run_config.environment.docker.gpu_support = True\n",
-        "# run_config.environment.docker.base_image = \"rapidsai/rapidsai:cuda9.2-runtime-ubuntu20.04\"\n",
+        "# run_config.environment.docker.base_image = \"rapidsai/rapidsai:cuda9.2-runtime-ubuntu18.04\"\n",
         "# # run_config.environment.docker.base_image_registry.address = '<registry_url>' # not required if the base_image is in Docker hub\n",
         "# # run_config.environment.docker.base_image_registry.username = '<user_name>' # needed only for private images\n",
         "# # run_config.environment.docker.base_image_registry.password = '<password>' # needed only for private images\n",
@@ -525,9 +532,9 @@
       }
     ],
     "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
+      "display_name": "Python 3.6",
       "language": "python",
-      "name": "python38-azureml"
+      "name": "python36"
     },
     "language_info": {
       "codemirror_mode": {

contrib/fairness/fairlearn-azureml-mitigation.ipynb

@@ -0,0 +1,621 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/contrib/fairness/fairlearn-azureml-mitigation.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Unfairness Mitigation with Fairlearn and Azure Machine Learning\n",
+        "**This notebook shows how to upload results from Fairlearn's GridSearch mitigation algorithm into a dashboard in Azure Machine Learning Studio**\n",
+        "\n",
+        "## Table of Contents\n",
+        "\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Loading the Data](#LoadingData)\n",
+        "1. [Training an Unmitigated Model](#UnmitigatedModel)\n",
+        "1. [Mitigation with GridSearch](#Mitigation)\n",
+        "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
+        "    1. Registering models\n",
+        "    1. Computing Fairness Metrics\n",
+        "    1. Uploading to Azure\n",
+        "1. [Conclusion](#Conclusion)\n",
+        "\n",
+        "<a id=\"Introduction\"></a>\n",
+        "## Introduction\n",
+        "This notebook shows how to use [Fairlearn (an open source fairness assessment and unfairness mitigation package)](http://fairlearn.org) and Azure Machine Learning Studio for a binary classification problem. This example uses the well-known adult census dataset. For the purposes of this notebook, we shall treat this as a loan decision problem. We will pretend that the label indicates whether or not each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan. Its purpose is purely illustrative of a workflow including a fairness dashboard - in particular, we do **not** include a full discussion of the detailed issues which arise when considering fairness in machine learning. For such discussions, please [refer to the Fairlearn website](http://fairlearn.org/).\n",
+        "\n",
+        "We will apply the [grid search algorithm](https://fairlearn.org/v0.4.6/api_reference/fairlearn.reductions.html#fairlearn.reductions.GridSearch) from the Fairlearn package using a specific notion of fairness called Demographic Parity. This produces a set of models, and we will view these in a dashboard both locally and in the Azure Machine Learning Studio.\n",
+        "\n",
+        "### Setup\n",
+        "\n",
+        "To use this notebook, an Azure Machine Learning workspace is required.\n",
+        "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
+        "This notebook also requires the following packages:\n",
+        "* `azureml-contrib-fairness`\n",
+        "* `fairlearn>=0.6.2` (pre-v0.5.0 will work with minor modifications)\n",
+        "* `joblib`\n",
+        "* `liac-arff`\n",
+        "* `raiwidgets~=0.7.0`\n",
+        "\n",
+        "Fairlearn relies on features introduced in v0.22.1 of `scikit-learn`. If you have an older version already installed, please uncomment and run the following cell:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# !pip install --upgrade scikit-learn>=0.22.1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, please ensure that when you downloaded this notebook, you also downloaded the `fairness_nb_utils.py` file from the same location, and placed it in the same directory as this notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LoadingData\"></a>\n",
+        "## Loading the Data\n",
+        "We use the well-known `adult` census dataset, which we will fetch from the OpenML website. We start with a fairly unremarkable set of imports:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairlearn.reductions import GridSearch, DemographicParity, ErrorRate\n",
+        "from raiwidgets import FairnessDashboard\n",
+        "\n",
+        "from sklearn.compose import ColumnTransformer\n",
+        "from sklearn.impute import SimpleImputer\n",
+        "from sklearn.linear_model import LogisticRegression\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn.compose import make_column_selector as selector\n",
+        "from sklearn.pipeline import Pipeline\n",
+        "\n",
+        "import pandas as pd"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can now load and inspect the data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairness_nb_utils import fetch_census_dataset\n",
+        "\n",
+        "data = fetch_census_dataset()\n",
+        "    \n",
+        "# Extract the items we want\n",
+        "X_raw = data.data\n",
+        "y = (data.target == '>50K') * 1\n",
+        "\n",
+        "X_raw[\"race\"].value_counts().to_dict()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We are going to treat the sex and race of each individual as protected attributes, and in this particular case we are going to remove these attributes from the main data (this is not always the best option - see the [Fairlearn website](http://fairlearn.github.io/) for further discussion). Protected attributes are often denoted by 'A' in the literature, and we follow that convention here:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "A = X_raw[['sex','race']]\n",
+        "X_raw = X_raw.drop(labels=['sex', 'race'], axis = 1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We now preprocess our data. To avoid the problem of data leakage, we split our data into training and test sets before performing any other transformations. Subsequent transformations (such as scalings) will be fit to the training data set, and then applied to the test dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "(X_train, X_test, y_train, y_test, A_train, A_test) = train_test_split(\n",
+        "    X_raw, y, A, test_size=0.3, random_state=12345, stratify=y\n",
+        ")\n",
+        "\n",
+        "# Ensure indices are aligned between X, y and A,\n",
+        "# after all the slicing and splitting of DataFrames\n",
+        "# and Series\n",
+        "\n",
+        "X_train = X_train.reset_index(drop=True)\n",
+        "X_test = X_test.reset_index(drop=True)\n",
+        "y_train = y_train.reset_index(drop=True)\n",
+        "y_test = y_test.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We have two types of column in the dataset - categorical columns which will need to be one-hot encoded, and numeric ones which will need to be rescaled. We also need to take care of missing values. We use a simple approach here, but please bear in mind that this is another way that bias could be introduced (especially if one subgroup tends to have more missing values).\n",
+        "\n",
+        "For this preprocessing, we make use of `Pipeline` objects from `sklearn`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "numeric_transformer = Pipeline(\n",
+        "    steps=[\n",
+        "        (\"impute\", SimpleImputer()),\n",
+        "        (\"scaler\", StandardScaler()),\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "categorical_transformer = Pipeline(\n",
+        "    [\n",
+        "        (\"impute\", SimpleImputer(strategy=\"most_frequent\")),\n",
+        "        (\"ohe\", OneHotEncoder(handle_unknown=\"ignore\", sparse=False)),\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "preprocessor = ColumnTransformer(\n",
+        "    transformers=[\n",
+        "        (\"num\", numeric_transformer, selector(dtype_exclude=\"category\")),\n",
+        "        (\"cat\", categorical_transformer, selector(dtype_include=\"category\")),\n",
+        "    ]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, the preprocessing pipeline is defined, we can run it on our training data, and apply the generated transform to our test data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_train = preprocessor.fit_transform(X_train)\n",
+        "X_test = preprocessor.transform(X_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"UnmitigatedModel\"></a>\n",
+        "## Training an Unmitigated Model\n",
+        "\n",
+        "So we have a point of comparison, we first train a model (specifically, logistic regression from scikit-learn) on the raw data, without applying any mitigation algorithm:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "unmitigated_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
+        "\n",
+        "unmitigated_predictor.fit(X_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can view this model in the fairness dashboard, and see the disparities which appear:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "FairnessDashboard(sensitive_features=A_test,\n",
+        "                  y_true=y_test,\n",
+        "                  y_pred={\"unmitigated\": unmitigated_predictor.predict(X_test)})"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Looking at the disparity in accuracy when we select 'Sex' as the sensitive feature, we see that males have an error rate about three times greater than the females. More interesting is the disparity in opportunitiy - males are offered loans at three times the rate of females.\n",
+        "\n",
+        "Despite the fact that we removed the feature from the training data, our predictor still discriminates based on sex. This demonstrates that simply ignoring a protected attribute when fitting a predictor rarely eliminates unfairness. There will generally be enough other features correlated with the removed attribute to lead to disparate impact."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Mitigation\"></a>\n",
+        "## Mitigation with GridSearch\n",
+        "\n",
+        "The `GridSearch` class in `Fairlearn` implements a simplified version of the exponentiated gradient reduction of [Agarwal et al. 2018](https://arxiv.org/abs/1803.02453). The user supplies a standard ML estimator, which is treated as a blackbox - for this simple example, we shall use the logistic regression estimator from scikit-learn. `GridSearch` works by generating a sequence of relabellings and reweightings, and trains a predictor for each.\n",
+        "\n",
+        "For this example, we specify demographic parity (on the protected attribute of sex) as the fairness metric. Demographic parity requires that individuals are offered the opportunity (a loan in this example) independent of membership in the protected class (i.e., females and males should be offered loans at the same rate). *We are using this metric for the sake of simplicity* in this example; the appropriate fairness metric can only be selected after *careful examination of the broader context* in which the model is to be used."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sweep = GridSearch(LogisticRegression(solver='liblinear', fit_intercept=True),\n",
+        "                   constraints=DemographicParity(),\n",
+        "                   grid_size=71)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "With our estimator created, we can fit it to the data. After `fit()` completes, we extract the full set of predictors from the `GridSearch` object.\n",
+        "\n",
+        "The following cell trains a many copies of the underlying estimator, and may take a minute or two to run:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sweep.fit(X_train, y_train,\n",
+        "          sensitive_features=A_train.sex)\n",
+        "\n",
+        "# For Fairlearn pre-v0.5.0, need sweep._predictors\n",
+        "predictors = sweep.predictors_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We could load these predictors into the Fairness dashboard now. However, the plot would be somewhat confusing due to their number. In this case, we are going to remove the predictors which are dominated in the error-disparity space by others from the sweep (note that the disparity will only be calculated for the protected attribute; other potentially protected attributes will *not* be mitigated). In general, one might not want to do this, since there may be other considerations beyond the strict optimisation of error and disparity (of the given protected attribute)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "errors, disparities = [], []\n",
+        "for predictor in predictors:\n",
+        "    error = ErrorRate()\n",
+        "    error.load_data(X_train, pd.Series(y_train), sensitive_features=A_train.sex)\n",
+        "    disparity = DemographicParity()\n",
+        "    disparity.load_data(X_train, pd.Series(y_train), sensitive_features=A_train.sex)\n",
+        "    \n",
+        "    errors.append(error.gamma(predictor.predict)[0])\n",
+        "    disparities.append(disparity.gamma(predictor.predict).max())\n",
+        "    \n",
+        "all_results = pd.DataFrame( {\"predictor\": predictors, \"error\": errors, \"disparity\": disparities})\n",
+        "\n",
+        "dominant_models_dict = dict()\n",
+        "base_name_format = \"census_gs_model_{0}\"\n",
+        "row_id = 0\n",
+        "for row in all_results.itertuples():\n",
+        "    model_name = base_name_format.format(row_id)\n",
+        "    errors_for_lower_or_eq_disparity = all_results[\"error\"][all_results[\"disparity\"]<=row.disparity]\n",
+        "    if row.error <= errors_for_lower_or_eq_disparity.min():\n",
+        "        dominant_models_dict[model_name] = row.predictor\n",
+        "    row_id = row_id + 1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can construct predictions for the dominant models (we include the unmitigated predictor as well, for comparison):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "predictions_dominant = {\"census_unmitigated\": unmitigated_predictor.predict(X_test)}\n",
+        "models_dominant = {\"census_unmitigated\": unmitigated_predictor}\n",
+        "for name, predictor in dominant_models_dict.items():\n",
+        "    value = predictor.predict(X_test)\n",
+        "    predictions_dominant[name] = value\n",
+        "    models_dominant[name] = predictor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "These predictions may then be viewed in the fairness dashboard. We include the race column from the dataset, as an alternative basis for assessing the models. However, since we have not based our mitigation on it, the variation in the models with respect to race can be large."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "FairnessDashboard(sensitive_features=A_test, \n",
+        "                  y_true=y_test.tolist(),\n",
+        "                  y_pred=predictions_dominant)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "When using sex as the sensitive feature and accuracy as the metric, we see a Pareto front forming - the set of predictors which represent optimal tradeoffs between accuracy and disparity in predictions. In the ideal case, we would have a predictor at (1,0) - perfectly accurate and without any unfairness under demographic parity (with respect to the protected attribute \"sex\"). The Pareto front represents the closest we can come to this ideal based on our data and choice of estimator. Note the range of the axes - the disparity axis covers more values than the accuracy, so we can reduce disparity substantially for a small loss in accuracy. Finally, we also see that the unmitigated model is towards the top right of the plot, with high accuracy, but worst disparity.\n",
+        "\n",
+        "By clicking on individual models on the plot, we can inspect their metrics for disparity and accuracy in greater detail. In a real example, we would then pick the model which represented the best trade-off between accuracy and disparity given the relevant business constraints."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"AzureUpload\"></a>\n",
+        "## Uploading a Fairness Dashboard to Azure\n",
+        "\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairnessDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. By default, the dashboard in Azure Machine Learning Studio also requires the models to be registered. The required stages are therefore:\n",
+        "1. Register the dominant models\n",
+        "1. Precompute all the required metrics\n",
+        "1. Upload to Azure\n",
+        "\n",
+        "Before that, we need to connect to Azure Machine Learning Studio:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace, Experiment, Model\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "ws.get_details()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"RegisterModels\"></a>\n",
+        "### Registering Models\n",
+        "\n",
+        "The fairness dashboard is designed to integrate with registered models, so we need to do this for the models we want in the Studio portal. The assumption is that the names of the models specified in the dashboard dictionary correspond to the `id`s (i.e. `<name>:<version>` pairs) of registered models in the workspace. We register each of the models in the `models_dominant` dictionary into the workspace. For this, we have to save each model to a file, and then register that file:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import joblib\n",
+        "import os\n",
+        "\n",
+        "os.makedirs('models', exist_ok=True)\n",
+        "def register_model(name, model):\n",
+        "    print(\"Registering \", name)\n",
+        "    model_path = \"models/{0}.pkl\".format(name)\n",
+        "    joblib.dump(value=model, filename=model_path)\n",
+        "    registered_model = Model.register(model_path=model_path,\n",
+        "                                      model_name=name,\n",
+        "                                      workspace=ws)\n",
+        "    print(\"Registered \", registered_model.id)\n",
+        "    return registered_model.id\n",
+        "\n",
+        "model_name_id_mapping = dict()\n",
+        "for name, model in models_dominant.items():\n",
+        "    m_id = register_model(name, model)\n",
+        "    model_name_id_mapping[name] = m_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, produce new predictions dictionaries, with the updated names:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "predictions_dominant_ids = dict()\n",
+        "for name, y_pred in predictions_dominant.items():\n",
+        "    predictions_dominant_ids[model_name_id_mapping[name]] = y_pred"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"PrecomputeMetrics\"></a>\n",
+        "### Precomputing Metrics\n",
+        "\n",
+        "We create a _dashboard dictionary_ using Fairlearn's `metrics` package. The `_create_group_metric_set` method has arguments similar to the Dashboard constructor, except that the sensitive features are passed as a dictionary (to ensure that names are available), and we must specify the type of prediction. Note that we use the `predictions_dominant_ids` dictionary we just created:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sf = { 'sex': A_test.sex, 'race': A_test.race }\n",
+        "\n",
+        "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
+        "\n",
+        "\n",
+        "dash_dict = _create_group_metric_set(y_true=y_test,\n",
+        "                                     predictions=predictions_dominant_ids,\n",
+        "                                     sensitive_features=sf,\n",
+        "                                     prediction_type='binary_classification')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"DashboardUpload\"></a>\n",
+        "### Uploading the Dashboard\n",
+        "\n",
+        "Now, we import our `contrib` package which contains the routine to perform the upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we can create an Experiment, then a Run, and upload our dashboard to it:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "exp = Experiment(ws, \"Test_Fairlearn_GridSearch_Census_Demo\")\n",
+        "print(exp)\n",
+        "\n",
+        "run = exp.start_logging()\n",
+        "try:\n",
+        "    dashboard_title = \"Dominant Models from GridSearch\"\n",
+        "    upload_id = upload_dashboard_dictionary(run,\n",
+        "                                            dash_dict,\n",
+        "                                            dashboard_name=dashboard_title)\n",
+        "    print(\"\\nUploaded to id: {0}\\n\".format(upload_id))\n",
+        "\n",
+        "    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)\n",
+        "finally:\n",
+        "    run.complete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The dashboard can be viewed in the Run Details page.\n",
+        "\n",
+        "Finally, we can verify that the dashboard dictionary which we downloaded matches our upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(dash_dict == downloaded_dict)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Conclusion\"></a>\n",
+        "## Conclusion\n",
+        "\n",
+        "In this notebook we have demonstrated how to use the `GridSearch` algorithm from Fairlearn to generate a collection of models, and then present them in the fairness dashboard in Azure Machine Learning Studio. Please remember that this notebook has not attempted to discuss the many considerations which should be part of any approach to unfairness mitigation. The [Fairlearn website](http://fairlearn.org/) provides that discussion"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "riedgar"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

contrib/fairness/fairlearn-azureml-mitigation.yml

@@ -0,0 +1,9 @@
+name: fairlearn-azureml-mitigation
+dependencies:
+- pip:
+  - azureml-sdk
+  - azureml-contrib-fairness
+  - fairlearn>=0.6.2
+  - joblib
+  - liac-arff
+  - raiwidgets~=0.16.0

contrib/fairness/fairness_nb_utils.py

@@ -0,0 +1,111 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+"""Utilities for azureml-contrib-fairness notebooks."""
+
+import arff
+from collections import OrderedDict
+from contextlib import closing
+import gzip
+import pandas as pd
+from sklearn.datasets import fetch_openml
+from sklearn.utils import Bunch
+import time
+
+
+def fetch_openml_with_retries(data_id, max_retries=4, retry_delay=60):
+    """Fetch a given dataset from OpenML with retries as specified."""
+    for i in range(max_retries):
+        try:
+            print("Download attempt {0} of {1}".format(i + 1, max_retries))
+            data = fetch_openml(data_id=data_id, as_frame=True)
+            break
+        except Exception as e:  # noqa: B902
+            print("Download attempt failed with exception:")
+            print(e)
+            if i + 1 != max_retries:
+                print("Will retry after {0} seconds".format(retry_delay))
+                time.sleep(retry_delay)
+                retry_delay = retry_delay * 2
+    else:
+        raise RuntimeError("Unable to download dataset from OpenML")
+
+    return data
+
+
+_categorical_columns = [
+    'workclass',
+    'education',
+    'marital-status',
+    'occupation',
+    'relationship',
+    'race',
+    'sex',
+    'native-country'
+]
+
+
+def fetch_census_dataset():
+    """Fetch the Adult Census Dataset.
+
+    This uses a particular URL for the Adult Census dataset. The code
+    is a simplified version of fetch_openml() in sklearn.
+
+    The data are copied from:
+    https://openml.org/data/v1/download/1595261.gz
+    (as of 2021-03-31)
+    """
+    try:
+        from urllib import urlretrieve
+    except ImportError:
+        from urllib.request import urlretrieve
+
+    filename = "1595261.gz"
+    data_url = "https://rainotebookscdn.blob.core.windows.net/datasets/"
+
+    remaining_attempts = 5
+    sleep_duration = 10
+    while remaining_attempts > 0:
+        try:
+            urlretrieve(data_url + filename, filename)
+
+            http_stream = gzip.GzipFile(filename=filename, mode='rb')
+
+            with closing(http_stream):
+                def _stream_generator(response):
+                    for line in response:
+                        yield line.decode('utf-8')
+
+                stream = _stream_generator(http_stream)
+                data = arff.load(stream)
+        except Exception as exc:  # noqa: B902
+            remaining_attempts -= 1
+            print("Error downloading dataset from {} ({} attempt(s) remaining)"
+                  .format(data_url, remaining_attempts))
+            print(exc)
+            time.sleep(sleep_duration)
+            sleep_duration *= 2
+            continue
+        else:
+            # dataset successfully downloaded
+            break
+    else:
+        raise Exception("Could not retrieve dataset from {}.".format(data_url))
+
+    attributes = OrderedDict(data['attributes'])
+    arff_columns = list(attributes)
+
+    raw_df = pd.DataFrame(data=data['data'], columns=arff_columns)
+
+    target_column_name = 'class'
+    target = raw_df.pop(target_column_name)
+    for col_name in _categorical_columns:
+        dtype = pd.api.types.CategoricalDtype(attributes[col_name])
+        raw_df[col_name] = raw_df[col_name].astype(dtype, copy=False)
+
+    result = Bunch()
+    result.data = raw_df
+    result.target = target
+
+    return result

contrib/fairness/upload-fairness-dashboard.ipynb

@@ -0,0 +1,545 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/contrib/fairness/upload-fairness-dashboard.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Upload a Fairness Dashboard to Azure Machine Learning Studio\n",
+        "**This notebook shows how to generate and upload a fairness assessment dashboard from Fairlearn to AzureML Studio**\n",
+        "\n",
+        "## Table of Contents\n",
+        "\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Loading the Data](#LoadingData)\n",
+        "1. [Processing the Data](#ProcessingData)\n",
+        "1. [Training Models](#TrainingModels)\n",
+        "1. [Logging in to AzureML](#LoginAzureML)\n",
+        "1. [Registering the Models](#RegisterModels)\n",
+        "1. [Using the Fairness Dashboard](#LocalDashboard)\n",
+        "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
+        "    1. Computing Fairness Metrics\n",
+        "    1. Uploading to Azure\n",
+        "1. [Conclusion](#Conclusion)\n",
+        "    \n",
+        "\n",
+        "<a id=\"Introduction\"></a>\n",
+        "## Introduction\n",
+        "\n",
+        "In this notebook, we walk through a simple example of using the `azureml-contrib-fairness` package to upload a collection of fairness statistics for a fairness dashboard. It is an example of integrating the [open source Fairlearn package](https://www.github.com/fairlearn/fairlearn) with Azure Machine Learning. This is not an example of fairness analysis or mitigation - this notebook simply shows how to get a fairness dashboard into the Azure Machine Learning portal. We will load the data and train a couple of simple models. We will then use Fairlearn to generate data for a Fairness dashboard, which we can upload to Azure Machine Learning portal and view there.\n",
+        "\n",
+        "### Setup\n",
+        "\n",
+        "To use this notebook, an Azure Machine Learning workspace is required.\n",
+        "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
+        "This notebook also requires the following packages:\n",
+        "* `azureml-contrib-fairness`\n",
+        "* `fairlearn>=0.6.2` (also works for pre-v0.5.0 with slight modifications)\n",
+        "* `joblib`\n",
+        "* `liac-arff`\n",
+        "* `raiwidgets~=0.7.0`\n",
+        "\n",
+        "Fairlearn relies on features introduced in v0.22.1 of `scikit-learn`. If you have an older version already installed, please uncomment and run the following cell:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# !pip install --upgrade scikit-learn>=0.22.1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, please ensure that when you downloaded this notebook, you also downloaded the `fairness_nb_utils.py` file from the same location, and placed it in the same directory as this notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LoadingData\"></a>\n",
+        "## Loading the Data\n",
+        "We use the well-known `adult` census dataset, which we fetch from the OpenML website. We start with a fairly unremarkable set of imports:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from sklearn import svm\n",
+        "from sklearn.compose import ColumnTransformer\n",
+        "from sklearn.impute import SimpleImputer\n",
+        "from sklearn.linear_model import LogisticRegression\n",
+        "from sklearn.model_selection import train_test_split\n",
+        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+        "from sklearn.compose import make_column_selector as selector\n",
+        "from sklearn.pipeline import Pipeline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we can load the data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from fairness_nb_utils import fetch_census_dataset\n",
+        "\n",
+        "data = fetch_census_dataset()\n",
+        "    \n",
+        "# Extract the items we want\n",
+        "X_raw = data.data\n",
+        "y = (data.target == '>50K') * 1"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can take a look at some of the data. For example, the next cells shows the counts of the different races identified in the dataset:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(X_raw[\"race\"].value_counts().to_dict())"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"ProcessingData\"></a>\n",
+        "## Processing the Data\n",
+        "\n",
+        "With the data loaded, we process it for our needs. First, we extract the sensitive features of interest into `A` (conventionally used in the literature) and leave the rest of the feature data in `X_raw`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "A = X_raw[['sex','race']]\n",
+        "X_raw = X_raw.drop(labels=['sex', 'race'],axis = 1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We now preprocess our data. To avoid the problem of data leakage, we split our data into training and test sets before performing any other transformations. Subsequent transformations (such as scalings) will be fit to the training data set, and then applied to the test dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "(X_train, X_test, y_train, y_test, A_train, A_test) = train_test_split(\n",
+        "    X_raw, y, A, test_size=0.3, random_state=12345, stratify=y\n",
+        ")\n",
+        "\n",
+        "# Ensure indices are aligned between X, y and A,\n",
+        "# after all the slicing and splitting of DataFrames\n",
+        "# and Series\n",
+        "\n",
+        "X_train = X_train.reset_index(drop=True)\n",
+        "X_test = X_test.reset_index(drop=True)\n",
+        "y_train = y_train.reset_index(drop=True)\n",
+        "y_test = y_test.reset_index(drop=True)\n",
+        "A_train = A_train.reset_index(drop=True)\n",
+        "A_test = A_test.reset_index(drop=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We have two types of column in the dataset - categorical columns which will need to be one-hot encoded, and numeric ones which will need to be rescaled. We also need to take care of missing values. We use a simple approach here, but please bear in mind that this is another way that bias could be introduced (especially if one subgroup tends to have more missing values).\n",
+        "\n",
+        "For this preprocessing, we make use of `Pipeline` objects from `sklearn`:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "numeric_transformer = Pipeline(\n",
+        "    steps=[\n",
+        "        (\"impute\", SimpleImputer()),\n",
+        "        (\"scaler\", StandardScaler()),\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "categorical_transformer = Pipeline(\n",
+        "    [\n",
+        "        (\"impute\", SimpleImputer(strategy=\"most_frequent\")),\n",
+        "        (\"ohe\", OneHotEncoder(handle_unknown=\"ignore\", sparse=False)),\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "preprocessor = ColumnTransformer(\n",
+        "    transformers=[\n",
+        "        (\"num\", numeric_transformer, selector(dtype_exclude=\"category\")),\n",
+        "        (\"cat\", categorical_transformer, selector(dtype_include=\"category\")),\n",
+        "    ]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, the preprocessing pipeline is defined, we can run it on our training data, and apply the generated transform to our test data:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_train = preprocessor.fit_transform(X_train)\n",
+        "X_test = preprocessor.transform(X_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"TrainingModels\"></a>\n",
+        "## Training Models\n",
+        "\n",
+        "We now train a couple of different models on our data. The `adult` census dataset is a classification problem - the goal is to predict whether a particular individual exceeds an income threshold. For the purpose of generating a dashboard to upload, it is sufficient to train two basic classifiers. First, a logistic regression classifier:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "lr_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
+        "\n",
+        "lr_predictor.fit(X_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "And for comparison, a support vector classifier:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "svm_predictor = svm.SVC()\n",
+        "\n",
+        "svm_predictor.fit(X_train, y_train)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LoginAzureML\"></a>\n",
+        "## Logging in to AzureML\n",
+        "\n",
+        "With our two classifiers trained, we can log into our AzureML workspace:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.core import Workspace, Experiment, Model\n",
+        "\n",
+        "ws = Workspace.from_config()\n",
+        "ws.get_details()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"RegisterModels\"></a>\n",
+        "## Registering the Models\n",
+        "\n",
+        "Next, we register our models. By default, the subroutine which uploads the models checks that the names provided correspond to registered models in the workspace. We define a utility routine to do the registering:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import joblib\n",
+        "import os\n",
+        "\n",
+        "os.makedirs('models', exist_ok=True)\n",
+        "def register_model(name, model):\n",
+        "    print(\"Registering \", name)\n",
+        "    model_path = \"models/{0}.pkl\".format(name)\n",
+        "    joblib.dump(value=model, filename=model_path)\n",
+        "    registered_model = Model.register(model_path=model_path,\n",
+        "                                      model_name=name,\n",
+        "                                      workspace=ws)\n",
+        "    print(\"Registered \", registered_model.id)\n",
+        "    return registered_model.id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now, we register the models. For convenience in subsequent method calls, we store the results in a dictionary, which maps the `id` of the registered model (a string in `name:version` format) to the predictor itself:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "model_dict = {}\n",
+        "\n",
+        "lr_reg_id = register_model(\"fairness_linear_regression\", lr_predictor)\n",
+        "model_dict[lr_reg_id] = lr_predictor\n",
+        "svm_reg_id = register_model(\"fairness_svm\", svm_predictor)\n",
+        "model_dict[svm_reg_id] = svm_predictor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"LocalDashboard\"></a>\n",
+        "## Using the Fairlearn Dashboard\n",
+        "\n",
+        "We can now examine the fairness of the two models we have training, both as a function of race and (binary) sex. Before uploading the dashboard to the AzureML portal, we will first instantiate a local instance of the Fairlearn dashboard.\n",
+        "\n",
+        "Regardless of the viewing location, the dashboard is based on three things - the true values, the model predictions and the sensitive feature values. The dashboard can use predictions from multiple models and multiple sensitive features if desired (as we are doing here).\n",
+        "\n",
+        "Our first step is to generate a dictionary mapping the `id` of the registered model to the corresponding array of predictions:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ys_pred = {}\n",
+        "for n, p in model_dict.items():\n",
+        "    ys_pred[n] = p.predict(X_test)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "We can examine these predictions in a locally invoked Fairlearn dashboard. This can be compared to the dashboard uploaded to the portal (in the next section):"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from raiwidgets import FairnessDashboard\n",
+        "\n",
+        "FairnessDashboard(sensitive_features=A_test, \n",
+        "                  y_true=y_test.tolist(),\n",
+        "                  y_pred=ys_pred)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"AzureUpload\"></a>\n",
+        "## Uploading a Fairness Dashboard to Azure\n",
+        "\n",
+        "Uploading a fairness dashboard to Azure is a two stage process. The `FairnessDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. The required stages are therefore:\n",
+        "1. Precompute all the required metrics\n",
+        "1. Upload to Azure\n",
+        "\n",
+        "\n",
+        "### Computing Fairness Metrics\n",
+        "We use Fairlearn to create a dictionary which contains all the data required to display a dashboard. This includes both the raw data (true values, predicted values and sensitive features), and also the fairness metrics. The API is similar to that used to invoke the Dashboard locally. However, there are a few minor changes to the API, and the type of problem being examined (binary classification, regression etc.) needs to be specified explicitly:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "sf = { 'Race': A_test.race, 'Sex': A_test.sex }\n",
+        "\n",
+        "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
+        "\n",
+        "dash_dict = _create_group_metric_set(y_true=y_test,\n",
+        "                                     predictions=ys_pred,\n",
+        "                                     sensitive_features=sf,\n",
+        "                                     prediction_type='binary_classification')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The `_create_group_metric_set()` method is currently underscored since its exact design is not yet final in Fairlearn."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Uploading to Azure\n",
+        "\n",
+        "We can now import the `azureml.contrib.fairness` package itself. We will round-trip the data, so there are two required subroutines:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, we can upload the generated dictionary to AzureML. The upload method requires a run, so we first create an experiment and a run. The uploaded dashboard can be seen on the corresponding Run Details page in AzureML Studio. For completeness, we also download the dashboard dictionary which we uploaded."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "exp = Experiment(ws, \"notebook-01\")\n",
+        "print(exp)\n",
+        "\n",
+        "run = exp.start_logging()\n",
+        "try:\n",
+        "    dashboard_title = \"Sample notebook upload\"\n",
+        "    upload_id = upload_dashboard_dictionary(run,\n",
+        "                                            dash_dict,\n",
+        "                                            dashboard_name=dashboard_title)\n",
+        "    print(\"\\nUploaded to id: {0}\\n\".format(upload_id))\n",
+        "\n",
+        "    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)\n",
+        "finally:\n",
+        "    run.complete()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Finally, we can verify that the dashboard dictionary which we downloaded matches our upload:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(dash_dict == downloaded_dict)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<a id=\"Conclusion\"></a>\n",
+        "## Conclusion\n",
+        "\n",
+        "In this notebook we have demonstrated how to generate and upload a fairness dashboard to AzureML Studio. We have not discussed how to analyse the results and apply mitigations. Those topics will be covered elsewhere."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "riedgar"
+      }
+    ],
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.10"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}

contrib/fairness/upload-fairness-dashboard.yml

@@ -0,0 +1,9 @@
+name: upload-fairness-dashboard
+dependencies:
+- pip:
+  - azureml-sdk
+  - azureml-contrib-fairness
+  - fairlearn>=0.6.2
+  - joblib
+  - liac-arff
+  - raiwidgets~=0.16.0

how-to-use-azureml/README.md

@@ -9,6 +9,7 @@ As a pre-requisite, run the [configuration Notebook](../configuration.ipynb) not
 * [train-on-amlcompute](./training/train-on-amlcompute): Use a 1-n node Azure ML managed compute cluster for remote runs on Azure CPU or GPU infrastructure.
 * [train-on-remote-vm](./training/train-on-remote-vm): Use Data Science Virtual Machine as a target for remote runs.
 * [logging-api](./track-and-monitor-experiments/logging-api): Learn about the details of logging metrics to run history.
+* [production-deploy-to-aks](./deployment/production-deploy-to-aks) Deploy a model to production at scale on Azure Kubernetes Service.
 * [enable-app-insights-in-production-service](./deployment/enable-app-insights-in-production-service) Learn how to use App Insights with production web service.
  
 Find quickstarts, end-to-end tutorials, and how-tos on the [official documentation site for Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/).

how-to-use-azureml/automated-machine-learning/automl_env.yml

@@ -1,26 +1,29 @@
 name: azure_automl
-channels:
-  - conda-forge
-  - pytorch
-  - main
 dependencies:
   # The python interpreter version.
-  # Azure ML only supports 3.8 and later.
+  # Currently Azure ML only supports 3.5.2 and later.
-- pip==22.3.1
+- pip==21.1.2
-- python>=3.10,<3.11
+- python>=3.5.2,<3.8
-- holidays==0.29
+- boto3==1.15.18
-- scipy==1.10.1
+- matplotlib==2.1.0
-- tqdm==4.66.1
+- numpy==1.18.5
+- cython
+- urllib3<1.24
+- scipy>=1.4.1,<=1.5.2
+- scikit-learn==0.22.1
+- pandas==0.25.1
+- py-xgboost<=0.90
+- conda-forge::fbprophet==0.5
+- holidays==0.9.11
+- pytorch::pytorch=1.4.0
+- cudatoolkit=10.1.243
+- tornado==6.1.0
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azureml-widgets~=1.59.0
+  - azureml-widgets~=1.38.0
-  - azureml-defaults~=1.59.0
+  - pytorch-transformers==1.0.0
-  - -r https://automlcesdkdataresources.blob.core.windows.net/validated-requirements/1.59.0/validated_win32_requirements.txt [--no-deps]
+  - spacy==2.1.8
-  - matplotlib==3.7.1
+  - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
-  - xgboost==1.5.2
+  - -r https://automlsdkdataresources.blob.core.windows.net/validated-requirements/1.38.0/validated_win32_requirements.txt [--no-deps]
-  - prophet==1.1.4
+  - arch==4.14
-  - onnx==1.16.1
-  - setuptools-git==1.2
-  - spacy==3.7.4
-  - https://aka.ms/automl-resources/packages/en_core_web_sm-3.7.1.tar.gz

how-to-use-azureml/automated-machine-learning/automl_env_linux.yml

@@ -1,30 +1,29 @@
 name: azure_automl
-channels:
-  - conda-forge
-  - pytorch
-  - main
 dependencies:
   # The python interpreter version.
-  # Azure ML only supports 3.7 and later.
+  # Currently Azure ML only supports 3.5.2 and later.
-- pip==22.3.1
+- pip==21.1.2
-- python>=3.10,<3.11
+- python>=3.5.2,<3.8
-- matplotlib==3.7.1
+- boto3==1.15.18
-- numpy>=1.21.6,<=1.23.5
+- matplotlib==2.1.0
-- urllib3==1.26.7
+- numpy==1.18.5
-- scipy==1.10.1
+- cython
-- scikit-learn==1.5.1
+- urllib3<1.24
-- holidays==0.29
+- scipy>=1.4.1,<=1.5.2
-- pytorch::pytorch=1.11.0
+- scikit-learn==0.22.1
+- pandas==0.25.1
+- py-xgboost<=0.90
+- conda-forge::fbprophet==0.5
+- holidays==0.9.11
+- pytorch::pytorch=1.4.0
 - cudatoolkit=10.1.243
-- notebook
+- tornado==6.1.0
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azureml-widgets~=1.59.0
+  - azureml-widgets~=1.38.0
-  - azureml-defaults~=1.59.0
   - pytorch-transformers==1.0.0
-  - spacy==3.7.4
+  - spacy==2.1.8
-  - xgboost==1.5.2
+  - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
-  - prophet==1.1.4
+  - -r https://automlsdkdataresources.blob.core.windows.net/validated-requirements/1.38.0/validated_linux_requirements.txt [--no-deps]
-  - https://aka.ms/automl-resources/packages/en_core_web_sm-3.7.1.tar.gz
+  - arch==4.14
-  - -r https://automlcesdkdataresources.blob.core.windows.net/validated-requirements/1.59.0/validated_linux_requirements.txt [--no-deps]

how-to-use-azureml/automated-machine-learning/automl_env_mac.yml

@@ -1,26 +1,30 @@
 name: azure_automl
-channels:
-  - conda-forge
-  - pytorch
-  - main
 dependencies:
   # The python interpreter version.
-  # Currently Azure ML only supports 3.7 and later.
+  # Currently Azure ML only supports 3.5.2 and later.
-- pip==22.3.1
+- pip==21.1.2
-- python>=3.10,<3.11
+- nomkl
-- numpy>=1.21.6,<=1.23.5
+- python>=3.5.2,<3.8
-- scipy==1.10.1
+- boto3==1.15.18
-- scikit-learn==1.5.1
+- matplotlib==2.1.0
-- holidays==0.29
+- numpy==1.18.5
+- cython
+- urllib3<1.24
+- scipy>=1.4.1,<=1.5.2
+- scikit-learn==0.22.1
+- pandas==0.25.1
+- py-xgboost<=0.90
+- conda-forge::fbprophet==0.5
+- holidays==0.9.11
+- pytorch::pytorch=1.4.0
+- cudatoolkit=9.0
+- tornado==6.1.0
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azureml-widgets~=1.59.0
+  - azureml-widgets~=1.38.0
-  - azureml-defaults~=1.59.0
   - pytorch-transformers==1.0.0
-  - prophet==1.1.4
+  - spacy==2.1.8
-  - xgboost==1.5.2
+  - https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
-  - spacy==3.7.4
+  - -r https://automlsdkdataresources.blob.core.windows.net/validated-requirements/1.38.0/validated_darwin_requirements.txt [--no-deps]
-  - matplotlib==3.7.1
+  - arch==4.14
-  - https://aka.ms/automl-resources/packages/en_core_web_sm-3.7.1.tar.gz
-  - -r https://automlcesdkdataresources.blob.core.windows.net/validated-requirements/1.59.0/validated_darwin_requirements.txt [--no-deps]

how-to-use-azureml/automated-machine-learning/automl_setup.cmd

@@ -33,8 +33,6 @@ if not errorlevel 1 (
   call conda env create -f %automl_env_file% -n %conda_env_name%
 )
 
-python "%conda_prefix%\scripts\pywin32_postinstall.py" -install
-
 call conda activate %conda_env_name% 2>nul:
 if errorlevel 1 goto ErrorExit
 

how-to-use-azureml/automated-machine-learning/check_conda_version.py

@@ -1,4 +1,4 @@
-from setuptools._vendor.packaging import version
+from distutils.version import LooseVersion
 import platform
 
 try:
@@ -17,7 +17,7 @@ if architecture != "64bit":
 
 minimumVersion = "4.7.8"
 
-versionInvalid = (version.parse(conda.__version__) < version.parse(minimumVersion))
+versionInvalid = (LooseVersion(conda.__version__) < LooseVersion(minimumVersion))
 
 if versionInvalid:
     print('Setup requires conda version ' + minimumVersion + ' or higher.')

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.yml

@@ -0,0 +1,4 @@
+name: auto-ml-classification-bank-marketing-all-features
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb

@@ -1,504 +1,483 @@
 {
-  "cells": [
+ "cells": [
-    {
+  {
-      "cell_type": "markdown",
+   "cell_type": "markdown",
-      "metadata": {},
+   "metadata": {},
-      "source": [
+   "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+    "# Automated Machine Learning\n",
-        "\n",
+    "_**Classification of credit card fraudulent transactions on remote compute **_\n",
-        "Licensed under the MIT License."
+    "\n",
-      ]
+    "## Contents\n",
-    },
+    "1. [Introduction](#Introduction)\n",
-    {
+    "1. [Setup](#Setup)\n",
-      "cell_type": "markdown",
+    "1. [Train](#Train)\n",
-      "metadata": {},
+    "1. [Results](#Results)\n",
-      "source": [
+    "1. [Test](#Test)\n",
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.png)"
+    "1. [Acknowledgements](#Acknowledgements)"
-      ]
+   ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Automated Machine Learning\n",
-        "_**Classification of credit card fraudulent transactions on remote compute **_\n",
-        "\n",
-        "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Setup](#Setup)\n",
-        "1. [Train](#Train)\n",
-        "1. [Results](#Results)\n",
-        "1. [Test](#Test)\n",
-        "1. [Acknowledgements](#Acknowledgements)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Introduction\n",
-        "\n",
-        "In this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.\n",
-        "\n",
-        "This notebook is using remote compute to train the model.\n",
-        "\n",
-        "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
-        "\n",
-        "In this notebook you will learn how to:\n",
-        "1. Create an experiment using an existing workspace.\n",
-        "2. Configure AutoML using `AutoMLConfig`.\n",
-        "3. Train the model using remote compute.\n",
-        "4. Explore the results.\n",
-        "5. Test the fitted model."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Setup\n",
-        "\n",
-        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import logging\n",
-        "\n",
-        "from matplotlib import pyplot as plt\n",
-        "import pandas as pd\n",
-        "import os\n",
-        "\n",
-        "import azureml.core\n",
-        "from azureml.core.experiment import Experiment\n",
-        "from azureml.core.workspace import Workspace\n",
-        "from azureml.core.dataset import Dataset\n",
-        "from azureml.train.automl import AutoMLConfig"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "\n",
-        "# choose a name for experiment\n",
-        "experiment_name = \"automl-classification-ccard-remote\"\n",
-        "\n",
-        "experiment = Experiment(ws, experiment_name)\n",
-        "\n",
-        "output = {}\n",
-        "output[\"Subscription ID\"] = ws.subscription_id\n",
-        "output[\"Workspace\"] = ws.name\n",
-        "output[\"Resource Group\"] = ws.resource_group\n",
-        "output[\"Location\"] = ws.location\n",
-        "output[\"Experiment Name\"] = experiment.name\n",
-        "output[\"SDK Version\"] = azureml.core.VERSION\n",
-        "pd.set_option(\"display.max_colwidth\", None)\n",
-        "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
-        "outputDf.T"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Create or Attach existing AmlCompute\n",
-        "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
-        "\n",
-        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
-        "\n",
-        "#### Creation of AmlCompute takes approximately 5 minutes. \n",
-        "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpu-cluster-1\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
-        "    print(\"Found existing cluster, use it.\")\n",
-        "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(\n",
-        "        vm_size=\"STANDARD_DS12_V2\", max_nodes=6\n",
-        "    )\n",
-        "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
-        "compute_target.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load Data\n",
-        "\n",
-        "Load the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "name": "load-data"
-      },
-      "outputs": [],
-      "source": [
-        "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv\"\n",
-        "dataset = Dataset.Tabular.from_delimited_files(data)\n",
-        "training_data, validation_data = dataset.random_split(percentage=0.8, seed=223)\n",
-        "label_column_name = \"Class\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Train\n",
-        "\n",
-        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
-        "\n",
-        "|Property|Description|\n",
-        "|-|-|\n",
-        "|**task**|classification or regression|\n",
-        "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
-        "|**enable_early_stopping**|Stop the run if the metric score is not showing improvement.|\n",
-        "|**n_cross_validations**|Number of cross validation splits.|\n",
-        "|**training_data**|Input dataset, containing both features and label column.|\n",
-        "|**label_column_name**|The name of the label column.|\n",
-        "\n",
-        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "name": "automl-config"
-      },
-      "outputs": [],
-      "source": [
-        "automl_settings = {\n",
-        "    \"n_cross_validations\": 3,\n",
-        "    \"primary_metric\": \"average_precision_score_weighted\",\n",
-        "    \"enable_early_stopping\": True,\n",
-        "    \"max_concurrent_iterations\": 2,  # This is a limit for testing purpose, please increase it as per cluster size\n",
-        "    \"experiment_timeout_hours\": 0.25,  # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ablity to find the best model possible\n",
-        "    \"verbosity\": logging.INFO,\n",
-        "}\n",
-        "\n",
-        "automl_config = AutoMLConfig(\n",
-        "    task=\"classification\",\n",
-        "    debug_log=\"automl_errors.log\",\n",
-        "    compute_target=compute_target,\n",
-        "    training_data=training_data,\n",
-        "    label_column_name=label_column_name,\n",
-        "    **automl_settings,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run = experiment.submit(automl_config, show_output=False)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# If you need to retrieve a run that already started, use the following code\n",
-        "# from azureml.train.automl.run import AutoMLRun\n",
-        "# remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Results"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Widget for Monitoring Runs\n",
-        "\n",
-        "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
-        "\n",
-        "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "tags": [
-          "widget-rundetails-sample"
-        ]
-      },
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "\n",
-        "RunDetails(remote_run).show()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run.wait_for_completion(show_output=False)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Explain model\n",
-        "\n",
-        "Automated ML models can be explained and visualized using the SDK Explainability library. "
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Analyze results\n",
-        "\n",
-        "### Retrieve the Best Model\n",
-        "\n",
-        "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model.  Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "best_run, fitted_model = remote_run.get_output()\n",
-        "fitted_model"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Print the properties of the model\n",
-        "The fitted_model is a python object and you can read the different properties of the object.\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Test the fitted model\n",
-        "\n",
-        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# convert the test data to dataframe\n",
-        "X_test_df = validation_data.drop_columns(\n",
-        "    columns=[label_column_name]\n",
-        ").to_pandas_dataframe()\n",
-        "y_test_df = validation_data.keep_columns(\n",
-        "    columns=[label_column_name], validate=True\n",
-        ").to_pandas_dataframe()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# call the predict functions on the model\n",
-        "y_pred = fitted_model.predict(X_test_df)\n",
-        "y_pred"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Calculate metrics for the prediction\n",
-        "\n",
-        "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
-        "from the trained model that was returned."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from sklearn.metrics import confusion_matrix\n",
-        "import numpy as np\n",
-        "import itertools\n",
-        "\n",
-        "cf = confusion_matrix(y_test_df.values, y_pred)\n",
-        "plt.imshow(cf, cmap=plt.cm.Blues, interpolation=\"nearest\")\n",
-        "plt.colorbar()\n",
-        "plt.title(\"Confusion Matrix\")\n",
-        "plt.xlabel(\"Predicted\")\n",
-        "plt.ylabel(\"Actual\")\n",
-        "class_labels = [\"False\", \"True\"]\n",
-        "tick_marks = np.arange(len(class_labels))\n",
-        "plt.xticks(tick_marks, class_labels)\n",
-        "plt.yticks([-0.5, 0, 1, 1.5], [\"\", \"False\", \"True\", \"\"])\n",
-        "# plotting text value inside cells\n",
-        "thresh = cf.max() / 2.0\n",
-        "for i, j in itertools.product(range(cf.shape[0]), range(cf.shape[1])):\n",
-        "    plt.text(\n",
-        "        j,\n",
-        "        i,\n",
-        "        format(cf[i, j], \"d\"),\n",
-        "        horizontalalignment=\"center\",\n",
-        "        color=\"white\" if cf[i, j] > thresh else \"black\",\n",
-        "    )\n",
-        "plt.show()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Acknowledgements"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
-        "\n",
-        "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Universit\u00c3\u00a9 Libre de Bruxelles) on big data mining and fraud detection.\n",
-        "More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
-        "\n",
-        "Please cite the following works:\n",
-        "\n",
-        "Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
-        "\n",
-        "Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
-        "\n",
-        "Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
-        "\n",
-        "Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
-        "\n",
-        "Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-A\u00c3\u00abl; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
-        "\n",
-        "Carcillo, Fabrizio; Le Borgne, Yann-A\u00c3\u00abl; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing\n",
-        "\n",
-        "Bertrand Lebichot, Yann-A\u00c3\u00abl Le Borgne, Liyun He, Frederic Obl\u00c3\u00a9, Gianluca Bontempi Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, pp 78-88, 2019\n",
-        "\n",
-        "Fabrizio Carcillo, Yann-A\u00c3\u00abl Le Borgne, Olivier Caelen, Frederic Obl\u00c3\u00a9, Gianluca Bontempi Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection Information Sciences, 2019"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "ratanase"
-      }
-    ],
-    "category": "tutorial",
-    "compute": [
-      "AML Compute"
-    ],
-    "datasets": [
-      "Creditcard"
-    ],
-    "deployment": [
-      "None"
-    ],
-    "exclude_from_index": false,
-    "file_extension": ".py",
-    "framework": [
-      "None"
-    ],
-    "friendly_name": "Classification of credit card fraudulent transactions using Automated ML",
-    "index_order": 5,
-    "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
-      "language": "python",
-      "name": "python38-azureml"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.7"
-    },
-    "mimetype": "text/x-python",
-    "name": "python",
-    "nbconvert_exporter": "python",
-    "pygments_lexer": "ipython3",
-    "tags": [
-      "remote_run",
-      "AutomatedML"
-    ],
-    "task": "Classification",
-    "version": "3.6.7"
   },
-  "nbformat": 4,
+  {
-  "nbformat_minor": 2
+   "cell_type": "markdown",
-}
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "In this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.\n",
+    "\n",
+    "This notebook is using remote compute to train the model.\n",
+    "\n",
+    "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
+    "\n",
+    "In this notebook you will learn how to:\n",
+    "1. Create an experiment using an existing workspace.\n",
+    "2. Configure AutoML using `AutoMLConfig`.\n",
+    "3. Train the model using remote compute.\n",
+    "4. Explore the results.\n",
+    "5. Test the fitted model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "\n",
+    "from matplotlib import pyplot as plt\n",
+    "import pandas as pd\n",
+    "import os\n",
+    "\n",
+    "import azureml.core\n",
+    "from azureml.core.experiment import Experiment\n",
+    "from azureml.core.workspace import Workspace\n",
+    "from azureml.core.dataset import Dataset\n",
+    "from azureml.train.automl import AutoMLConfig"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ws = Workspace.from_config()\n",
+    "\n",
+    "# choose a name for experiment\n",
+    "experiment_name = \"automl-classification-ccard-remote\"\n",
+    "\n",
+    "experiment = Experiment(ws, experiment_name)\n",
+    "\n",
+    "output = {}\n",
+    "output[\"Subscription ID\"] = ws.subscription_id\n",
+    "output[\"Workspace\"] = ws.name\n",
+    "output[\"Resource Group\"] = ws.resource_group\n",
+    "output[\"Location\"] = ws.location\n",
+    "output[\"Experiment Name\"] = experiment.name\n",
+    "pd.set_option(\"display.max_colwidth\", -1)\n",
+    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
+    "outputDf.T"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create or Attach existing AmlCompute\n",
+    "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
+    "\n",
+    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+    "\n",
+    "#### Creation of AmlCompute takes approximately 5 minutes. \n",
+    "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
+    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+    "from azureml.core.compute_target import ComputeTargetException\n",
+    "\n",
+    "# Choose a name for your CPU cluster\n",
+    "cpu_cluster_name = \"cpu-cluster-1\"\n",
+    "\n",
+    "# Verify that cluster does not exist already\n",
+    "try:\n",
+    "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
+    "    print(\"Found existing cluster, use it.\")\n",
+    "except ComputeTargetException:\n",
+    "    compute_config = AmlCompute.provisioning_configuration(\n",
+    "        vm_size=\"STANDARD_DS12_V2\", max_nodes=6\n",
+    "    )\n",
+    "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
+    "compute_target.wait_for_completion(show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load Data\n",
+    "\n",
+    "Load the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv\"\n",
+    "dataset = Dataset.Tabular.from_delimited_files(data)\n",
+    "training_data, validation_data = dataset.random_split(percentage=0.8, seed=223)\n",
+    "label_column_name = \"Class\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Train\n",
+    "\n",
+    "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
+    "\n",
+    "|Property|Description|\n",
+    "|-|-|\n",
+    "|**task**|classification or regression|\n",
+    "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
+    "|**enable_early_stopping**|Stop the run if the metric score is not showing improvement.|\n",
+    "|**n_cross_validations**|Number of cross validation splits.|\n",
+    "|**training_data**|Input dataset, containing both features and label column.|\n",
+    "|**label_column_name**|The name of the label column.|\n",
+    "\n",
+    "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_settings = {\n",
+    "    \"n_cross_validations\": 3,\n",
+    "    \"primary_metric\": \"average_precision_score_weighted\",\n",
+    "    \"enable_early_stopping\": True,\n",
+    "    \"max_concurrent_iterations\": 2,  # This is a limit for testing purpose, please increase it as per cluster size\n",
+    "    \"experiment_timeout_hours\": 0.25,  # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ablity to find the best model possible\n",
+    "    \"verbosity\": logging.INFO,\n",
+    "}\n",
+    "\n",
+    "automl_config = AutoMLConfig(\n",
+    "    task=\"classification\",\n",
+    "    debug_log=\"automl_errors.log\",\n",
+    "    compute_target=compute_target,\n",
+    "    training_data=training_data,\n",
+    "    label_column_name=label_column_name,\n",
+    "    **automl_settings,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "remote_run = experiment.submit(automl_config, show_output=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# If you need to retrieve a run that already started, use the following code\n",
+    "# from azureml.train.automl.run import AutoMLRun\n",
+    "# remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Widget for Monitoring Runs\n",
+    "\n",
+    "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
+    "\n",
+    "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "widget-rundetails-sample"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "from azureml.widgets import RunDetails\n",
+    "\n",
+    "RunDetails(remote_run).show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "remote_run.wait_for_completion(show_output=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Explain model\n",
+    "\n",
+    "Automated ML models can be explained and visualized using the SDK Explainability library. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analyze results\n",
+    "\n",
+    "### Retrieve the Best Model\n",
+    "\n",
+    "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model.  Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "best_run, fitted_model = remote_run.get_output()\n",
+    "fitted_model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Print the properties of the model\n",
+    "The fitted_model is a python object and you can read the different properties of the object.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Test the fitted model\n",
+    "\n",
+    "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# convert the test data to dataframe\n",
+    "X_test_df = validation_data.drop_columns(\n",
+    "    columns=[label_column_name]\n",
+    ").to_pandas_dataframe()\n",
+    "y_test_df = validation_data.keep_columns(\n",
+    "    columns=[label_column_name], validate=True\n",
+    ").to_pandas_dataframe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# call the predict functions on the model\n",
+    "y_pred = fitted_model.predict(X_test_df)\n",
+    "y_pred"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Calculate metrics for the prediction\n",
+    "\n",
+    "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
+    "from the trained model that was returned."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics import confusion_matrix\n",
+    "import numpy as np\n",
+    "import itertools\n",
+    "\n",
+    "cf = confusion_matrix(y_test_df.values, y_pred)\n",
+    "plt.imshow(cf, cmap=plt.cm.Blues, interpolation=\"nearest\")\n",
+    "plt.colorbar()\n",
+    "plt.title(\"Confusion Matrix\")\n",
+    "plt.xlabel(\"Predicted\")\n",
+    "plt.ylabel(\"Actual\")\n",
+    "class_labels = [\"False\", \"True\"]\n",
+    "tick_marks = np.arange(len(class_labels))\n",
+    "plt.xticks(tick_marks, class_labels)\n",
+    "plt.yticks([-0.5, 0, 1, 1.5], [\"\", \"False\", \"True\", \"\"])\n",
+    "# plotting text value inside cells\n",
+    "thresh = cf.max() / 2.0\n",
+    "for i, j in itertools.product(range(cf.shape[0]), range(cf.shape[1])):\n",
+    "    plt.text(\n",
+    "        j,\n",
+    "        i,\n",
+    "        format(cf[i, j], \"d\"),\n",
+    "        horizontalalignment=\"center\",\n",
+    "        color=\"white\" if cf[i, j] > thresh else \"black\",\n",
+    "    )\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Acknowledgements"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
+    "\n",
+    "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection.\n",
+    "More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
+    "\n",
+    "Please cite the following works:\n",
+    "\n",
+    "Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
+    "\n",
+    "Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
+    "\n",
+    "Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
+    "\n",
+    "Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
+    "\n",
+    "Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-Aël; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
+    "\n",
+    "Carcillo, Fabrizio; Le Borgne, Yann-Aël; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing\n",
+    "\n",
+    "Bertrand Lebichot, Yann-Aël Le Borgne, Liyun He, Frederic Oblé, Gianluca Bontempi Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, pp 78-88, 2019\n",
+    "\n",
+    "Fabrizio Carcillo, Yann-Aël Le Borgne, Olivier Caelen, Frederic Oblé, Gianluca Bontempi Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection Information Sciences, 2019"
+   ]
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": "ratanase"
+   }
+  ],
+  "category": "tutorial",
+  "compute": [
+   "AML Compute"
+  ],
+  "datasets": [
+   "Creditcard"
+  ],
+  "deployment": [
+   "None"
+  ],
+  "exclude_from_index": false,
+  "file_extension": ".py",
+  "framework": [
+   "None"
+  ],
+  "friendly_name": "Classification of credit card fraudulent transactions using Automated ML",
+  "index_order": 5,
+  "kernelspec": {
+   "display_name": "Python 3.6 - AzureML",
+   "language": "python",
+   "name": "python3-azureml"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.7"
+  },
+  "mimetype": "text/x-python",
+  "name": "python",
+  "nbconvert_exporter": "python",
+  "pygments_lexer": "ipython3",
+  "tags": [
+   "remote_run",
+   "AutomatedML"
+  ],
+  "task": "Classification",
+  "version": "3.6.7"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.yml

@@ -0,0 +1,4 @@
+name: auto-ml-classification-credit-card-fraud
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb

@@ -0,0 +1,591 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Automated Machine Learning\n",
+    "_**Text Classification Using Deep Learning**_\n",
+    "\n",
+    "## Contents\n",
+    "1. [Introduction](#Introduction)\n",
+    "1. [Setup](#Setup)\n",
+    "1. [Data](#Data)\n",
+    "1. [Train](#Train)\n",
+    "1. [Evaluate](#Evaluate)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "This notebook demonstrates classification with text data using deep learning in AutoML.\n",
+    "\n",
+    "AutoML highlights here include using deep neural networks (DNNs) to create embedded features from text data. Depending on the compute cluster the user provides, AutoML tried out Bidirectional Encoder Representations from Transformers (BERT) when a GPU compute is used, and Bidirectional Long-Short Term neural network (BiLSTM) when a CPU compute is used, thereby optimizing the choice of DNN for the uesr's setup.\n",
+    "\n",
+    "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
+    "\n",
+    "Notebook synopsis:\n",
+    "\n",
+    "1. Creating an Experiment in an existing Workspace\n",
+    "2. Configuration and remote run of AutoML for a text dataset (20 Newsgroups dataset from scikit-learn) for classification\n",
+    "3. Registering the best model for future use\n",
+    "4. Evaluating the final model on a test set"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import logging\n",
+    "import os\n",
+    "import shutil\n",
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "import azureml.core\n",
+    "from azureml.core.experiment import Experiment\n",
+    "from azureml.core.workspace import Workspace\n",
+    "from azureml.core.dataset import Dataset\n",
+    "from azureml.core.compute import AmlCompute\n",
+    "from azureml.core.compute import ComputeTarget\n",
+    "from azureml.core.run import Run\n",
+    "from azureml.widgets import RunDetails\n",
+    "from azureml.core.model import Model\n",
+    "from helper import run_inference, get_result_df\n",
+    "from azureml.train.automl import AutoMLConfig\n",
+    "from sklearn.datasets import fetch_20newsgroups"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ws = Workspace.from_config()\n",
+    "\n",
+    "# Choose an experiment name.\n",
+    "experiment_name = \"automl-classification-text-dnn\"\n",
+    "\n",
+    "experiment = Experiment(ws, experiment_name)\n",
+    "\n",
+    "output = {}\n",
+    "output[\"Subscription ID\"] = ws.subscription_id\n",
+    "output[\"Workspace Name\"] = ws.name\n",
+    "output[\"Resource Group\"] = ws.resource_group\n",
+    "output[\"Location\"] = ws.location\n",
+    "output[\"Experiment Name\"] = experiment.name\n",
+    "pd.set_option(\"display.max_colwidth\", -1)\n",
+    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
+    "outputDf.T"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set up a compute cluster\n",
+    "This section uses a user-provided compute cluster (named \"dnntext-cluster\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments.\n",
+    "\n",
+    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
+    "\n",
+    "Whether you provide/select a CPU or GPU cluster, AutoML will choose the appropriate DNN for that setup - BiLSTM or BERT text featurizer will be included in the candidate featurizers on CPU and GPU respectively.  If your goal is to obtain the most accurate model, we recommend you use GPU clusters since BERT featurizers usually outperform BiLSTM featurizers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+    "from azureml.core.compute_target import ComputeTargetException\n",
+    "\n",
+    "num_nodes = 2\n",
+    "\n",
+    "# Choose a name for your cluster.\n",
+    "amlcompute_cluster_name = \"dnntext-cluster\"\n",
+    "\n",
+    "# Verify that cluster does not exist already\n",
+    "try:\n",
+    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
+    "    print(\"Found existing cluster, use it.\")\n",
+    "except ComputeTargetException:\n",
+    "    compute_config = AmlCompute.provisioning_configuration(\n",
+    "        vm_size=\"STANDARD_NC6\",  # CPU for BiLSTM, such as \"STANDARD_D2_V2\"\n",
+    "        # To use BERT (this is recommended for best performance), select a GPU such as \"STANDARD_NC6\"\n",
+    "        # or similar GPU option\n",
+    "        # available in your workspace\n",
+    "        max_nodes=num_nodes,\n",
+    "    )\n",
+    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
+    "\n",
+    "compute_target.wait_for_completion(show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Get data\n",
+    "For this notebook we will use 20 Newsgroups data from scikit-learn. We filter the data to contain four classes and take a sample as training data. Please note that for accuracy improvement, more data is needed. For this notebook we provide a small-data example so that you can use this template to use with your larger sized data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_dir = \"text-dnn-data\"  # Local directory to store data\n",
+    "blobstore_datadir = data_dir  # Blob store directory to store data in\n",
+    "target_column_name = \"y\"\n",
+    "feature_column_name = \"X\"\n",
+    "\n",
+    "\n",
+    "def get_20newsgroups_data():\n",
+    "    \"\"\"Fetches 20 Newsgroups data from scikit-learn\n",
+    "    Returns them in form of pandas dataframes\n",
+    "    \"\"\"\n",
+    "    remove = (\"headers\", \"footers\", \"quotes\")\n",
+    "    categories = [\n",
+    "        \"rec.sport.baseball\",\n",
+    "        \"rec.sport.hockey\",\n",
+    "        \"comp.graphics\",\n",
+    "        \"sci.space\",\n",
+    "    ]\n",
+    "\n",
+    "    data = fetch_20newsgroups(\n",
+    "        subset=\"train\",\n",
+    "        categories=categories,\n",
+    "        shuffle=True,\n",
+    "        random_state=42,\n",
+    "        remove=remove,\n",
+    "    )\n",
+    "    data = pd.DataFrame(\n",
+    "        {feature_column_name: data.data, target_column_name: data.target}\n",
+    "    )\n",
+    "\n",
+    "    data_train = data[:200]\n",
+    "    data_test = data[200:300]\n",
+    "\n",
+    "    data_train = remove_blanks_20news(\n",
+    "        data_train, feature_column_name, target_column_name\n",
+    "    )\n",
+    "    data_test = remove_blanks_20news(data_test, feature_column_name, target_column_name)\n",
+    "\n",
+    "    return data_train, data_test\n",
+    "\n",
+    "\n",
+    "def remove_blanks_20news(data, feature_column_name, target_column_name):\n",
+    "\n",
+    "    data[feature_column_name] = (\n",
+    "        data[feature_column_name]\n",
+    "        .replace(r\"\\n\", \" \", regex=True)\n",
+    "        .apply(lambda x: x.strip())\n",
+    "    )\n",
+    "    data = data[data[feature_column_name] != \"\"]\n",
+    "\n",
+    "    return data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Fetch data and upload to datastore for use in training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_train, data_test = get_20newsgroups_data()\n",
+    "\n",
+    "if not os.path.isdir(data_dir):\n",
+    "    os.mkdir(data_dir)\n",
+    "\n",
+    "train_data_fname = data_dir + \"/train_data.csv\"\n",
+    "test_data_fname = data_dir + \"/test_data.csv\"\n",
+    "\n",
+    "data_train.to_csv(train_data_fname, index=False)\n",
+    "data_test.to_csv(test_data_fname, index=False)\n",
+    "\n",
+    "datastore = ws.get_default_datastore()\n",
+    "datastore.upload(src_dir=data_dir, target_path=blobstore_datadir, overwrite=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_dataset = Dataset.Tabular.from_delimited_files(\n",
+    "    path=[(datastore, blobstore_datadir + \"/train_data.csv\")]\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Prepare AutoML run"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook uses the blocked_models parameter to exclude some models that can take a longer time to train on some text datasets. You can choose to remove models from the blocked_models list but you may need to increase the experiment_timeout_hours parameter value to get results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_settings = {\n",
+    "    \"experiment_timeout_minutes\": 30,\n",
+    "    \"primary_metric\": \"accuracy\",\n",
+    "    \"max_concurrent_iterations\": num_nodes,\n",
+    "    \"max_cores_per_iteration\": -1,\n",
+    "    \"enable_dnn\": True,\n",
+    "    \"enable_early_stopping\": True,\n",
+    "    \"validation_size\": 0.3,\n",
+    "    \"verbosity\": logging.INFO,\n",
+    "    \"enable_voting_ensemble\": False,\n",
+    "    \"enable_stack_ensemble\": False,\n",
+    "}\n",
+    "\n",
+    "automl_config = AutoMLConfig(\n",
+    "    task=\"classification\",\n",
+    "    debug_log=\"automl_errors.log\",\n",
+    "    compute_target=compute_target,\n",
+    "    training_data=train_dataset,\n",
+    "    label_column_name=target_column_name,\n",
+    "    blocked_models=[\"LightGBM\", \"XGBoostClassifier\"],\n",
+    "    **automl_settings,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Submit AutoML Run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_run = experiment.submit(automl_config, show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Displaying the run objects gives you links to the visual tools in the Azure Portal. Go try them!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Retrieve the Best Model\n",
+    "Below we select the best model pipeline from our iterations, use it to test on test data on the same compute cluster."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For local inferencing, you can load the model locally via. the method `remote_run.get_output()`. For more information on the arguments expected by this method, you can run `remote_run.get_output??`.\n",
+    "Note that when the model contains BERT, this step will require pytorch and pytorch-transformers installed in your local environment. The exact versions of these packages can be found in the **automl_env.yml** file located in the local copy of your azureml-examples folder here: \"azureml-examples/python-sdk/tutorials/automl-with-azureml\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieve the best Run object\n",
+    "best_run = automl_run.get_best_child()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can now see what text transformations are used to convert text data to features for this dataset, including deep learning transformations based on BiLSTM or Transformer (BERT is one implementation of a Transformer) models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Download the featurization summary JSON file locally\n",
+    "best_run.download_file(\n",
+    "    \"outputs/featurization_summary.json\", \"featurization_summary.json\"\n",
+    ")\n",
+    "\n",
+    "# Render the JSON as a pandas DataFrame\n",
+    "with open(\"featurization_summary.json\", \"r\") as f:\n",
+    "    records = json.load(f)\n",
+    "\n",
+    "featurization_summary = pd.DataFrame.from_records(records)\n",
+    "featurization_summary[\"Transformations\"].tolist()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Registering the best model\n",
+    "We now register the best fitted model from the AutoML Run for use in future deployments.  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Get results stats, extract the best model from AutoML run, download and register the resultant best model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "summary_df = get_result_df(automl_run)\n",
+    "best_dnn_run_id = summary_df[\"run_id\"].iloc[0]\n",
+    "best_dnn_run = Run(experiment, best_dnn_run_id)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_dir = \"Model\"  # Local folder where the model will be stored temporarily\n",
+    "if not os.path.isdir(model_dir):\n",
+    "    os.mkdir(model_dir)\n",
+    "\n",
+    "best_dnn_run.download_file(\"outputs/model.pkl\", model_dir + \"/model.pkl\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Register the model in your Azure Machine Learning Workspace. If you previously registered a model, please make sure to delete it so as to replace it with this new model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Register the model\n",
+    "model_name = \"textDNN-20News\"\n",
+    "model = Model.register(\n",
+    "    model_path=model_dir + \"/model.pkl\", model_name=model_name, tags=None, workspace=ws\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluate on Test Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We now use the best fitted model from the AutoML Run to make predictions on the test set.  \n",
+    "\n",
+    "Test set schema should match that of the training set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_dataset = Dataset.Tabular.from_delimited_files(\n",
+    "    path=[(datastore, blobstore_datadir + \"/test_data.csv\")]\n",
+    ")\n",
+    "\n",
+    "# preview the first 3 rows of the dataset\n",
+    "test_dataset.take(3).to_pandas_dataframe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_experiment = Experiment(ws, experiment_name + \"_test\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "script_folder = os.path.join(os.getcwd(), \"inference\")\n",
+    "os.makedirs(script_folder, exist_ok=True)\n",
+    "shutil.copy(\"infer.py\", script_folder)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_run = run_inference(\n",
+    "    test_experiment,\n",
+    "    compute_target,\n",
+    "    script_folder,\n",
+    "    best_dnn_run,\n",
+    "    test_dataset,\n",
+    "    target_column_name,\n",
+    "    model_name,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Display computed metrics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "RunDetails(test_run).show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_run.wait_for_completion()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pd.Series(test_run.get_metrics())"
+   ]
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": "anshirga"
+   }
+  ],
+  "compute": [
+   "AML Compute"
+  ],
+  "datasets": [
+   "None"
+  ],
+  "deployment": [
+   "None"
+  ],
+  "exclude_from_index": false,
+  "framework": [
+   "None"
+  ],
+  "friendly_name": "DNN Text Featurization",
+  "index_order": 2,
+  "kernelspec": {
+   "display_name": "Python 3.6 - AzureML",
+   "language": "python",
+   "name": "python3-azureml"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.7"
+  },
+  "tags": [
+   "None"
+  ],
+  "task": "Text featurization using DNNs for classification"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.yml

@@ -0,0 +1,4 @@
+name: auto-ml-classification-text-dnn
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/classification-text-dnn/helper.py

@@ -0,0 +1,68 @@
+import pandas as pd
+from azureml.core import Environment
+from azureml.train.estimator import Estimator
+from azureml.core.run import Run
+
+
+def run_inference(
+    test_experiment,
+    compute_target,
+    script_folder,
+    train_run,
+    test_dataset,
+    target_column_name,
+    model_name,
+):
+
+    inference_env = train_run.get_environment()
+
+    est = Estimator(
+        source_directory=script_folder,
+        entry_script="infer.py",
+        script_params={
+            "--target_column_name": target_column_name,
+            "--model_name": model_name,
+        },
+        inputs=[test_dataset.as_named_input("test_data")],
+        compute_target=compute_target,
+        environment_definition=inference_env,
+    )
+
+    run = test_experiment.submit(
+        est,
+        tags={
+            "training_run_id": train_run.id,
+            "run_algorithm": train_run.properties["run_algorithm"],
+            "valid_score": train_run.properties["score"],
+            "primary_metric": train_run.properties["primary_metric"],
+        },
+    )
+
+    run.log("run_algorithm", run.tags["run_algorithm"])
+    return run
+
+
+def get_result_df(remote_run):
+
+    children = list(remote_run.get_children(recursive=True))
+    summary_df = pd.DataFrame(
+        index=["run_id", "run_algorithm", "primary_metric", "Score"]
+    )
+    goal_minimize = False
+    for run in children:
+        if "run_algorithm" in run.properties and "score" in run.properties:
+            summary_df[run.id] = [
+                run.id,
+                run.properties["run_algorithm"],
+                run.properties["primary_metric"],
+                float(run.properties["score"]),
+            ]
+            if "goal" in run.properties:
+                goal_minimize = run.properties["goal"].split("_")[-1] == "min"
+
+    summary_df = summary_df.T.sort_values(
+        "Score", ascending=goal_minimize
+    ).drop_duplicates(["run_algorithm"])
+    summary_df = summary_df.set_index("run_algorithm")
+
+    return summary_df

how-to-use-azureml/automated-machine-learning/classification-text-dnn/infer.py

@@ -0,0 +1,68 @@
+import argparse
+
+import pandas as pd
+import numpy as np
+
+from sklearn.externals import joblib
+
+from azureml.automl.runtime.shared.score import scoring, constants
+from azureml.core import Run
+from azureml.core.model import Model
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--target_column_name",
+    type=str,
+    dest="target_column_name",
+    help="Target Column Name",
+)
+parser.add_argument(
+    "--model_name", type=str, dest="model_name", help="Name of registered model"
+)
+
+args = parser.parse_args()
+target_column_name = args.target_column_name
+model_name = args.model_name
+
+print("args passed are: ")
+print("Target column name: ", target_column_name)
+print("Name of registered model: ", model_name)
+
+model_path = Model.get_model_path(model_name)
+# deserialize the model file back into a sklearn model
+model = joblib.load(model_path)
+
+run = Run.get_context()
+# get input dataset by name
+test_dataset = run.input_datasets["test_data"]
+
+X_test_df = test_dataset.drop_columns(
+    columns=[target_column_name]
+).to_pandas_dataframe()
+y_test_df = (
+    test_dataset.with_timestamp_columns(None)
+    .keep_columns(columns=[target_column_name])
+    .to_pandas_dataframe()
+)
+
+predicted = model.predict_proba(X_test_df)
+
+if isinstance(predicted, pd.DataFrame):
+    predicted = predicted.values
+
+# Use the AutoML scoring module
+train_labels = model.classes_
+class_labels = np.unique(
+    np.concatenate((y_test_df.values, np.reshape(train_labels, (-1, 1))))
+)
+classification_metrics = list(constants.CLASSIFICATION_SCALAR_SET)
+scores = scoring.score_classification(
+    y_test_df.values, predicted, classification_metrics, class_labels, train_labels
+)
+
+print("scores:")
+print(scores)
+
+for key, value in scores.items():
+    run.log(key, value)

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.yml

@@ -0,0 +1,4 @@
+name: auto-ml-continuous-retraining
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/continuous-retraining/check_data.py

@@ -31,15 +31,12 @@ try:
     model = Model(ws, args.model_name)
     last_train_time = model.created_time
     print("Model was last trained on {0}.".format(last_train_time))
-except Exception:
+except Exception as e:
     print("Could not get last model train time.")
     last_train_time = datetime.min.replace(tzinfo=pytz.UTC)
 
 train_ds = Dataset.get_by_name(ws, args.ds_name)
-dataset_changed_time = train_ds.data_changed_time.replace(tzinfo=pytz.UTC)
+dataset_changed_time = train_ds.data_changed_time
-
-print("dataset_changed_time=" + str(dataset_changed_time))
-print("last_train_time=" + str(last_train_time))
 
 if not dataset_changed_time > last_train_time:
     print("Cancelling run since there is no new data.")

how-to-use-azureml/automated-machine-learning/continuous-retraining/upload_weather_data.py

@@ -120,13 +120,9 @@ except Exception:
     end_time = datetime(2021, 5, 1, 0, 0)
     end_time_last_slice = end_time - relativedelta(weeks=2)
 
-try:
+train_df = get_noaa_data(end_time_last_slice, end_time)
-    train_df = get_noaa_data(end_time_last_slice, end_time)
-except Exception as ex:
-    print("get_noaa_data failed:", ex)
-    train_df = None
 
-if train_df is not None and train_df.size > 0:
+if train_df.size > 0:
     print(
         "Received {0} rows of new data after {1}.".format(
             train_df.shape[0], end_time_last_slice

how-to-use-azureml/automated-machine-learning/experimental/README.md

@@ -9,7 +9,7 @@ To run these notebook on your own notebook server, use these installation instru
 The instructions below will install everything you need and then start a Jupyter notebook.
 If you would like to use a lighter-weight version of the client that does not install all of the machine learning libraries locally, you can leverage the [experimental notebooks.](experimental/README.md)
 
-### 1. Install mini-conda from [here](https://conda.io/miniconda.html), choose 64-bit Python 3.8 or higher.
+### 1. Install mini-conda from [here](https://conda.io/miniconda.html), choose 64-bit Python 3.7 or higher.
 - **Note**: if you already have conda installed, you can keep using it but it should be version 4.4.10 or later (as shown by: conda -V).  If you have a previous version installed, you can update it using the command: conda update conda.
 There's no need to install mini-conda specifically.
 

how-to-use-azureml/automated-machine-learning/experimental/autofeaturization-codegen/codegen-for-autofeaturization.ipynb

@@ -1,346 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/custom-model-training-from-autofeaturization-run.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Automated Machine Learning - Codegen for AutoFeaturization \n",
-        "_**Autofeaturization of credit card fraudulent transactions dataset on remote compute and codegen functionality**_\n",
-        "\n",
-        "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Setup](#Setup)\n",
-        "1. [Data](#Data)\n",
-        "1. [Autofeaturization](#Autofeaturization)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Introduction'></a>\n",
-        "## Introduction"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "**Autofeaturization** lets you run an AutoML experiment to only featurize the datasets. These datasets along with the transformer are stored in AML Storage and linked to the run which can later be retrieved and used to train models. \n",
-        "\n",
-        "**To run Autofeaturization, set the number of iterations to zero and featurization as auto.**\n",
-        "\n",
-        "Please refer to [Autofeaturization and custom model training](../autofeaturization-custom-model-training/custom-model-training-from-autofeaturization-run.ipynb) for more details on the same.\n",
-        "\n",
-        "[Codegen](https://github.com/Azure/automl-codegen-preview) is a feature, which when enabled, provides a user with the script of the underlying functionality and a notebook to tweak inputs or code and rerun the same.\n",
-        "\n",
-        "In this example we use the credit card fraudulent transactions dataset to showcase how you can use AutoML for autofeaturization and further how you can enable the `Codegen` feature.\n",
-        "\n",
-        "This notebook is using remote compute to complete the featurization.\n",
-        "\n",
-        "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../configuration.ipynb) notebook first if you haven't already, to establish your connection to the AzureML Workspace. \n",
-        "\n",
-        "Here you will learn how to create an autofeaturization experiment using an existing workspace with codegen feature enabled."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Setup'></a>\n",
-        "## Setup\n",
-        "\n",
-        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import logging\n",
-        "import pandas as pd\n",
-        "import azureml.core\n",
-        "from azureml.core.experiment import Experiment\n",
-        "from azureml.core.workspace import Workspace\n",
-        "from azureml.core.dataset import Dataset\n",
-        "from azureml.train.automl import AutoMLConfig"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print(\"This notebook was created using version 1.59.0 of the Azure ML SDK\")\n",
-        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "\n",
-        "# choose a name for experiment\n",
-        "experiment_name = 'automl-autofeaturization-ccard-codegen-remote'\n",
-        "\n",
-        "experiment=Experiment(ws, experiment_name)\n",
-        "\n",
-        "output = {}\n",
-        "output['Subscription ID'] = ws.subscription_id\n",
-        "output['Workspace'] = ws.name\n",
-        "output['Resource Group'] = ws.resource_group\n",
-        "output['Location'] = ws.location\n",
-        "output['Experiment Name'] = experiment.name\n",
-        "outputDf = pd.DataFrame(data = output, index = [''])\n",
-        "outputDf.T"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Create or Attach existing AmlCompute\n",
-        "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
-        "\n",
-        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
-        "\n",
-        "#### Creation of AmlCompute takes approximately 5 minutes. \n",
-        "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpu-codegen\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
-        "    print('Found existing cluster, use it.')\n",
-        "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
-        "                                                           max_nodes=6)\n",
-        "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
-        "\n",
-        "compute_target.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Data'></a>\n",
-        "## Data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load Data\n",
-        "\n",
-        "Load the credit card fraudulent transactions dataset from a CSV file, containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. \n",
-        "\n",
-        "Here the autofeaturization run will featurize the training data passed in."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "##### Training Dataset"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard_train.csv\"\n",
-        "training_dataset = Dataset.Tabular.from_delimited_files(training_data) # Tabular dataset\n",
-        "\n",
-        "label_column_name = 'Class' # output label"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Autofeaturization'></a>\n",
-        "## AutoFeaturization\n",
-        "\n",
-        "Instantiate an AutoMLConfig object. This defines the settings and data used to run the autofeaturization experiment.\n",
-        "\n",
-        "|Property|Description|\n",
-        "|-|-|\n",
-        "|**task**|classification or regression or forecasting|\n",
-        "|**training_data**|Input training dataset, containing both features and label column.|\n",
-        "|**iterations**|For an autofeaturization run, iterations will be 0.|\n",
-        "|**featurization**|For an autofeaturization run, featurization can be 'auto' or 'custom'.|\n",
-        "|**label_column_name**|The name of the label column.|\n",
-        "|**enable_code_generation**|For enabling codegen for the run, value would be True|\n",
-        "\n",
-        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "automl_config = AutoMLConfig(task = 'classification',\n",
-        "                             debug_log = 'automl_errors.log',\n",
-        "                             iterations = 0, # autofeaturization run can be triggered by setting iterations to 0\n",
-        "                             compute_target = compute_target,\n",
-        "                             training_data = training_dataset,\n",
-        "                             label_column_name = label_column_name,\n",
-        "                             featurization = 'auto',\n",
-        "                             verbosity = logging.INFO,\n",
-        "                             enable_code_generation = True # enable codegen\n",
-        "                            )"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run = experiment.submit(automl_config, show_output = False)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Results"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Widget for Monitoring Runs\n",
-        "\n",
-        "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
-        "\n",
-        "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(remote_run).show()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run.wait_for_completion(show_output=False)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Codegen Script and Notebook"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Codegen script and notebook can be found under the `Outputs + logs` section from the details page of the remote run. Please check for the `autofeaturization_notebook.ipynb` under `/outputs/generated_code`. To modify the featurization code, open `script.py` and make changes. The codegen notebook can be run with the same environment configuration as the above AutoML run."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Experiment Complete!"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "bhavanatumma"
-      }
-    ],
-    "interpreter": {
-      "hash": "adb464b67752e4577e3dc163235ced27038d19b7d88def00d75d1975bde5d9ab"
-    },
-    "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
-      "language": "python",
-      "name": "python38-azureml"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.9"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}

how-to-use-azureml/automated-machine-learning/experimental/autofeaturization-custom-model-training/custom-model-training-from-autofeaturization-run.ipynb

@@ -1,729 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/custom-model-training-from-autofeaturization-run.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Automated Machine Learning - AutoFeaturization (Part 1)\n",
-        "_**Autofeaturization of credit card fraudulent transactions dataset on remote compute**_\n",
-        "\n",
-        "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Setup](#Setup)\n",
-        "1. [Data](#Data)\n",
-        "1. [Autofeaturization](#Autofeaturization)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Introduction'></a>\n",
-        "## Introduction"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Autofeaturization is a new feature to let you as the user run an AutoML experiment to only featurize the datasets. These datasets along with the transformer will be stored in the experiment which can later be retrieved and used to train models, either via AutoML or custom training. \n",
-        "\n",
-        "**To run Autofeaturization, pass in zero iterations and featurization as auto. This will featurize the datasets and terminate the experiment. Training will not occur.**\n",
-        "\n",
-        "*Limitations - Sparse data cannot be supported at the moment. Any dataset that has extensive categorical data might be featurized into sparse data which will not be allowed as input to AutoML. Efforts are underway to support sparse data and will be updated soon.* \n",
-        "\n",
-        "In this example we use the credit card fraudulent transactions dataset to showcase how you can use AutoML for autofeaturization. The goal is to clean and featurize the training dataset.\n",
-        "\n",
-        "This notebook is using remote compute to complete the featurization.\n",
-        "\n",
-        "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../configuration.ipynb) notebook first if you haven't already, to establish your connection to the AzureML Workspace. \n",
-        "\n",
-        "In the below steps, you will learn how to:\n",
-        "1. Create an autofeaturization experiment using an existing workspace.\n",
-        "2. View the featurized datasets and transformer"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Setup'></a>\n",
-        "## Setup\n",
-        "\n",
-        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import logging\n",
-        "import pandas as pd\n",
-        "import azureml.core\n",
-        "from azureml.core.experiment import Experiment\n",
-        "from azureml.core.workspace import Workspace\n",
-        "from azureml.core.dataset import Dataset\n",
-        "from azureml.train.automl import AutoMLConfig"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print(\"This notebook was created using version 1.59.0 of the Azure ML SDK\")\n",
-        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "\n",
-        "# choose a name for experiment\n",
-        "experiment_name = 'automl-autofeaturization-ccard-remote'\n",
-        "\n",
-        "experiment=Experiment(ws, experiment_name)\n",
-        "\n",
-        "output = {}\n",
-        "output['Subscription ID'] = ws.subscription_id\n",
-        "output['Workspace'] = ws.name\n",
-        "output['Resource Group'] = ws.resource_group\n",
-        "output['Location'] = ws.location\n",
-        "output['Experiment Name'] = experiment.name\n",
-        "outputDf = pd.DataFrame(data = output, index = [''])\n",
-        "outputDf.T"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Create or Attach existing AmlCompute\n",
-        "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
-        "\n",
-        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
-        "\n",
-        "#### Creation of AmlCompute takes approximately 5 minutes. \n",
-        "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your CPU cluster\n",
-        "cpu_cluster_name = \"cpu-cluster\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
-        "    print('Found existing cluster, use it.')\n",
-        "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
-        "                                                           max_nodes=6)\n",
-        "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
-        "\n",
-        "compute_target.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Data'></a>\n",
-        "## Data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load Data\n",
-        "\n",
-        "Load the credit card fraudulent transactions dataset from a CSV file, containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. \n",
-        "\n",
-        "Here the autofeaturization run will featurize the training data passed in."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "##### Training Dataset"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard_train.csv\"\n",
-        "training_dataset = Dataset.Tabular.from_delimited_files(training_data) # Tabular dataset\n",
-        "\n",
-        "label_column_name = 'Class' # output label"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Autofeaturization'></a>\n",
-        "## AutoFeaturization\n",
-        "\n",
-        "Instantiate an AutoMLConfig object. This defines the settings and data used to run the autofeaturization experiment.\n",
-        "\n",
-        "|Property|Description|\n",
-        "|-|-|\n",
-        "|**task**|classification or regression|\n",
-        "|**training_data**|Input training dataset, containing both features and label column.|\n",
-        "|**iterations**|For an autofeaturization run, iterations will be 0.|\n",
-        "|**featurization**|For an autofeaturization run, featurization will be 'auto'.|\n",
-        "|**label_column_name**|The name of the label column.|\n",
-        "\n",
-        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "automl_config = AutoMLConfig(task = 'classification',\n",
-        "                             debug_log = 'automl_errors.log',\n",
-        "                             iterations = 0, # autofeaturization run can be triggered by setting iterations to 0\n",
-        "                             compute_target = compute_target,\n",
-        "                             training_data = training_dataset,\n",
-        "                             label_column_name = label_column_name,\n",
-        "                             featurization = 'auto',\n",
-        "                             verbosity = logging.INFO\n",
-        "                            )"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run = experiment.submit(automl_config, show_output = False)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Transformer and Featurized Datasets\n",
-        "The given datasets have been featurized and stored under `Outputs + logs` from the details page of the remote run. The structure is shown below. The featurized dataset is stored under `/outputs/featurization/data` and the transformer is saved under `/outputs/featurization/pipeline` \n",
-        "\n",
-        "Below you will learn how to refer to the data saved in your run and retrieve the same."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Featurized Data](https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/autofeaturization_img.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Results"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Widget for Monitoring Runs\n",
-        "\n",
-        "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
-        "\n",
-        "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.widgets import RunDetails\n",
-        "RunDetails(remote_run).show()"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "remote_run.wait_for_completion(show_output=False)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Automated Machine Learning - AutoFeaturization (Part 2)\n",
-        "_**Training using a custom model with the featurized data from Autofeaturization run of credit card fraudulent transactions dataset**_\n",
-        "\n",
-        "## Contents\n",
-        "1. [Introduction](#Introduction)\n",
-        "1. [Data Setup](#DataSetup)\n",
-        "1. [Autofeaturization Data](#AutofeaturizationData)\n",
-        "1. [Train](#Train)\n",
-        "1. [Results](#Results)\n",
-        "1. [Test](#Test)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Introduction'></a>\n",
-        "## Introduction\n",
-        "\n",
-        "Here we use the featurized dataset saved in the above run to showcase how you can perform custom training by using the transformer from an autofeaturization run to transform validation / test datasets. \n",
-        "\n",
-        "The goal is to use autofeaturized run data and transformer to transform and run a custom training experiment independently\n",
-        "\n",
-        "In the below steps, you will learn how to:\n",
-        "1. Read transformer from a completed autofeaturization run and transform data\n",
-        "2. Pull featurized data from a completed autofeaturization run\n",
-        "3. Run a custom training experiment with the above data\n",
-        "4. Check results"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='DataSetup'></a>\n",
-        "## Data Setup"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "We will load the featurized training data and also load the transformer from the above autofeaturized run. This transformer can then be used to transform the test data to check the accuracy of the custom model after training."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load Test Data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "load test dataset from CSV and split into X and y columns to featurize with the transformer going forward."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "test_data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard_test.csv\"\n",
-        "\n",
-        "test_dataset = pd.read_csv(test_data)\n",
-        "label_column_name = 'Class'\n",
-        "\n",
-        "X_test_data = test_dataset[test_dataset.columns.difference([label_column_name])]\n",
-        "y_test_data = test_dataset[label_column_name].values\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load data_transformer from the above remote run artifact"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### (Method 1)\n",
-        "\n",
-        "Method 1 allows you to read the transformer from the remote storage."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import mlflow\n",
-        "mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())\n",
-        "\n",
-        "# Set uri to fetch data transformer from remote parent run.\n",
-        "artifact_path = \"/outputs/featurization/pipeline/\"\n",
-        "uri = \"runs:/\" + remote_run.id + artifact_path\n",
-        "\n",
-        "print(uri)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### (Method 2)\n",
-        "\n",
-        "Method 2 downloads the transformer to the local directory and then can be used to transform the data. Uncomment to use."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "''' import pathlib\n",
-        "\n",
-        "# Download the transformer to the local directory\n",
-        "transformers_file_path = \"/outputs/featurization/pipeline/\"\n",
-        "local_path = \"./transformer\"\n",
-        "remote_run.download_files(prefix=transformers_file_path, output_directory=local_path, batch_size=500)\n",
-        "\n",
-        "path = pathlib.Path(\"transformer\") \n",
-        "path = str(path.absolute()) + transformers_file_path\n",
-        "str_uri = \"file:///\" + path\n",
-        "\n",
-        "print(str_uri) '''"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Transform Data"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "**Note:** Not all datasets produce a y_transformer. The dataset used in the current notebook requires a transformer as the y column data is categorical. \n",
-        "\n",
-        "We will go ahead and download the mlflow transformer model and use it to transform test data that can be used for further experimentation below. To run the commented code, make sure the environment requirement is satisfied. You can go ahead and create the environment from the `conda.yaml` file under `/outputs/featurization/pipeline/` and run the given code in it."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "''' from azureml.automl.core.shared.constants import Transformers\n",
-        "\n",
-        "transformers = mlflow.sklearn.load_model(uri) # Using method 1\n",
-        "data_transformers = transformers.get_transformers()\n",
-        "x_transformer = data_transformers[Transformers.X_TRANSFORMER]\n",
-        "y_transformer = data_transformers[Transformers.Y_TRANSFORMER]\n",
-        "\n",
-        "X_test = x_transformer.transform(X_test_data)\n",
-        "y_test = y_transformer.transform(y_test_data) '''"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Run the following cell to see the featurization summary of X and y transformers. Uncomment to use.  "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "''' X_data_summary = x_transformer.get_featurization_summary(is_user_friendly=False)\n",
-        "\n",
-        "summary_df = pd.DataFrame.from_records(X_data_summary)\n",
-        "summary_df '''"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Load Datastore\n",
-        "\n",
-        "The below data store holds the featurized datasets, hence we load and access the data. Check the path and file names according to the saved structure in your experiment `Outputs + logs` as seen in <i>Autofeaturization Part 1</i>"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.datastore import Datastore\n",
-        "\n",
-        "ds = Datastore.get(ws, \"workspaceartifactstore\")\n",
-        "experiment_loc = \"ExperimentRun/dcid.\" + remote_run.id\n",
-        "\n",
-        "remote_data_path = \"/outputs/featurization/data/\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='AutofeaturizationData'></a>\n",
-        "## Autofeaturization Data\n",
-        "\n",
-        "We will load the training data from the previously completed Autofeaturization experiment. The resulting featurized dataframe can be passed into the custom model for training. Here we are saving the file to local from the experiment storage and reading the data."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "train_data_file_path = \"full_training_dataset.df.parquet\"\n",
-        "local_data_path = \"./data/\" + train_data_file_path\n",
-        "\n",
-        "remote_run.download_file(remote_data_path + train_data_file_path, local_data_path)\n",
-        "\n",
-        "full_training_data = pd.read_parquet(local_data_path)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Another way to load the data is to go to the above autofeaturization experiment and check for the featurized dataset ids under `Output datasets`. Uncomment and replace them accordingly below, to use."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# train_data = Dataset.get_by_id(ws, 'cb4418ee-bac4-45ac-b055-600653bdf83a') # replace the featurized full_training_dataset id\n",
-        "# full_training_data = train_data.to_pandas_dataframe()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Training Data"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "We are dropping the y column and weights column from the featurized training dataset."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "Y_COLUMN = \"automl_y\"\n",
-        "SW_COLUMN = \"automl_weights\"\n",
-        "\n",
-        "X_train = full_training_data[full_training_data.columns.difference([Y_COLUMN, SW_COLUMN])]\n",
-        "y_train = full_training_data[Y_COLUMN].values\n",
-        "sample_weight = full_training_data[SW_COLUMN].values"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Train'></a>\n",
-        "## Train"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Here we are passing our training data to the lightgbm classifier, any custom model can be used with your data. Let us first install lightgbm."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "! pip install lightgbm"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import lightgbm as lgb\n",
-        "\n",
-        "model = lgb.LGBMClassifier(learning_rate=0.08,max_depth=-5,random_state=42)\n",
-        "model.fit(X_train, y_train, sample_weight=sample_weight)"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Once training is done, the test data obtained after transforming from the above downloaded transformer can be used to calculate the accuracy "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print('Training accuracy {:.4f}'.format(model.score(X_train, y_train)))\n",
-        "\n",
-        "# Uncomment below to test the model on test data \n",
-        "# print('Testing accuracy {:.4f}'.format(model.score(X_test, y_test)))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Results'></a>\n",
-        "## Analyze results\n",
-        "\n",
-        "### Retrieve the Model"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "model"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<a id='Test'></a>\n",
-        "## Test the fitted model\n",
-        "\n",
-        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Uncomment below to test the model on test data\n",
-        "# y_pred = model.predict(X_test)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Experiment Complete!"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "bhavanatumma"
-      }
-    ],
-    "interpreter": {
-      "hash": "adb464b67752e4577e3dc163235ced27038d19b7d88def00d75d1975bde5d9ab"
-    },
-    "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
-      "language": "python",
-      "name": "python38-azureml"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.6.9"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 2
-}

how-to-use-azureml/automated-machine-learning/experimental/automl_thin_client_env.yml

@@ -1,15 +1,17 @@
 name: azure_automl_experimental
 dependencies:
   # The python interpreter version.
-  # Currently Azure ML only supports 3.7.0 and later.
+  # Currently Azure ML only supports 3.5.2 and later.
-- pip<=22.3.1
+- pip<=19.3.1
-- python>=3.7.0,<3.11
+- python>=3.5.2,<3.8
+- cython
+- urllib3<1.24
+- PyJWT < 2.0.0
+- numpy==1.18.5
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
   - azureml-defaults
   - azureml-sdk
   - azureml-widgets
-  - azureml-mlflow
   - pandas
-  - mlflow

how-to-use-azureml/automated-machine-learning/experimental/automl_thin_client_env_mac.yml

@@ -1,24 +1,18 @@
 name: azure_automl_experimental
-channels:
-  - conda-forge
-  - main
 dependencies:
   # The python interpreter version.
-  # Currently Azure ML only supports 3.7.0 and later.
+  # Currently Azure ML only supports 3.5.2 and later.
-- pip<=20.2.4
+- pip<=19.3.1
 - nomkl
-- python>=3.7.0,<3.11
+- python>=3.5.2,<3.8
-- urllib3==1.26.7
+- cython
+- urllib3<1.24
 - PyJWT < 2.0.0
-- numpy>=1.21.6,<=1.22.3
+- numpy==1.18.5
 
 - pip:
   # Required packages for AzureML execution, history, and data preparation.
-  - azure-core==1.24.1
-  - azure-identity==1.7.0
   - azureml-defaults
   - azureml-sdk
   - azureml-widgets
-  - azureml-mlflow
   - pandas
-  - mlflow

how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud-local-managed/auto-ml-classification-credit-card-fraud-local-managed.ipynb

@@ -0,0 +1,420 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Automated Machine Learning\n",
+        "_**Classification of credit card fraudulent transactions on local managed compute **_\n",
+        "\n",
+        "## Contents\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Setup](#Setup)\n",
+        "1. [Train](#Train)\n",
+        "1. [Results](#Results)\n",
+        "1. [Test](#Test)\n",
+        "1. [Acknowledgements](#Acknowledgements)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Introduction\n",
+        "\n",
+        "In this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.\n",
+        "\n",
+        "This notebook is using local managed compute to train the model.\n",
+        "\n",
+        "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../../../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
+        "\n",
+        "In this notebook you will learn how to:\n",
+        "1. Create an experiment using an existing workspace.\n",
+        "2. Configure AutoML using `AutoMLConfig`.\n",
+        "3. Train the model using local managed compute.\n",
+        "4. Explore the results.\n",
+        "5. Test the fitted model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Setup\n",
+        "\n",
+        "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import logging\n",
+        "\n",
+        "import pandas as pd\n",
+        "\n",
+        "import azureml.core\n",
+        "from azureml.core.compute_target import LocalTarget\n",
+        "from azureml.core.experiment import Experiment\n",
+        "from azureml.core.workspace import Workspace\n",
+        "from azureml.core.dataset import Dataset\n",
+        "from azureml.train.automl import AutoMLConfig"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(\"This notebook was created using version 1.38.0 of the Azure ML SDK\")\n",
+        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "\n",
+        "# choose a name for experiment\n",
+        "experiment_name = 'automl-local-managed'\n",
+        "\n",
+        "experiment=Experiment(ws, experiment_name)\n",
+        "\n",
+        "output = {}\n",
+        "output['Subscription ID'] = ws.subscription_id\n",
+        "output['Workspace'] = ws.name\n",
+        "output['Resource Group'] = ws.resource_group\n",
+        "output['Location'] = ws.location\n",
+        "output['Experiment Name'] = experiment.name\n",
+        "pd.set_option('display.max_colwidth', -1)\n",
+        "outputDf = pd.DataFrame(data = output, index = [''])\n",
+        "outputDf.T"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Determine if local docker is configured for Linux images\n",
+        "\n",
+        "Local managed runs will leverage a Linux docker container to submit the run to. Due to this, the docker needs to be configured to use Linux containers."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# Check if Docker is installed and Linux containers are enabled\n",
+        "import subprocess\n",
+        "from subprocess import CalledProcessError\n",
+        "try:\n",
+        "    assert subprocess.run(\"docker -v\", shell=True).returncode == 0, 'Local Managed runs require docker to be installed.'\n",
+        "    out = subprocess.check_output(\"docker system info\", shell=True).decode('ascii')\n",
+        "    assert \"OSType: linux\" in out, 'Docker engine needs to be configured to use Linux containers.' \\\n",
+        "        'https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers'\n",
+        "except CalledProcessError as ex:\n",
+        "    raise Exception('Local Managed runs require docker to be installed.') from ex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Load Data\n",
+        "\n",
+        "Load the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv\"\n",
+        "dataset = Dataset.Tabular.from_delimited_files(data)\n",
+        "training_data, validation_data = dataset.random_split(percentage=0.8, seed=223)\n",
+        "label_column_name = 'Class'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train\n",
+        "\n",
+        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**task**|classification or regression|\n",
+        "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
+        "|**enable_early_stopping**|Stop the run if the metric score is not showing improvement.|\n",
+        "|**n_cross_validations**|Number of cross validation splits.|\n",
+        "|**training_data**|Input dataset, containing both features and label column.|\n",
+        "|**label_column_name**|The name of the label column.|\n",
+        "|**enable_local_managed**|Enable the experimental local-managed scenario.|\n",
+        "\n",
+        "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "automl_settings = {\n",
+        "    \"n_cross_validations\": 3,\n",
+        "    \"primary_metric\": 'average_precision_score_weighted',\n",
+        "    \"enable_early_stopping\": True,\n",
+        "    \"experiment_timeout_hours\": 0.3, #for real scenarios we recommend a timeout of at least one hour \n",
+        "    \"verbosity\": logging.INFO,\n",
+        "}\n",
+        "\n",
+        "automl_config = AutoMLConfig(task = 'classification',\n",
+        "                             debug_log = 'automl_errors.log',\n",
+        "                             compute_target = LocalTarget(),\n",
+        "                             enable_local_managed = True,\n",
+        "                             training_data = training_data,\n",
+        "                             label_column_name = label_column_name,\n",
+        "                             **automl_settings\n",
+        "                            )"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "parent_run = experiment.submit(automl_config, show_output = True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# If you need to retrieve a run that already started, use the following code\n",
+        "#from azureml.train.automl.run import AutoMLRun\n",
+        "#parent_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "parent_run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Results"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Explain model\n",
+        "\n",
+        "Automated ML models can be explained and visualized using the SDK Explainability library. "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Analyze results\n",
+        "\n",
+        "### Retrieve the Best Child Run\n",
+        "\n",
+        "Below we select the best pipeline from our iterations. The `get_best_child` method returns the best run. Overloads on `get_best_child` allow you to retrieve the best run for *any* logged metric."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "best_run = parent_run.get_best_child()\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Test the fitted model\n",
+        "\n",
+        "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "X_test_df = validation_data.drop_columns(columns=[label_column_name])\n",
+        "y_test_df = validation_data.keep_columns(columns=[label_column_name], validate=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### Creating ModelProxy for submitting prediction runs to the training environment.\n",
+        "We will create a ModelProxy for the best child run, which will allow us to submit a run that does the prediction in the training environment. Unlike the local client, which can have different versions of some libraries, the training environment will have all the compatible libraries for the model already."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from azureml.train.automl.model_proxy import ModelProxy\n",
+        "best_model_proxy = ModelProxy(best_run)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# call the predict functions on the model proxy\n",
+        "y_pred = best_model_proxy.predict(X_test_df).to_pandas_dataframe()\n",
+        "y_pred"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Acknowledgements"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
+        "\n",
+        "\n",
+        "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Universit\u00c3\u0192\u00c2\u00a9 Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
+        "Please cite the following works: \n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tAndrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tDal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
+        "o\tDal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
+        "\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00a2\tCarcillo, Fabrizio; Le Borgne, Yann-A\u00c3\u0192\u00c2\u00abl; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing"
+      ]
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "sekrupa"
+      }
+    ],
+    "category": "tutorial",
+    "compute": [
+      "AML Compute"
+    ],
+    "datasets": [
+      "Creditcard"
+    ],
+    "deployment": [
+      "None"
+    ],
+    "exclude_from_index": false,
+    "file_extension": ".py",
+    "framework": [
+      "None"
+    ],
+    "friendly_name": "Classification of credit card fraudulent transactions using Automated ML",
+    "index_order": 5,
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.7"
+    },
+    "mimetype": "text/x-python",
+    "name": "python",
+    "nbconvert_exporter": "python",
+    "pygments_lexer": "ipython3",
+    "tags": [
+      "AutomatedML"
+    ],
+    "task": "Classification",
+    "version": "3.6.7"
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/experimental/classification-credit-card-fraud-local-managed/auto-ml-classification-credit-card-fraud-local-managed.yml

@@ -0,0 +1,4 @@
+name: auto-ml-classification-credit-card-fraud-local-managed
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/experimental/regression-model-proxy/auto-ml-regression-model-proxy.ipynb

@@ -91,7 +91,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "print(\"This notebook was created using version 1.59.0 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.38.0 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },
@@ -180,29 +180,6 @@
         "label = \"ERP\"\n"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "The split data will be used in the remote compute by ModelProxy and locally to compare results.\n",
-        "So, we need to persist the split data to avoid descrepencies from different package versions in the local and remote."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ds = ws.get_default_datastore()\n",
-        "\n",
-        "train_data = Dataset.Tabular.register_pandas_dataframe(\n",
-        "    train_data.to_pandas_dataframe(), target=(ds, \"machineTrainData\"), name=\"train_data\")\n",
-        "\n",
-        "test_data = Dataset.Tabular.register_pandas_dataframe(\n",
-        "    test_data.to_pandas_dataframe(), target=(ds, \"machineTestData\"), name=\"test_data\")"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -327,8 +304,7 @@
       "metadata": {},
       "source": [
         "#### Show hyperparameters\n",
-        "Show the model pipeline used for the best run with its hyperparameters.\n",
+        "Show the model pipeline used for the best run with its hyperparameters."
-        "For ensemble pipelines it shows the iterations and algorithms that are ensembled."
       ]
     },
     {
@@ -337,19 +313,8 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "run_properties = best_run.get_details()['properties']\n",
+        "run_properties = json.loads(best_run.get_details()['properties']['pipeline_script'])\n",
-        "pipeline_script = json.loads(run_properties['pipeline_script'])\n",
+        "print(json.dumps(run_properties, indent = 1)) "
-        "print(json.dumps(pipeline_script, indent = 1)) \n",
-        "\n",
-        "if 'ensembled_iterations' in run_properties:\n",
-        "    print(\"\")\n",
-        "    print(\"Ensembled Iterations\")\n",
-        "    print(run_properties['ensembled_iterations'])\n",
-        "    \n",
-        "if 'ensembled_algorithms' in run_properties:\n",
-        "    print(\"\")\n",
-        "    print(\"Ensembled Algorithms\")\n",
-        "    print(run_properties['ensembled_algorithms'])"
       ]
     },
     {
@@ -448,9 +413,9 @@
       "automated-machine-learning"
     ],
     "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
+      "display_name": "Python 3.6",
       "language": "python",
-      "name": "python38-azureml"
+      "name": "python36"
     },
     "language_info": {
       "codemirror_mode": {

how-to-use-azureml/automated-machine-learning/experimental/regression-model-proxy/auto-ml-regression-model-proxy.yml

@@ -0,0 +1,4 @@
+name: auto-ml-regression-model-proxy
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-backtest-many-models/assets/score.py

@@ -5,7 +5,6 @@ import json
 import os
 import re
 
-import numpy as np
 import pandas as pd
 
 from matplotlib import pyplot as plt
@@ -122,10 +121,7 @@ def calculate_scores_and_build_plots(
     input_dir: str, output_dir: str, automl_settings: Dict[str, Any]
 ):
     os.makedirs(output_dir, exist_ok=True)
-    grains = automl_settings.get(
+    grains = automl_settings.get(constants.TimeSeries.GRAIN_COLUMN_NAMES)
-        constants.TimeSeries.TIME_SERIES_ID_COLUMN_NAMES,
-        automl_settings.get(constants.TimeSeries.GRAIN_COLUMN_NAMES, None),
-    )
     time_column_name = automl_settings.get(constants.TimeSeries.TIME_COLUMN_NAME)
     if grains is None:
         grains = []
@@ -150,9 +146,6 @@ def calculate_scores_and_build_plots(
         _draw_one_plot(one_forecast, time_column_name, grains, pdf)
     pdf.close()
     forecast_df.to_csv(os.path.join(output_dir, FORECASTS_FILE), index=False)
-    # Remove np.NaN and np.inf from the prediction and actuals data.
-    forecast_df.replace([np.inf, -np.inf], np.nan, inplace=True)
-    forecast_df.dropna(subset=[ACTUALS, PREDICTIONS], inplace=True)
     metrics = compute_all_metrics(forecast_df, grains + [BACKTEST_ITER])
     metrics.to_csv(os.path.join(output_dir, SCORES_FILE), index=False)
 

how-to-use-azureml/automated-machine-learning/forecasting-backtest-many-models/auto-ml-forecasting-backtest-many-models.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-backtest-many-models
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-backtest-many-models/update_env.yml

@@ -0,0 +1,3 @@
+dependencies:
+- pip:
+  - azureml-contrib-automl-pipeline-steps

how-to-use-azureml/automated-machine-learning/forecasting-backtest-single-model/assets/retrain_models.py

@@ -43,20 +43,11 @@ def init():
     global output_dir
     global automl_settings
     global model_uid
-    global forecast_quantiles
-
     logger.info("Initialization of the run.")
     parser = argparse.ArgumentParser("Parsing input arguments.")
     parser.add_argument("--output-dir", dest="out", required=True)
     parser.add_argument("--model-name", dest="model", default=None)
     parser.add_argument("--model-uid", dest="model_uid", default=None)
-    parser.add_argument(
-        "--forecast_quantiles",
-        nargs="*",
-        type=float,
-        help="forecast quantiles list",
-        default=None,
-    )
 
     parsed_args, _ = parser.parse_known_args()
     model_name = parsed_args.model
@@ -64,7 +55,6 @@ def init():
     target_column_name = automl_settings.get("label_column_name")
     output_dir = parsed_args.out
     model_uid = parsed_args.model_uid
-    forecast_quantiles = parsed_args.forecast_quantiles
     os.makedirs(output_dir, exist_ok=True)
     os.environ["AUTOML_IGNORE_PACKAGE_VERSION_INCOMPATIBILITIES".lower()] = "True"
 
@@ -136,18 +126,23 @@ def run_backtest(data_input_name: str, file_name: str, experiment: Experiment):
         )
         print(f"The model {best_run.properties['model_name']} was registered.")
 
-    # By default we will have forecast quantiles of 0.5, which is our target
+    _, x_pred = fitted_model.forecast(X_test)
-    if forecast_quantiles:
+    x_pred.reset_index(inplace=True, drop=False)
-        if 0.5 not in forecast_quantiles:
+    columns = [automl_settings[constants.TimeSeries.TIME_COLUMN_NAME]]
-            forecast_quantiles.append(0.5)
+    if automl_settings.get(constants.TimeSeries.GRAIN_COLUMN_NAMES):
-        fitted_model.quantiles = forecast_quantiles
+        # We know that fitted_model.grain_column_names is a list.
-
+        columns.extend(fitted_model.grain_column_names)
-    x_pred = fitted_model.forecast_quantiles(X_test)
+    columns.append(constants.TimeSeriesInternal.DUMMY_TARGET_COLUMN)
+    # Remove featurized columns.
+    x_pred = x_pred[columns]
+    x_pred.rename(
+        {constants.TimeSeriesInternal.DUMMY_TARGET_COLUMN: "predicted_level"},
+        axis=1,
+        inplace=True,
+    )
     x_pred["actual_level"] = y_test
     x_pred["backtest_iteration"] = f"iteration_{last_training_date}"
-    x_pred.rename({0.5: "predicted_level"}, axis=1, inplace=True)
     date_safe = RE_INVALID_SYMBOLS.sub("_", last_training_date)
-
     x_pred.to_csv(os.path.join(output_dir, f"iteration_{date_safe}.csv"), index=False)
     return x_pred
 

how-to-use-azureml/automated-machine-learning/forecasting-backtest-single-model/assets/score.py

@@ -5,7 +5,6 @@ import json
 import os
 import re
 
-import numpy as np
 import pandas as pd
 
 from matplotlib import pyplot as plt
@@ -147,9 +146,6 @@ def calculate_scores_and_build_plots(
         _draw_one_plot(one_forecast, time_column_name, grains, pdf)
     pdf.close()
     forecast_df.to_csv(os.path.join(output_dir, FORECASTS_FILE), index=False)
-    # Remove np.NaN and np.inf from the prediction and actuals data.
-    forecast_df.replace([np.inf, -np.inf], np.nan, inplace=True)
-    forecast_df.dropna(subset=[ACTUALS, PREDICTIONS], inplace=True)
     metrics = compute_all_metrics(forecast_df, grains + [BACKTEST_ITER])
     metrics.to_csv(os.path.join(output_dir, SCORES_FILE), index=False)
 

how-to-use-azureml/automated-machine-learning/forecasting-backtest-single-model/auto-ml-forecasting-backtest-single-model.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-backtest-single-model
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-backtest-single-model/pipeline_helper.py

@@ -31,7 +31,6 @@ def get_backtest_pipeline(
     step_number: int,
     model_name: Optional[str] = None,
     model_uid: Optional[str] = None,
-    forecast_quantiles: Optional[list] = None,
 ) -> Pipeline:
     """
     :param experiment: The experiment used to run the pipeline.
@@ -45,7 +44,6 @@ def get_backtest_pipeline(
     :param step_size: The number of periods to step back in backtesting.
     :param step_number: The number of backtesting iterations.
     :param model_uid: The uid to mark models from this run of the experiment.
-    :param forecast_quantiles: The forecast quantiles that are required in the inference.
     :return: The pipeline to be used for model retraining.
              **Note:** The output will be uploaded in the pipeline output
              called 'score'.
@@ -74,8 +72,6 @@ def get_backtest_pipeline(
     run_config.docker.use_docker = True
     run_config.environment = env
 
-    utilities.set_environment_variables_for_run(run_config)
-
     split_data = PipelineData(name="split_data_output", datastore=None).as_dataset()
     split_step = PythonScriptStep(
         name="split_data_for_backtest",
@@ -118,7 +114,6 @@ def get_backtest_pipeline(
         run_invocation_timeout=3600,
         node_count=node_count,
     )
-    utilities.set_environment_variables_for_run(back_test_config)
     forecasts = PipelineData(name="forecasts", datastore=None)
     if model_name:
         parallel_step_name = "{}-backtest".format(model_name.replace("_", "-"))
@@ -137,9 +132,6 @@ def get_backtest_pipeline(
     if model_uid is not None:
         prs_args.append("--model-uid")
         prs_args.append(model_uid)
-    if forecast_quantiles:
-        prs_args.append("--forecast_quantiles")
-        prs_args.extend(forecast_quantiles)
     backtest_prs = ParallelRunStep(
         name=parallel_step_name,
         parallel_run_config=back_test_config,
@@ -157,7 +149,12 @@ def get_backtest_pipeline(
         inputs=[forecasts.as_mount()],
         outputs=[data_results],
         source_directory=PROJECT_FOLDER,
-        arguments=["--forecasts", forecasts, "--output-dir", data_results],
+        arguments=[
+            "--forecasts",
+            forecasts,
+            "--output-dir",
+            data_results,
+        ],
         runconfig=run_config,
         compute_target=compute_target,
         allow_reuse=False,

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/Beer_no_valid_split_test.csv

@@ -0,0 +1,20 @@
+DATE,grain,BeerProduction
+2017-01-01,grain,9049
+2017-02-01,grain,10458
+2017-03-01,grain,12489
+2017-04-01,grain,11499
+2017-05-01,grain,13553
+2017-06-01,grain,14740
+2017-07-01,grain,11424
+2017-08-01,grain,13412
+2017-09-01,grain,11917
+2017-10-01,grain,12721
+2017-11-01,grain,13272
+2017-12-01,grain,14278
+2018-01-01,grain,9572
+2018-02-01,grain,10423
+2018-03-01,grain,12667
+2018-04-01,grain,11904
+2018-05-01,grain,14120
+2018-06-01,grain,14565
+2018-07-01,grain,12622

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/Beer_no_valid_split_train.csv

@@ -0,0 +1,301 @@
+DATE,grain,BeerProduction
+1992-01-01,grain,3459
+1992-02-01,grain,3458
+1992-03-01,grain,4002
+1992-04-01,grain,4564
+1992-05-01,grain,4221
+1992-06-01,grain,4529
+1992-07-01,grain,4466
+1992-08-01,grain,4137
+1992-09-01,grain,4126
+1992-10-01,grain,4259
+1992-11-01,grain,4240
+1992-12-01,grain,4936
+1993-01-01,grain,3031
+1993-02-01,grain,3261
+1993-03-01,grain,4160
+1993-04-01,grain,4377
+1993-05-01,grain,4307
+1993-06-01,grain,4696
+1993-07-01,grain,4458
+1993-08-01,grain,4457
+1993-09-01,grain,4364
+1993-10-01,grain,4236
+1993-11-01,grain,4500
+1993-12-01,grain,4974
+1994-01-01,grain,3075
+1994-02-01,grain,3377
+1994-03-01,grain,4443
+1994-04-01,grain,4261
+1994-05-01,grain,4460
+1994-06-01,grain,4985
+1994-07-01,grain,4324
+1994-08-01,grain,4719
+1994-09-01,grain,4374
+1994-10-01,grain,4248
+1994-11-01,grain,4784
+1994-12-01,grain,4971
+1995-01-01,grain,3370
+1995-02-01,grain,3484
+1995-03-01,grain,4269
+1995-04-01,grain,3994
+1995-05-01,grain,4715
+1995-06-01,grain,4974
+1995-07-01,grain,4223
+1995-08-01,grain,5000
+1995-09-01,grain,4235
+1995-10-01,grain,4554
+1995-11-01,grain,4851
+1995-12-01,grain,4826
+1996-01-01,grain,3699
+1996-02-01,grain,3983
+1996-03-01,grain,4262
+1996-04-01,grain,4619
+1996-05-01,grain,5219
+1996-06-01,grain,4836
+1996-07-01,grain,4941
+1996-08-01,grain,5062
+1996-09-01,grain,4365
+1996-10-01,grain,5012
+1996-11-01,grain,4850
+1996-12-01,grain,5097
+1997-01-01,grain,3758
+1997-02-01,grain,3825
+1997-03-01,grain,4454
+1997-04-01,grain,4635
+1997-05-01,grain,5210
+1997-06-01,grain,5057
+1997-07-01,grain,5231
+1997-08-01,grain,5034
+1997-09-01,grain,4970
+1997-10-01,grain,5342
+1997-11-01,grain,4831
+1997-12-01,grain,5965
+1998-01-01,grain,3796
+1998-02-01,grain,4019
+1998-03-01,grain,4898
+1998-04-01,grain,5090
+1998-05-01,grain,5237
+1998-06-01,grain,5447
+1998-07-01,grain,5435
+1998-08-01,grain,5107
+1998-09-01,grain,5515
+1998-10-01,grain,5583
+1998-11-01,grain,5346
+1998-12-01,grain,6286
+1999-01-01,grain,4032
+1999-02-01,grain,4435
+1999-03-01,grain,5479
+1999-04-01,grain,5483
+1999-05-01,grain,5587
+1999-06-01,grain,6176
+1999-07-01,grain,5621
+1999-08-01,grain,5889
+1999-09-01,grain,5828
+1999-10-01,grain,5849
+1999-11-01,grain,6180
+1999-12-01,grain,6771
+2000-01-01,grain,4243
+2000-02-01,grain,4952
+2000-03-01,grain,6008
+2000-04-01,grain,5353
+2000-05-01,grain,6435
+2000-06-01,grain,6673
+2000-07-01,grain,5636
+2000-08-01,grain,6630
+2000-09-01,grain,5887
+2000-10-01,grain,6322
+2000-11-01,grain,6520
+2000-12-01,grain,6678
+2001-01-01,grain,5082
+2001-02-01,grain,5216
+2001-03-01,grain,5893
+2001-04-01,grain,5894
+2001-05-01,grain,6799
+2001-06-01,grain,6667
+2001-07-01,grain,6374
+2001-08-01,grain,6840
+2001-09-01,grain,5575
+2001-10-01,grain,6545
+2001-11-01,grain,6789
+2001-12-01,grain,7180
+2002-01-01,grain,5117
+2002-02-01,grain,5442
+2002-03-01,grain,6337
+2002-04-01,grain,6525
+2002-05-01,grain,7216
+2002-06-01,grain,6761
+2002-07-01,grain,6958
+2002-08-01,grain,7070
+2002-09-01,grain,6148
+2002-10-01,grain,6924
+2002-11-01,grain,6716
+2002-12-01,grain,7975
+2003-01-01,grain,5326
+2003-02-01,grain,5609
+2003-03-01,grain,6414
+2003-04-01,grain,6741
+2003-05-01,grain,7144
+2003-06-01,grain,7133
+2003-07-01,grain,7568
+2003-08-01,grain,7266
+2003-09-01,grain,6634
+2003-10-01,grain,7626
+2003-11-01,grain,6843
+2003-12-01,grain,8540
+2004-01-01,grain,5629
+2004-02-01,grain,5898
+2004-03-01,grain,7045
+2004-04-01,grain,7094
+2004-05-01,grain,7333
+2004-06-01,grain,7918
+2004-07-01,grain,7289
+2004-08-01,grain,7396
+2004-09-01,grain,7259
+2004-10-01,grain,7268
+2004-11-01,grain,7731
+2004-12-01,grain,9058
+2005-01-01,grain,5557
+2005-02-01,grain,6237
+2005-03-01,grain,7723
+2005-04-01,grain,7262
+2005-05-01,grain,8241
+2005-06-01,grain,8757
+2005-07-01,grain,7352
+2005-08-01,grain,8496
+2005-09-01,grain,7741
+2005-10-01,grain,7710
+2005-11-01,grain,8247
+2005-12-01,grain,8902
+2006-01-01,grain,6066
+2006-02-01,grain,6590
+2006-03-01,grain,7923
+2006-04-01,grain,7335
+2006-05-01,grain,8843
+2006-06-01,grain,9327
+2006-07-01,grain,7792
+2006-08-01,grain,9156
+2006-09-01,grain,8037
+2006-10-01,grain,8640
+2006-11-01,grain,9128
+2006-12-01,grain,9545
+2007-01-01,grain,6627
+2007-02-01,grain,6743
+2007-03-01,grain,8195
+2007-04-01,grain,7828
+2007-05-01,grain,9570
+2007-06-01,grain,9484
+2007-07-01,grain,8608
+2007-08-01,grain,9543
+2007-09-01,grain,8123
+2007-10-01,grain,9649
+2007-11-01,grain,9390
+2007-12-01,grain,10065
+2008-01-01,grain,7093
+2008-02-01,grain,7483
+2008-03-01,grain,8365
+2008-04-01,grain,8895
+2008-05-01,grain,9794
+2008-06-01,grain,9977
+2008-07-01,grain,9553
+2008-08-01,grain,9375
+2008-09-01,grain,9225
+2008-10-01,grain,9948
+2008-11-01,grain,8758
+2008-12-01,grain,10839
+2009-01-01,grain,7266
+2009-02-01,grain,7578
+2009-03-01,grain,8688
+2009-04-01,grain,9162
+2009-05-01,grain,9369
+2009-06-01,grain,10167
+2009-07-01,grain,9507
+2009-08-01,grain,8923
+2009-09-01,grain,9272
+2009-10-01,grain,9075
+2009-11-01,grain,8949
+2009-12-01,grain,10843
+2010-01-01,grain,6558
+2010-02-01,grain,7481
+2010-03-01,grain,9475
+2010-04-01,grain,9424
+2010-05-01,grain,9351
+2010-06-01,grain,10552
+2010-07-01,grain,9077
+2010-08-01,grain,9273
+2010-09-01,grain,9420
+2010-10-01,grain,9413
+2010-11-01,grain,9866
+2010-12-01,grain,11455
+2011-01-01,grain,6901
+2011-02-01,grain,8014
+2011-03-01,grain,9832
+2011-04-01,grain,9281
+2011-05-01,grain,9967
+2011-06-01,grain,11344
+2011-07-01,grain,9106
+2011-08-01,grain,10469
+2011-09-01,grain,10085
+2011-10-01,grain,9612
+2011-11-01,grain,10328
+2011-12-01,grain,11483
+2012-01-01,grain,7486
+2012-02-01,grain,8641
+2012-03-01,grain,9709
+2012-04-01,grain,9423
+2012-05-01,grain,11342
+2012-06-01,grain,11274
+2012-07-01,grain,9845
+2012-08-01,grain,11163
+2012-09-01,grain,9532
+2012-10-01,grain,10754
+2012-11-01,grain,10953
+2012-12-01,grain,11922
+2013-01-01,grain,8395
+2013-02-01,grain,8888
+2013-03-01,grain,10110
+2013-04-01,grain,10493
+2013-05-01,grain,12218
+2013-06-01,grain,11385
+2013-07-01,grain,11186
+2013-08-01,grain,11462
+2013-09-01,grain,10494
+2013-10-01,grain,11540
+2013-11-01,grain,11138
+2013-12-01,grain,12709
+2014-01-01,grain,8557
+2014-02-01,grain,9059
+2014-03-01,grain,10055
+2014-04-01,grain,10977
+2014-05-01,grain,11792
+2014-06-01,grain,11904
+2014-07-01,grain,10965
+2014-08-01,grain,10981
+2014-09-01,grain,10828
+2014-10-01,grain,11817
+2014-11-01,grain,10470
+2014-12-01,grain,13310
+2015-01-01,grain,8400
+2015-02-01,grain,9062
+2015-03-01,grain,10722
+2015-04-01,grain,11107
+2015-05-01,grain,11508
+2015-06-01,grain,12904
+2015-07-01,grain,11869
+2015-08-01,grain,11224
+2015-09-01,grain,12022
+2015-10-01,grain,11983
+2015-11-01,grain,11506
+2015-12-01,grain,14183
+2016-01-01,grain,8650
+2016-02-01,grain,10323
+2016-03-01,grain,12110
+2016-04-01,grain,11424
+2016-05-01,grain,12243
+2016-06-01,grain,13686
+2016-07-01,grain,10956
+2016-08-01,grain,12706
+2016-09-01,grain,12279
+2016-10-01,grain,11914
+2016-11-01,grain,13025
+2016-12-01,grain,14431

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.ipynb

@@ -0,0 +1,728 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+        "\n",
+        "Licensed under the MIT License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.png)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "# Automated Machine Learning\n",
+        "**Beer Production Forecasting**\n",
+        "\n",
+        "## Contents\n",
+        "1. [Introduction](#Introduction)\n",
+        "1. [Setup](#Setup)\n",
+        "1. [Data](#Data)\n",
+        "1. [Train](#Train)\n",
+        "1. [Evaluate](#Evaluate)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "## Introduction\n",
+        "This notebook demonstrates demand forecasting for Beer Production Dataset using AutoML.\n",
+        "\n",
+        "AutoML highlights here include using Deep Learning forecasts, Arima, Prophet,  Remote Execution and Remote Inferencing, and working with the `forecast` function. Please also look at the additional forecasting notebooks, which document lagging, rolling windows, forecast quantiles, other ways to use the forecast function, and forecaster deployment.\n",
+        "\n",
+        "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
+        "\n",
+        "Notebook synopsis:\n",
+        "\n",
+        "1. Creating an Experiment in an existing Workspace\n",
+        "2. Configuration and remote run of AutoML for a time-series model exploring Regression learners, Arima, Prophet and DNNs\n",
+        "4. Evaluating the fitted model using a rolling test "
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "## Setup\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import azureml.core\n",
+        "import pandas as pd\n",
+        "import numpy as np\n",
+        "import logging\n",
+        "import warnings\n",
+        "\n",
+        "from pandas.tseries.frequencies import to_offset\n",
+        "\n",
+        "# Squash warning messages for cleaner output in the notebook\n",
+        "warnings.showwarning = lambda *args, **kwargs: None\n",
+        "\n",
+        "from azureml.core.workspace import Workspace\n",
+        "from azureml.core.experiment import Experiment\n",
+        "from azureml.train.automl import AutoMLConfig\n",
+        "from matplotlib import pyplot as plt\n",
+        "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+        "from azureml.train.estimator import Estimator"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "print(\"This notebook was created using version 1.38.0 of the Azure ML SDK\")\n",
+        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "ws = Workspace.from_config()\n",
+        "\n",
+        "# choose a name for the run history container in the workspace\n",
+        "experiment_name = \"beer-remote-cpu\"\n",
+        "\n",
+        "experiment = Experiment(ws, experiment_name)\n",
+        "\n",
+        "output = {}\n",
+        "output[\"Subscription ID\"] = ws.subscription_id\n",
+        "output[\"Workspace\"] = ws.name\n",
+        "output[\"Resource Group\"] = ws.resource_group\n",
+        "output[\"Location\"] = ws.location\n",
+        "output[\"Run History Name\"] = experiment_name\n",
+        "pd.set_option(\"display.max_colwidth\", -1)\n",
+        "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
+        "outputDf.T"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "### Using AmlCompute\n",
+        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you use `AmlCompute` as your training compute resource.\n",
+        "\n",
+        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
+        "from azureml.core.compute_target import ComputeTargetException\n",
+        "\n",
+        "# Choose a name for your CPU cluster\n",
+        "cpu_cluster_name = \"beer-cluster\"\n",
+        "\n",
+        "# Verify that cluster does not exist already\n",
+        "try:\n",
+        "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
+        "    print(\"Found existing cluster, use it.\")\n",
+        "except ComputeTargetException:\n",
+        "    compute_config = AmlCompute.provisioning_configuration(\n",
+        "        vm_size=\"STANDARD_DS12_V2\", max_nodes=4\n",
+        "    )\n",
+        "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
+        "\n",
+        "compute_target.wait_for_completion(show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "## Data\n",
+        "Read Beer demand data from file, and preview data."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "Let's set up what we know about the dataset. \n",
+        "\n",
+        "**Target column** is what we want to forecast.\n",
+        "\n",
+        "**Time column** is the time axis along which to predict.\n",
+        "\n",
+        "**Time series identifier columns** are identified by values of the columns listed `time_series_id_column_names`, for example \"store\" and \"item\" if your data has multiple time series of sales, one series for each combination of store and item sold.\n",
+        "\n",
+        "**Forecast frequency (freq)** This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
+        "\n",
+        "This dataset has only one time series. Please see the [orange juice notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales) for an example of a multi-time series dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "import pandas as pd\n",
+        "from pandas import DataFrame\n",
+        "from pandas import Grouper\n",
+        "from pandas import concat\n",
+        "from pandas.plotting import register_matplotlib_converters\n",
+        "\n",
+        "register_matplotlib_converters()\n",
+        "plt.figure(figsize=(20, 10))\n",
+        "plt.tight_layout()\n",
+        "\n",
+        "plt.subplot(2, 1, 1)\n",
+        "plt.title(\"Beer Production By Year\")\n",
+        "df = pd.read_csv(\n",
+        "    \"Beer_no_valid_split_train.csv\", parse_dates=True, index_col=\"DATE\"\n",
+        ").drop(columns=\"grain\")\n",
+        "test_df = pd.read_csv(\n",
+        "    \"Beer_no_valid_split_test.csv\", parse_dates=True, index_col=\"DATE\"\n",
+        ").drop(columns=\"grain\")\n",
+        "plt.plot(df)\n",
+        "\n",
+        "plt.subplot(2, 1, 2)\n",
+        "plt.title(\"Beer Production By Month\")\n",
+        "groups = df.groupby(df.index.month)\n",
+        "months = concat([DataFrame(x[1].values) for x in groups], axis=1)\n",
+        "months = DataFrame(months)\n",
+        "months.columns = range(1, 13)\n",
+        "months.boxplot()\n",
+        "\n",
+        "plt.show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "target_column_name = \"BeerProduction\"\n",
+        "time_column_name = \"DATE\"\n",
+        "time_series_id_column_names = []\n",
+        "freq = \"M\"  # Monthly data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Split Training data into Train and Validation set and Upload to Datastores"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from helper import split_fraction_by_grain\n",
+        "from helper import split_full_for_forecasting\n",
+        "\n",
+        "train, valid = split_full_for_forecasting(df, time_column_name)\n",
+        "train.to_csv(\"train.csv\")\n",
+        "valid.to_csv(\"valid.csv\")\n",
+        "test_df.to_csv(\"test.csv\")\n",
+        "\n",
+        "datastore = ws.get_default_datastore()\n",
+        "datastore.upload_files(\n",
+        "    files=[\"./train.csv\"],\n",
+        "    target_path=\"beer-dataset/tabular/\",\n",
+        "    overwrite=True,\n",
+        "    show_progress=True,\n",
+        ")\n",
+        "datastore.upload_files(\n",
+        "    files=[\"./valid.csv\"],\n",
+        "    target_path=\"beer-dataset/tabular/\",\n",
+        "    overwrite=True,\n",
+        "    show_progress=True,\n",
+        ")\n",
+        "datastore.upload_files(\n",
+        "    files=[\"./test.csv\"],\n",
+        "    target_path=\"beer-dataset/tabular/\",\n",
+        "    overwrite=True,\n",
+        "    show_progress=True,\n",
+        ")\n",
+        "\n",
+        "from azureml.core import Dataset\n",
+        "\n",
+        "train_dataset = Dataset.Tabular.from_delimited_files(\n",
+        "    path=[(datastore, \"beer-dataset/tabular/train.csv\")]\n",
+        ")\n",
+        "valid_dataset = Dataset.Tabular.from_delimited_files(\n",
+        "    path=[(datastore, \"beer-dataset/tabular/valid.csv\")]\n",
+        ")\n",
+        "test_dataset = Dataset.Tabular.from_delimited_files(\n",
+        "    path=[(datastore, \"beer-dataset/tabular/test.csv\")]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "### Setting forecaster maximum horizon \n",
+        "\n",
+        "The forecast horizon is the number of periods into the future that the model should predict. Here, we set the horizon to 12 periods (i.e. 12 months). Notice that this is much shorter than the number of months in the test set; we will need to use a rolling test to evaluate the performance on the whole test set. For more discussion of forecast horizons and guiding principles for setting them, please see the [energy demand notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand).  "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "forecast_horizon = 12"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "## Train\n",
+        "\n",
+        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
+        "\n",
+        "|Property|Description|\n",
+        "|-|-|\n",
+        "|**task**|forecasting|\n",
+        "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>\n",
+        "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
+        "|**training_data**|Input dataset, containing both features and label column.|\n",
+        "|**label_column_name**|The name of the label column.|\n",
+        "|**enable_dnn**|Enable Forecasting DNNs|\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
+        "\n",
+        "forecasting_parameters = ForecastingParameters(\n",
+        "    time_column_name=time_column_name,\n",
+        "    forecast_horizon=forecast_horizon,\n",
+        "    freq=\"MS\",  # Set the forecast frequency to be monthly (start of the month)\n",
+        ")\n",
+        "\n",
+        "# We will disable the enable_early_stopping flag to ensure the DNN model is recommended for demonstration purpose.\n",
+        "automl_config = AutoMLConfig(\n",
+        "    task=\"forecasting\",\n",
+        "    primary_metric=\"normalized_root_mean_squared_error\",\n",
+        "    experiment_timeout_hours=1,\n",
+        "    training_data=train_dataset,\n",
+        "    label_column_name=target_column_name,\n",
+        "    validation_data=valid_dataset,\n",
+        "    verbosity=logging.INFO,\n",
+        "    compute_target=compute_target,\n",
+        "    max_concurrent_iterations=4,\n",
+        "    max_cores_per_iteration=-1,\n",
+        "    enable_dnn=True,\n",
+        "    enable_early_stopping=False,\n",
+        "    forecasting_parameters=forecasting_parameters,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "We will now run the experiment, starting with 10 iterations of model search. The experiment can be continued for more iterations if more accurate results are required. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "remote_run = experiment.submit(automl_config, show_output=True)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "# If you need to retrieve a run that already started, use the following code\n",
+        "# from azureml.train.automl.run import AutoMLRun\n",
+        "# remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "Displaying the run objects gives you links to the visual tools in the Azure Portal. Go try them!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "### Retrieve the Best Model for Each Algorithm\n",
+        "Below we select the best pipeline from our iterations. The get_output method on automl_classifier returns the best run and the fitted model for the last fit invocation. There are overloads on get_output that allow you to retrieve the best run and fitted model for any logged metric or a particular iteration."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from helper import get_result_df\n",
+        "\n",
+        "summary_df = get_result_df(remote_run)\n",
+        "summary_df"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.core.run import Run\n",
+        "from azureml.widgets import RunDetails\n",
+        "\n",
+        "forecast_model = \"TCNForecaster\"\n",
+        "if not forecast_model in summary_df[\"run_id\"]:\n",
+        "    forecast_model = \"ForecastTCN\"\n",
+        "\n",
+        "best_dnn_run_id = summary_df[\"run_id\"][forecast_model]\n",
+        "best_dnn_run = Run(experiment, best_dnn_run_id)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "best_dnn_run.parent\n",
+        "RunDetails(best_dnn_run.parent).show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "best_dnn_run\n",
+        "RunDetails(best_dnn_run).show()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "## Evaluate on Test Data"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "source": [
+        "We now use the best fitted model from the AutoML Run to make forecasts for the test set.  \n",
+        "\n",
+        "We always score on the original dataset whose schema matches the training set schema."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "from azureml.core import Dataset\n",
+        "\n",
+        "test_dataset = Dataset.Tabular.from_delimited_files(\n",
+        "    path=[(datastore, \"beer-dataset/tabular/test.csv\")]\n",
+        ")\n",
+        "# preview the first 3 rows of the dataset\n",
+        "test_dataset.take(5).to_pandas_dataframe()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "compute_target = ws.compute_targets[\"beer-cluster\"]\n",
+        "test_experiment = Experiment(ws, experiment_name + \"_test\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import shutil\n",
+        "\n",
+        "script_folder = os.path.join(os.getcwd(), \"inference\")\n",
+        "os.makedirs(script_folder, exist_ok=True)\n",
+        "shutil.copy(\"infer.py\", script_folder)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from helper import run_inference\n",
+        "\n",
+        "test_run = run_inference(\n",
+        "    test_experiment,\n",
+        "    compute_target,\n",
+        "    script_folder,\n",
+        "    best_dnn_run,\n",
+        "    test_dataset,\n",
+        "    valid_dataset,\n",
+        "    forecast_horizon,\n",
+        "    target_column_name,\n",
+        "    time_column_name,\n",
+        "    freq,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "RunDetails(test_run).show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from helper import run_multiple_inferences\n",
+        "\n",
+        "summary_df = run_multiple_inferences(\n",
+        "    summary_df,\n",
+        "    experiment,\n",
+        "    test_experiment,\n",
+        "    compute_target,\n",
+        "    script_folder,\n",
+        "    test_dataset,\n",
+        "    valid_dataset,\n",
+        "    forecast_horizon,\n",
+        "    target_column_name,\n",
+        "    time_column_name,\n",
+        "    freq,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "for run_name, run_summary in summary_df.iterrows():\n",
+        "    print(run_name)\n",
+        "    print(run_summary)\n",
+        "    run_id = run_summary.run_id\n",
+        "    test_run_id = run_summary.test_run_id\n",
+        "    test_run = Run(test_experiment, test_run_id)\n",
+        "    test_run.wait_for_completion()\n",
+        "    test_score = test_run.get_metrics()[run_summary.primary_metric]\n",
+        "    summary_df.loc[summary_df.run_id == run_id, \"Test Score\"] = test_score\n",
+        "    print(\"Test Score: \", test_score)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "hideCode": false,
+        "hidePrompt": false
+      },
+      "outputs": [],
+      "source": [
+        "summary_df"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "authors": [
+      {
+        "name": "jialiu"
+      }
+    ],
+    "hide_code_all_hidden": false,
+    "kernelspec": {
+      "display_name": "Python 3.6",
+      "language": "python",
+      "name": "python36"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.6.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-beer-remote
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/helper.py

@@ -0,0 +1,183 @@
+import pandas as pd
+from azureml.core import Environment
+from azureml.core.conda_dependencies import CondaDependencies
+from azureml.train.estimator import Estimator
+from azureml.core.run import Run
+from azureml.automl.core.shared import constants
+
+
+def split_fraction_by_grain(df, fraction, time_column_name, grain_column_names=None):
+    if not grain_column_names:
+        df["tmp_grain_column"] = "grain"
+        grain_column_names = ["tmp_grain_column"]
+
+    """Group df by grain and split on last n rows for each group."""
+    df_grouped = df.sort_values(time_column_name).groupby(
+        grain_column_names, group_keys=False
+    )
+
+    df_head = df_grouped.apply(
+        lambda dfg: dfg.iloc[: -int(len(dfg) * fraction)] if fraction > 0 else dfg
+    )
+
+    df_tail = df_grouped.apply(
+        lambda dfg: dfg.iloc[-int(len(dfg) * fraction) :] if fraction > 0 else dfg[:0]
+    )
+
+    if "tmp_grain_column" in grain_column_names:
+        for df2 in (df, df_head, df_tail):
+            df2.drop("tmp_grain_column", axis=1, inplace=True)
+
+        grain_column_names.remove("tmp_grain_column")
+
+    return df_head, df_tail
+
+
+def split_full_for_forecasting(
+    df, time_column_name, grain_column_names=None, test_split=0.2
+):
+    index_name = df.index.name
+
+    # Assumes that there isn't already a column called tmpindex
+
+    df["tmpindex"] = df.index
+
+    train_df, test_df = split_fraction_by_grain(
+        df, test_split, time_column_name, grain_column_names
+    )
+
+    train_df = train_df.set_index("tmpindex")
+    train_df.index.name = index_name
+
+    test_df = test_df.set_index("tmpindex")
+    test_df.index.name = index_name
+
+    df.drop("tmpindex", axis=1, inplace=True)
+
+    return train_df, test_df
+
+
+def get_result_df(remote_run):
+    children = list(remote_run.get_children(recursive=True))
+    summary_df = pd.DataFrame(
+        index=["run_id", "run_algorithm", "primary_metric", "Score"]
+    )
+    goal_minimize = False
+    for run in children:
+        if (
+            run.get_status().lower() == constants.RunState.COMPLETE_RUN
+            and "run_algorithm" in run.properties
+            and "score" in run.properties
+        ):
+            # We only count in the completed child runs.
+            summary_df[run.id] = [
+                run.id,
+                run.properties["run_algorithm"],
+                run.properties["primary_metric"],
+                float(run.properties["score"]),
+            ]
+            if "goal" in run.properties:
+                goal_minimize = run.properties["goal"].split("_")[-1] == "min"
+
+    summary_df = summary_df.T.sort_values(
+        "Score", ascending=goal_minimize
+    ).drop_duplicates(["run_algorithm"])
+    summary_df = summary_df.set_index("run_algorithm")
+    return summary_df
+
+
+def run_inference(
+    test_experiment,
+    compute_target,
+    script_folder,
+    train_run,
+    test_dataset,
+    lookback_dataset,
+    max_horizon,
+    target_column_name,
+    time_column_name,
+    freq,
+):
+    model_base_name = "model.pkl"
+    if "model_data_location" in train_run.properties:
+        model_location = train_run.properties["model_data_location"]
+        _, model_base_name = model_location.rsplit("/", 1)
+    train_run.download_file(
+        "outputs/{}".format(model_base_name), "inference/{}".format(model_base_name)
+    )
+    train_run.download_file("outputs/conda_env_v_1_0_0.yml", "inference/condafile.yml")
+
+    inference_env = Environment("myenv")
+    inference_env.docker.enabled = True
+    inference_env.python.conda_dependencies = CondaDependencies(
+        conda_dependencies_file_path="inference/condafile.yml"
+    )
+
+    est = Estimator(
+        source_directory=script_folder,
+        entry_script="infer.py",
+        script_params={
+            "--max_horizon": max_horizon,
+            "--target_column_name": target_column_name,
+            "--time_column_name": time_column_name,
+            "--frequency": freq,
+            "--model_path": model_base_name,
+        },
+        inputs=[
+            test_dataset.as_named_input("test_data"),
+            lookback_dataset.as_named_input("lookback_data"),
+        ],
+        compute_target=compute_target,
+        environment_definition=inference_env,
+    )
+
+    run = test_experiment.submit(
+        est,
+        tags={
+            "training_run_id": train_run.id,
+            "run_algorithm": train_run.properties["run_algorithm"],
+            "valid_score": train_run.properties["score"],
+            "primary_metric": train_run.properties["primary_metric"],
+        },
+    )
+
+    run.log("run_algorithm", run.tags["run_algorithm"])
+    return run
+
+
+def run_multiple_inferences(
+    summary_df,
+    train_experiment,
+    test_experiment,
+    compute_target,
+    script_folder,
+    test_dataset,
+    lookback_dataset,
+    max_horizon,
+    target_column_name,
+    time_column_name,
+    freq,
+):
+    for run_name, run_summary in summary_df.iterrows():
+        print(run_name)
+        print(run_summary)
+        run_id = run_summary.run_id
+        train_run = Run(train_experiment, run_id)
+
+        test_run = run_inference(
+            test_experiment,
+            compute_target,
+            script_folder,
+            train_run,
+            test_dataset,
+            lookback_dataset,
+            max_horizon,
+            target_column_name,
+            time_column_name,
+            freq,
+        )
+
+        print(test_run)
+        summary_df.loc[summary_df.run_id == run_id, "test_run_id"] = test_run.id
+
+    return summary_df

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/infer.py

@@ -0,0 +1,386 @@
+import argparse
+import os
+
+import numpy as np
+import pandas as pd
+
+from pandas.tseries.frequencies import to_offset
+from sklearn.externals import joblib
+from sklearn.metrics import mean_absolute_error, mean_squared_error
+
+from azureml.automl.runtime.shared.score import scoring, constants
+from azureml.core import Run
+
+try:
+    import torch
+
+    _torch_present = True
+except ImportError:
+    _torch_present = False
+
+
+def align_outputs(
+    y_predicted,
+    X_trans,
+    X_test,
+    y_test,
+    predicted_column_name="predicted",
+    horizon_colname="horizon_origin",
+):
+    """
+    Demonstrates how to get the output aligned to the inputs
+    using pandas indexes. Helps understand what happened if
+    the output's shape differs from the input shape, or if
+    the data got re-sorted by time and grain during forecasting.
+
+    Typical causes of misalignment are:
+    * we predicted some periods that were missing in actuals -> drop from eval
+    * model was asked to predict past max_horizon -> increase max horizon
+    * data at start of X_test was needed for lags -> provide previous periods
+    """
+    if horizon_colname in X_trans:
+        df_fcst = pd.DataFrame(
+            {
+                predicted_column_name: y_predicted,
+                horizon_colname: X_trans[horizon_colname],
+            }
+        )
+    else:
+        df_fcst = pd.DataFrame({predicted_column_name: y_predicted})
+
+    # y and X outputs are aligned by forecast() function contract
+    df_fcst.index = X_trans.index
+
+    # align original X_test to y_test
+    X_test_full = X_test.copy()
+    X_test_full[target_column_name] = y_test
+
+    # X_test_full's index does not include origin, so reset for merge
+    df_fcst.reset_index(inplace=True)
+    X_test_full = X_test_full.reset_index().drop(columns="index")
+    together = df_fcst.merge(X_test_full, how="right")
+
+    # drop rows where prediction or actuals are nan
+    # happens because of missing actuals
+    # or at edges of time due to lags/rolling windows
+    clean = together[
+        together[[target_column_name, predicted_column_name]].notnull().all(axis=1)
+    ]
+    return clean
+
+
+def do_rolling_forecast_with_lookback(
+    fitted_model, X_test, y_test, max_horizon, X_lookback, y_lookback, freq="D"
+):
+    """
+    Produce forecasts on a rolling origin over the given test set.
+
+    Each iteration makes a forecast for the next 'max_horizon' periods
+    with respect to the current origin, then advances the origin by the
+    horizon time duration. The prediction context for each forecast is set so
+    that the forecaster uses the actual target values prior to the current
+    origin time for constructing lag features.
+
+    This function returns a concatenated DataFrame of rolling forecasts.
+    """
+    print("Using lookback of size: ", y_lookback.size)
+    df_list = []
+    origin_time = X_test[time_column_name].min()
+    X = X_lookback.append(X_test)
+    y = np.concatenate((y_lookback, y_test), axis=0)
+    while origin_time <= X_test[time_column_name].max():
+        # Set the horizon time - end date of the forecast
+        horizon_time = origin_time + max_horizon * to_offset(freq)
+
+        # Extract test data from an expanding window up-to the horizon
+        expand_wind = X[time_column_name] < horizon_time
+        X_test_expand = X[expand_wind]
+        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
+        y_query_expand.fill(np.NaN)
+
+        if origin_time != X[time_column_name].min():
+            # Set the context by including actuals up-to the origin time
+            test_context_expand_wind = X[time_column_name] < origin_time
+            context_expand_wind = X_test_expand[time_column_name] < origin_time
+            y_query_expand[context_expand_wind] = y[test_context_expand_wind]
+
+        # Print some debug info
+        print(
+            "Horizon_time:",
+            horizon_time,
+            " origin_time: ",
+            origin_time,
+            " max_horizon: ",
+            max_horizon,
+            " freq: ",
+            freq,
+        )
+        print("expand_wind: ", expand_wind)
+        print("y_query_expand")
+        print(y_query_expand)
+        print("X_test")
+        print(X)
+        print("X_test_expand")
+        print(X_test_expand)
+        print("Type of X_test_expand: ", type(X_test_expand))
+        print("Type of y_query_expand: ", type(y_query_expand))
+
+        print("y_query_expand")
+        print(y_query_expand)
+
+        # Make a forecast out to the maximum horizon
+        # y_fcst, X_trans = y_query_expand, X_test_expand
+        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
+
+        print("y_fcst")
+        print(y_fcst)
+
+        # Align forecast with test set for dates within
+        # the current rolling window
+        trans_tindex = X_trans.index.get_level_values(time_column_name)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
+        test_roll_wind = expand_wind & (X[time_column_name] >= origin_time)
+        df_list.append(
+            align_outputs(
+                y_fcst[trans_roll_wind],
+                X_trans[trans_roll_wind],
+                X[test_roll_wind],
+                y[test_roll_wind],
+            )
+        )
+
+        # Advance the origin time
+        origin_time = horizon_time
+
+    return pd.concat(df_list, ignore_index=True)
+
+
+def do_rolling_forecast(fitted_model, X_test, y_test, max_horizon, freq="D"):
+    """
+    Produce forecasts on a rolling origin over the given test set.
+
+    Each iteration makes a forecast for the next 'max_horizon' periods
+    with respect to the current origin, then advances the origin by the
+    horizon time duration. The prediction context for each forecast is set so
+    that the forecaster uses the actual target values prior to the current
+    origin time for constructing lag features.
+
+    This function returns a concatenated DataFrame of rolling forecasts.
+    """
+    df_list = []
+    origin_time = X_test[time_column_name].min()
+    while origin_time <= X_test[time_column_name].max():
+        # Set the horizon time - end date of the forecast
+        horizon_time = origin_time + max_horizon * to_offset(freq)
+
+        # Extract test data from an expanding window up-to the horizon
+        expand_wind = X_test[time_column_name] < horizon_time
+        X_test_expand = X_test[expand_wind]
+        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
+        y_query_expand.fill(np.NaN)
+
+        if origin_time != X_test[time_column_name].min():
+            # Set the context by including actuals up-to the origin time
+            test_context_expand_wind = X_test[time_column_name] < origin_time
+            context_expand_wind = X_test_expand[time_column_name] < origin_time
+            y_query_expand[context_expand_wind] = y_test[test_context_expand_wind]
+
+        # Print some debug info
+        print(
+            "Horizon_time:",
+            horizon_time,
+            " origin_time: ",
+            origin_time,
+            " max_horizon: ",
+            max_horizon,
+            " freq: ",
+            freq,
+        )
+        print("expand_wind: ", expand_wind)
+        print("y_query_expand")
+        print(y_query_expand)
+        print("X_test")
+        print(X_test)
+        print("X_test_expand")
+        print(X_test_expand)
+        print("Type of X_test_expand: ", type(X_test_expand))
+        print("Type of y_query_expand: ", type(y_query_expand))
+        print("y_query_expand")
+        print(y_query_expand)
+
+        # Make a forecast out to the maximum horizon
+        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
+
+        print("y_fcst")
+        print(y_fcst)
+
+        # Align forecast with test set for dates within the
+        # current rolling window
+        trans_tindex = X_trans.index.get_level_values(time_column_name)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
+        test_roll_wind = expand_wind & (X_test[time_column_name] >= origin_time)
+        df_list.append(
+            align_outputs(
+                y_fcst[trans_roll_wind],
+                X_trans[trans_roll_wind],
+                X_test[test_roll_wind],
+                y_test[test_roll_wind],
+            )
+        )
+
+        # Advance the origin time
+        origin_time = horizon_time
+
+    return pd.concat(df_list, ignore_index=True)
+
+
+def APE(actual, pred):
+    """
+    Calculate absolute percentage error.
+    Returns a vector of APE values with same length as actual/pred.
+    """
+    return 100 * np.abs((actual - pred) / actual)
+
+
+def MAPE(actual, pred):
+    """
+    Calculate mean absolute percentage error.
+    Remove NA and values where actual is close to zero
+    """
+    not_na = ~(np.isnan(actual) | np.isnan(pred))
+    not_zero = ~np.isclose(actual, 0.0)
+    actual_safe = actual[not_na & not_zero]
+    pred_safe = pred[not_na & not_zero]
+    return np.mean(APE(actual_safe, pred_safe))
+
+
+def map_location_cuda(storage, loc):
+    return storage.cuda()
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "--max_horizon",
+    type=int,
+    dest="max_horizon",
+    default=10,
+    help="Max Horizon for forecasting",
+)
+parser.add_argument(
+    "--target_column_name",
+    type=str,
+    dest="target_column_name",
+    help="Target Column Name",
+)
+parser.add_argument(
+    "--time_column_name", type=str, dest="time_column_name", help="Time Column Name"
+)
+parser.add_argument(
+    "--frequency", type=str, dest="freq", help="Frequency of prediction"
+)
+parser.add_argument(
+    "--model_path",
+    type=str,
+    dest="model_path",
+    default="model.pkl",
+    help="Filename of model to be loaded",
+)
+
+args = parser.parse_args()
+max_horizon = args.max_horizon
+target_column_name = args.target_column_name
+time_column_name = args.time_column_name
+freq = args.freq
+model_path = args.model_path
+
+print("args passed are: ")
+print(max_horizon)
+print(target_column_name)
+print(time_column_name)
+print(freq)
+print(model_path)
+
+run = Run.get_context()
+# get input dataset by name
+test_dataset = run.input_datasets["test_data"]
+lookback_dataset = run.input_datasets["lookback_data"]
+
+grain_column_names = []
+
+df = test_dataset.to_pandas_dataframe()
+
+print("Read df")
+print(df)
+
+X_test_df = test_dataset.drop_columns(columns=[target_column_name])
+y_test_df = test_dataset.with_timestamp_columns(None).keep_columns(
+    columns=[target_column_name]
+)
+
+X_lookback_df = lookback_dataset.drop_columns(columns=[target_column_name])
+y_lookback_df = lookback_dataset.with_timestamp_columns(None).keep_columns(
+    columns=[target_column_name]
+)
+
+_, ext = os.path.splitext(model_path)
+if ext == ".pt":
+    # Load the fc-tcn torch model.
+    assert _torch_present
+    if torch.cuda.is_available():
+        map_location = map_location_cuda
+    else:
+        map_location = "cpu"
+    with open(model_path, "rb") as fh:
+        fitted_model = torch.load(fh, map_location=map_location)
+else:
+    # Load the sklearn pipeline.
+    fitted_model = joblib.load(model_path)
+
+if hasattr(fitted_model, "get_lookback"):
+    lookback = fitted_model.get_lookback()
+    df_all = do_rolling_forecast_with_lookback(
+        fitted_model,
+        X_test_df.to_pandas_dataframe(),
+        y_test_df.to_pandas_dataframe().values.T[0],
+        max_horizon,
+        X_lookback_df.to_pandas_dataframe()[-lookback:],
+        y_lookback_df.to_pandas_dataframe().values.T[0][-lookback:],
+        freq,
+    )
+else:
+    df_all = do_rolling_forecast(
+        fitted_model,
+        X_test_df.to_pandas_dataframe(),
+        y_test_df.to_pandas_dataframe().values.T[0],
+        max_horizon,
+        freq,
+    )
+
+print(df_all)
+
+print("target values:::")
+print(df_all[target_column_name])
+print("predicted values:::")
+print(df_all["predicted"])
+
+# Use the AutoML scoring module
+regression_metrics = list(constants.REGRESSION_SCALAR_SET)
+y_test = np.array(df_all[target_column_name])
+y_pred = np.array(df_all["predicted"])
+scores = scoring.score_regression(y_test, y_pred, regression_metrics)
+
+print("scores:")
+print(scores)
+
+for key, value in scores.items():
+    run.log(key, value)
+
+print("Simple forecasting model")
+rmse = np.sqrt(mean_squared_error(df_all[target_column_name], df_all["predicted"]))
+print("[Test Data] \nRoot Mean squared error: %.2f" % rmse)
+mae = mean_absolute_error(df_all[target_column_name], df_all["predicted"])
+print("mean_absolute_error score: %.2f" % mae)
+print("MAPE: %.2f" % MAPE(df_all[target_column_name], df_all["predicted"]))
+
+run.log("rmse", rmse)
+run.log("mae", mae)

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-bike-share
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/forecasting_script.py

@@ -1,6 +1,6 @@
 import argparse
 from azureml.core import Dataset, Run
-import joblib
+from sklearn.externals import joblib
 
 parser = argparse.ArgumentParser()
 parser.add_argument(
@@ -36,18 +36,18 @@ y_test_df = (
 
 fitted_model = joblib.load("model.pkl")
 
-X_rf = fitted_model.rolling_forecast(X_test_df, y_test_df.values, step=1)
+y_pred, X_trans = fitted_model.rolling_evaluation(X_test_df, y_test_df.values)
 
 # Add predictions, actuals, and horizon relative to rolling origin to the test feature data
 assign_dict = {
-    fitted_model.forecast_origin_column_name: "forecast_origin",
+    "horizon_origin": X_trans["horizon_origin"].values,
-    fitted_model.forecast_column_name: "predicted",
+    "predicted": y_pred,
-    fitted_model.actual_column_name: target_column_name,
+    target_column_name: y_test_df[target_column_name].values,
 }
-X_rf.rename(columns=assign_dict, inplace=True)
+df_all = X_test_df.assign(**assign_dict)
 
 file_name = "outputs/predictions.csv"
-export_csv = X_rf.to_csv(file_name, header=True)
+export_csv = df_all.to_csv(file_name, header=True)
 
 # Upload the predictions into artifacts
 run.upload_file(name=file_name, path_or_stream=file_name)

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-energy-demand
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/forecasting_script.py

@@ -6,7 +6,7 @@ compute instance.
 
 import argparse
 from azureml.core import Dataset, Run
-import joblib
+from sklearn.externals import joblib
 from pandas.tseries.frequencies import to_offset
 
 parser = argparse.ArgumentParser()

how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-function
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-github-dau/helper.py

@@ -79,7 +79,9 @@ def get_result_df(remote_run):
             if "goal" in run.properties:
                 goal_minimize = run.properties["goal"].split("_")[-1] == "min"
 
-    summary_df = summary_df.T.sort_values("Score", ascending=goal_minimize)
+    summary_df = summary_df.T.sort_values(
+        "Score", ascending=goal_minimize
+    ).drop_duplicates(["run_algorithm"])
     summary_df = summary_df.set_index("run_algorithm")
     return summary_df
 
@@ -103,8 +105,13 @@ def run_inference(
     train_run.download_file(
         "outputs/{}".format(model_base_name), "inference/{}".format(model_base_name)
     )
+    train_run.download_file("outputs/conda_env_v_1_0_0.yml", "inference/condafile.yml")
 
-    inference_env = train_run.get_environment()
+    inference_env = Environment("myenv")
+    inference_env.docker.enabled = True
+    inference_env.python.conda_dependencies = CondaDependencies(
+        conda_dependencies_file_path="inference/condafile.yml"
+    )
 
     est = Estimator(
         source_directory=script_folder,

how-to-use-azureml/automated-machine-learning/forecasting-github-dau/infer.py

@@ -4,7 +4,8 @@ import os
 import numpy as np
 import pandas as pd
 
-import joblib
+from pandas.tseries.frequencies import to_offset
+from sklearn.externals import joblib
 from sklearn.metrics import mean_absolute_error, mean_squared_error
 
 from azureml.automl.runtime.shared.score import scoring, constants
@@ -18,8 +19,219 @@ except ImportError:
     _torch_present = False
 
 
-def map_location_cuda(storage, loc):
+def align_outputs(
-    return storage.cuda()
+    y_predicted,
+    X_trans,
+    X_test,
+    y_test,
+    predicted_column_name="predicted",
+    horizon_colname="horizon_origin",
+):
+    """
+    Demonstrates how to get the output aligned to the inputs
+    using pandas indexes. Helps understand what happened if
+    the output's shape differs from the input shape, or if
+    the data got re-sorted by time and grain during forecasting.
+
+    Typical causes of misalignment are:
+    * we predicted some periods that were missing in actuals -> drop from eval
+    * model was asked to predict past max_horizon -> increase max horizon
+    * data at start of X_test was needed for lags -> provide previous periods
+    """
+    if horizon_colname in X_trans:
+        df_fcst = pd.DataFrame(
+            {
+                predicted_column_name: y_predicted,
+                horizon_colname: X_trans[horizon_colname],
+            }
+        )
+    else:
+        df_fcst = pd.DataFrame({predicted_column_name: y_predicted})
+
+    # y and X outputs are aligned by forecast() function contract
+    df_fcst.index = X_trans.index
+
+    # align original X_test to y_test
+    X_test_full = X_test.copy()
+    X_test_full[target_column_name] = y_test
+
+    # X_test_full's index does not include origin, so reset for merge
+    df_fcst.reset_index(inplace=True)
+    X_test_full = X_test_full.reset_index().drop(columns="index")
+    together = df_fcst.merge(X_test_full, how="right")
+
+    # drop rows where prediction or actuals are nan
+    # happens because of missing actuals
+    # or at edges of time due to lags/rolling windows
+    clean = together[
+        together[[target_column_name, predicted_column_name]].notnull().all(axis=1)
+    ]
+    return clean
+
+
+def do_rolling_forecast_with_lookback(
+    fitted_model, X_test, y_test, max_horizon, X_lookback, y_lookback, freq="D"
+):
+    """
+    Produce forecasts on a rolling origin over the given test set.
+
+    Each iteration makes a forecast for the next 'max_horizon' periods
+    with respect to the current origin, then advances the origin by the
+    horizon time duration. The prediction context for each forecast is set so
+    that the forecaster uses the actual target values prior to the current
+    origin time for constructing lag features.
+
+    This function returns a concatenated DataFrame of rolling forecasts.
+    """
+    print("Using lookback of size: ", y_lookback.size)
+    df_list = []
+    origin_time = X_test[time_column_name].min()
+    X = X_lookback.append(X_test)
+    y = np.concatenate((y_lookback, y_test), axis=0)
+    while origin_time <= X_test[time_column_name].max():
+        # Set the horizon time - end date of the forecast
+        horizon_time = origin_time + max_horizon * to_offset(freq)
+
+        # Extract test data from an expanding window up-to the horizon
+        expand_wind = X[time_column_name] < horizon_time
+        X_test_expand = X[expand_wind]
+        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
+        y_query_expand.fill(np.NaN)
+
+        if origin_time != X[time_column_name].min():
+            # Set the context by including actuals up-to the origin time
+            test_context_expand_wind = X[time_column_name] < origin_time
+            context_expand_wind = X_test_expand[time_column_name] < origin_time
+            y_query_expand[context_expand_wind] = y[test_context_expand_wind]
+
+        # Print some debug info
+        print(
+            "Horizon_time:",
+            horizon_time,
+            " origin_time: ",
+            origin_time,
+            " max_horizon: ",
+            max_horizon,
+            " freq: ",
+            freq,
+        )
+        print("expand_wind: ", expand_wind)
+        print("y_query_expand")
+        print(y_query_expand)
+        print("X_test")
+        print(X)
+        print("X_test_expand")
+        print(X_test_expand)
+        print("Type of X_test_expand: ", type(X_test_expand))
+        print("Type of y_query_expand: ", type(y_query_expand))
+
+        print("y_query_expand")
+        print(y_query_expand)
+
+        # Make a forecast out to the maximum horizon
+        # y_fcst, X_trans = y_query_expand, X_test_expand
+        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
+
+        print("y_fcst")
+        print(y_fcst)
+
+        # Align forecast with test set for dates within
+        # the current rolling window
+        trans_tindex = X_trans.index.get_level_values(time_column_name)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
+        test_roll_wind = expand_wind & (X[time_column_name] >= origin_time)
+        df_list.append(
+            align_outputs(
+                y_fcst[trans_roll_wind],
+                X_trans[trans_roll_wind],
+                X[test_roll_wind],
+                y[test_roll_wind],
+            )
+        )
+
+        # Advance the origin time
+        origin_time = horizon_time
+
+    return pd.concat(df_list, ignore_index=True)
+
+
+def do_rolling_forecast(fitted_model, X_test, y_test, max_horizon, freq="D"):
+    """
+    Produce forecasts on a rolling origin over the given test set.
+
+    Each iteration makes a forecast for the next 'max_horizon' periods
+    with respect to the current origin, then advances the origin by the
+    horizon time duration. The prediction context for each forecast is set so
+    that the forecaster uses the actual target values prior to the current
+    origin time for constructing lag features.
+
+    This function returns a concatenated DataFrame of rolling forecasts.
+    """
+    df_list = []
+    origin_time = X_test[time_column_name].min()
+    while origin_time <= X_test[time_column_name].max():
+        # Set the horizon time - end date of the forecast
+        horizon_time = origin_time + max_horizon * to_offset(freq)
+
+        # Extract test data from an expanding window up-to the horizon
+        expand_wind = X_test[time_column_name] < horizon_time
+        X_test_expand = X_test[expand_wind]
+        y_query_expand = np.zeros(len(X_test_expand)).astype(np.float)
+        y_query_expand.fill(np.NaN)
+
+        if origin_time != X_test[time_column_name].min():
+            # Set the context by including actuals up-to the origin time
+            test_context_expand_wind = X_test[time_column_name] < origin_time
+            context_expand_wind = X_test_expand[time_column_name] < origin_time
+            y_query_expand[context_expand_wind] = y_test[test_context_expand_wind]
+
+        # Print some debug info
+        print(
+            "Horizon_time:",
+            horizon_time,
+            " origin_time: ",
+            origin_time,
+            " max_horizon: ",
+            max_horizon,
+            " freq: ",
+            freq,
+        )
+        print("expand_wind: ", expand_wind)
+        print("y_query_expand")
+        print(y_query_expand)
+        print("X_test")
+        print(X_test)
+        print("X_test_expand")
+        print(X_test_expand)
+        print("Type of X_test_expand: ", type(X_test_expand))
+        print("Type of y_query_expand: ", type(y_query_expand))
+        print("y_query_expand")
+        print(y_query_expand)
+
+        # Make a forecast out to the maximum horizon
+        y_fcst, X_trans = fitted_model.forecast(X_test_expand, y_query_expand)
+
+        print("y_fcst")
+        print(y_fcst)
+
+        # Align forecast with test set for dates within the
+        # current rolling window
+        trans_tindex = X_trans.index.get_level_values(time_column_name)
+        trans_roll_wind = (trans_tindex >= origin_time) & (trans_tindex < horizon_time)
+        test_roll_wind = expand_wind & (X_test[time_column_name] >= origin_time)
+        df_list.append(
+            align_outputs(
+                y_fcst[trans_roll_wind],
+                X_trans[trans_roll_wind],
+                X_test[test_roll_wind],
+                y_test[test_roll_wind],
+            )
+        )
+
+        # Advance the origin time
+        origin_time = horizon_time
+
+    return pd.concat(df_list, ignore_index=True)
 
 
 def APE(actual, pred):
@@ -42,6 +254,10 @@ def MAPE(actual, pred):
     return np.mean(APE(actual_safe, pred_safe))
 
 
+def map_location_cuda(storage, loc):
+    return storage.cuda()
+
+
 parser = argparse.ArgumentParser()
 parser.add_argument(
     "--max_horizon",
@@ -87,6 +303,7 @@ print(model_path)
 run = Run.get_context()
 # get input dataset by name
 test_dataset = run.input_datasets["test_data"]
+lookback_dataset = run.input_datasets["lookback_data"]
 
 grain_column_names = []
 
@@ -95,8 +312,15 @@ df = test_dataset.to_pandas_dataframe()
 print("Read df")
 print(df)
 
-X_test_df = df
+X_test_df = test_dataset.drop_columns(columns=[target_column_name])
-y_test = df.pop(target_column_name).to_numpy()
+y_test_df = test_dataset.with_timestamp_columns(None).keep_columns(
+    columns=[target_column_name]
+)
+
+X_lookback_df = lookback_dataset.drop_columns(columns=[target_column_name])
+y_lookback_df = lookback_dataset.with_timestamp_columns(None).keep_columns(
+    columns=[target_column_name]
+)
 
 _, ext = os.path.splitext(model_path)
 if ext == ".pt":
@@ -112,20 +336,37 @@ else:
     # Load the sklearn pipeline.
     fitted_model = joblib.load(model_path)
 
-X_rf = fitted_model.rolling_forecast(X_test_df, y_test, step=1)
+if hasattr(fitted_model, "get_lookback"):
-assign_dict = {
+    lookback = fitted_model.get_lookback()
-    fitted_model.forecast_origin_column_name: "forecast_origin",
+    df_all = do_rolling_forecast_with_lookback(
-    fitted_model.forecast_column_name: "predicted",
+        fitted_model,
-    fitted_model.actual_column_name: target_column_name,
+        X_test_df.to_pandas_dataframe(),
-}
+        y_test_df.to_pandas_dataframe().values.T[0],
-X_rf.rename(columns=assign_dict, inplace=True)
+        max_horizon,
+        X_lookback_df.to_pandas_dataframe()[-lookback:],
+        y_lookback_df.to_pandas_dataframe().values.T[0][-lookback:],
+        freq,
+    )
+else:
+    df_all = do_rolling_forecast(
+        fitted_model,
+        X_test_df.to_pandas_dataframe(),
+        y_test_df.to_pandas_dataframe().values.T[0],
+        max_horizon,
+        freq,
+    )
 
-print(X_rf.head())
+print(df_all)
+
+print("target values:::")
+print(df_all[target_column_name])
+print("predicted values:::")
+print(df_all["predicted"])
 
 # Use the AutoML scoring module
 regression_metrics = list(constants.REGRESSION_SCALAR_SET)
-y_test = np.array(X_rf[target_column_name])
+y_test = np.array(df_all[target_column_name])
-y_pred = np.array(X_rf["predicted"])
+y_pred = np.array(df_all["predicted"])
 scores = scoring.score_regression(y_test, y_pred, regression_metrics)
 
 print("scores:")
@@ -135,11 +376,11 @@ for key, value in scores.items():
     run.log(key, value)
 
 print("Simple forecasting model")
-rmse = np.sqrt(mean_squared_error(X_rf[target_column_name], X_rf["predicted"]))
+rmse = np.sqrt(mean_squared_error(df_all[target_column_name], df_all["predicted"]))
 print("[Test Data] \nRoot Mean squared error: %.2f" % rmse)
-mae = mean_absolute_error(X_rf[target_column_name], X_rf["predicted"])
+mae = mean_absolute_error(df_all[target_column_name], df_all["predicted"])
 print("mean_absolute_error score: %.2f" % mae)
-print("MAPE: %.2f" % MAPE(X_rf[target_column_name], X_rf["predicted"]))
+print("MAPE: %.2f" % MAPE(df_all[target_column_name], df_all["predicted"]))
 
 run.log("rmse", rmse)
 run.log("mae", mae)

how-to-use-azureml/automated-machine-learning/forecasting-hierarchical-timeseries/README.md

@@ -0,0 +1,94 @@
+---
+page_type: sample
+languages:
+- python
+products:
+- azure-machine-learning
+description: Tutorial showing how to solve a complex machine learning time series forecasting problems at scale by using Azure Automated ML and Hierarchical time series accelerator.
+---
+
+## Microsoft Solution Accelerator: Hierachical Time Series Forecasting
+
+In most applications, customers have a need to understand their forecasts at a macro and micro level of the business. Whether that be predicting sales of products at different geographic locations, or understanding the expected workforce demand for different organizations at a company, the ability to train a machine learning model to intelligently forecast on hierarchy data is essential.
+
+This business pattern is common across a wide variety of industries and applicable to many real world use cases. Below are some examples of where the hierarchical time series pattern is useful.
+
+| Industry       | Scenario                                |
+|----------------|--------------------------------------------|
+| *Restaurant Chain* | Building demand forecasting models across thousands of restaurants and several countries. |
+| *Retail Organization* | Building workforce optimization models for thousands of stores. |
+| *Retail Organization*| Price optimization models for hundreds of thousands of products available. |
+
+
+### Technical Summary
+
+A hierarchical time series is a structure in which each of the unique series are arranged into a hierarchy based on dimensions such as geography, or product type. The table below shows an example of data whose unique attributes form a hierarchy. Our hierarchy is defined by the `product type` such as headphones or tablets, the `product category` which splits product types into accessories and devices, and the `region` the products are sold in. The table below demonstrates the first input of each unique series in the hierarchy.
+
+![data-table](./media/data-table.png)
+
+To further visualize this, the leaf levels of the hierarchy contain all the time series with unique combinations of attribute values. Each higher level in the hierarchy will consider one less dimension for defining the time series and will aggregate each set of `child nodes` from the lower level into a `parent node`.
+
+![hierachy-sample](./media/hierarchy-sample-ms.PNG)
+
+> **Note:** If no unique root level exists in the data, Automated Machine Learning will create a node `automl_top_level` for users to train or forecasts totals.
+
+## Prerequisites
+
+To use this solution accelerator, all you need is access to an [Azure subscription](https://azure.microsoft.com/free/) and an [Azure Machine Learning Workspace](https://docs.microsoft.com/azure/machine-learning/how-to-manage-workspace) that you'll create below.
+
+A basic understanding of Azure Machine Learning and hierarchical time series concepts will be helpful for understanding the solution. The following resources can help introduce you to these concepts:
+
+1. [Azure Machine Learning Overview](https://azure.microsoft.com/services/machine-learning/)
+2. [Azure Machine Learning Tutorials](https://docs.microsoft.com/azure/machine-learning/tutorial-1st-experiment-sdk-setup)
+3. [Azure Machine Learning Sample Notebooks on Github](https://github.com/Azure/azureml-examples/)
+4. [Forecasting: Principles and Practice, Hierarchical time series](https://otexts.com/fpp2/hts.html)
+
+## Getting started
+
+### 1. Set up the Compute Instance
+Please create a [Compute Instance](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-instance#create) and clone the git repo to your workspace.
+
+### 2. Run the Notebook
+
+Once your environment is set up, go to JupyterLab and run the notebook auto-ml-hierarchical-timeseries.ipynb on Compute Instance you created. It would run through the steps outlined sequentially. By the end, you'll know how to train, score, and make predictions using the hierarchical time series model pattern on Azure Machine Learning.
+
+| Notebook       | Description                                |
+|----------------|--------------------------------------------|
+| `auto-ml-forecasting-hierarchical-timeseries.ipynb`|Creates a pipeline to train machine learning models for the defined hierarchy and forecast at the desired hierarchy level using Automated ML. |
+
+
+![Work Flow](./media/workflow.PNG)
+
+## Key Concepts
+
+### Automated Machine Learning
+
+[Automated Machine Learning](https://docs.microsoft.com/azure/machine-learning/concept-automated-ml) also referred to as automated ML or AutoML, is the process of automating the time consuming, iterative tasks of machine learning model development. It allows data scientists, analysts, and developers to build ML models with high scale, efficiency, and productivity all while sustaining model quality.
+
+### Pipelines
+
+[Pipelines](https://docs.microsoft.com/azure/machine-learning/concept-ml-pipelines) allow you to create workflows in your machine learning projects. These workflows have a number of benefits including speed, simplicity, repeatability, and modularity.
+
+### ParallelRunStep
+
+[ParallelRunStep](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.parallel_run_step.parallelrunstep?view=azure-ml-py) enables the parallel training of models and is commonly used for batch inferencing. This [document](https://docs.microsoft.com/azure/machine-learning/how-to-use-parallel-run-step) walks through some of the key concepts around ParallelRunStep.
+
+### Other Concepts
+
+In additional to ParallelRunStep, Pipelines and Automated Machine Learning, you'll also be working with the following concepts including [workspace](https://docs.microsoft.com/azure/machine-learning/concept-workspace), [datasets](https://docs.microsoft.com/azure/machine-learning/concept-data#datasets), [compute targets](https://docs.microsoft.com/azure/machine-learning/concept-compute-target#train), [python script steps](https://docs.microsoft.com/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py), and [Azure Open Datasets](https://azure.microsoft.com/services/open-datasets/).
+
+## Contributing
+
+This project welcomes contributions and suggestions. To learn more visit the [contributing](CONTRIBUTING.md) section.
+
+Most contributions require you to agree to a Contributor License Agreement (CLA)
+declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+
+When you submit a pull request, a CLA bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

how-to-use-azureml/automated-machine-learning/forecasting-hierarchical-timeseries/auto-ml-forecasting-hierarchical-timeseries.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-hierarchical-timeseries
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-hierarchical-timeseries/update_env.yml

@@ -0,0 +1,3 @@
+dependencies:
+- pip:
+  - azureml-contrib-automl-pipeline-steps

how-to-use-azureml/automated-machine-learning/forecasting-many-models/auto-ml-forecasting-many-models.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-many-models
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-many-models/scripts/data_preprocessing_file.py

@@ -1,39 +0,0 @@
-from pathlib import Path
-from azureml.core import Run
-
-import argparse
-import os
-
-
-def main(args):
-    output = Path(args.output)
-    output.mkdir(parents=True, exist_ok=True)
-
-    run_context = Run.get_context()
-    input_path = run_context.input_datasets["train_10_models"]
-
-    for file_name in os.listdir(input_path):
-        input_file = os.path.join(input_path, file_name)
-        with open(input_file, "r") as f:
-            content = f.read()
-
-            # Apply any data pre-processing techniques here
-
-            output_file = os.path.join(output, file_name)
-            with open(output_file, "w") as f:
-                f.write(content)
-
-
-def my_parse_args():
-    parser = argparse.ArgumentParser("Test")
-
-    parser.add_argument("--input", type=str)
-    parser.add_argument("--output", type=str)
-
-    args = parser.parse_args()
-    return args
-
-
-if __name__ == "__main__":
-    args = my_parse_args()
-    main(args)

how-to-use-azureml/automated-machine-learning/forecasting-many-models/scripts/data_preprocessing_tabular.py

@@ -1,37 +0,0 @@
-from pathlib import Path
-from azureml.core import Run
-import argparse
-
-
-def main(args):
-    output = Path(args.output)
-    output.mkdir(parents=True, exist_ok=True)
-
-    run_context = Run.get_context()
-    dataset = run_context.input_datasets["train_10_models"]
-    df = dataset.to_pandas_dataframe()
-
-    # Drop the column "Revenue" from the dataset to avoid information leak as
-    # "Quantity" = "Revenue" / "Price". Please modify the logic based on your data.
-    drop_column_name = "Revenue"
-    if drop_column_name in df.columns:
-        df.drop(drop_column_name, axis=1, inplace=True)
-
-    # Apply any data pre-processing techniques here
-
-    df.to_parquet(output / "data_prepared_result.parquet", compression=None)
-
-
-def my_parse_args():
-    parser = argparse.ArgumentParser("Test")
-
-    parser.add_argument("--input", type=str)
-    parser.add_argument("--output", type=str)
-
-    args = parser.parse_args()
-    return args
-
-
-if __name__ == "__main__":
-    args = my_parse_args()
-    main(args)

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-orange-juice-sales
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/forecasting_script.py

@@ -6,7 +6,7 @@ compute instance.
 
 import argparse
 from azureml.core import Dataset, Run
-import joblib
+from sklearn.externals import joblib
 from pandas.tseries.frequencies import to_offset
 
 parser = argparse.ArgumentParser()

how-to-use-azureml/automated-machine-learning/forecasting-pipelines/auto-ml-forecasting-pipelines.ipynb

@@ -1,834 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
-        "\n",
-        "Licensed under the MIT License."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-pipelines/auto-ml-forecasting-pipelines.png)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "<font color=\"red\" size=\"5\"><strong>!Important!</strong> </br>This notebook is outdated and is not supported by the AutoML Team. Please use the supported version ([link](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1h_automl_in_pipeline/automl-forecasting-in-pipeline)).</font>\n",
-        "</br>\n",
-        "</br>\n",
-        "<font color=\"red\" size=\"5\">\n",
-        "For examples illustrating how to build pipelines with components, please use the following links:</font>\n",
-        "<ul>\n",
-        "    <li><a href=\"https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline\">Many Models</a></li>\n",
-        "    <li><a href=\"https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-hierarchical-timeseries-in-pipeline\">Hierarchical Time Series</a></li>\n",
-        "    <li><a href=\"https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/automl-standalone-jobs/automl-forecasting-distributed-tcn\">Distributed TCN</a></li>\n",
-        "</ul>"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Training and Inferencing AutoML Forecasting Model Using Pipelines"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Introduction\n",
-        "\n",
-        "In this notebook, we demonstrate how to use piplines to train and inference on AutoML Forecasting model. Two pipelines will be created: one for training AutoML model, and the other is for inference on AutoML model. We'll also demonstrate how to schedule the inference pipeline so you can get inference results periodically (with refreshed test dataset). Make sure you have executed the [configuration notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) before running this notebook. In this notebook you will learn how to:\n",
-        "\n",
-        "- Configure AutoML using AutoMLConfig for forecasting tasks using pipeline AutoMLSteps.\n",
-        "- Create and register an AutoML model using AzureML pipeline.\n",
-        "- Inference and schdelue the pipeline using registered model."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Setup\n",
-        "\n",
-        "As part of the setup you have already created an Azure ML `Workspace` object. For AutoML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import json\n",
-        "import logging\n",
-        "import os\n",
-        "\n",
-        "import pandas as pd\n",
-        "\n",
-        "import azureml.core\n",
-        "from azureml.core.experiment import Experiment\n",
-        "from azureml.core.workspace import Workspace\n",
-        "from azureml.train.automl import AutoMLConfig"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "print(\"This notebook was created using version 1.38.0 of the Azure ML SDK\")\n",
-        "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Accessing the Azure ML workspace requires authentication with Azure.\n",
-        "\n",
-        "The default authentication is interactive authentication using the default tenant. Executing the ws = Workspace.from_config() line in the cell below will prompt for authentication the first time that it is run.\n",
-        "\n",
-        "If you have multiple Azure tenants, you can specify the tenant by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
-        "```\n",
-        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
-        "auth = InteractiveLoginAuthentication(tenant_id = 'mytenantid')\n",
-        "ws = Workspace.from_config(auth = auth)\n",
-        "```\n",
-        "If you need to run in an environment where interactive login is not possible, you can use Service Principal authentication by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
-        "```\n",
-        "from azureml.core.authentication import ServicePrincipalAuthentication\n",
-        "auth = ServicePrincipalAuthentication('mytenantid', 'myappid', 'mypassword')\n",
-        "ws = Workspace.from_config(auth = auth)\n",
-        "```\n",
-        "For more details, see aka.ms/aml-notebook-auth"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "ws = Workspace.from_config()\n",
-        "dstor = ws.get_default_datastore()\n",
-        "\n",
-        "# Choose a name for the run history container in the workspace.\n",
-        "experiment_name = \"forecasting-pipeline\"\n",
-        "experiment = Experiment(ws, experiment_name)\n",
-        "\n",
-        "output = {}\n",
-        "output[\"Subscription ID\"] = ws.subscription_id\n",
-        "output[\"Workspace\"] = ws.name\n",
-        "output[\"Resource Group\"] = ws.resource_group\n",
-        "output[\"Location\"] = ws.location\n",
-        "output[\"Run History Name\"] = experiment_name\n",
-        "pd.set_option(\"display.max_colwidth\", None)\n",
-        "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
-        "outputDf.T"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Compute"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Compute \n",
-        "\n",
-        "#### Create or Attach existing AmlCompute\n",
-        "\n",
-        "You will need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
-        "\n",
-        "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
-        "\n",
-        "#### Creation of AmlCompute takes approximately 5 minutes. \n",
-        "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
-        "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
-        "from azureml.core.compute_target import ComputeTargetException\n",
-        "\n",
-        "# Choose a name for your CPU cluster\n",
-        "amlcompute_cluster_name = \"forecast-step-cluster\"\n",
-        "\n",
-        "# Verify that cluster does not exist already\n",
-        "try:\n",
-        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
-        "    print(\"Found existing cluster, use it.\")\n",
-        "except ComputeTargetException:\n",
-        "    compute_config = AmlCompute.provisioning_configuration(\n",
-        "        vm_size=\"STANDARD_DS12_V2\", max_nodes=4\n",
-        "    )\n",
-        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
-        "compute_target.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Data\n",
-        "You are now ready to load the historical orange juice sales data. For demonstration purposes, we extract sales time-series for just a few of the stores. We will load the CSV file into a plain pandas DataFrame; the time column in the CSV is called _WeekStarting_, so it will be specially parsed into the datetime type."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "time_column_name = \"WeekStarting\"\n",
-        "train = pd.read_csv(\"oj-train.csv\", parse_dates=[time_column_name])\n",
-        "\n",
-        "train.head()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "Each row in the DataFrame holds a quantity of weekly sales for an OJ brand at a single store. The data also includes the sales price, a flag indicating if the OJ brand was advertised in the store that week, and some customer demographic information based on the store location. For historical reasons, the data also include the logarithm of the sales quantity. The Dominick's grocery data is commonly used to illustrate econometric modeling techniques where logarithms of quantities are generally preferred.    \n",
-        "\n",
-        "The task is now to build a time-series model for the _Quantity_ column. It is important to note that this dataset is comprised of many individual time-series - one for each unique combination of _Store_ and _Brand_. To distinguish the individual time-series, we define the **time_series_id_column_names** - the columns whose values determine the boundaries between time-series: "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "time_series_id_column_names = [\"Store\", \"Brand\"]\n",
-        "nseries = train.groupby(time_series_id_column_names).ngroups\n",
-        "print(\"Data contains {0} individual time-series.\".format(nseries))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Test Splitting\n",
-        "We now split the data into a training and a testing set for later forecast prediction. The test set will contain the final 4 weeks of observed sales for each time-series. The splits should be stratified by series, so we use a group-by statement on the time series identifier columns."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "n_test_periods = 4\n",
-        "\n",
-        "test = pd.read_csv(\"oj-test.csv\", parse_dates=[time_column_name])"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Upload data to datastore\n",
-        "The [Machine Learning service workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-workspace), is paired with the storage account, which contains the default data store. We will use it to upload the train and test data and create [tabular datasets](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) for training and testing. A tabular dataset defines a series of lazily-evaluated, immutable operations to load data from the data source into tabular representation."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.data.dataset_factory import TabularDatasetFactory\n",
-        "\n",
-        "datastore = ws.get_default_datastore()\n",
-        "train_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
-        "    train, target=(datastore, \"dataset/\"), name=\"dominicks_OJ_train_pipeline\"\n",
-        ")\n",
-        "\n",
-        "test_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
-        "    test, target=(datastore, \"dataset/\"), name=\"dominicks_OJ_test_pipeline\"\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Training"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Modeling\n",
-        "\n",
-        "For forecasting tasks, AutoML uses pre-processing and estimation steps that are specific to time-series. AutoML will undertake the following pre-processing steps:\n",
-        "* Detect time-series sample frequency (e.g. hourly, daily, weekly) and create new records for absent time points to make the series regular. A regular time series has a well-defined frequency and has a value at every sample point in a contiguous time span \n",
-        "* Impute missing values in the target (via forward-fill) and feature columns (using median column values) \n",
-        "* Create features based on time series identifiers to enable fixed effects across different series\n",
-        "* Create time-based features to assist in learning seasonal patterns\n",
-        "* Encode categorical variables to numeric quantities\n",
-        "\n",
-        "In this notebook, AutoML will train a single, regression-type model across **all** time-series in a given training set. This allows the model to generalize across related series. If you're looking for training multiple models for different time-series, please see the many-models notebook.\n",
-        "\n",
-        "You are almost ready to start an AutoML training job. First, we need to define the target column."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "target_column_name = \"Quantity\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Forecasting Parameters\n",
-        "To define forecasting parameters for your experiment training, you can leverage the ForecastingParameters class. The table below details the forecasting parameter we will be passing into our experiment.\n",
-        "\n",
-        "\n",
-        "|Property|Description|\n",
-        "|-|-|\n",
-        "|**time_column_name**|The name of your time column.|\n",
-        "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
-        "|**time_series_id_column_names**|The column names used to uniquely identify the time series in data that has multiple rows with the same timestamp. If the time series identifiers are not defined, the data set is assumed to be one time series.|\n",
-        "|**freq**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
-        "|**cv_step_size**|Number of periods between two consecutive cross-validation folds. The default value is \"auto\", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
-        "\n",
-        "forecasting_parameters = ForecastingParameters(\n",
-        "    time_column_name=time_column_name,\n",
-        "    forecast_horizon=n_test_periods,\n",
-        "    time_series_id_column_names=time_series_id_column_names,\n",
-        "    freq=\"W-THU\",  # Set the forecast frequency to be weekly (start on each Thursday),\n",
-        "    cv_step_size=\"auto\",\n",
-        ")\n",
-        "\n",
-        "automl_config = AutoMLConfig(\n",
-        "    task=\"forecasting\",\n",
-        "    debug_log=\"automl_oj_sales_errors.log\",\n",
-        "    primary_metric=\"normalized_mean_absolute_error\",\n",
-        "    experiment_timeout_hours=0.25,\n",
-        "    training_data=train_dataset,\n",
-        "    label_column_name=target_column_name,\n",
-        "    compute_target=compute_target,\n",
-        "    enable_early_stopping=True,\n",
-        "    n_cross_validations=\"auto\",  # Feel free to set to a small integer (>=2) if runtime is an issue.\n",
-        "    verbosity=logging.INFO,\n",
-        "    max_cores_per_iteration=-1,\n",
-        "    forecasting_parameters=forecasting_parameters,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
-        "from azureml.pipeline.steps import AutoMLStep\n",
-        "from azureml.pipeline.core import Pipeline, PipelineParameter\n",
-        "from azureml.pipeline.steps import PythonScriptStep\n",
-        "\n",
-        "metrics_output_name = \"metrics_output\"\n",
-        "best_model_output_name = \"best_model_output\"\n",
-        "model_file_name = \"model_file\"\n",
-        "metrics_data_name = \"metrics_data\"\n",
-        "\n",
-        "metrics_data = PipelineData(\n",
-        "    name=metrics_data_name,\n",
-        "    datastore=datastore,\n",
-        "    pipeline_output_name=metrics_output_name,\n",
-        "    training_output=TrainingOutput(type=\"Metrics\"),\n",
-        ")\n",
-        "model_data = PipelineData(\n",
-        "    name=model_file_name,\n",
-        "    datastore=datastore,\n",
-        "    pipeline_output_name=best_model_output_name,\n",
-        "    training_output=TrainingOutput(type=\"Model\"),\n",
-        ")\n",
-        "\n",
-        "automl_step = AutoMLStep(\n",
-        "    name=\"automl_module\",\n",
-        "    automl_config=automl_config,\n",
-        "    outputs=[metrics_data, model_data],\n",
-        "    allow_reuse=False,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Register Model Step"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Run Configuration and Environment\n",
-        "To have a pipeline step run, we first need an environment to run the jobs. The environment can be build using the following code."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core.runconfig import CondaDependencies, RunConfiguration\n",
-        "\n",
-        "# create a new RunConfig object\n",
-        "conda_run_config = RunConfiguration(framework=\"python\")\n",
-        "\n",
-        "# Set compute target to AmlCompute\n",
-        "conda_run_config.target = compute_target\n",
-        "\n",
-        "conda_run_config.docker.use_docker = True\n",
-        "\n",
-        "cd = CondaDependencies.create(\n",
-        "    pip_packages=[\n",
-        "        \"azureml-sdk[automl]\",\n",
-        "        \"applicationinsights\",\n",
-        "        \"azureml-opendatasets\",\n",
-        "        \"azureml-defaults\",\n",
-        "    ],\n",
-        "    conda_packages=[\"numpy==1.19.5\"],\n",
-        "    pin_sdk_version=False,\n",
-        ")\n",
-        "conda_run_config.environment.python.conda_dependencies = cd\n",
-        "\n",
-        "print(\"run config is ready\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "#### Step to register the model.\n",
-        "The following code generates a step to register the model to the workspace from previous step. "
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# The model name with which to register the trained model in the workspace.\n",
-        "model_name_str = \"ojmodel\"\n",
-        "model_name = PipelineParameter(\"model_name\", default_value=model_name_str)\n",
-        "\n",
-        "\n",
-        "register_model_step = PythonScriptStep(\n",
-        "    script_name=\"register_model.py\",\n",
-        "    name=\"register_model\",\n",
-        "    source_directory=\"scripts\",\n",
-        "    allow_reuse=False,\n",
-        "    arguments=[\n",
-        "        \"--model_name\",\n",
-        "        model_name,\n",
-        "        \"--model_path\",\n",
-        "        model_data,\n",
-        "        \"--ds_name\",\n",
-        "        \"dominicks_OJ_train\",\n",
-        "    ],\n",
-        "    inputs=[model_data],\n",
-        "    compute_target=compute_target,\n",
-        "    runconfig=conda_run_config,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Build the Pipeline"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_pipeline = Pipeline(\n",
-        "    description=\"training_pipeline\",\n",
-        "    workspace=ws,\n",
-        "    steps=[automl_step, register_model_step],\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Submit Pipeline Run"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_pipeline_run = experiment.submit(training_pipeline)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "training_pipeline_run.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Get metrics for each runs"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "output_dir = \"train_output\"\n",
-        "pipeline_output = training_pipeline_run.get_pipeline_output(\"metrics_output\")\n",
-        "pipeline_output.download(output_dir)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "file_path = os.path.join(output_dir, pipeline_output.path_on_datastore)\n",
-        "with open(file_path) as f:\n",
-        "    metrics = json.load(f)\n",
-        "for run_id, metrics in metrics.items():\n",
-        "    print(\"{}: {}\".format(run_id, metrics[\"normalized_root_mean_squared_error\"][0]))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Inference"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "There are several ways to do the inference, for here we will demonstrate how to use the registered model and pipeline to do the inference. (how to register a model https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.model.model?view=azure-ml-py)."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Get Inference Pipeline Environment\n",
-        "To trigger an inference pipeline run, we first need a running environment for run that contains all the appropriate packages for the model unpickling. This environment can be either assess from the training run or using the `yml` file that comes with the model."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.train.automl.run import AutoMLRun\n",
-        "\n",
-        "for step in training_pipeline_run.get_steps():\n",
-        "    if step.properties.get(\"StepType\") == \"AutoMLStep\":\n",
-        "        automl_run = AutoMLRun(experiment, step.id)\n",
-        "        break\n",
-        "\n",
-        "best_run = automl_run.get_best_child()\n",
-        "inference_env = best_run.get_environment()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "After we have the environment for the inference, we could build run config based on this environment."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "run_config = RunConfiguration()\n",
-        "run_config.environment = inference_env"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Build and submit the inference pipeline"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "The inference pipeline will create two different format of outputs, 1) a tabular dataset that contains the prediction and 2) an `OutputFileDatasetConfig` that can be used for the sequential pipeline steps."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.data import OutputFileDatasetConfig\n",
-        "\n",
-        "output_data = OutputFileDatasetConfig(name=\"prediction_result\")\n",
-        "\n",
-        "output_ds_name = \"oj-output\"\n",
-        "\n",
-        "inference_step = PythonScriptStep(\n",
-        "    name=\"infer-results\",\n",
-        "    source_directory=\"scripts\",\n",
-        "    script_name=\"infer.py\",\n",
-        "    arguments=[\n",
-        "        \"--model_name\",\n",
-        "        model_name_str,\n",
-        "        \"--ouput_dataset_name\",\n",
-        "        output_ds_name,\n",
-        "        \"--test_dataset_name\",\n",
-        "        test_dataset.name,\n",
-        "        \"--target_column_name\",\n",
-        "        target_column_name,\n",
-        "        \"--output_path\",\n",
-        "        output_data,\n",
-        "    ],\n",
-        "    compute_target=compute_target,\n",
-        "    allow_reuse=False,\n",
-        "    runconfig=run_config,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "inference_pipeline = Pipeline(ws, [inference_step])\n",
-        "inference_run = experiment.submit(inference_pipeline)"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "inference_run.wait_for_completion(show_output=True)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### Get the predicted data"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "from azureml.core import Dataset\n",
-        "\n",
-        "inference_ds = Dataset.get_by_name(ws, output_ds_name)\n",
-        "inference_df = inference_ds.to_pandas_dataframe()\n",
-        "inference_df.tail(5)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Schedule Pipeline"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "This section is about how to schedule a pipeline for periodically predictions. For more info about pipeline schedule and pipeline endpoint, please follow this [notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb)."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "inference_published_pipeline = inference_pipeline.publish(\n",
-        "    name=\"OJ Inference Test\", description=\"OJ Inference Test\"\n",
-        ")\n",
-        "print(\"Newly published pipeline id: {}\".format(inference_published_pipeline.id))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "If `test_dataset` is going to refresh every 4 weeks before Friday 16:00 and we want to predict every 4 weeks (forecast_horizon), we can schedule our pipeline to run every 4 weeks at 16:00 to get daily inference results. You can refresh your test dataset (a newer version will be created) periodically when new data is available (i.e. target column in test dataset would have values in the beginning as context data, and followed by NaNs to be predicted). The inference pipeline will pick up context to further improve the forecast accuracy."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# schedule\n",
-        "\n",
-        "from azureml.pipeline.core.schedule import ScheduleRecurrence, Schedule\n",
-        "\n",
-        "recurrence = ScheduleRecurrence(\n",
-        "    frequency=\"Week\", interval=4, week_days=[\"Friday\"], hours=[16], minutes=[0]\n",
-        ")\n",
-        "\n",
-        "schedule = Schedule.create(\n",
-        "    workspace=ws,\n",
-        "    name=\"OJ_Inference_schedule\",\n",
-        "    pipeline_id=inference_published_pipeline.id,\n",
-        "    experiment_name=\"Schedule-run-OJ\",\n",
-        "    recurrence=recurrence,\n",
-        "    wait_for_provisioning=True,\n",
-        "    description=\"Schedule Run\",\n",
-        ")\n",
-        "\n",
-        "# You may want to make sure that the schedule is provisioned properly\n",
-        "# before making any further changes to the schedule\n",
-        "\n",
-        "print(\"Created schedule with id: {}\".format(schedule.id))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "### [Optional] Disable schedule"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "schedule.disable()"
-      ]
-    }
-  ],
-  "metadata": {
-    "authors": [
-      {
-        "name": "jialiu"
-      }
-    ],
-    "category": "tutorial",
-    "celltoolbar": "Raw Cell Format",
-    "compute": [
-      "Remote"
-    ],
-    "datasets": [
-      "Orange Juice Sales"
-    ],
-    "deployment": [
-      "Azure Container Instance"
-    ],
-    "exclude_from_index": false,
-    "framework": [
-      "Azure ML AutoML"
-    ],
-    "friendly_name": "Forecasting orange juice sales with deployment",
-    "index_order": 1,
-    "kernelspec": {
-      "display_name": "Python 3.8 - AzureML",
-      "language": "python",
-      "name": "python38-azureml"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.8.5"
-    },
-    "tags": [
-      "None"
-    ],
-    "task": "Forecasting",
-    "vscode": {
-      "interpreter": {
-        "hash": "6bd77c88278e012ef31757c15997a7bea8c943977c43d6909403c00ae11d43ca"
-      }
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 4
-}

how-to-use-azureml/automated-machine-learning/forecasting-pipelines/oj-test.csv

@@ -1,37 +0,0 @@
-WeekStarting,Store,Brand,Advert,Price,Age60,COLLEGE,INCOME,Hincome150,Large HH,Minorities,WorkingWoman,SSTRDIST,SSTRVOL,CPDIST5,CPWVOL5
-1992-09-10,2,dominicks,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-10,2,minute.maid,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-10,2,tropicana,0,2.64,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-10,5,dominicks,0,1.85,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-10,5,minute.maid,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-10,5,tropicana,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-10,8,dominicks,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-10,8,minute.maid,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-10,8,tropicana,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-17,2,dominicks,0,1.77,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-17,2,minute.maid,0,2.83,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-17,2,tropicana,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-17,5,dominicks,0,1.85,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-17,5,minute.maid,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-17,5,tropicana,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-17,8,dominicks,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-17,8,minute.maid,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-17,8,tropicana,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-24,2,dominicks,0,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-24,2,minute.maid,0,2.67,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-24,2,tropicana,1,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-24,5,dominicks,0,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-24,5,minute.maid,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-24,5,tropicana,1,2.78,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-24,8,dominicks,0,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-24,8,minute.maid,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-24,8,tropicana,1,2.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-10-01,2,dominicks,0,1.82,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-10-01,2,minute.maid,1,2.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-10-01,2,tropicana,0,2.97,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-10-01,5,dominicks,0,1.85,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-10-01,5,minute.maid,1,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-10-01,5,tropicana,0,2.78,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-10-01,8,dominicks,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-10-01,8,minute.maid,1,2.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-10-01,8,tropicana,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947

how-to-use-azureml/automated-machine-learning/forecasting-pipelines/oj-train.csv

@@ -1,997 +0,0 @@
-WeekStarting,Store,Brand,Quantity,Advert,Price,Age60,COLLEGE,INCOME,Hincome150,Large HH,Minorities,WorkingWoman,SSTRDIST,SSTRVOL,CPDIST5,CPWVOL5
-1990-06-14,2,dominicks,10560,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-06-14,2,minute.maid,4480,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-06-14,2,tropicana,8256,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-06-14,5,dominicks,1792,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-14,5,minute.maid,4224,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-14,5,tropicana,5888,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-14,8,dominicks,14336,1,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-14,8,minute.maid,6080,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-14,8,tropicana,8896,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-21,8,dominicks,6400,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-21,8,minute.maid,51968,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-21,8,tropicana,7296,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-28,5,dominicks,2496,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-28,5,minute.maid,4352,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-28,5,tropicana,6976,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-06-28,8,dominicks,3968,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-28,8,minute.maid,4928,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-06-28,8,tropicana,10368,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-05,5,dominicks,2944,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-05,5,minute.maid,4928,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-05,5,tropicana,6528,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-05,8,dominicks,4352,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-05,8,minute.maid,5312,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-05,8,tropicana,6976,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-12,5,dominicks,1024,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-12,5,minute.maid,31168,1,2.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-12,5,tropicana,4928,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-12,8,dominicks,3520,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-12,8,minute.maid,39424,1,2.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-12,8,tropicana,6464,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-19,8,dominicks,6464,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-19,8,minute.maid,5568,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-19,8,tropicana,8192,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-26,2,dominicks,8000,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-07-26,2,minute.maid,4672,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-07-26,2,tropicana,6144,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-07-26,5,dominicks,4224,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-26,5,minute.maid,10048,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-26,5,tropicana,5312,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-07-26,8,dominicks,5952,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-26,8,minute.maid,14592,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-07-26,8,tropicana,7936,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-02,2,dominicks,6848,1,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-02,2,minute.maid,20160,1,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-02,2,tropicana,3840,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-02,5,dominicks,4544,1,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-02,5,minute.maid,21760,1,2.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-02,5,tropicana,5120,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-02,8,dominicks,8832,1,2.09,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-02,8,minute.maid,22208,1,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-02,8,tropicana,6656,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-09,2,dominicks,2880,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-09,2,minute.maid,2688,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-09,2,tropicana,8000,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-09,5,dominicks,1728,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-09,5,minute.maid,4544,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-09,5,tropicana,7936,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-09,8,dominicks,7232,0,2.09,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-09,8,minute.maid,5760,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-09,8,tropicana,8256,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-16,5,dominicks,1216,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-16,5,minute.maid,52224,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-16,5,tropicana,6080,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-16,8,dominicks,5504,0,2.09,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-16,8,minute.maid,54016,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-16,8,tropicana,5568,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-23,2,dominicks,1600,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-23,2,minute.maid,3008,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-23,2,tropicana,8896,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-23,5,dominicks,1152,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-23,5,minute.maid,3584,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-23,5,tropicana,4160,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-23,8,dominicks,4800,0,2.09,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-23,8,minute.maid,5824,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-23,8,tropicana,7488,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-30,2,dominicks,25344,1,1.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-30,2,minute.maid,4672,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-30,2,tropicana,7168,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-08-30,5,dominicks,30144,1,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-30,5,minute.maid,5120,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-30,5,tropicana,5888,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-08-30,8,dominicks,52672,1,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-30,8,minute.maid,6528,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-08-30,8,tropicana,6144,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-06,2,dominicks,10752,0,1.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-06,2,minute.maid,2752,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-06,2,tropicana,10880,0,3.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-06,5,dominicks,8960,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-06,5,minute.maid,4416,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-06,5,tropicana,9536,0,3.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-06,8,dominicks,16448,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-06,8,minute.maid,5440,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-06,8,tropicana,11008,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-13,2,dominicks,6656,0,1.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-13,2,minute.maid,26176,1,2.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-13,2,tropicana,7744,0,3.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-13,5,dominicks,8192,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-13,5,minute.maid,30208,1,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-13,5,tropicana,8320,0,3.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-13,8,dominicks,19072,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-13,8,minute.maid,36544,1,2.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-13,8,tropicana,5760,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-20,2,dominicks,6592,0,1.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-20,2,minute.maid,3712,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-20,2,tropicana,8512,0,3.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-09-20,5,dominicks,6528,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-20,5,minute.maid,4160,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-20,5,tropicana,8000,0,3.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-20,8,dominicks,13376,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-20,8,minute.maid,3776,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-20,8,tropicana,10112,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-27,5,dominicks,34688,1,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-27,5,minute.maid,4992,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-27,5,tropicana,5824,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-09-27,8,dominicks,61440,1,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-27,8,minute.maid,5504,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-09-27,8,tropicana,8448,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-04,5,dominicks,4672,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-04,5,minute.maid,13952,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-04,5,tropicana,10624,1,3.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-04,8,dominicks,13760,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-04,8,minute.maid,12416,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-04,8,tropicana,8448,1,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-11,2,dominicks,1728,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-11,2,minute.maid,30656,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-11,2,tropicana,5504,0,3.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-11,5,dominicks,1088,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-11,5,minute.maid,47680,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-11,5,tropicana,6656,0,3.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-11,8,dominicks,3136,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-11,8,minute.maid,53696,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-11,8,tropicana,7424,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-18,2,dominicks,33792,1,1.24,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-18,2,minute.maid,3840,0,2.98,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-18,2,tropicana,5888,0,3.56,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-18,5,dominicks,69440,1,1.24,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-18,5,minute.maid,7616,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-18,5,tropicana,5184,0,3.51,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-18,8,dominicks,186176,1,1.14,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-18,8,minute.maid,5696,0,2.51,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-18,8,tropicana,5824,0,3.04,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-25,2,dominicks,1920,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-25,2,minute.maid,2816,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-25,2,tropicana,8384,0,3.56,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-10-25,5,dominicks,1280,0,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-25,5,minute.maid,8896,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-25,5,tropicana,4928,0,3.51,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-10-25,8,dominicks,3712,0,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-25,8,minute.maid,4864,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-10-25,8,tropicana,6656,0,3.04,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-01,2,dominicks,8960,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-01,2,minute.maid,23104,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-01,2,tropicana,5952,0,3.56,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-01,5,dominicks,35456,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-01,5,minute.maid,28544,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-01,5,tropicana,5888,0,3.51,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-01,8,dominicks,35776,1,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-01,8,minute.maid,37184,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-01,8,tropicana,6272,0,3.04,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-08,2,dominicks,11392,0,1.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-08,2,minute.maid,3392,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-08,2,tropicana,6848,0,3.56,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-08,5,dominicks,13824,0,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-08,5,minute.maid,5440,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-08,5,tropicana,5312,0,3.51,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-08,8,dominicks,26880,0,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-08,8,minute.maid,5504,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-08,8,tropicana,6912,0,3.04,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-15,2,dominicks,28416,0,0.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-15,2,minute.maid,26304,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-15,2,tropicana,9216,0,3.87,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-15,5,dominicks,14208,0,0.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-15,5,minute.maid,52416,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-15,5,tropicana,9984,0,3.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-15,8,dominicks,71680,0,0.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-15,8,minute.maid,51008,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-15,8,tropicana,10496,0,3.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-22,2,dominicks,17152,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-22,2,minute.maid,6336,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-22,2,tropicana,12160,0,2.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-22,5,dominicks,29312,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-22,5,minute.maid,11712,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-22,5,tropicana,8448,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-22,8,dominicks,25088,1,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-22,8,minute.maid,11072,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-22,8,tropicana,11840,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-29,2,dominicks,26560,1,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-29,2,minute.maid,9920,0,3.17,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-29,2,tropicana,12672,0,2.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-11-29,5,dominicks,52992,1,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-29,5,minute.maid,13952,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-29,5,tropicana,10880,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-11-29,8,dominicks,91456,1,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-29,8,minute.maid,12160,0,2.62,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-11-29,8,tropicana,9664,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-06,2,dominicks,6336,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-06,2,minute.maid,25280,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-06,2,tropicana,6528,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-06,5,dominicks,15680,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-06,5,minute.maid,36160,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-06,5,tropicana,5696,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-06,8,dominicks,23808,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-06,8,minute.maid,30528,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-06,8,tropicana,6272,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-13,2,dominicks,26368,1,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-13,2,minute.maid,14848,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-13,2,tropicana,6144,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-13,5,dominicks,43520,1,1.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-13,5,minute.maid,12864,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-13,5,tropicana,5696,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-13,8,dominicks,89856,1,1.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-13,8,minute.maid,12096,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-13,8,tropicana,7168,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-20,2,dominicks,896,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-20,2,minute.maid,12288,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-20,2,tropicana,21120,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-20,5,dominicks,3904,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-20,5,minute.maid,22208,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-20,5,tropicana,32384,0,2.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-20,8,dominicks,12224,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-20,8,minute.maid,16448,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-20,8,tropicana,29504,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-27,2,dominicks,1472,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-27,2,minute.maid,6272,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-27,2,tropicana,12416,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1990-12-27,5,dominicks,896,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-27,5,minute.maid,9984,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-27,5,tropicana,10752,0,2.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1990-12-27,8,dominicks,3776,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-27,8,minute.maid,9344,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1990-12-27,8,tropicana,8704,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-03,2,dominicks,1344,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-03,2,minute.maid,9152,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-03,2,tropicana,9472,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-03,5,dominicks,2240,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-03,5,minute.maid,14016,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-03,5,tropicana,6912,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-03,8,dominicks,13824,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-03,8,minute.maid,16128,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-03,8,tropicana,9280,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-10,2,dominicks,111680,1,0.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-10,2,minute.maid,4160,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-10,2,tropicana,17920,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-10,5,dominicks,125760,1,0.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-10,5,minute.maid,6080,0,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-10,5,tropicana,13440,0,2.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-10,8,dominicks,251072,1,0.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-10,8,minute.maid,5376,0,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-10,8,tropicana,12224,0,2.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-17,2,dominicks,1856,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-17,2,minute.maid,10176,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-17,2,tropicana,9408,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-17,5,dominicks,1408,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-17,5,minute.maid,7808,0,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-17,5,tropicana,7808,0,2.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-17,8,dominicks,4864,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-17,8,minute.maid,6656,0,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-17,8,tropicana,10368,0,2.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-24,2,dominicks,5568,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-24,2,minute.maid,29056,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-24,2,tropicana,6272,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-24,5,dominicks,7232,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-24,5,minute.maid,40896,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-24,5,tropicana,5248,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-24,8,dominicks,10176,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-24,8,minute.maid,59712,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-24,8,tropicana,8128,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-31,2,dominicks,32064,1,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-31,2,minute.maid,7104,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-31,2,tropicana,6912,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-01-31,5,dominicks,41216,1,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-31,5,minute.maid,6272,0,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-31,5,tropicana,6208,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-01-31,8,dominicks,105344,1,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-31,8,minute.maid,9856,0,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-01-31,8,tropicana,5952,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-07,2,dominicks,4352,0,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-07,2,minute.maid,7488,0,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-07,2,tropicana,16768,0,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-07,5,dominicks,9024,0,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-07,5,minute.maid,7872,0,2.41,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-07,5,tropicana,21440,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-07,8,dominicks,33600,0,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-07,8,minute.maid,6720,0,2.12,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-07,8,tropicana,21696,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-14,2,dominicks,704,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-14,2,minute.maid,4224,0,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-14,2,tropicana,6272,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-14,5,dominicks,1600,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-14,5,minute.maid,6144,0,2.41,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-14,5,tropicana,7360,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-14,8,dominicks,4736,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-14,8,minute.maid,4224,0,2.12,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-14,8,tropicana,7808,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-21,2,dominicks,13760,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-21,2,minute.maid,8960,0,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-21,2,tropicana,7936,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-21,5,dominicks,2496,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-21,5,minute.maid,8448,0,2.41,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-21,5,tropicana,6720,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-21,8,dominicks,10304,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-21,8,minute.maid,9728,0,2.12,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-21,8,tropicana,8128,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-28,2,dominicks,43328,1,1.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-28,2,minute.maid,22464,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-28,2,tropicana,6144,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-02-28,5,dominicks,6336,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-28,5,minute.maid,18688,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-28,5,tropicana,6656,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-02-28,8,dominicks,5056,1,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-28,8,minute.maid,40320,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-02-28,8,tropicana,7424,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-07,2,dominicks,57600,1,1.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-07,2,minute.maid,3840,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-07,2,tropicana,7936,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-07,5,dominicks,56384,1,1.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-07,5,minute.maid,6272,0,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-07,5,tropicana,6016,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-07,8,dominicks,179968,1,0.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-07,8,minute.maid,5120,0,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-07,8,tropicana,5952,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-14,2,dominicks,704,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-14,2,minute.maid,12992,0,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-14,2,tropicana,7808,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-14,5,dominicks,1600,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-14,5,minute.maid,12096,0,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-14,5,tropicana,6144,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-14,8,dominicks,4992,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-14,8,minute.maid,19264,0,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-14,8,tropicana,7616,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-21,2,dominicks,6016,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-21,2,minute.maid,70144,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-21,2,tropicana,6080,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-21,5,dominicks,2944,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-21,5,minute.maid,73216,1,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-21,5,tropicana,4928,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-21,8,dominicks,6400,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-21,8,minute.maid,170432,1,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-21,8,tropicana,5312,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-28,2,dominicks,10368,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-28,2,minute.maid,21248,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-28,2,tropicana,42176,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-03-28,5,dominicks,13504,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-28,5,minute.maid,18944,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-28,5,tropicana,67712,1,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-03-28,8,dominicks,14912,1,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-28,8,minute.maid,39680,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-03-28,8,tropicana,161792,1,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-04,2,dominicks,12608,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-04,2,minute.maid,5696,1,2.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-04,2,tropicana,4928,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-04,5,dominicks,5376,0,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-04,5,minute.maid,6400,1,2.46,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-04,5,tropicana,8640,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-04,8,dominicks,34624,0,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-04,8,minute.maid,8128,1,2.17,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-04,8,tropicana,17280,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-11,2,dominicks,6336,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-11,2,minute.maid,7680,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-11,2,tropicana,29504,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-11,5,dominicks,6656,0,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-11,5,minute.maid,8640,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-11,5,tropicana,35520,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-11,8,dominicks,10368,0,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-11,8,minute.maid,9088,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-11,8,tropicana,47040,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-18,2,dominicks,140736,1,0.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-18,2,minute.maid,6336,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-18,2,tropicana,9984,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-18,5,dominicks,95680,1,0.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-18,5,minute.maid,7296,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-18,5,tropicana,9664,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-18,8,dominicks,194880,1,0.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-18,8,minute.maid,6720,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-18,8,tropicana,14464,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-25,2,dominicks,960,1,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-25,2,minute.maid,8576,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-25,2,tropicana,35200,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-04-25,5,dominicks,896,1,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-25,5,minute.maid,12480,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-25,5,tropicana,49088,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-04-25,8,dominicks,5696,1,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-25,8,minute.maid,7552,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-04-25,8,tropicana,52928,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-02,2,dominicks,1216,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-02,2,minute.maid,15104,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-02,2,tropicana,23936,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-02,5,dominicks,1728,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-02,5,minute.maid,14144,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-02,5,tropicana,14912,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-02,8,dominicks,7168,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-02,8,minute.maid,24768,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-02,8,tropicana,21184,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-09,2,dominicks,1664,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-09,2,minute.maid,76480,1,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-09,2,tropicana,7104,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-09,5,dominicks,1280,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-09,5,minute.maid,88256,1,1.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-09,5,tropicana,6464,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-09,8,dominicks,2880,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-09,8,minute.maid,183296,1,1.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-09,8,tropicana,7360,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-16,2,dominicks,4992,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-16,2,minute.maid,5056,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-16,2,tropicana,24512,1,2.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-16,5,dominicks,5696,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-16,5,minute.maid,6848,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-16,5,tropicana,25024,1,2.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-16,8,dominicks,12288,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-16,8,minute.maid,8896,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-16,8,tropicana,15744,1,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-05-23,2,dominicks,27968,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-23,2,minute.maid,4736,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-23,2,tropicana,6336,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-23,5,dominicks,28288,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-23,5,minute.maid,7808,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-23,5,tropicana,6272,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-30,2,dominicks,12160,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-30,2,minute.maid,4480,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-30,2,tropicana,6080,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-05-30,5,dominicks,4864,0,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-30,5,minute.maid,6272,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-05-30,5,tropicana,5056,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-06,2,dominicks,2240,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-06,2,minute.maid,4032,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-06,2,tropicana,33536,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-06,5,dominicks,2880,0,2.09,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-06,5,minute.maid,6144,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-06,5,tropicana,47616,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-06,8,dominicks,9280,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-06,8,minute.maid,6656,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-06,8,tropicana,46912,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-13,2,dominicks,5504,1,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-13,2,minute.maid,14784,1,1.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-13,2,tropicana,13248,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-13,5,dominicks,5760,1,1.41,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-13,5,minute.maid,27776,1,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-13,5,tropicana,13888,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-13,8,dominicks,25856,1,1.26,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-13,8,minute.maid,35456,1,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-13,8,tropicana,18240,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-20,2,dominicks,8832,0,1.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-20,2,minute.maid,12096,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-20,2,tropicana,6208,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-20,5,dominicks,15040,0,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-20,5,minute.maid,20800,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-20,5,tropicana,6144,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-20,8,dominicks,19264,0,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-20,8,minute.maid,17408,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-20,8,tropicana,6464,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-27,2,dominicks,2624,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-27,2,minute.maid,41792,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-27,2,tropicana,10624,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-06-27,5,dominicks,5120,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-27,5,minute.maid,45696,1,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-27,5,tropicana,9344,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-06-27,8,dominicks,6848,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-27,8,minute.maid,75520,1,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-06-27,8,tropicana,8512,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-04,2,dominicks,10432,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-04,2,minute.maid,10560,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-04,2,tropicana,44672,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-04,5,dominicks,3264,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-04,5,minute.maid,14336,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-04,5,tropicana,32896,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-04,8,dominicks,12928,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-04,8,minute.maid,21632,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-04,8,tropicana,28416,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-11,5,dominicks,9536,1,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-11,5,minute.maid,4928,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-11,5,tropicana,21056,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-11,8,dominicks,44032,1,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-11,8,minute.maid,8384,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-11,8,tropicana,16960,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-18,2,dominicks,8320,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-18,2,minute.maid,4224,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-18,2,tropicana,20096,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-18,5,dominicks,6208,0,1.59,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-18,5,minute.maid,4608,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-18,5,tropicana,15360,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-18,8,dominicks,25408,0,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-18,8,minute.maid,9920,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-18,8,tropicana,8320,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-25,2,dominicks,6784,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-25,2,minute.maid,2880,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-25,2,tropicana,9152,1,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-07-25,5,dominicks,6592,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-25,5,minute.maid,5248,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-25,5,tropicana,8000,1,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-07-25,8,dominicks,38336,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-25,8,minute.maid,6592,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-07-25,8,tropicana,11136,1,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-01,2,dominicks,60544,1,0.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-01,2,minute.maid,3968,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-01,2,tropicana,21952,0,2.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-01,5,dominicks,63552,1,0.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-01,5,minute.maid,4224,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-01,5,tropicana,21120,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-01,8,dominicks,152384,1,0.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-01,8,minute.maid,7168,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-01,8,tropicana,27712,0,2.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-08,2,dominicks,20608,0,0.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-08,2,minute.maid,3712,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-08,2,tropicana,13568,0,2.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-08,5,dominicks,27968,0,0.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-08,5,minute.maid,4288,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-08,5,tropicana,11904,0,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-08,8,dominicks,54464,0,0.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-08,8,minute.maid,6208,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-08,8,tropicana,7744,0,2.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-15,5,dominicks,21760,1,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-15,5,minute.maid,16896,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-15,5,tropicana,5056,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-15,8,dominicks,47680,1,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-15,8,minute.maid,30528,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-15,8,tropicana,5184,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-22,5,dominicks,2688,0,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-22,5,minute.maid,77184,1,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-22,5,tropicana,4608,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-22,8,dominicks,14720,0,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-22,8,minute.maid,155840,1,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-22,8,tropicana,6272,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-29,2,dominicks,16064,0,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-29,2,minute.maid,2816,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-29,2,tropicana,4160,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-08-29,5,dominicks,10432,0,1.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-29,5,minute.maid,5184,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-29,5,tropicana,6016,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-08-29,8,dominicks,53248,0,1.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-29,8,minute.maid,10752,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-08-29,8,tropicana,7744,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-05,2,dominicks,12480,0,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-05,2,minute.maid,4288,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-05,2,tropicana,39424,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-05,5,dominicks,9792,0,1.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-05,5,minute.maid,5248,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-05,5,tropicana,50752,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-05,8,dominicks,40576,0,1.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-05,8,minute.maid,6976,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-05,8,tropicana,53184,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-12,2,dominicks,17024,0,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-12,2,minute.maid,18240,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-12,2,tropicana,5632,0,3.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-12,5,dominicks,8448,0,1.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-12,5,minute.maid,20672,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-12,5,tropicana,5632,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-12,8,dominicks,25856,0,1.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-12,8,minute.maid,31872,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-12,8,tropicana,6784,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-19,2,dominicks,13440,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-19,2,minute.maid,7360,0,1.95,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-19,2,tropicana,9024,1,2.68,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-19,8,dominicks,24064,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-19,8,minute.maid,5312,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-19,8,tropicana,8000,1,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-26,2,dominicks,10112,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-26,2,minute.maid,7808,0,1.83,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-26,2,tropicana,6016,0,3.44,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-09-26,5,dominicks,6912,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-26,5,minute.maid,12352,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-26,5,tropicana,6400,0,3.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-09-26,8,dominicks,15680,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-26,8,minute.maid,33344,0,1.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-09-26,8,tropicana,6592,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-03,2,dominicks,9088,0,1.56,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-03,2,minute.maid,13504,0,1.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-03,2,tropicana,7744,0,3.14,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-03,5,dominicks,8256,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-03,5,minute.maid,12032,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-03,5,tropicana,5440,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-03,8,dominicks,16576,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-03,8,minute.maid,13504,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-03,8,tropicana,5248,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-10,2,dominicks,22848,1,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-10,2,minute.maid,10048,0,1.91,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-10,2,tropicana,6784,0,3.07,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-10,5,dominicks,28672,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-10,5,minute.maid,13440,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-10,5,tropicana,8128,0,2.94,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-10,8,dominicks,49664,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-10,8,minute.maid,13504,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-10,8,tropicana,6592,0,2.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-17,2,dominicks,6976,0,1.65,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-17,2,minute.maid,135936,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-17,2,tropicana,6784,0,3.07,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-17,8,dominicks,10752,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-17,8,minute.maid,335808,1,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-17,8,tropicana,5888,0,2.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-24,2,dominicks,4160,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-24,2,minute.maid,5056,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-24,2,tropicana,6272,0,3.07,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-24,5,dominicks,4416,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-24,5,minute.maid,5824,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-24,5,tropicana,7232,0,2.94,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-24,8,dominicks,9792,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-24,8,minute.maid,13120,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-24,8,tropicana,6336,0,2.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-31,2,dominicks,3328,0,1.83,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-31,2,minute.maid,27968,0,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-31,2,tropicana,5312,0,3.07,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-10-31,5,dominicks,1856,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-31,5,minute.maid,50112,0,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-31,5,tropicana,7168,0,2.94,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-10-31,8,dominicks,7104,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-31,8,minute.maid,49664,0,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-10-31,8,tropicana,5888,0,2.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-07,2,dominicks,12096,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-07,2,minute.maid,4736,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-07,2,tropicana,9216,0,3.11,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-07,5,dominicks,6528,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-07,5,minute.maid,5184,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-07,5,tropicana,7872,0,2.94,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-07,8,dominicks,9216,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-07,8,minute.maid,10880,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-07,8,tropicana,6080,0,2.94,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-14,2,dominicks,6208,0,1.76,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-14,2,minute.maid,7808,0,2.14,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-14,2,tropicana,7296,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-14,5,dominicks,6080,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-14,5,minute.maid,8384,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-14,5,tropicana,7552,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-14,8,dominicks,12608,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-14,8,minute.maid,9984,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-14,8,tropicana,6848,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-21,2,dominicks,3008,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-21,2,minute.maid,12480,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-21,2,tropicana,34240,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-21,5,dominicks,3456,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-21,5,minute.maid,10112,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-21,5,tropicana,69504,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-21,8,dominicks,16448,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-21,8,minute.maid,9216,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-21,8,tropicana,54016,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-28,2,dominicks,19456,1,1.5,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-28,2,minute.maid,9664,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-28,2,tropicana,7168,0,2.64,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-11-28,5,dominicks,25856,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-28,5,minute.maid,8384,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-28,5,tropicana,8960,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-11-28,8,dominicks,27968,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-28,8,minute.maid,7680,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-11-28,8,tropicana,10368,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-05,2,dominicks,16768,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-05,2,minute.maid,7168,0,2.06,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-05,2,tropicana,6080,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-05,5,dominicks,25728,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-05,5,minute.maid,11456,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-05,5,tropicana,6912,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-05,8,dominicks,37824,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-05,8,minute.maid,7296,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-05,8,tropicana,5568,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-12,2,dominicks,13568,1,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-12,2,minute.maid,4480,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-12,2,tropicana,5120,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-12,5,dominicks,23552,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-12,5,minute.maid,5952,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-12,5,tropicana,6656,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-12,8,dominicks,33664,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-12,8,minute.maid,8192,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-12,8,tropicana,4864,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-19,2,dominicks,6080,0,1.61,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-19,2,minute.maid,5952,0,2.22,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-19,2,tropicana,8320,0,2.74,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-19,5,dominicks,2944,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-19,5,minute.maid,8512,0,2.26,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-19,5,tropicana,8192,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-19,8,dominicks,17728,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-19,8,minute.maid,6080,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-19,8,tropicana,7232,0,2.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-26,2,dominicks,10432,1,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-26,2,minute.maid,21696,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-26,2,tropicana,17728,0,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1991-12-26,5,dominicks,5888,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-26,5,minute.maid,27968,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-26,5,tropicana,13440,0,2.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1991-12-26,8,dominicks,25088,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-26,8,minute.maid,15040,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1991-12-26,8,tropicana,15232,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-02,2,dominicks,11712,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-02,2,minute.maid,12032,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-02,2,tropicana,13120,0,2.35,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-02,5,dominicks,6848,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-02,5,minute.maid,24000,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-02,5,tropicana,12160,0,2.39,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-02,8,dominicks,13184,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-02,8,minute.maid,9472,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-02,8,tropicana,47040,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-09,2,dominicks,4032,0,1.76,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-09,2,minute.maid,7040,0,2.12,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-09,2,tropicana,13120,0,2.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-09,5,dominicks,1792,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-09,5,minute.maid,6848,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-09,5,tropicana,11840,0,2.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-09,8,dominicks,3136,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-09,8,minute.maid,5888,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-09,8,tropicana,9280,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-16,2,dominicks,6336,0,1.82,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-16,2,minute.maid,10240,1,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-16,2,tropicana,9792,0,2.43,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-16,5,dominicks,5248,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-16,5,minute.maid,15104,1,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-16,5,tropicana,8640,0,2.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-16,8,dominicks,5696,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-16,8,minute.maid,14336,1,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-16,8,tropicana,6720,0,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-23,2,dominicks,13632,0,1.47,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-23,2,minute.maid,6848,1,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-23,2,tropicana,3520,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-23,5,dominicks,16768,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-23,5,minute.maid,11392,1,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-23,5,tropicana,5888,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-23,8,dominicks,19008,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-23,8,minute.maid,11712,1,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-23,8,tropicana,5056,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-30,2,dominicks,45120,0,1.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-30,2,minute.maid,3968,0,2.61,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-30,2,tropicana,5504,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-01-30,5,dominicks,52160,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-30,5,minute.maid,5824,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-30,5,tropicana,7424,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-01-30,8,dominicks,121664,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-30,8,minute.maid,7936,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-01-30,8,tropicana,6080,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-06,2,dominicks,9984,0,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-06,2,minute.maid,5888,0,2.26,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-06,2,tropicana,6720,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-06,5,dominicks,16640,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-06,5,minute.maid,7488,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-06,5,tropicana,5632,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-06,8,dominicks,38848,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-06,8,minute.maid,5184,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-06,8,tropicana,10496,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-13,2,dominicks,4800,0,1.82,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-13,2,minute.maid,6208,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-13,2,tropicana,20224,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-13,5,dominicks,1344,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-13,5,minute.maid,8320,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-13,5,tropicana,33600,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-13,8,dominicks,6144,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-13,8,minute.maid,7168,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-13,8,tropicana,39040,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-20,2,dominicks,11776,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-20,2,minute.maid,72256,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-20,2,tropicana,5056,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-20,5,dominicks,4608,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-20,5,minute.maid,99904,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-20,5,tropicana,5376,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-20,8,dominicks,13632,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-20,8,minute.maid,216064,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-20,8,tropicana,4480,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-27,2,dominicks,11584,0,1.54,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-27,2,minute.maid,11520,0,2.11,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-27,2,tropicana,43584,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-02-27,5,dominicks,12672,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-27,5,minute.maid,6976,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-27,5,tropicana,54272,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-02-27,8,dominicks,9792,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-27,8,minute.maid,15040,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-02-27,8,tropicana,61760,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-05,2,dominicks,51264,1,1.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-05,2,minute.maid,5824,0,2.35,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-05,2,tropicana,25728,0,1.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-05,5,dominicks,48640,1,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-05,5,minute.maid,9984,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-05,5,tropicana,33600,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-05,8,dominicks,86912,1,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-05,8,minute.maid,11840,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-05,8,tropicana,15360,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-12,2,dominicks,14976,0,1.44,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-12,2,minute.maid,19392,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-12,2,tropicana,31808,0,1.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-12,5,dominicks,13248,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-12,5,minute.maid,32832,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-12,5,tropicana,24448,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-12,8,dominicks,24512,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-12,8,minute.maid,25472,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-12,8,tropicana,54976,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-19,2,dominicks,30784,0,1.59,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-19,2,minute.maid,9536,0,2.1,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-19,2,tropicana,20736,0,1.91,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-19,5,dominicks,29248,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-19,5,minute.maid,8128,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-19,5,tropicana,22784,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-19,8,dominicks,58048,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-19,8,minute.maid,16384,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-19,8,tropicana,34368,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-26,2,dominicks,12480,0,1.6,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-26,2,minute.maid,5312,0,2.28,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-26,2,tropicana,15168,0,2.81,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-03-26,5,dominicks,4608,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-26,5,minute.maid,6464,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-26,5,tropicana,19008,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-03-26,8,dominicks,13952,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-26,8,minute.maid,20480,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-03-26,8,tropicana,10752,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-02,2,dominicks,3264,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-02,2,minute.maid,14528,1,1.9,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-02,2,tropicana,28096,1,2.5,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-02,5,dominicks,3136,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-02,5,minute.maid,36800,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-02,5,tropicana,15808,1,2.5,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-02,8,dominicks,15168,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-02,8,minute.maid,34688,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-02,8,tropicana,20096,1,2.5,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-09,2,dominicks,8768,0,1.48,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-09,2,minute.maid,12416,0,2.12,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-09,2,tropicana,12416,0,2.58,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-09,5,dominicks,13184,0,1.58,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-09,5,minute.maid,12928,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-09,5,tropicana,14144,0,2.5,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-09,8,dominicks,14592,0,1.58,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-09,8,minute.maid,22400,0,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-09,8,tropicana,16192,0,2.5,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-16,2,dominicks,70848,1,1.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-16,2,minute.maid,5376,0,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-16,2,tropicana,5376,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-16,5,dominicks,67712,1,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-16,5,minute.maid,7424,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-16,5,tropicana,9600,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-16,8,dominicks,145088,1,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-16,8,minute.maid,7808,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-16,8,tropicana,6528,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-23,2,dominicks,18560,0,1.42,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-23,2,minute.maid,19008,1,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-23,2,tropicana,9792,0,2.67,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-23,5,dominicks,18880,0,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-23,5,minute.maid,34176,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-23,5,tropicana,10112,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-23,8,dominicks,43712,0,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-23,8,minute.maid,48064,1,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-23,8,tropicana,8320,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-30,2,dominicks,9152,0,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-30,2,minute.maid,3904,0,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-30,2,tropicana,16960,1,2.39,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-04-30,5,dominicks,6208,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-30,5,minute.maid,4160,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-30,5,tropicana,31872,1,2.24,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-04-30,8,dominicks,20608,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-30,8,minute.maid,7360,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-04-30,8,tropicana,30784,1,2.16,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-07,2,dominicks,9600,0,2.0,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-07,2,minute.maid,6336,0,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-07,2,tropicana,8320,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-07,5,dominicks,5952,0,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-07,5,minute.maid,5952,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-07,5,tropicana,9280,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-07,8,dominicks,18752,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-07,8,minute.maid,6272,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-07,8,tropicana,18048,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-14,2,dominicks,4800,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-14,2,minute.maid,5440,0,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-14,2,tropicana,6912,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-14,5,dominicks,4160,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-14,5,minute.maid,6528,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-14,5,tropicana,7680,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-14,8,dominicks,20160,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-14,8,minute.maid,6400,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-14,8,tropicana,12864,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-21,2,dominicks,9664,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-21,2,minute.maid,22400,1,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-21,2,tropicana,6976,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-21,5,dominicks,23488,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-21,5,minute.maid,30656,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-21,5,tropicana,8704,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-21,8,dominicks,18688,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-21,8,minute.maid,54592,1,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-21,8,tropicana,7168,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-28,2,dominicks,45568,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-28,2,minute.maid,3968,0,2.84,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-28,2,tropicana,7232,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-05-28,5,dominicks,60480,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-28,5,minute.maid,6656,0,2.66,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-28,5,tropicana,9920,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-05-28,8,dominicks,133824,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-28,8,minute.maid,8128,0,2.39,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-05-28,8,tropicana,9024,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-04,2,dominicks,20992,0,1.74,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-04,2,minute.maid,3264,0,2.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-04,2,tropicana,51520,1,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-04,5,dominicks,20416,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-04,5,minute.maid,4416,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-04,5,tropicana,91968,1,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-04,8,dominicks,63488,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-04,8,minute.maid,4928,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-04,8,tropicana,84992,1,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-11,2,dominicks,6592,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-11,2,minute.maid,4352,0,2.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-11,2,tropicana,22272,0,2.21,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-11,5,dominicks,6336,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-11,5,minute.maid,5696,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-11,5,tropicana,44096,0,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-11,8,dominicks,71040,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-11,8,minute.maid,5440,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-11,8,tropicana,14144,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-18,2,dominicks,4992,0,2.05,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-18,2,minute.maid,4480,0,2.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-18,2,tropicana,46144,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-25,2,dominicks,8064,0,1.24,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-25,2,minute.maid,3840,0,2.52,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-25,2,tropicana,4352,1,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-06-25,5,dominicks,1408,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-25,5,minute.maid,5696,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-25,5,tropicana,7296,1,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-06-25,8,dominicks,15360,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-25,8,minute.maid,5888,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-06-25,8,tropicana,7488,1,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-02,2,dominicks,7360,0,1.61,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-02,2,minute.maid,13312,1,2.0,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-02,2,tropicana,17280,0,2.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-02,5,dominicks,4672,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-02,5,minute.maid,39680,1,2.01,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-02,5,tropicana,12928,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-02,8,dominicks,17728,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-02,8,minute.maid,23872,1,2.02,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-02,8,tropicana,12352,0,2.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-09,2,dominicks,10048,0,1.4,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-09,2,minute.maid,3776,1,2.33,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-09,2,tropicana,5696,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-09,5,dominicks,19520,0,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-09,5,minute.maid,6208,1,2.19,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-09,5,tropicana,6848,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-09,8,dominicks,24256,0,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-09,8,minute.maid,6848,1,2.19,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-09,8,tropicana,5696,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-16,2,dominicks,10112,0,1.91,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-16,2,minute.maid,4800,0,2.89,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-16,2,tropicana,6848,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-16,5,dominicks,7872,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-16,5,minute.maid,7872,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-16,5,tropicana,8064,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-16,8,dominicks,19968,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-16,8,minute.maid,8192,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-16,8,tropicana,7680,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-23,2,dominicks,9152,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-23,2,minute.maid,24960,1,2.29,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-23,2,tropicana,4416,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-23,5,dominicks,5184,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-23,5,minute.maid,54528,1,2.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-23,5,tropicana,4992,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-23,8,dominicks,15936,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-23,8,minute.maid,55040,1,2.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-23,8,tropicana,5440,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-30,2,dominicks,36288,1,1.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-30,2,minute.maid,4544,0,2.86,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-30,2,tropicana,4672,0,3.16,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-07-30,5,dominicks,42240,1,1.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-30,5,minute.maid,6400,0,2.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-30,5,tropicana,7360,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-07-30,8,dominicks,76352,1,1.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-30,8,minute.maid,6528,0,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-07-30,8,tropicana,5632,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-06,2,dominicks,3776,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-06,2,minute.maid,3968,1,2.81,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-06,2,tropicana,7168,1,3.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-06,5,dominicks,6592,1,1.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-06,5,minute.maid,5888,1,2.65,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-06,5,tropicana,8384,1,2.89,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-06,8,dominicks,17408,1,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-06,8,minute.maid,6208,1,2.45,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-06,8,tropicana,8960,1,2.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-13,2,dominicks,3328,0,1.97,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-13,2,minute.maid,49600,1,1.99,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-13,2,tropicana,5056,0,3.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-13,5,dominicks,2112,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-13,5,minute.maid,56384,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-13,5,tropicana,8832,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-13,8,dominicks,17536,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-13,8,minute.maid,94720,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-13,8,tropicana,6080,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-20,2,dominicks,13824,0,1.36,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-20,2,minute.maid,23488,1,1.94,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-20,2,tropicana,13376,1,2.79,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-20,5,dominicks,21248,0,1.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-20,5,minute.maid,27072,1,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-20,5,tropicana,17728,1,2.79,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-20,8,dominicks,31232,0,1.59,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-20,8,minute.maid,55552,1,1.99,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-20,8,tropicana,8576,1,2.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-27,2,dominicks,9024,0,1.19,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-27,2,minute.maid,19008,0,1.69,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-27,2,tropicana,8128,0,2.75,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-08-27,5,dominicks,1856,0,1.29,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-27,5,minute.maid,3840,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-27,5,tropicana,9600,0,2.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-08-27,8,dominicks,19200,0,1.29,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-27,8,minute.maid,18688,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-08-27,8,tropicana,8000,0,2.89,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-03,2,dominicks,2048,0,2.09,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-03,2,minute.maid,11584,0,1.81,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-03,2,tropicana,19456,1,2.49,0.232864734,0.248934934,10.55320518,0.463887065,0.103953406,0.114279949,0.303585347,2.110122129,1.142857143,1.927279669,0.37692661299999997
-1992-09-03,5,dominicks,3712,0,1.99,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-03,5,minute.maid,6144,0,1.69,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-03,5,tropicana,25664,1,2.49,0.117368032,0.32122573,10.92237097,0.535883355,0.103091585,0.053875277,0.410568032,3.801997814,0.681818182,1.600573425,0.736306837
-1992-09-03,8,dominicks,12800,0,1.79,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-03,8,minute.maid,14656,0,1.69,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947
-1992-09-03,8,tropicana,21760,1,2.49,0.252394035,0.095173274,10.59700966,0.054227156,0.131749698,0.035243328,0.283074736,2.636332801,1.5,2.905384316,0.641015947

how-to-use-azureml/automated-machine-learning/forecasting-pipelines/scripts/infer.py

@@ -1,155 +0,0 @@
-import argparse
-from datetime import datetime
-import os
-import uuid
-import numpy as np
-import pandas as pd
-
-from pandas.tseries.frequencies import to_offset
-import joblib
-from sklearn.metrics import mean_absolute_error, mean_squared_error
-
-from azureml.data.dataset_factory import TabularDatasetFactory
-from azureml.automl.runtime.shared.score import scoring, constants as metrics_constants
-import azureml.automl.core.shared.constants as constants
-from azureml.core import Run, Dataset, Model
-
-try:
-    import torch
-
-    _torch_present = True
-except ImportError:
-    _torch_present = False
-
-
-def infer_forecasting_dataset_tcn(
-    X_test, y_test, model, output_path, output_dataset_name="results"
-):
-
-    y_pred, df_all = model.forecast(X_test, y_test)
-
-    run = Run.get_context()
-
-    TabularDatasetFactory.register_pandas_dataframe(
-        df_all,
-        target=(
-            run.experiment.workspace.get_default_datastore(),
-            datetime.now().strftime("%Y-%m-%d-") + str(uuid.uuid4())[:6],
-        ),
-        name=output_dataset_name,
-    )
-    df_all.to_csv(os.path.join(output_path, output_dataset_name + ".csv"), index=False)
-
-
-def map_location_cuda(storage, loc):
-    return storage.cuda()
-
-
-def get_model(model_path, model_file_name):
-    # _, ext = os.path.splitext(model_path)
-    model_full_path = os.path.join(model_path, model_file_name)
-    print(model_full_path)
-    if model_file_name.endswith("pt"):
-        # Load the fc-tcn torch model.
-        assert _torch_present, "Loading DNN models needs torch to be presented."
-        if torch.cuda.is_available():
-            map_location = map_location_cuda
-        else:
-            map_location = "cpu"
-        with open(model_full_path, "rb") as fh:
-            fitted_model = torch.load(fh, map_location=map_location)
-    else:
-        # Load the sklearn pipeline.
-        fitted_model = joblib.load(model_full_path)
-    return fitted_model
-
-
-def get_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "--model_name", type=str, dest="model_name", help="Model to be loaded"
-    )
-
-    parser.add_argument(
-        "--ouput_dataset_name",
-        type=str,
-        dest="ouput_dataset_name",
-        default="results",
-        help="Dataset name of the final output",
-    )
-    parser.add_argument(
-        "--target_column_name",
-        type=str,
-        dest="target_column_name",
-        help="The target column name.",
-    )
-    parser.add_argument(
-        "--test_dataset_name",
-        type=str,
-        dest="test_dataset_name",
-        default="results",
-        help="Dataset name of the final output",
-    )
-    parser.add_argument(
-        "--output_path",
-        type=str,
-        dest="output_path",
-        default="results",
-        help="The output path",
-    )
-    args = parser.parse_args()
-    return args
-
-
-def get_data(run, fitted_model, target_column_name, test_dataset_name):
-
-    # get input dataset by name
-    test_dataset = Dataset.get_by_name(run.experiment.workspace, test_dataset_name)
-    test_df = test_dataset.to_pandas_dataframe()
-    if target_column_name in test_df:
-        y_test = test_df.pop(target_column_name).values
-    else:
-        y_test = np.full(test_df.shape[0], np.nan)
-
-    return test_df, y_test
-
-
-def get_model_filename(run, model_name, model_path):
-    model = Model(run.experiment.workspace, model_name)
-    if "model_file_name" in model.tags:
-        return model.tags["model_file_name"]
-    is_pkl = True
-    if model.tags.get("algorithm") == "TCNForecaster" or os.path.exists(
-        os.path.join(model_path, "model.pt")
-    ):
-        is_pkl = False
-    return "model.pkl" if is_pkl else "model.pt"
-
-
-if __name__ == "__main__":
-    run = Run.get_context()
-
-    args = get_args()
-    model_name = args.model_name
-    ouput_dataset_name = args.ouput_dataset_name
-    test_dataset_name = args.test_dataset_name
-    target_column_name = args.target_column_name
-    print("args passed are: ")
-
-    print(model_name)
-    print(test_dataset_name)
-    print(ouput_dataset_name)
-    print(target_column_name)
-
-    model_path = Model.get_model_path(model_name)
-    model_file_name = get_model_filename(run, model_name, model_path)
-    print(model_file_name)
-    fitted_model = get_model(model_path, model_file_name)
-
-    X_test_df, y_test = get_data(
-        run, fitted_model, target_column_name, test_dataset_name
-    )
-
-    infer_forecasting_dataset_tcn(
-        X_test_df, y_test, fitted_model, args.output_path, ouput_dataset_name
-    )

how-to-use-azureml/automated-machine-learning/forecasting-pipelines/scripts/register_model.py

@@ -1,64 +0,0 @@
-import argparse
-import os
-import uuid
-import shutil
-from azureml.core.model import Model, Dataset
-from azureml.core.run import Run, _OfflineRun
-from azureml.core import Workspace
-import azureml.automl.core.shared.constants as constants
-from azureml.train.automl.run import AutoMLRun
-
-
-def get_best_automl_run(pipeline_run):
-    all_children = [c for c in pipeline_run.get_children()]
-    automl_step = [
-        c for c in all_children if c.properties.get("runTemplate") == "AutoML"
-    ]
-    for c in all_children:
-        print(c, c.properties)
-    automlrun = AutoMLRun(pipeline_run.experiment, automl_step[0].id)
-    best = automlrun.get_best_child()
-    return best
-
-
-def get_model_path(model_artifact_path):
-    return model_artifact_path.split("/")[1]
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--model_name")
-    parser.add_argument("--model_path")
-    parser.add_argument("--ds_name")
-    args = parser.parse_args()
-
-    print("Argument 1(model_name): %s" % args.model_name)
-    print("Argument 2(model_path): %s" % args.model_path)
-    print("Argument 3(ds_name): %s" % args.ds_name)
-
-    run = Run.get_context()
-    ws = None
-    if type(run) == _OfflineRun:
-        ws = Workspace.from_config()
-    else:
-        ws = run.experiment.workspace
-
-    train_ds = Dataset.get_by_name(ws, args.ds_name)
-    datasets = [(Dataset.Scenario.TRAINING, train_ds)]
-    new_dir = str(uuid.uuid4())
-    os.makedirs(new_dir)
-
-    # Register model with training dataset
-    best_run = get_best_automl_run(run.parent)
-    model_artifact_path = best_run.properties[constants.PROPERTY_KEY_OF_MODEL_PATH]
-    algo = best_run.properties.get("run_algorithm")
-    model_artifact_dir = model_artifact_path.split("/")[0]
-    model_file_name = model_artifact_path.split("/")[1]
-    model = best_run.register_model(
-        args.model_name,
-        model_path=model_artifact_dir,
-        datasets=datasets,
-        tags={"algorithm": algo, "model_file_name": model_file_name},
-    )
-
-    print("Registered version {0} of model {1}".format(model.version, model.name))

how-to-use-azureml/automated-machine-learning/forecasting-recipes-univariate/auto-ml-forecasting-univariate-recipe-experiment-settings.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-univariate-recipe-experiment-settings
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-recipes-univariate/auto-ml-forecasting-univariate-recipe-run-experiment.yml

@@ -0,0 +1,4 @@
+name: auto-ml-forecasting-univariate-recipe-run-experiment
+dependencies:
+- pip:
+  - azureml-sdk

how-to-use-azureml/automated-machine-learning/forecasting-recipes-univariate/forecasting_script.py

@@ -7,7 +7,7 @@ compute instance.
 import argparse
 from azureml.core import Dataset, Run
 from azureml.automl.core.shared.constants import TimeSeriesInternal
-import joblib
+from sklearn.externals import joblib
 
 parser = argparse.ArgumentParser()
 parser.add_argument(

how-to-use-azureml/automated-machine-learning/forecasting-recipes-univariate/helper_functions.py

@@ -46,11 +46,11 @@ def kpss_test(series, **kw):
     """
     if kw["store"]:
         statistic, p_value, critical_values, rstore = stattools.kpss(
-            series, regression=kw["reg_type"], nlags=kw["lags"], store=kw["store"]
+            series, regression=kw["reg_type"], lags=kw["lags"], store=kw["store"]
         )
     else:
         statistic, p_value, lags, critical_values = stattools.kpss(
-            series, regression=kw["reg_type"], nlags=kw["lags"]
+            series, regression=kw["reg_type"], lags=kw["lags"]
         )
     output = {
         "statistic": statistic,

how-to-use-azureml/automated-machine-learning/image-classification-multiclass-batch-scoring/README.md

@@ -0,0 +1,18 @@
+---
+page_type: sample
+languages:
+- python
+products:
+- azure-machine-learning
+description: Notebook showing how to use Azure Machine Learning pipelines to do Batch Predictions with an Image Classification model trained using AutoML.
+---
+
+# Batch Scoring with an Image Classification Model
+- Dataset: Toy dataset with images of products found in a fridge
+    - **[Jupyter Notebook](auto-ml-image-classification-multiclass-batch-scoring.ipynb)**
+        - register an Image Classification Multi-Class model already trained using AutoML
+        - create an Inference Dataset
+        - provision compute targets and create a Batch Scoring script
+        - use ParallelRunStep to do batch scoring
+        - build, run, and publish a pipeline
+        - enable a REST endpoint for the pipeline

how-to-use-azureml/automated-machine-learning/image-classification-multiclass-batch-scoring/auto-ml-image-classification-multiclass-batch-scoring.ipynb

@@ -0,0 +1,950 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+    "\n",
+    "Licensed under the MIT License.\n",
+    "\n",
+    "# Batch Predictions for an Image Classification model trained using AutoML\n",
+    "In this notebook, we go over how you can use [Azure Machine Learning pipelines](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-pipeline-batch-scoring-classification) to run a batch scoring image classification job.\n",
+    "\n",
+    "**Please note:** For this notebook you can use an existing image classification model trained using AutoML for Images or use the simple model training we included below for convenience. For detailed instructions on how to train an image classification model with AutoML, please refer to the official [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models) and to the [image classification multiclass notebook](https://github.com/Azure/azureml-examples/blob/main/python-sdk/tutorials/automl-with-azureml/image-classification-multiclass/auto-ml-image-classification-multiclass.ipynb)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Important:** This feature is currently in public preview. This preview version is provided without a service-level agreement. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/en-us/support/legal/preview-supplemental-terms/)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Environment Setup\n",
+    "Please follow the [\"Setup a new conda environment\"](https://github.com/Azure/azureml-examples/tree/main/python-sdk/tutorials/automl-with-azureml#3-setup-a-new-conda-environment) instructions to get started."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import azureml.core\n",
+    "\n",
+    "print(\"This notebook was created using version 1.35.0 of the Azure ML SDK.\")\n",
+    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK.\")\n",
+    "assert (\n",
+    "    azureml.core.VERSION >= \"1.35\"\n",
+    "), \"Please upgrade the Azure ML SDK by running '!pip install --upgrade azureml-sdk' then restart the kernel.\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## You will perform the following tasks:\n",
+    "\n",
+    "* Register a Model already trained using AutoML for Image Classification.\n",
+    "* Create an Inference Dataset.\n",
+    "* Provision compute targets and create a Batch Scoring script.\n",
+    "* Use ParallelRunStep to do batch scoring.\n",
+    "* Build, run, and publish a pipeline.\n",
+    "* Enable a REST endpoint for the pipeline."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Workspace setup\n",
+    "\n",
+    "An [Azure ML Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#workspace) is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inference, and the monitoring of deployed models.\n",
+    "\n",
+    "Create an Azure ML Workspace within your Azure subscription or load an existing workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.workspace import Workspace\n",
+    "\n",
+    "ws = Workspace.from_config()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Workspace default datastore is used to store inference input images and outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def_data_store = ws.get_default_datastore()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Compute target setup\n",
+    "You will need to provide a [Compute Target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#computes) that will be used for your AutoML model training. AutoML models for image tasks require [GPU SKUs](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) such as the ones from the NC, NCv2, NCv3, ND, NDv2 and NCasT4 series. We recommend using the NCsv3-series (with v100 GPUs) for faster training. Using a compute target with a multi-GPU VM SKU will leverage the multiple GPUs to speed up training. Additionally, setting up a compute target with multiple nodes will allow for faster model training by leveraging parallelism, when tuning hyperparameters for your model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
+    "\n",
+    "cluster_name = \"gpu-cluster-nc6\"\n",
+    "\n",
+    "try:\n",
+    "    compute_target = ws.compute_targets[cluster_name]\n",
+    "    print(\"Found existing compute target.\")\n",
+    "except KeyError:\n",
+    "    print(\"Creating a new compute target...\")\n",
+    "    compute_config = AmlCompute.provisioning_configuration(\n",
+    "        vm_size=\"Standard_NC6\",\n",
+    "        idle_seconds_before_scaledown=600,\n",
+    "        min_nodes=0,\n",
+    "        max_nodes=4,\n",
+    "    )\n",
+    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+    "# Can poll for a minimum number of nodes and for a specific timeout.\n",
+    "# If no min_node_count is provided, it will use the scale settings for the cluster.\n",
+    "compute_target.wait_for_completion(\n",
+    "    show_output=True, min_node_count=None, timeout_in_minutes=20\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Train an Image Classification model\n",
+    "\n",
+    "In this section we will do a quick model train to use for the batch scoring. For a datailed example on how to train an image classification model, please refer to the official [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models) or to the [image classification multiclass notebook](https://github.com/Azure/azureml-examples/blob/main/python-sdk/tutorials/automl-with-azureml/image-classification-multiclass/auto-ml-image-classification-multiclass.ipynb). If you already have a model trained in the same workspace, you can skip to section [\"Create data objects\"](#Create-data-objects)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Experiment Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Experiment\n",
+    "\n",
+    "experiment_name = \"automl-image-batchscoring\"\n",
+    "experiment = Experiment(ws, name=experiment_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Download dataset with input Training Data\n",
+    "\n",
+    "All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import urllib\n",
+    "from zipfile import ZipFile\n",
+    "\n",
+    "# download data\n",
+    "download_url = \"https://cvbp-secondary.z19.web.core.windows.net/datasets/image_classification/fridgeObjects.zip\"\n",
+    "data_file = \"./fridgeObjects.zip\"\n",
+    "urllib.request.urlretrieve(download_url, filename=data_file)\n",
+    "\n",
+    "# extract files\n",
+    "with ZipFile(data_file, \"r\") as zip:\n",
+    "    print(\"extracting files...\")\n",
+    "    zip.extractall()\n",
+    "    print(\"done\")\n",
+    "# delete zip file\n",
+    "os.remove(data_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Convert the downloaded data to JSONL"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "\n",
+    "src = \"./fridgeObjects/\"\n",
+    "train_validation_ratio = 5\n",
+    "\n",
+    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
+    "workspaceblobstore = ws.get_default_datastore().name\n",
+    "\n",
+    "# Path to the training and validation files\n",
+    "train_annotations_file = os.path.join(src, \"train_annotations.jsonl\")\n",
+    "validation_annotations_file = os.path.join(src, \"validation_annotations.jsonl\")\n",
+    "\n",
+    "# sample json line dictionary\n",
+    "json_line_sample = {\n",
+    "    \"image_url\": \"AmlDatastore://\"\n",
+    "    + workspaceblobstore\n",
+    "    + \"/\"\n",
+    "    + os.path.basename(os.path.dirname(src)),\n",
+    "    \"label\": \"\",\n",
+    "}\n",
+    "\n",
+    "index = 0\n",
+    "# Scan each sub directary and generate jsonl line\n",
+    "with open(train_annotations_file, \"w\") as train_f:\n",
+    "    with open(validation_annotations_file, \"w\") as validation_f:\n",
+    "        for className in os.listdir(src):\n",
+    "            subDir = src + className\n",
+    "            if not os.path.isdir(subDir):\n",
+    "                continue\n",
+    "            # Scan each sub directary\n",
+    "            print(\"Parsing \" + subDir)\n",
+    "            for image in os.listdir(subDir):\n",
+    "                json_line = dict(json_line_sample)\n",
+    "                json_line[\"image_url\"] += f\"/{className}/{image}\"\n",
+    "                json_line[\"label\"] = className\n",
+    "\n",
+    "                if index % train_validation_ratio == 0:\n",
+    "                    # validation annotation\n",
+    "                    validation_f.write(json.dumps(json_line) + \"\\n\")\n",
+    "                else:\n",
+    "                    # train annotation\n",
+    "                    train_f.write(json.dumps(json_line) + \"\\n\")\n",
+    "                index += 1"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Upload the JSONL file and images to Datastore"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
+    "ds = ws.get_default_datastore()\n",
+    "ds.upload(src_dir=\"./fridgeObjects\", target_path=\"fridgeObjects\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Create and register datasets in workspace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Dataset\n",
+    "from azureml.data import DataType\n",
+    "\n",
+    "# get existing training dataset\n",
+    "training_dataset_name = \"fridgeObjectsTrainingDataset\"\n",
+    "if training_dataset_name in ws.datasets:\n",
+    "    training_dataset = ws.datasets.get(training_dataset_name)\n",
+    "    print(\"Found the training dataset\", training_dataset_name)\n",
+    "else:\n",
+    "    # create training dataset\n",
+    "    training_dataset = Dataset.Tabular.from_json_lines_files(\n",
+    "        path=ds.path(\"fridgeObjects/train_annotations.jsonl\"),\n",
+    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
+    "    )\n",
+    "    training_dataset = training_dataset.register(\n",
+    "        workspace=ws, name=training_dataset_name\n",
+    "    )\n",
+    "# get existing validation dataset\n",
+    "validation_dataset_name = \"fridgeObjectsValidationDataset\"\n",
+    "if validation_dataset_name in ws.datasets:\n",
+    "    validation_dataset = ws.datasets.get(validation_dataset_name)\n",
+    "    print(\"Found the validation dataset\", validation_dataset_name)\n",
+    "else:\n",
+    "    # create validation dataset\n",
+    "    validation_dataset = Dataset.Tabular.from_json_lines_files(\n",
+    "        path=ds.path(\"fridgeObjects/validation_annotations.jsonl\"),\n",
+    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
+    "    )\n",
+    "    validation_dataset = validation_dataset.register(\n",
+    "        workspace=ws, name=validation_dataset_name\n",
+    "    )\n",
+    "print(\"Training dataset name: \" + training_dataset.name)\n",
+    "print(\"Validation dataset name: \" + validation_dataset.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Submit training 1 training run with default hyperparameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.automl.core.shared.constants import ImageTask\n",
+    "from azureml.train.automl import AutoMLImageConfig\n",
+    "from azureml.train.hyperdrive import GridParameterSampling, choice\n",
+    "\n",
+    "image_config_vit = AutoMLImageConfig(\n",
+    "    task=ImageTask.IMAGE_CLASSIFICATION,\n",
+    "    compute_target=compute_target,\n",
+    "    training_data=training_dataset,\n",
+    "    validation_data=validation_dataset,\n",
+    "    hyperparameter_sampling=GridParameterSampling({\"model_name\": choice(\"vitb16r224\")}),\n",
+    "    iterations=1,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run = experiment.submit(image_config_vit)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run.wait_for_completion(wait_post_processing=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create data objects\n",
+    "\n",
+    "When building pipelines, `Dataset` objects are used for reading data from workspace datastores, and `PipelineData` objects are used for transferring intermediate data between pipeline steps.\n",
+    "\n",
+    "This batch scoring example only uses one pipeline step, but in use-cases with multiple steps, the typical flow will include:\n",
+    "\n",
+    "1. Using `Dataset` objects as inputs to fetch raw data, performing some transformations, then output a `PipelineData` object. \n",
+    "1. Use the previous step's `PipelineData` **output object** as an **input object**, repeated for subsequent steps.\n",
+    "\n",
+    "For this scenario you create `Dataset` objects corresponding to the datastore directories for the input images. You also create a `PipelineData` object for the batch scoring output data. An object reference in the `outputs` array becomes available as an **input** for a subsequent pipeline step, for scenarios where there is more than one step. In this case we are just going to build a single step pipeline.\n",
+    "\n",
+    "It is assumed that an image classification training run was already performed in this workspace and the files are already in the datastore. If this is not the case, please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models) to know how to train an image classification model with AutoML.\n",
+    "\n",
+    "All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.dataset import Dataset\n",
+    "from azureml.pipeline.core import PipelineData\n",
+    "\n",
+    "input_images = Dataset.File.from_files((def_data_store, \"fridgeObjects/**/*.jpg\"))\n",
+    "\n",
+    "output_dir = PipelineData(name=\"scores\", datastore=def_data_store)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we need to register the input datasets for batch scoring with the workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_images = input_images.register(\n",
+    "    workspace=ws, name=\"fridgeObjects_scoring_images\", create_new_version=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Retrieve the environment and metrics from the training run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.experiment import Experiment\n",
+    "from azureml.core import Run\n",
+    "\n",
+    "experiment_name = \"automl-image-batchscoring\"\n",
+    "# If your model was not trained with this notebook, replace the id below\n",
+    "# with the run id of the child training run (i.e., the one ending with HD_0)\n",
+    "training_run_id = automl_image_run.id + \"_HD_0\"\n",
+    "exp = Experiment(ws, experiment_name)\n",
+    "training_run = Run(exp, training_run_id)\n",
+    "\n",
+    "# The below will give only the requested metric\n",
+    "metrics = training_run.get_metrics(\"accuracy\")\n",
+    "best_metric = max(metrics[\"accuracy\"])\n",
+    "print(\"best_metric:\", best_metric)\n",
+    "\n",
+    "# Retrieve the training environment\n",
+    "env = training_run.get_environment()\n",
+    "print(env)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Register model with metric and environment tags\n",
+    "\n",
+    "Now you register the model to your workspace, which allows you to easily retrieve it in the pipeline process. In the `register()` static function, the `model_name` parameter is the key you use to locate your model throughout the SDK.\n",
+    "Tag the model with the metrics and the environment used to train the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.model import Model\n",
+    "\n",
+    "tags = dict()\n",
+    "tags[\"accuracy\"] = best_metric\n",
+    "tags[\"env_name\"] = env.name\n",
+    "tags[\"env_version\"] = env.version\n",
+    "\n",
+    "model_name = \"fridgeObjectsClassifier\"\n",
+    "model = training_run.register_model(\n",
+    "    model_name=model_name, model_path=\"train_artifacts\", tags=tags\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# List the models from the workspace\n",
+    "models = Model.list(ws, name=model_name, latest=True)\n",
+    "print(model.name)\n",
+    "print(model.tags)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Write a scoring script"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To do the scoring, you create a batch scoring script `batch_scoring.py`, and write it to the scripts folder in current directory. The script takes a minibatch of input images, applies the classification model, and outputs the predictions to a results file.\n",
+    "\n",
+    "The script `batch_scoring.py` takes the following parameters, which get passed from the `ParallelRunStep` that you create later:\n",
+    "\n",
+    "- `--model_name`: the name of the model being used\n",
+    "\n",
+    "While creating the batch scoring script, refer to the scoring scripts generated under the outputs folder of the Automl training runs. This will help to identify the right model settings to be used in the batch scoring script init method while loading the model.\n",
+    "Note: The batch scoring script we generate in the subsequent step is different from the scoring script generated by the training runs in the below screenshot. We refer to it just to identify the right model settings to be used in the batch scoring script.\n",
+    "\n",
+    "![Training run outputs](ui_outputs.PNG \"Training run outputs\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# View the batch scoring script. Use the model settings as appropriate for your model.\n",
+    "with open(\"./scripts/batch_scoring.py\", \"r\") as f:\n",
+    "    print(f.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Build and run the pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create the parallel-run configuration to wrap the inference script\n",
+    "Create the pipeline run configuration specifying the script, environment configuration, and parameters. Specify the compute target you already attached to your workspace as the target of execution of the script. This will set the run configuration of the ParallelRunStep we will define next.\n",
+    "\n",
+    "Refer this [site](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/machine-learning-pipelines/parallel-run) for more details on ParallelRunStep of Azure Machine Learning Pipelines."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.pipeline.steps import ParallelRunConfig\n",
+    "\n",
+    "parallel_run_config = ParallelRunConfig(\n",
+    "    environment=env,\n",
+    "    entry_script=\"batch_scoring.py\",\n",
+    "    source_directory=\"scripts\",\n",
+    "    output_action=\"append_row\",\n",
+    "    append_row_file_name=\"parallel_run_step.txt\",\n",
+    "    mini_batch_size=\"20\",  # Num files to process in one call\n",
+    "    error_threshold=1,\n",
+    "    compute_target=compute_target,\n",
+    "    process_count_per_node=2,\n",
+    "    node_count=1,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create the pipeline step\n",
+    "\n",
+    "A pipeline step is an object that encapsulates everything you need for running a pipeline including:\n",
+    "\n",
+    "* environment and dependency settings\n",
+    "* the compute resource to run the pipeline on\n",
+    "* input and output data, and any custom parameters\n",
+    "* reference to a script to run during the step\n",
+    "\n",
+    "There are multiple classes that inherit from the parent class [`PipelineStep`](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/?view=azure-ml-py) to assist with building a step using certain frameworks and stacks. In this example, you use the [`ParallelRunStep`](https://docs.microsoft.com/en-us/python/api/azureml-contrib-pipeline-steps/azureml.contrib.pipeline.steps.parallelrunstep?view=azure-ml-py) class to define your step logic using a scoring script. `ParallelRunStep` executes the script in a distributed fashion.\n",
+    "\n",
+    "The pipelines infrastructure uses the `ArgumentParser` class to pass parameters into pipeline steps. For example, in the code below the first argument `--model_name` is given the property identifier `model_name`. In the `main()` function, this property is accessed using `Model.get_model_path(args.model_name)`.\n",
+    "\n",
+    "Note: The pipeline in this tutorial only has one step and writes the output to a file, but for multi-step pipelines, you also use `ArgumentParser` to define a directory to write output data for input to subsequent steps. See the [notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/nyc-taxi-data-regression-model-building.ipynb) for an example of passing data between multiple pipeline steps using the `ArgumentParser` design pattern."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.pipeline.steps import ParallelRunStep\n",
+    "from datetime import datetime\n",
+    "\n",
+    "parallel_step_name = \"batchscoring-\" + datetime.now().strftime(\"%Y%m%d%H%M\")\n",
+    "\n",
+    "arguments = [\"--model_name\", model_name]\n",
+    "\n",
+    "# Specify inference batch_size, otherwise uses default value. (This is different from the mini_batch_size above)\n",
+    "# NOTE: Large batch sizes may result in OOM errors.\n",
+    "# arguments = arguments + [\"--batch_size\", \"20\"]\n",
+    "\n",
+    "batch_score_step = ParallelRunStep(\n",
+    "    name=parallel_step_name,\n",
+    "    inputs=[input_images.as_named_input(\"input_images\")],\n",
+    "    output=output_dir,\n",
+    "    arguments=arguments,\n",
+    "    parallel_run_config=parallel_run_config,\n",
+    "    allow_reuse=False,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For a list of all classes for different step types, see the [steps package](https://docs.microsoft.com/python/api/azureml-pipeline-steps/azureml.pipeline.steps?view=azure-ml-py)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run the pipeline\n",
+    "\n",
+    "Now you run the pipeline. First create a `Pipeline` object with your workspace reference and the pipeline step you created. The `steps` parameter is an array of steps, and in this case, there is only one step for batch scoring. To build pipelines with multiple steps, you place the steps in order in this array.\n",
+    "\n",
+    "Next use the `Experiment.submit()` function to submit the pipeline for execution. You also specify the custom parameter `param_batch_size`. The `wait_for_completion` function will output logs during the pipeline build process, which allows you to see current progress.\n",
+    "\n",
+    "Note: The first pipeline run takes roughly **15 minutes**, as all dependencies must be downloaded, a Docker image is created, and the Python environment is provisioned/created. Running it again takes significantly less time as those resources are reused. However, total run time depends on the workload of your scripts and processes running in each pipeline step."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Experiment\n",
+    "from azureml.pipeline.core import Pipeline\n",
+    "\n",
+    "pipeline = Pipeline(workspace=ws, steps=[batch_score_step])\n",
+    "pipeline_run = Experiment(ws, \"batch_scoring_automl_image\").submit(pipeline)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This will output information of the pipeline run, including the link to the details page of portal.\n",
+    "pipeline_run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Wait the run for completion and show output log to console\n",
+    "pipeline_run.wait_for_completion(show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Download and review output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tempfile\n",
+    "import os\n",
+    "\n",
+    "batch_run = pipeline_run.find_step_run(batch_score_step.name)[0]\n",
+    "batch_output = batch_run.get_output_data(output_dir.name)\n",
+    "\n",
+    "target_dir = tempfile.mkdtemp()\n",
+    "batch_output.download(local_path=target_dir)\n",
+    "result_file = os.path.join(\n",
+    "    target_dir, batch_output.path_on_datastore, parallel_run_config.append_row_file_name\n",
+    ")\n",
+    "result_file\n",
+    "\n",
+    "# Print the first five lines of the output\n",
+    "with open(result_file) as f:\n",
+    "    for x in range(5):\n",
+    "        print(next(f))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Choose a random file for visualization"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "import json\n",
+    "\n",
+    "with open(result_file, \"r\") as f:\n",
+    "    contents = f.readlines()\n",
+    "rand_file = contents[random.randrange(len(contents))]\n",
+    "prediction = json.loads(rand_file)\n",
+    "print(prediction[\"filename\"])\n",
+    "print(prediction[\"probs\"])\n",
+    "print(prediction[\"labels\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Download the image file from the datastore\n",
+    "path = (\n",
+    "    \"fridgeObjects\"\n",
+    "    + \"/\"\n",
+    "    + prediction[\"filename\"].split(\"/\")[-2]\n",
+    "    + \"/\"\n",
+    "    + prediction[\"filename\"].split(\"/\")[-1]\n",
+    ")\n",
+    "path_on_datastore = def_data_store.path(path)\n",
+    "single_image_ds = Dataset.File.from_files(path=path_on_datastore, validate=False)\n",
+    "image = single_image_ds.download()[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.image as mpimg\n",
+    "from PIL import Image\n",
+    "import numpy as np\n",
+    "import json\n",
+    "\n",
+    "IMAGE_SIZE = (18, 12)\n",
+    "plt.figure(figsize=IMAGE_SIZE)\n",
+    "img_np = mpimg.imread(image)\n",
+    "img = Image.fromarray(img_np.astype(\"uint8\"), \"RGB\")\n",
+    "x, y = img.size\n",
+    "\n",
+    "fig, ax = plt.subplots(1, figsize=(15, 15))\n",
+    "# Display the image\n",
+    "ax.imshow(img_np)\n",
+    "\n",
+    "label_index = np.argmax(prediction[\"probs\"])\n",
+    "label = prediction[\"labels\"][label_index]\n",
+    "conf_score = prediction[\"probs\"][label_index]\n",
+    "\n",
+    "display_text = \"{} ({})\".format(label, round(conf_score, 3))\n",
+    "print(display_text)\n",
+    "\n",
+    "color = \"red\"\n",
+    "plt.text(30, 30, display_text, color=color, fontsize=30)\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Publish and run from REST endpoint"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
+    "\n",
+    "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "published_pipeline = pipeline_run.publish_pipeline(\n",
+    "    name=\"automl-image-batch-scoring\",\n",
+    "    description=\"Batch scoring using Automl for Image\",\n",
+    "    version=\"1.0\",\n",
+    ")\n",
+    "\n",
+    "published_pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To run the pipeline from the REST endpoint, you first need an OAuth2 Bearer-type authentication header. This example uses interactive authentication for illustration purposes, but for most production scenarios requiring automated or headless authentication, use service principal authentication as [described in this notebook](https://aka.ms/pl-restep-auth).\n",
+    "\n",
+    "Service principal authentication involves creating an **App Registration** in **Azure Active Directory**, generating a client secret, and then granting your service principal **role access** to your machine learning workspace. You then use the [`ServicePrincipalAuthentication`](https://docs.microsoft.com/python/api/azureml-core/azureml.core.authentication.serviceprincipalauthentication?view=azure-ml-py) class to manage your auth flow.\n",
+    "\n",
+    "Both `InteractiveLoginAuthentication` and `ServicePrincipalAuthentication` inherit from `AbstractAuthentication`, and in both cases you use the `get_authentication_header()` function in the same way to fetch the header."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
+    "\n",
+    "interactive_auth = InteractiveLoginAuthentication()\n",
+    "auth_header = interactive_auth.get_authentication_header()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Get the REST url from the `endpoint` property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the `process_count_per_node` is passed through to `ParallelRunStep` because you defined it is defined as a `PipelineParameter` object in the step configuration.\n",
+    "\n",
+    "Make the request to trigger the run. Access the `Id` key from the response dictionary to get the value of the run id."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "rest_endpoint = published_pipeline.endpoint\n",
+    "response = requests.post(\n",
+    "    rest_endpoint,\n",
+    "    headers=auth_header,\n",
+    "    json={\n",
+    "        \"ExperimentName\": \"batch_scoring\",\n",
+    "        \"ParameterAssignments\": {\"process_count_per_node\": 2},\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    response.raise_for_status()\n",
+    "except Exception:\n",
+    "    raise Exception(\n",
+    "        \"Received bad response from the endpoint: {}\\n\"\n",
+    "        \"Response Code: {}\\n\"\n",
+    "        \"Headers: {}\\n\"\n",
+    "        \"Content: {}\".format(\n",
+    "            rest_endpoint, response.status_code, response.headers, response.content\n",
+    "        )\n",
+    "    )\n",
+    "run_id = response.json().get(\"Id\")\n",
+    "print(\"Submitted pipeline run: \", run_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.pipeline.core.run import PipelineRun\n",
+    "\n",
+    "published_pipeline_run = PipelineRun(ws.experiments[\"batch_scoring\"], run_id)\n",
+    "published_pipeline_run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Wait the run for completion and show output log to console\n",
+    "published_pipeline_run.wait_for_completion(show_output=True)"
+   ]
+  }
+ ],
+ "metadata": {
+  "authors": [
+   {
+    "name": [
+     "sanpil",
+     "trmccorm",
+     "pansav"
+    ]
+   }
+  ],
+  "categories": [
+   "tutorials"
+  ],
+  "kernelspec": {
+   "display_name": "Python 3.6 - AzureML",
+   "language": "python",
+   "name": "python3-azureml"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.8"
+  },
+  "metadata": {
+   "interpreter": {
+    "hash": "0f25b6eb4724eea488a4edd67dd290abce7d142c09986fc811384b5aebc0585a"
+   }
+  },
+  "msauthor": "trbye"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

how-to-use-azureml/automated-machine-learning/image-classification-multiclass-batch-scoring/scripts/batch_scoring.py

@@ -0,0 +1,69 @@
+# Copyright (c) Microsoft. All rights reserved.
+# Licensed under the MIT license.
+
+import os
+import argparse
+import json
+
+from azureml.core.model import Model
+from azureml.automl.core.shared import logging_utilities
+
+try:
+    from azureml.automl.dnn.vision.common.logging_utils import get_logger
+    from azureml.automl.dnn.vision.common.model_export_utils import (
+        load_model,
+        run_inference_batch,
+    )
+    from azureml.automl.dnn.vision.classification.inference.score import (
+        _score_with_model,
+    )
+    from azureml.automl.dnn.vision.common.utils import _set_logging_parameters
+except ImportError:
+    from azureml.contrib.automl.dnn.vision.common.logging_utils import get_logger
+    from azureml.contrib.automl.dnn.vision.common.model_export_utils import (
+        load_model,
+        run_inference_batch,
+    )
+    from azureml.contrib.automl.dnn.vision.classification.inference.score import (
+        _score_with_model,
+    )
+    from azureml.contrib.automl.dnn.vision.common.utils import _set_logging_parameters
+
+TASK_TYPE = "image-classification"
+logger = get_logger("azureml.automl.core.scoring_script_images")
+
+
+def init():
+    global model
+    global batch_size
+
+    # Set up logging
+    _set_logging_parameters(TASK_TYPE, {})
+
+    parser = argparse.ArgumentParser(
+        description="Retrieve model_name and batch_size from arguments."
+    )
+    parser.add_argument("--model_name", dest="model_name", required=True)
+    parser.add_argument("--batch_size", dest="batch_size", type=int, required=False)
+    args, _ = parser.parse_known_args()
+
+    batch_size = args.batch_size
+
+    model_path = os.path.join(Model.get_model_path(args.model_name), "model.pt")
+    print(model_path)
+
+    try:
+        logger.info("Loading model from path: {}.".format(model_path))
+        model_settings = {}
+        model = load_model(TASK_TYPE, model_path, **model_settings)
+        logger.info("Loading successful.")
+    except Exception as e:
+        logging_utilities.log_traceback(e, logger)
+        raise
+
+
+def run(mini_batch):
+    logger.info("Running inference.")
+    result = run_inference_batch(model, mini_batch, _score_with_model, batch_size)
+    logger.info("Finished inferencing.")
+    return result

how-to-use-azureml/automated-machine-learning/image-classification-multiclass/README.md

@@ -0,0 +1,15 @@
+---
+page_type: sample
+languages:
+- python
+products:
+- azure-machine-learning
+description: Notebook showing how to use AutoML for training an Image Classification Multi-Class model. We will use a small dataset to train the model, demonstrate how you can tune hyperparameters of the model to optimize model performance and deploy the model to use in inference scenarios.
+---
+
+# Image Classification Multi-Class using AutoML for Images
+- Dataset: Toy dataset with images of products found in a fridge
+    - **[Jupyter Notebook](auto-ml-image-classification-multiclass.ipynb)**
+        - train an Image Classification Multi-Class model using AutoML
+        - tune hyperparameters of the model to optimize model performance
+        - deploy the model to use in inference scenarios

how-to-use-azureml/automated-machine-learning/image-classification-multiclass/auto-ml-image-classification-multiclass.ipynb

@@ -0,0 +1,744 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
+    "\n",
+    "Licensed under the MIT License.\n",
+    "\n",
+    "# Training an Image Classification Multi-Class model using AutoML\n",
+    "In this notebook, we go over how you can use AutoML for training an Image Classification Multi-Class model. We will use a small dataset to train the model, demonstrate how you can tune hyperparameters of the model to optimize model performance and deploy the model to use in inference scenarios. For detailed information please refer to the [documentation of AutoML for Images](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![img](example_image_classification_multiclass_predictions.jpg)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Important:** This feature is currently in public preview. This preview version is provided without a service-level agreement. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/en-us/support/legal/preview-supplemental-terms/)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Environment Setup\n",
+    "Please follow the [\"Setup a new conda environment\"](https://github.com/Azure/azureml-examples/tree/main/python-sdk/tutorials/automl-with-azureml#3-setup-a-new-conda-environment) instructions to get started."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import azureml.core\n",
+    "\n",
+    "print(\"This notebook was created using version 1.35.0 of the Azure ML SDK.\")\n",
+    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK.\")\n",
+    "assert (\n",
+    "    azureml.core.VERSION >= \"1.35\"\n",
+    "), \"Please upgrade the Azure ML SDK by running '!pip install --upgrade azureml-sdk' then restart the kernel.\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Workspace setup\n",
+    "In order to train and deploy models in Azure ML, you will first need to set up a workspace.\n",
+    "\n",
+    "An [Azure ML Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#workspace) is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inference, and the monitoring of deployed models.\n",
+    "\n",
+    "Create an Azure ML Workspace within your Azure subscription or load an existing workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.workspace import Workspace\n",
+    "\n",
+    "ws = Workspace.from_config()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Compute target setup\n",
+    "You will need to provide a [Compute Target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#computes) that will be used for your AutoML model training. AutoML models for image tasks require [GPU SKUs](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) such as the ones from the NC, NCv2, NCv3, ND, NDv2 and NCasT4 series. We recommend using the NCsv3-series (with v100 GPUs) for faster training. Using a compute target with a multi-GPU VM SKU will leverage the multiple GPUs to speed up training. Additionally, setting up a compute target with multiple nodes will allow for faster model training by leveraging parallelism, when tuning hyperparameters for your model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
+    "\n",
+    "cluster_name = \"gpu-cluster-nc6\"\n",
+    "\n",
+    "try:\n",
+    "    compute_target = ws.compute_targets[cluster_name]\n",
+    "    print(\"Found existing compute target.\")\n",
+    "except KeyError:\n",
+    "    print(\"Creating a new compute target...\")\n",
+    "    compute_config = AmlCompute.provisioning_configuration(\n",
+    "        vm_size=\"Standard_NC6\",\n",
+    "        idle_seconds_before_scaledown=600,\n",
+    "        min_nodes=0,\n",
+    "        max_nodes=4,\n",
+    "    )\n",
+    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
+    "# Can poll for a minimum number of nodes and for a specific timeout.\n",
+    "# If no min_node_count is provided, it will use the scale settings for the cluster.\n",
+    "compute_target.wait_for_completion(\n",
+    "    show_output=True, min_node_count=None, timeout_in_minutes=20\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Experiment Setup\n",
+    "Create an [Experiment](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#experiments) in your workspace to track your model training runs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Experiment\n",
+    "\n",
+    "experiment_name = \"automl-image-multiclass\"\n",
+    "experiment = Experiment(ws, name=experiment_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Dataset with input Training Data\n",
+    "\n",
+    "In order to generate models for computer vision, you will need to bring in labeled image data as input for model training in the form of an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset). You can either use a dataset that you have exported from a [Data Labeling](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-label-data) project, or create a new Tabular Dataset with your labeled training data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this notebook, we use a toy dataset called Fridge Objects, which consists of 134 images of 4 classes of beverage container {can, carton, milk bottle, water bottle} photos taken on different backgrounds.\n",
+    "\n",
+    "All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE).\n",
+    "\n",
+    "We first download and unzip the data locally."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import urllib\n",
+    "from zipfile import ZipFile\n",
+    "\n",
+    "# download data\n",
+    "download_url = \"https://cvbp-secondary.z19.web.core.windows.net/datasets/image_classification/fridgeObjects.zip\"\n",
+    "data_file = \"./fridgeObjects.zip\"\n",
+    "urllib.request.urlretrieve(download_url, filename=data_file)\n",
+    "\n",
+    "# extract files\n",
+    "with ZipFile(data_file, \"r\") as zip:\n",
+    "    print(\"extracting files...\")\n",
+    "    zip.extractall()\n",
+    "    print(\"done\")\n",
+    "# delete zip file\n",
+    "os.remove(data_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is a sample image from this dataset:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import Image\n",
+    "\n",
+    "sample_image = \"./fridgeObjects/milk_bottle/99.jpg\"\n",
+    "Image(filename=sample_image)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Convert the downloaded data to JSONL\n",
+    "In this example, the fridge object dataset is stored in a directory. There are four different folders inside:\n",
+    "\n",
+    "- /water_bottle\n",
+    "- /milk_bottle\n",
+    "- /carton\n",
+    "- /can\n",
+    "\n",
+    "This is the most common data format for multiclass image classification. Each folder title corresponds to the image label for the images contained inside.\n",
+    "\n",
+    "In order to use this data to create an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset), we first need to convert it to the required JSONL format. Please refer to the [documentation on how to prepare datasets](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-prepare-datasets-for-automl-images).\n",
+    "\n",
+    "The following script is creating two .jsonl files (one for training and one for validation) in the parent folder of the dataset. The train / validation ratio corresponds to 20% of the data going into the validation file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "\n",
+    "src = \"./fridgeObjects/\"\n",
+    "train_validation_ratio = 5\n",
+    "\n",
+    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
+    "workspaceblobstore = ws.get_default_datastore().name\n",
+    "\n",
+    "# Path to the training and validation files\n",
+    "train_annotations_file = os.path.join(src, \"train_annotations.jsonl\")\n",
+    "validation_annotations_file = os.path.join(src, \"validation_annotations.jsonl\")\n",
+    "\n",
+    "# sample json line dictionary\n",
+    "json_line_sample = {\n",
+    "    \"image_url\": \"AmlDatastore://\"\n",
+    "    + workspaceblobstore\n",
+    "    + \"/\"\n",
+    "    + os.path.basename(os.path.dirname(src)),\n",
+    "    \"label\": \"\",\n",
+    "}\n",
+    "\n",
+    "index = 0\n",
+    "# Scan each sub directary and generate jsonl line\n",
+    "with open(train_annotations_file, \"w\") as train_f:\n",
+    "    with open(validation_annotations_file, \"w\") as validation_f:\n",
+    "        for className in os.listdir(src):\n",
+    "            subDir = src + className\n",
+    "            if not os.path.isdir(subDir):\n",
+    "                continue\n",
+    "            # Scan each sub directary\n",
+    "            print(\"Parsing \" + subDir)\n",
+    "            for image in os.listdir(subDir):\n",
+    "                json_line = dict(json_line_sample)\n",
+    "                json_line[\"image_url\"] += f\"/{className}/{image}\"\n",
+    "                json_line[\"label\"] = className\n",
+    "\n",
+    "                if index % train_validation_ratio == 0:\n",
+    "                    # validation annotation\n",
+    "                    validation_f.write(json.dumps(json_line) + \"\\n\")\n",
+    "                else:\n",
+    "                    # train annotation\n",
+    "                    train_f.write(json.dumps(json_line) + \"\\n\")\n",
+    "                index += 1"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Upload the JSONL file and images to Datastore\n",
+    "In order to use the data for training in Azure ML, we upload it to our Azure ML Workspace via a [Datastore](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#datasets-and-datastores). The datastore provides a mechanism for you to upload/download data and interact with it from your remote compute targets. It is an abstraction over Azure Storage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
+    "ds = ws.get_default_datastore()\n",
+    "ds.upload(src_dir=\"./fridgeObjects\", target_path=\"fridgeObjects\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, we need to create an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset) from the data we uploaded to the Datastore. We create one dataset for training and one for validation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Dataset\n",
+    "from azureml.data import DataType\n",
+    "\n",
+    "# get existing training dataset\n",
+    "training_dataset_name = \"fridgeObjectsTrainingDataset\"\n",
+    "if training_dataset_name in ws.datasets:\n",
+    "    training_dataset = ws.datasets.get(training_dataset_name)\n",
+    "    print(\"Found the training dataset\", training_dataset_name)\n",
+    "else:\n",
+    "    # create training dataset\n",
+    "    training_dataset = Dataset.Tabular.from_json_lines_files(\n",
+    "        path=ds.path(\"fridgeObjects/train_annotations.jsonl\"),\n",
+    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
+    "    )\n",
+    "    training_dataset = training_dataset.register(\n",
+    "        workspace=ws, name=training_dataset_name\n",
+    "    )\n",
+    "# get existing validation dataset\n",
+    "validation_dataset_name = \"fridgeObjectsValidationDataset\"\n",
+    "if validation_dataset_name in ws.datasets:\n",
+    "    validation_dataset = ws.datasets.get(validation_dataset_name)\n",
+    "    print(\"Found the validation dataset\", validation_dataset_name)\n",
+    "else:\n",
+    "    # create validation dataset\n",
+    "    validation_dataset = Dataset.Tabular.from_json_lines_files(\n",
+    "        path=ds.path(\"fridgeObjects/validation_annotations.jsonl\"),\n",
+    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
+    "    )\n",
+    "    validation_dataset = validation_dataset.register(\n",
+    "        workspace=ws, name=validation_dataset_name\n",
+    "    )\n",
+    "print(\"Training dataset name: \" + training_dataset.name)\n",
+    "print(\"Validation dataset name: \" + validation_dataset.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Validation dataset is optional. If no validation dataset is specified, by default 20% of your training data will be used for validation. You can control the percentage using the `split_ratio` argument - please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#model-agnostic-hyperparameters) for more details.\n",
+    "\n",
+    "This is what the training dataset looks like:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "training_dataset.to_pandas_dataframe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configuring your AutoML run for image tasks\n",
+    "AutoML allows you to easily train models for Image Classification, Object Detection & Instance Segmentation on your image data. You can control the model algorithm to be used, specify hyperparameter values for your model as well as perform a sweep across the hyperparameter space to generate an optimal model. Parameters for configuring your AutoML Image run are specified using the `AutoMLImageConfig` - please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#configure-your-experiment-settings) for the details on the parameters that can be used and their values."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When using AutoML for image tasks, you need to specify the model algorithms using the `model_name` parameter. You can either specify a single model or choose to sweep over multiple models. Please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#configure-model-algorithms-and-hyperparameters) for the list of supported model algorithms.\n",
+    "\n",
+    "### Using default hyperparameter values for the specified algorithm\n",
+    "Before doing a large sweep to search for the optimal models and hyperparameters, we recommend trying the default values for a given model to get a first baseline. Next, you can explore multiple hyperparameters for the same model before sweeping over multiple models and their parameters. This allows an iterative approach, as with multiple models and multiple hyperparameters for each (as we showcase in the next section), the search space grows exponentially, and  you need more iterations to find optimal configurations.\n",
+    "\n",
+    "If you wish to use the default hyperparameter values for a given algorithm (say `vitb16r224`), you can specify the config for your AutoML Image runs as follows:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.automl.core.shared.constants import ImageTask\n",
+    "from azureml.train.automl import AutoMLImageConfig\n",
+    "from azureml.train.hyperdrive import GridParameterSampling, choice\n",
+    "\n",
+    "image_config_vit = AutoMLImageConfig(\n",
+    "    task=ImageTask.IMAGE_CLASSIFICATION,\n",
+    "    compute_target=compute_target,\n",
+    "    training_data=training_dataset,\n",
+    "    validation_data=validation_dataset,\n",
+    "    hyperparameter_sampling=GridParameterSampling({\"model_name\": choice(\"vitb16r224\")}),\n",
+    "    iterations=1,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Submitting an AutoML run for Computer Vision tasks\n",
+    "Once you've created the config settings for your run, you can submit an AutoML run using the config in order to train a vision model using your training dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run = experiment.submit(image_config_vit)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run.wait_for_completion(wait_post_processing=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Hyperparameter sweeping for your AutoML models for computer vision tasks\n",
+    "In this example, we use the AutoMLImageConfig to train an Image Classification model using the following model algorithms: `seresnext`, `resnet50`, `vitb16r224`, and `vits16r224`.\n",
+    "\n",
+    "When using AutoML for Images, you can perform a hyperparameter sweep over a defined parameter space to find the optimal model. In this example, we sweep over the hyperparameters for each algorithm, choosing from a range of values for learning_rate, number_of_epochs, layers_to_freeze, etc., to generate a model with the optimal 'accuracy'. If hyperparameter values are not specified, then default values are used for the specified algorithm.\n",
+    "\n",
+    "We use Random Sampling to pick samples from this parameter space and try a total of 10 iterations with these different samples, running 2 iterations at a time on our compute target, which has been previously set up using 4 nodes. Please note that the more parameters the space has, the more iterations you need to find optimal models.\n",
+    "\n",
+    "We leverage the Bandit early termination policy which will terminate poor performing configs (those that are not within 20% slack of the best performing config), thus significantly saving compute resources.\n",
+    "\n",
+    "For more details on model and hyperparameter sweeping, please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.automl.core.shared.constants import ImageTask\n",
+    "from azureml.train.automl import AutoMLImageConfig\n",
+    "from azureml.train.hyperdrive import BanditPolicy, RandomParameterSampling\n",
+    "from azureml.train.hyperdrive import choice, uniform\n",
+    "\n",
+    "parameter_space = {\n",
+    "    \"learning_rate\": uniform(0.001, 0.01),\n",
+    "    \"model\": choice(\n",
+    "        {\n",
+    "            \"model_name\": choice(\"vitb16r224\", \"vits16r224\"),\n",
+    "            \"number_of_epochs\": choice(15, 30),\n",
+    "        },\n",
+    "        {\n",
+    "            \"model_name\": choice(\"seresnext\", \"resnest50\"),\n",
+    "            \"layers_to_freeze\": choice(0, 2),\n",
+    "        },\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "tuning_settings = {\n",
+    "    \"iterations\": 10,\n",
+    "    \"max_concurrent_iterations\": 2,\n",
+    "    \"hyperparameter_sampling\": RandomParameterSampling(parameter_space),\n",
+    "    \"early_termination_policy\": BanditPolicy(\n",
+    "        evaluation_interval=2, slack_factor=0.2, delay_evaluation=6\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "automl_image_config = AutoMLImageConfig(\n",
+    "    task=ImageTask.IMAGE_CLASSIFICATION,\n",
+    "    compute_target=compute_target,\n",
+    "    training_data=training_dataset,\n",
+    "    validation_data=validation_dataset,\n",
+    "    **tuning_settings,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run = experiment.submit(automl_image_config)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "automl_image_run.wait_for_completion(wait_post_processing=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When doing a hyperparameter sweep, it can be useful to visualize the different configurations that were tried using the HyperDrive UI. You can navigate to this UI by going to the 'Child runs' tab in the UI of the main `automl_image_run` from above, which is the HyperDrive parent run. Then you can go into the 'Child runs' tab of this HyperDrive parent run. Alternatively, here below you can see directly the HyperDrive parent run and navigate to its 'Child runs' tab:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core import Run\n",
+    "\n",
+    "hyperdrive_run = Run(experiment=experiment, run_id=automl_image_run.id + \"_HD\")\n",
+    "hyperdrive_run"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Register the optimal vision model from the AutoML run\n",
+    "Once the run completes, we can register the model that was created from the best run (configuration that resulted in the best primary metric)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Register the model from the best run\n",
+    "\n",
+    "best_child_run = automl_image_run.get_best_child()\n",
+    "model_name = best_child_run.properties[\"model_name\"]\n",
+    "model = best_child_run.register_model(\n",
+    "    model_name=model_name, model_path=\"outputs/model.pt\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Deploy model as a web service\n",
+    "Once you have your trained model, you can deploy the model on Azure. You can deploy your trained model as a web service on Azure Container Instances ([ACI](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-container-instance)) or Azure Kubernetes Service ([AKS](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-kubernetes-service)). Please note that ACI only supports small models under 1 GB in size. For testing larger models or for the high-scale production stage, we recommend using AKS.\n",
+    "In this tutorial, we will deploy the model as a web service in AKS."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You will need to first create an AKS compute cluster or use an existing AKS cluster. You can use either GPU or CPU VM SKUs for your deployment cluster"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.compute import ComputeTarget, AksCompute\n",
+    "from azureml.exceptions import ComputeTargetException\n",
+    "\n",
+    "# Choose a name for your cluster\n",
+    "aks_name = \"aks-cpu-mc\"\n",
+    "# Check to see if the cluster already exists\n",
+    "try:\n",
+    "    aks_target = ComputeTarget(workspace=ws, name=aks_name)\n",
+    "    print(\"Found existing compute target\")\n",
+    "except ComputeTargetException:\n",
+    "    print(\"Creating a new compute target...\")\n",
+    "    # Provision AKS cluster with a CPU machine\n",
+    "    prov_config = AksCompute.provisioning_configuration(vm_size=\"STANDARD_D3_V2\")\n",
+    "    # Create the cluster\n",
+    "    aks_target = ComputeTarget.create(\n",
+    "        workspace=ws, name=aks_name, provisioning_configuration=prov_config\n",
+    "    )\n",
+    "    aks_target.wait_for_completion(show_output=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, you will need to define the [inference configuration](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#update-inference-configuration), that describes how to set up the web-service containing your model. You can use the scoring script and the environment from the training run in your inference config.\n",
+    "\n",
+    "<b>Note:</b> To change the model's settings, open the downloaded scoring script and modify the model_settings variable <i>before</i> deploying the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azureml.core.model import InferenceConfig\n",
+    "\n",
+    "best_child_run.download_file(\n",
+    "    \"outputs/scoring_file_v_1_0_0.py\", output_file_path=\"score.py\"\n",
+    ")\n",
+    "environment = best_child_run.get_environment()\n",
+    "inference_config = InferenceConfig(entry_script=\"score.py\", environment=environment)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can then deploy the model as an AKS web service."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Deploy the model from the best run as an AKS web service\n",
+    "from azureml.core.webservice import AksWebservice\n",
+    "from azureml.core.model import Model\n",
+    "\n",
+    "aks_config = AksWebservice.deploy_configuration(\n",
+    "    autoscale_enabled=True, cpu_cores=1, memory_gb=5, enable_app_insights=True\n",
+    ")\n",
+    "\n",
+    "aks_service = Model.deploy(\n",
+    "    ws,\n",
+    "    models=[model],\n",
+    "    inference_config=inference_config,\n",
+    "    deployment_config=aks_config,\n",
+    "    deployment_target=aks_target,\n",
+    "    name=\"automl-image-test-cpu-mc\",\n",
+    "    overwrite=True,\n",
+    ")\n",
+    "aks_service.wait_for_deployment(show_output=True)\n",
+    "print(aks_service.state)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Test the web service\n",
+    "Finally, let's test our deployed web service to predict new images. You can pass in any image. In this case, we'll use a random image from the dataset and pass it to the scoring URI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "# URL for the web service\n",
+    "scoring_uri = aks_service.scoring_uri\n",
+    "\n",
+    "# If the service is authenticated, set the key or token\n",
+    "key, _ = aks_service.get_keys()\n",
+    "\n",
+    "sample_image = \"./test_image.jpg\"\n",
+    "\n",
+    "# Load image data\n",
+    "data = open(sample_image, \"rb\").read()\n",
+    "\n",
+    "# Set the content type\n",
+    "headers = {\"Content-Type\": \"application/octet-stream\"}\n",
+    "\n",
+    "# If authentication is enabled, set the authorization header\n",
+    "headers[\"Authorization\"] = f\"Bearer {key}\"\n",
+    "\n",
+    "# Make the request and display the response\n",
+    "resp = requests.post(scoring_uri, data, headers=headers)\n",
+    "print(resp.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Visualize predictions\n",
+    "Now that we have scored a test image, we can visualize the prediction for this image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.image as mpimg\n",
+    "from PIL import Image\n",
+    "import numpy as np\n",
+    "import json\n",
+    "\n",
+    "IMAGE_SIZE = (18, 12)\n",
+    "plt.figure(figsize=IMAGE_SIZE)\n",
+    "img_np = mpimg.imread(sample_image)\n",
+    "img = Image.fromarray(img_np.astype(\"uint8\"), \"RGB\")\n",
+    "x, y = img.size\n",
+    "\n",
+    "fig, ax = plt.subplots(1, figsize=(15, 15))\n",
+    "# Display the image\n",
+    "ax.imshow(img_np)\n",
+    "\n",
+    "prediction = json.loads(resp.text)\n",
+    "label_index = np.argmax(prediction[\"probs\"])\n",
+    "label = prediction[\"labels\"][label_index]\n",
+    "conf_score = prediction[\"probs\"][label_index]\n",
+    "\n",
+    "display_text = \"{} ({})\".format(label, round(conf_score, 3))\n",
+    "print(display_text)\n",
+    "\n",
+    "color = \"red\"\n",
+    "plt.text(30, 30, display_text, color=color, fontsize=30)\n",
+    "\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.6 - AzureML",
+   "language": "python",
+   "name": "python3-azureml"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.10"
+  },
+  "nteract": {
+   "version": "nteract-front-end@1.0.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

how-to-use-azureml/automated-machine-learning/image-classification-multilabel/README.md

@@ -0,0 +1,15 @@
+---
+page_type: sample
+languages:
+- python
+products:
+- azure-machine-learning
+description: Notebook showing how to use AutoML for training an Image Classification Multi-Label model. We will use a small dataset to train the model, demonstrate how you can tune hyperparameters of the model to optimize model performance and deploy the model to use in inference scenarios.
+---
+
+# Image Classification Multi-Label using AutoML for Images
+- Dataset: Toy dataset with images of products found in a fridge
+    - **[Jupyter Notebook](auto-ml-image-classification-multilabel.ipynb)**
+        - train an Image Classification Multi-Label model using AutoML
+        - tune hyperparameters of the model to optimize model performance
+        - deploy the model to use in inference scenarios