{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/machine-learning-pipelines/nyc-taxi-data-regression-model-building/nyc-taxi-data-regression-model-building.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# NYC Taxi Data Regression Model\n",
        "This is an [Azure Machine Learning Pipelines](https://aka.ms/aml-pipelines) version of two-part tutorial ([Part 1](https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-data-prep), [Part 2](https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-auto-train-models)) available for Azure Machine Learning.\n",
        "\n",
        "You can combine the two part tutorial into one using AzureML Pipelines as Pipelines provide a way to stitch together various steps involved (like data preparation and training in this case) in a machine learning workflow.\n",
        "\n",
        "In this notebook, you learn how to prepare data for regression modeling by using the [Azure Machine Learning Data Prep SDK](https://aka.ms/data-prep-sdk) for Python. You run various transformations to filter and combine two different NYC taxi data sets. Once you prepare the NYC taxi data for regression modeling, then you will use [AutoMLStep](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlstep?view=azure-ml-py) available with [Azure Machine Learning Pipelines](https://aka.ms/aml-pipelines) to define your machine learning goals and constraints as well as to launch the automated machine learning process. The automated machine learning technique iterates over many combinations of algorithms and hyperparameters until it finds the best model based on your criterion.\n",
        "\n",
        "After you complete building the model, you can predict the cost of a taxi trip by training a model on data features. These features include the pickup day and time, the number of passengers, and the pickup location.\n",
        "\n",
        "## Prerequisite\n",
        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you go through the configuration Notebook located at https://github.com/Azure/MachineLearningNotebooks first if you haven't. This sets you up with a working config file that has information on your workspace, subscription id, etc.\n",
        "\n",
        "We will run various transformations to filter and combine two different NYC taxi data sets. We will use DataPrep SDK for this preparing data. \n",
        "\n",
        "Perform `pip install azureml-dataprep` if you have't already done so."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prepare data for regression modeling\n",
        "First, we will prepare data for regression modeling. We will leverage the convenience of Azure Open Datasets along with the power of Azure Machine Learning service to create a regression model to predict NYC taxi fare prices. Perform `pip install azureml-opendatasets` to get the open dataset package.  The Open Datasets package contains a class representing each data source (NycTlcGreen and NycTlcYellow) to easily filter date parameters before downloading.\n",
        "\n",
        "\n",
        "### Load data\n",
        "Begin by creating a dataframe to hold the taxi data. When working in a non-Spark environment, Open Datasets only allows downloading one month of data at a time with certain classes to avoid MemoryError with large datasets. To download a year of taxi data, iteratively fetch one month at a time, and before appending it to green_df_raw, randomly sample 500 records from each month to avoid bloating the dataframe. Then preview the data. To keep this process short, we are sampling data of only 1 month.\n",
        "\n",
        "Note: Open Datasets has mirroring classes for working in Spark environments where data size and memory aren't a concern."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import azureml.core\n",
        "# Check core SDK version number\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.opendatasets import NycTlcGreen, NycTlcYellow\n",
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "from dateutil.relativedelta import relativedelta\n",
        "\n",
        "green_df_raw = pd.DataFrame([])\n",
        "start = datetime.strptime(\"1/1/2016\",\"%m/%d/%Y\")\n",
        "end = datetime.strptime(\"1/31/2016\",\"%m/%d/%Y\")\n",
        "\n",
        "number_of_months = 1\n",
        "sample_size = 5000\n",
        "\n",
        "for sample_month in range(number_of_months):\n",
        "    temp_df_green = NycTlcGreen(start + relativedelta(months=sample_month), end + relativedelta(months=sample_month)) \\\n",
        "        .to_pandas_dataframe()\n",
        "    green_df_raw = green_df_raw.append(temp_df_green.sample(sample_size))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "yellow_df_raw = pd.DataFrame([])\n",
        "start = datetime.strptime(\"1/1/2016\",\"%m/%d/%Y\")\n",
        "end = datetime.strptime(\"1/31/2016\",\"%m/%d/%Y\")\n",
        "\n",
        "sample_size = 500\n",
        "\n",
        "for sample_month in range(number_of_months):\n",
        "    temp_df_yellow = NycTlcYellow(start + relativedelta(months=sample_month), end + relativedelta(months=sample_month)) \\\n",
        "        .to_pandas_dataframe()\n",
        "    yellow_df_raw = yellow_df_raw.append(temp_df_yellow.sample(sample_size))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### See the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import azureml.dataprep as dprep\n",
        "from IPython.display import display\n",
        "\n",
        "display(green_df_raw.head(5))\n",
        "display(yellow_df_raw.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Download data locally and then upload to Azure Blob\n",
        "This is a one-time process to save the dave in the default datastore. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "dataDir = \"data\"\n",
        "\n",
        "if not os.path.exists(dataDir):\n",
        "    os.mkdir(dataDir)\n",
        "\n",
        "greenDir = dataDir + \"/green\"\n",
        "yelloDir = dataDir + \"/yellow\"\n",
        "\n",
        "if not os.path.exists(greenDir):\n",
        "    os.mkdir(greenDir)\n",
        "    \n",
        "if not os.path.exists(yelloDir):\n",
        "    os.mkdir(yelloDir)\n",
        "    \n",
        "greenTaxiData = greenDir + \"/part-00000\"\n",
        "yellowTaxiData = yelloDir + \"/part-00000\"\n",
        "\n",
        "green_df_raw.to_csv(greenTaxiData, index=False)\n",
        "yellow_df_raw.to_csv(yellowTaxiData, index=False)\n",
        "\n",
        "print(\"Data written to local folder.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Workspace\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "print(\"Workspace: \" + ws.name, \"Region: \" + ws.location, sep = '\\n')\n",
        "\n",
        "# Default datastore\n",
        "default_store = ws.get_default_datastore() \n",
        "\n",
        "default_store.upload_files([greenTaxiData], \n",
        "                           target_path = 'green', \n",
        "                           overwrite = False, \n",
        "                           show_progress = True)\n",
        "\n",
        "default_store.upload_files([yellowTaxiData], \n",
        "                           target_path = 'yellow', \n",
        "                           overwrite = False, \n",
        "                           show_progress = True)\n",
        "\n",
        "print(\"Upload calls completed.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setup Compute\n",
        "#### Create new or use an existing compute"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.compute import AmlCompute\n",
        "from azureml.core.compute import ComputeTarget\n",
        "\n",
        "aml_compute = ws.get_default_compute_target(\"CPU\")\n",
        "\n",
        "if aml_compute is None:\n",
        "    amlcompute_cluster_name = \"cpu-cluster\"\n",
        "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = \"STANDARD_D2_V2\",\n",
        "                                                                max_nodes = 4)\n",
        "\n",
        "    aml_compute = ComputeTarget.create(ws, amlcompute_cluster_name, provisioning_config)\n",
        "    aml_compute.wait_for_completion(show_output = True, min_node_count = None, timeout_in_minutes = 20)\n",
        "\n",
        "aml_compute"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define RunConfig for the compute\n",
        "We need `azureml-dataprep` SDK for all the steps below. We will also use `pandas`, `scikit-learn` and `automl` for the training step. Defining the `runconfig` for that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.runconfig import RunConfiguration\n",
        "from azureml.core.conda_dependencies import CondaDependencies\n",
        "\n",
        "# Create a new runconfig object\n",
        "aml_run_config = RunConfiguration()\n",
        "\n",
        "# Use the aml_compute you created above. \n",
        "aml_run_config.target = aml_compute\n",
        "\n",
        "# Enable Docker\n",
        "aml_run_config.environment.docker.enabled = True\n",
        "\n",
        "# Set Docker base image to the default CPU-based image\n",
        "aml_run_config.environment.docker.base_image = \"mcr.microsoft.com/azureml/base:0.2.1\"\n",
        "\n",
        "# Use conda_dependencies.yml to create a conda environment in the Docker image for execution\n",
        "aml_run_config.environment.python.user_managed_dependencies = False\n",
        "\n",
        "# Auto-prepare the Docker image when used for execution (if it is not already prepared)\n",
        "aml_run_config.auto_prepare_environment = True\n",
        "\n",
        "# Specify CondaDependencies obj, add necessary packages\n",
        "aml_run_config.environment.python.conda_dependencies = CondaDependencies.create(\n",
        "    conda_packages=['pandas','scikit-learn'], \n",
        "    pip_packages=['azureml-sdk', 'azureml-dataprep', 'azureml-train-automl==1.0.33'], \n",
        "    pin_sdk_version=False)\n",
        "\n",
        "print (\"Run configuration created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Prepare data\n",
        "Now we will prepare for regression modeling by using the `Azure Machine Learning Data Prep SDK for Python`. We run various transformations to filter and combine two different NYC taxi data sets.\n",
        "\n",
        "We achieve this by creating a separate step for each transformation as this allows us to reuse the steps and saves us from running all over again in case of any change. We will keep data preparation scripts in one subfolder and training scripts in another.\n",
        "\n",
        "> The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define Useful Colums\n",
        "Here we are defining a set of \"useful\" columns for both Green and Yellow taxi data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "display(green_df_raw.columns)\n",
        "display(yellow_df_raw.columns)\n",
        "\n",
        "# useful columns needed for the Azure Machine Learning NYC Taxi tutorial\n",
        "useful_columns = str([\"cost\", \"distance\", \"dropoff_datetime\", \"dropoff_latitude\", \n",
        "                      \"dropoff_longitude\", \"passengers\", \"pickup_datetime\", \n",
        "                      \"pickup_latitude\", \"pickup_longitude\", \"store_forward\", \"vendor\"]).replace(\",\", \";\")\n",
        "\n",
        "print(\"Useful columns defined.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Cleanse Green taxi data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.data.data_reference import DataReference \n",
        "from azureml.pipeline.core import PipelineData\n",
        "from azureml.pipeline.steps import PythonScriptStep\n",
        "\n",
        "# python scripts folder\n",
        "prepare_data_folder = './scripts/prepdata'\n",
        "\n",
        "blob_green_data = DataReference(\n",
        "    datastore=default_store,\n",
        "    data_reference_name=\"green_taxi_data\",\n",
        "    path_on_datastore=\"green/part-00000\")\n",
        "\n",
        "# rename columns as per Azure Machine Learning NYC Taxi tutorial\n",
        "green_columns = str({ \n",
        "    \"vendorID\": \"vendor\",\n",
        "    \"lpepPickupDatetime\": \"pickup_datetime\",\n",
        "    \"lpepDropoffDatetime\": \"dropoff_datetime\",\n",
        "    \"storeAndFwdFlag\": \"store_forward\",\n",
        "    \"pickupLongitude\": \"pickup_longitude\",\n",
        "    \"pickupLatitude\": \"pickup_latitude\",\n",
        "    \"dropoffLongitude\": \"dropoff_longitude\",\n",
        "    \"dropoffLatitude\": \"dropoff_latitude\",\n",
        "    \"passengerCount\": \"passengers\",\n",
        "    \"fareAmount\": \"cost\",\n",
        "    \"tripDistance\": \"distance\"\n",
        "}).replace(\",\", \";\")\n",
        "\n",
        "# Define output after cleansing step\n",
        "cleansed_green_data = PipelineData(\"green_taxi_data\", datastore=default_store)\n",
        "\n",
        "print('Cleanse script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# cleansing step creation\n",
        "# See the cleanse.py for details about input and output\n",
        "cleansingStepGreen = PythonScriptStep(\n",
        "    name=\"Cleanse Green Taxi Data\",\n",
        "    script_name=\"cleanse.py\", \n",
        "    arguments=[\"--input_cleanse\", blob_green_data, \n",
        "               \"--useful_columns\", useful_columns,\n",
        "               \"--columns\", green_columns,\n",
        "               \"--output_cleanse\", cleansed_green_data],\n",
        "    inputs=[blob_green_data],\n",
        "    outputs=[cleansed_green_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig=aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"cleansingStepGreen created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Cleanse Yellow taxi data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "blob_yellow_data = DataReference(\n",
        "    datastore=default_store,\n",
        "    data_reference_name=\"yellow_taxi_data\",\n",
        "    path_on_datastore=\"yellow/part-00000\")\n",
        "\n",
        "yellow_columns = str({\n",
        "    \"vendorID\": \"vendor\",\n",
        "    \"tpepPickupDateTime\": \"pickup_datetime\",\n",
        "    \"tpepDropoffDateTime\": \"dropoff_datetime\",\n",
        "    \"storeAndFwdFlag\": \"store_forward\",\n",
        "    \"startLon\": \"pickup_longitude\",\n",
        "    \"startLat\": \"pickup_latitude\",\n",
        "    \"endLon\": \"dropoff_longitude\",\n",
        "    \"endLat\": \"dropoff_latitude\",\n",
        "    \"passengerCount\": \"passengers\",\n",
        "    \"fareAmount\": \"cost\",\n",
        "    \"tripDistance\": \"distance\"\n",
        "}).replace(\",\", \";\")\n",
        "\n",
        "# Define output after cleansing step\n",
        "cleansed_yellow_data = PipelineData(\"yellow_taxi_data\", datastore=default_store)\n",
        "\n",
        "print('Cleanse script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# cleansing step creation\n",
        "# See the cleanse.py for details about input and output\n",
        "cleansingStepYellow = PythonScriptStep(\n",
        "    name=\"Cleanse Yellow Taxi Data\",\n",
        "    script_name=\"cleanse.py\", \n",
        "    arguments=[\"--input_cleanse\", blob_yellow_data, \n",
        "               \"--useful_columns\", useful_columns,\n",
        "               \"--columns\", yellow_columns,\n",
        "               \"--output_cleanse\", cleansed_yellow_data],\n",
        "    inputs=[blob_yellow_data],\n",
        "    outputs=[cleansed_yellow_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig=aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"cleansingStepYellow created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Merge cleansed Green and Yellow datasets\n",
        "We are creating a single data source by merging the cleansed versions of Green and Yellow taxi data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Define output after merging step\n",
        "merged_data = PipelineData(\"merged_data\", datastore=default_store)\n",
        "\n",
        "print('Merge script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# merging step creation\n",
        "# See the merge.py for details about input and output\n",
        "mergingStep = PythonScriptStep(\n",
        "    name=\"Merge Taxi Data\",\n",
        "    script_name=\"merge.py\", \n",
        "    arguments=[\"--input_green_merge\", cleansed_green_data, \n",
        "               \"--input_yellow_merge\", cleansed_yellow_data,\n",
        "               \"--output_merge\", merged_data],\n",
        "    inputs=[cleansed_green_data, cleansed_yellow_data],\n",
        "    outputs=[merged_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig=aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"mergingStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Filter data\n",
        "This step filters out coordinates for locations that are outside the city border. We use a TypeConverter object to change the latitude and longitude fields to decimal type. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Define output after merging step\n",
        "filtered_data = PipelineData(\"filtered_data\", datastore=default_store)\n",
        "\n",
        "print('Filter script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# filter step creation\n",
        "# See the filter.py for details about input and output\n",
        "filterStep = PythonScriptStep(\n",
        "    name=\"Filter Taxi Data\",\n",
        "    script_name=\"filter.py\", \n",
        "    arguments=[\"--input_filter\", merged_data, \n",
        "               \"--output_filter\", filtered_data],\n",
        "    inputs=[merged_data],\n",
        "    outputs=[filtered_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"FilterStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Normalize data\n",
        "In this step, we split the pickup and dropoff datetime values into the respective date and time columns and then we rename the columns to use meaningful names."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Define output after normalize step\n",
        "normalized_data = PipelineData(\"normalized_data\", datastore=default_store)\n",
        "\n",
        "print('Normalize script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# normalize step creation\n",
        "# See the normalize.py for details about input and output\n",
        "normalizeStep = PythonScriptStep(\n",
        "    name=\"Normalize Taxi Data\",\n",
        "    script_name=\"normalize.py\", \n",
        "    arguments=[\"--input_normalize\", filtered_data, \n",
        "               \"--output_normalize\", normalized_data],\n",
        "    inputs=[filtered_data],\n",
        "    outputs=[normalized_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"normalizeStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Transform data\n",
        "Transform the normalized taxi data to final required format. This steps does the following:\n",
        "\n",
        "- Split the pickup and dropoff date further into the day of the week, day of the month, and month values. \n",
        "- To get the day of the week value, uses the derive_column_by_example() function. The function takes an array parameter of example objects that define the input data, and the preferred output. The function automatically determines the preferred transformation. For the pickup and dropoff time columns, split the time into the hour, minute, and second by using the split_column_by_example() function with no example parameter.\n",
        "- After new features are generated, use the drop_columns() function to delete the original fields as the newly generated features are preferred. \n",
        "- Rename the rest of the fields to use meaningful descriptions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Define output after transforme step\n",
        "transformed_data = PipelineData(\"transformed_data\", datastore=default_store)\n",
        "\n",
        "print('Transform script is in {}.'.format(os.path.realpath(prepare_data_folder)))\n",
        "\n",
        "# transform step creation\n",
        "# See the transform.py for details about input and output\n",
        "transformStep = PythonScriptStep(\n",
        "    name=\"Transform Taxi Data\",\n",
        "    script_name=\"transform.py\", \n",
        "    arguments=[\"--input_transform\", normalized_data,\n",
        "               \"--output_transform\", transformed_data],\n",
        "    inputs=[normalized_data],\n",
        "    outputs=[transformed_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=prepare_data_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"transformStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Extract features\n",
        "Add the following columns to be features for our model creation. The prediction value will be *cost*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "feature_columns = str(['pickup_weekday','pickup_hour', 'distance','passengers', 'vendor']).replace(\",\", \";\")\n",
        "\n",
        "train_model_folder = './scripts/trainmodel'\n",
        "\n",
        "print('Extract script is in {}.'.format(os.path.realpath(train_model_folder)))\n",
        "\n",
        "# features data after transform step\n",
        "features_data = PipelineData(\"features_data\", datastore=default_store)\n",
        "\n",
        "# featurization step creation\n",
        "# See the featurization.py for details about input and output\n",
        "featurizationStep = PythonScriptStep(\n",
        "    name=\"Extract Features\",\n",
        "    script_name=\"featurization.py\", \n",
        "    arguments=[\"--input_featurization\", transformed_data, \n",
        "               \"--useful_columns\", feature_columns,\n",
        "               \"--output_featurization\", features_data],\n",
        "    inputs=[transformed_data],\n",
        "    outputs=[features_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=train_model_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"featurizationStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Extract label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "label_columns = str(['cost']).replace(\",\", \";\")\n",
        "\n",
        "# label data after transform step\n",
        "label_data = PipelineData(\"label_data\", datastore=default_store)\n",
        "\n",
        "print('Extract script is in {}.'.format(os.path.realpath(train_model_folder)))\n",
        "\n",
        "# label step creation\n",
        "# See the featurization.py for details about input and output\n",
        "labelStep = PythonScriptStep(\n",
        "    name=\"Extract Labels\",\n",
        "    script_name=\"featurization.py\", \n",
        "    arguments=[\"--input_featurization\", transformed_data, \n",
        "               \"--useful_columns\", label_columns,\n",
        "               \"--output_featurization\", label_data],\n",
        "    inputs=[transformed_data],\n",
        "    outputs=[label_data],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=train_model_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"labelStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Split the data into train and test sets\n",
        "This function segregates the data into the **x**, features, dataset for model training and **y**, values to predict, dataset for testing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# train and test splits output\n",
        "output_split_train_x = PipelineData(\"output_split_train_x\", datastore=default_store)\n",
        "output_split_train_y = PipelineData(\"output_split_train_y\", datastore=default_store)\n",
        "output_split_test_x = PipelineData(\"output_split_test_x\", datastore=default_store)\n",
        "output_split_test_y = PipelineData(\"output_split_test_y\", datastore=default_store)\n",
        "\n",
        "print('Data spilt script is in {}.'.format(os.path.realpath(train_model_folder)))\n",
        "\n",
        "# test train split step creation\n",
        "# See the train_test_split.py for details about input and output\n",
        "testTrainSplitStep = PythonScriptStep(\n",
        "    name=\"Train Test Data Split\",\n",
        "    script_name=\"train_test_split.py\", \n",
        "    arguments=[\"--input_split_features\", features_data, \n",
        "               \"--input_split_labels\", label_data,\n",
        "               \"--output_split_train_x\", output_split_train_x,\n",
        "               \"--output_split_train_y\", output_split_train_y,\n",
        "               \"--output_split_test_x\", output_split_test_x,\n",
        "               \"--output_split_test_y\", output_split_test_y],\n",
        "    inputs=[features_data, label_data],\n",
        "    outputs=[output_split_train_x, output_split_train_y, output_split_test_x, output_split_test_y],\n",
        "    compute_target=aml_compute,\n",
        "    runconfig = aml_run_config,\n",
        "    source_directory=train_model_folder,\n",
        "    allow_reuse=True\n",
        ")\n",
        "\n",
        "print(\"testTrainSplitStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Use automated machine learning to build regression model\n",
        "Now we will use **automated machine learning** to build the regression model. We will use [AutoMLStep](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlstep?view=azure-ml-py) in AML Pipelines for this part. These functions use various features from the data set and allow an automated model to build relationships between the features and the price of a taxi trip."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Automatically train a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Create experiment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "\n",
        "experiment = Experiment(ws, 'NYCTaxi_Tutorial_Pipelines')\n",
        "\n",
        "print(\"Experiment created\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Create get_data script\n",
        "\n",
        "A script with `get_data()` function is necessary to fetch training features(X) and labels(Y) on remote compute, from input data. Here we use mounted path of `train_test_split` step to get the x and y train values. They are added as environment variable on compute machine by default\n",
        "\n",
        "Note: Every DataReference are added as environment variable on compute machine since the defualt mode is mount"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print('get_data.py will be written to {}.'.format(os.path.realpath(train_model_folder)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile $train_model_folder/get_data.py\n",
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "def get_data():\n",
        "    print(\"In get_data\")\n",
        "    print(os.environ['AZUREML_DATAREFERENCE_output_split_train_x'])\n",
        "    X_train  = pd.read_csv(os.environ['AZUREML_DATAREFERENCE_output_split_train_x'] + \"/part-00000\", header=0)\n",
        "    y_train  = pd.read_csv(os.environ['AZUREML_DATAREFERENCE_output_split_train_y'] + \"/part-00000\", header=0)\n",
        "    \n",
        "    return { \"X\" : X_train.values, \"y\" : y_train.values.flatten() }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define settings for autogeneration and tuning\n",
        "\n",
        "Here we define the experiment parameter and model settings for autogeneration and tuning. We can specify automl_settings as **kwargs as well. Also note that we have to use a get_data() function for remote excutions. See get_data script for more details.\n",
        "\n",
        "Use your defined training settings as a parameter to an `AutoMLConfig` object. Additionally, specify your training data and the type of model, which is `regression` in this case.\n",
        "\n",
        "Note: When using AmlCompute, we can't pass Numpy arrays directly to the fit method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import logging\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "\n",
        "# Change iterations to a reasonable number (50) to get better accuracy\n",
        "automl_settings = {\n",
        "    \"iteration_timeout_minutes\" : 10,\n",
        "    \"iterations\" : 2,\n",
        "    \"primary_metric\" : 'spearman_correlation',\n",
        "    \"preprocess\" : True,\n",
        "    \"verbosity\" : logging.INFO,\n",
        "    \"n_cross_validations\": 5\n",
        "}\n",
        "\n",
        "automl_config = AutoMLConfig(task = 'regression',\n",
        "                             debug_log = 'automated_ml_errors.log',\n",
        "                             path = train_model_folder,\n",
        "                             compute_target=aml_compute,\n",
        "                             run_configuration=aml_run_config,\n",
        "                             data_script = train_model_folder + \"/get_data.py\",\n",
        "                             **automl_settings)\n",
        "                             \n",
        "print(\"AutoML config created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Define AutoMLStep"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.automl import AutoMLStep\n",
        "\n",
        "trainWithAutomlStep = AutoMLStep(\n",
        "    name='AutoML_Regression',\n",
        "    automl_config=automl_config,\n",
        "    inputs=[output_split_train_x, output_split_train_y],\n",
        "    allow_reuse=True,\n",
        "    hash_paths=[os.path.realpath(train_model_folder)])\n",
        "\n",
        "print(\"trainWithAutomlStep created.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Build and run the pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.pipeline.core import Pipeline\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "pipeline_steps = [trainWithAutomlStep]\n",
        "\n",
        "pipeline = Pipeline(workspace = ws, steps=pipeline_steps)\n",
        "print(\"Pipeline is built.\")\n",
        "\n",
        "pipeline_run = experiment.submit(pipeline, regenerate_outputs=False)\n",
        "\n",
        "print(\"Pipeline submitted for execution.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "RunDetails(pipeline_run).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Explore the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Before we proceed we need to wait for the run to complete.\n",
        "pipeline_run.wait_for_completion()\n",
        "\n",
        "# functions to download output to local and fetch as dataframe\n",
        "def get_download_path(download_path, output_name):\n",
        "    output_folder = os.listdir(download_path + '/azureml')[0]\n",
        "    path =  download_path + '/azureml/' + output_folder + '/' + output_name\n",
        "    return path\n",
        "\n",
        "def fetch_df(step, output_name):\n",
        "    output_data = step.get_output_data(output_name)\n",
        "    \n",
        "    download_path = './outputs/' + output_name\n",
        "    output_data.download(download_path)\n",
        "    df_path = get_download_path(download_path, output_name) + '/part-00000'\n",
        "    return dprep.auto_read_file(path=df_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View cleansed taxi data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "green_cleanse_step = pipeline_run.find_step_run(cleansingStepGreen.name)[0]\n",
        "yellow_cleanse_step = pipeline_run.find_step_run(cleansingStepYellow.name)[0]\n",
        "\n",
        "cleansed_green_df = fetch_df(green_cleanse_step, cleansed_green_data.name)\n",
        "cleansed_yellow_df = fetch_df(yellow_cleanse_step, cleansed_yellow_data.name)\n",
        "\n",
        "display(cleansed_green_df.head(5))\n",
        "display(cleansed_yellow_df.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View the combined taxi data profile"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "merge_step = pipeline_run.find_step_run(mergingStep.name)[0]\n",
        "combined_df = fetch_df(merge_step, merged_data.name)\n",
        "\n",
        "display(combined_df.get_profile())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View the filtered taxi data profile"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "filter_step = pipeline_run.find_step_run(filterStep.name)[0]\n",
        "filtered_df = fetch_df(filter_step, filtered_data.name)\n",
        "\n",
        "display(filtered_df.get_profile())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View normalized taxi data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "normalize_step = pipeline_run.find_step_run(normalizeStep.name)[0]\n",
        "normalized_df = fetch_df(normalize_step, normalized_data.name)\n",
        "\n",
        "display(normalized_df.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View transformed taxi data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "transform_step = pipeline_run.find_step_run(transformStep.name)[0]\n",
        "transformed_df = fetch_df(transform_step, transformed_data.name)\n",
        "\n",
        "display(transformed_df.get_profile())\n",
        "display(transformed_df.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View training data used by AutoML"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "split_step = pipeline_run.find_step_run(testTrainSplitStep.name)[0]\n",
        "train_split_x = fetch_df(split_step, output_split_train_x.name)\n",
        "train_split_y = fetch_df(split_step, output_split_train_y.name)\n",
        "\n",
        "display_x_train = train_split_x.keep_columns(columns=[\"vendor\", \"pickup_weekday\", \"pickup_hour\", \"passengers\", \"distance\"])\n",
        "display_y_train = train_split_y.rename_columns(column_pairs={\"Column1\": \"cost\"})\n",
        "\n",
        "display(display_x_train.get_profile())\n",
        "display(display_x_train.head(5))\n",
        "display(display_y_train.get_profile())\n",
        "display(display_y_train.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### View the details of the AutoML run"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.automl.run import AutoMLRun\n",
        "#from azureml.widgets import RunDetails\n",
        "\n",
        "# workaround to get the automl run as its the last step in the pipeline \n",
        "# and get_steps() returns the steps from latest to first\n",
        "\n",
        "for step in pipeline_run.get_steps():\n",
        "    automl_step_run_id = step.id\n",
        "    print(step.name)\n",
        "    print(automl_step_run_id)\n",
        "    break\n",
        "\n",
        "automl_run = AutoMLRun(experiment = experiment, run_id=automl_step_run_id)\n",
        "#RunDetails(automl_run).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Retrieve all Child runs\n",
        "\n",
        "We use SDK methods to fetch all the child runs and see individual metrics that we log."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "children = list(automl_run.get_children())\n",
        "metricslist = {}\n",
        "for run in children:\n",
        "    properties = run.get_properties()\n",
        "    metrics = {k: v for k, v in run.get_metrics().items() if isinstance(v, float)}\n",
        "    metricslist[int(properties['iteration'])] = metrics\n",
        "\n",
        "rundata = pd.DataFrame(metricslist).sort_index(1)\n",
        "rundata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retreive the best model\n",
        "\n",
        "Uncomment the below cell to retrieve the best model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# best_run, fitted_model = automl_run.get_output()\n",
        "# print(best_run)\n",
        "# print(fitted_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Test the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Get test data\n",
        "\n",
        "Uncomment the below cell to get test data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# split_step = pipeline_run.find_step_run(testTrainSplitStep.name)[0]\n",
        "\n",
        "# x_test = fetch_df(split_step, output_split_test_x.name)\n",
        "# y_test = fetch_df(split_step, output_split_test_y.name)\n",
        "\n",
        "# display(x_test.keep_columns(columns=[\"vendor\", \"pickup_weekday\", \"pickup_hour\", \"passengers\", \"distance\"]).head(5))\n",
        "# display(y_test.rename_columns(column_pairs={\"Column1\": \"cost\"}).head(5))\n",
        "\n",
        "# x_test = x_test.to_pandas_dataframe()\n",
        "# y_test = y_test.to_pandas_dataframe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Test the best fitted model\n",
        "\n",
        "Uncomment the below cell to test the best fitted model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# y_predict = fitted_model.predict(x_test.values)\n",
        "\n",
        "# y_actual =  y_test.iloc[:,0].values.tolist()\n",
        "\n",
        "# display(pd.DataFrame({'Actual':y_actual, 'Predicted':y_predict}).head(5))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# import matplotlib.pyplot as plt\n",
        "\n",
        "# fig = plt.figure(figsize=(14, 10))\n",
        "# ax1 = fig.add_subplot(111)\n",
        "\n",
        "# distance_vals = [x[4] for x in x_test.values]\n",
        "\n",
        "# ax1.scatter(distance_vals[:100], y_predict[:100], s=18, c='b', marker=\"s\", label='Predicted')\n",
        "# ax1.scatter(distance_vals[:100], y_actual[:100], s=18, c='r', marker=\"o\", label='Actual')\n",
        "\n",
        "# ax1.set_xlabel('distance (mi)')\n",
        "# ax1.set_title('Predicted and Actual Cost/Distance')\n",
        "# ax1.set_ylabel('Cost ($)')\n",
        "\n",
        "# plt.legend(loc='upper left', prop={'size': 12})\n",
        "# plt.rcParams.update({'font.size': 14})\n",
        "# plt.show()"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "sanpil"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}