MachineLearningNotebooks/tutorials/regression-automated-ml.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/tutorials/regression-part2-automated-ml.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Tutorial: Use automated machine learning to predict taxi fares"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this tutorial, you use automated machine learning in Azure Machine Learning service to create a regression model to predict NYC taxi fare prices. This process accepts training data and configuration settings, and automatically iterates through combinations of different feature normalization/standardization methods, models, and hyperparameter settings to arrive at the best model.\n",
        "\n",
        "In this tutorial you learn the following tasks:\n",
        "\n",
        "* Download, transform, and clean data using Azure Open Datasets\n",
        "* Train an automated machine learning regression model\n",
        "* Calculate model accuracy\n",
        "\n",
        "If you don\u00e2\u20ac\u2122t have an Azure subscription, create a free account before you begin. Try the [free or paid version](https://aka.ms/AMLFree) of Azure Machine Learning service today."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "* Complete the [setup tutorial](https://docs.microsoft.com/azure/machine-learning/service/tutorial-1st-experiment-sdk-setup) if you don't already have an Azure Machine Learning service workspace or notebook virtual machine.\n",
        "* After you complete the setup tutorial, open the **tutorials/regression-automated-ml.ipynb** notebook using the same notebook server.\n",
        "\n",
        "This tutorial is also available on [GitHub](https://github.com/Azure/MachineLearningNotebooks/tree/master/tutorials) if you wish to run it in your own [local environment](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment#local). Run `pip install azureml-sdk[automl] azureml-opendatasets azureml-widgets` to get the required packages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Download and prepare data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Import the necessary packages. The Open Datasets package contains a class representing each data source (`NycTlcGreen` for example) to easily filter date parameters before downloading."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.opendatasets import NycTlcGreen\n",
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "from dateutil.relativedelta import relativedelta"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Begin by creating a dataframe to hold the taxi data. When working in a non-Spark environment, Open Datasets only allows downloading one month of data at a time with certain classes to avoid `MemoryError` with large datasets. To download taxi data, iteratively fetch one month at a time, and before appending it to `green_taxi_df` randomly sample 2,000 records from each month to avoid bloating the dataframe. Then preview the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "green_taxi_df = pd.DataFrame([])\n",
        "start = datetime.strptime(\"1/1/2015\",\"%m/%d/%Y\")\n",
        "end = datetime.strptime(\"1/31/2015\",\"%m/%d/%Y\")\n",
        "\n",
        "for sample_month in range(12):\n",
        "    temp_df_green = NycTlcGreen(start + relativedelta(months=sample_month), end + relativedelta(months=sample_month)) \\\n",
        "        .to_pandas_dataframe()\n",
        "    green_taxi_df = green_taxi_df.append(temp_df_green.sample(2000))\n",
        "    \n",
        "green_taxi_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that the initial data is loaded, define a function to create various time-based features from the pickup datetime field. This will create new fields for the month number, day of month, day of week, and hour of day, and will allow the model to factor in time-based seasonality. \n",
        "\n",
        "Use the `apply()` function on the dataframe to iteratively apply the `build_time_features()` function to each row in the taxi data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def build_time_features(vector):\n",
        "    pickup_datetime = vector[0]\n",
        "    month_num = pickup_datetime.month\n",
        "    day_of_month = pickup_datetime.day\n",
        "    day_of_week = pickup_datetime.weekday()\n",
        "    hour_of_day = pickup_datetime.hour\n",
        "    \n",
        "    return pd.Series((month_num, day_of_month, day_of_week, hour_of_day))\n",
        "\n",
        "green_taxi_df[[\"month_num\", \"day_of_month\",\"day_of_week\", \"hour_of_day\"]] = green_taxi_df[[\"lpepPickupDatetime\"]].apply(build_time_features, axis=1)\n",
        "green_taxi_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Remove some of the columns that you won't need for training or additional feature building."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "columns_to_remove = [\"lpepPickupDatetime\", \"lpepDropoffDatetime\", \"puLocationId\", \"doLocationId\", \"extra\", \"mtaTax\",\n",
        "                     \"improvementSurcharge\", \"tollsAmount\", \"ehailFee\", \"tripType\", \"rateCodeID\", \n",
        "                     \"storeAndFwdFlag\", \"paymentType\", \"fareAmount\", \"tipAmount\"\n",
        "                    ]\n",
        "for col in columns_to_remove:\n",
        "    green_taxi_df.pop(col)\n",
        "    \n",
        "green_taxi_df.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Cleanse data "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run the `describe()` function on the new dataframe to see summary statistics for each field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "green_taxi_df.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "From the summary statistics, you see that there are several fields that have outliers or values that will reduce model accuracy. First filter the lat/long fields to be within the bounds of the Manhattan area. This will filter out longer taxi trips or trips that are outliers in respect to their relationship with other features. \n",
        "\n",
        "Additionally filter the `tripDistance` field to be greater than zero but less than 31 miles (the haversine distance between the two lat/long pairs). This eliminates long outlier trips that have inconsistent trip cost.\n",
        "\n",
        "Lastly, the `totalAmount` field has negative values for the taxi fares, which don't make sense in the context of our model, and the `passengerCount` field has bad data with the minimum values being zero.\n",
        "\n",
        "Filter out these anomalies using query functions, and then remove the last few columns unnecessary for training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "final_df = green_taxi_df.query(\"pickupLatitude>=40.53 and pickupLatitude<=40.88\")\n",
        "final_df = final_df.query(\"pickupLongitude>=-74.09 and pickupLongitude<=-73.72\")\n",
        "final_df = final_df.query(\"tripDistance>=0.25 and tripDistance<31\")\n",
        "final_df = final_df.query(\"passengerCount>0 and totalAmount>0\")\n",
        "\n",
        "columns_to_remove_for_training = [\"pickupLongitude\", \"pickupLatitude\", \"dropoffLongitude\", \"dropoffLatitude\"]\n",
        "for col in columns_to_remove_for_training:\n",
        "    final_df.pop(col)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Call `describe()` again on the data to ensure cleansing worked as expected. You now have a prepared and cleansed set of taxi, holiday, and weather data to use for machine learning model training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "final_df.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Configure workspace\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create a workspace object from the existing workspace. A [Workspace](https://docs.microsoft.com/python/api/azureml-core/azureml.core.workspace.workspace?view=azure-ml-py) is a class that accepts your Azure subscription and resource information. It also creates a cloud resource to monitor and track your model runs. `Workspace.from_config()` reads the file **config.json** and loads the authentication details into an object named `ws`. `ws` is used throughout the rest of the code in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.workspace import Workspace\n",
        "ws = Workspace.from_config()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Split the data into train and test sets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Split the data into training and test sets by using the `train_test_split` function in the `scikit-learn` library. This function segregates the data into the x (**features**) data set for model training and the y (**values to predict**) data set for testing. The `test_size` parameter determines the percentage of data to allocate to testing. The `random_state` parameter sets a seed to the random generator, so that your train-test splits are deterministic."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "y_df = final_df.pop(\"totalAmount\")\n",
        "x_df = final_df\n",
        "\n",
        "x_train, x_test, y_train, y_test = train_test_split(x_df, y_df, test_size=0.2, random_state=223)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The purpose of this step is to have data points to test the finished model that haven't been used to train the model, in order to measure true accuracy. \n",
        "\n",
        "In other words, a well-trained model should be able to accurately make predictions from data it hasn't already seen. You now have data prepared for auto-training a machine learning model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Automatically train a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To automatically train a model, take the following steps:\n",
        "1. Define settings for the experiment run. Attach your training data to the configuration, and modify settings that control the training process.\n",
        "1. Submit the experiment for model tuning. After submitting the experiment, the process iterates through different machine learning algorithms and hyperparameter settings, adhering to your defined constraints. It chooses the best-fit model by optimizing an accuracy metric."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Define training settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Define the experiment parameter and model settings for training. View the full list of [settings](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-auto-train). Submitting the experiment with these default settings will take approximately 5-10 min, but if you want a shorter run time, reduce the `iterations` parameter.\n",
        "\n",
        "\n",
        "|Property| Value in this tutorial |Description|\n",
        "|----|----|---|\n",
        "|**iteration_timeout_minutes**|2|Time limit in minutes for each iteration. Reduce this value to decrease total runtime.|\n",
        "|**iterations**|20|Number of iterations. In each iteration, a new machine learning model is trained with your data. This is the primary value that affects total run time.|\n",
        "|**primary_metric**| spearman_correlation | Metric that you want to optimize. The best-fit model will be chosen based on this metric.|\n",
        "|**preprocess**| True | By using **True**, the experiment can preprocess the input data (handling missing data, converting text to numeric, etc.)|\n",
        "|**verbosity**| logging.INFO | Controls the level of logging.|\n",
        "|**n_cross_validations**|5|Number of cross-validation splits to perform when validation data is not specified.|"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import logging\n",
        "\n",
        "automl_settings = {\n",
        "    \"iteration_timeout_minutes\": 2,\n",
        "    \"iterations\": 20,\n",
        "    \"primary_metric\": 'spearman_correlation',\n",
        "    \"preprocess\": True,\n",
        "    \"verbosity\": logging.INFO,\n",
        "    \"n_cross_validations\": 5\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use your defined training settings as a `**kwargs` parameter to an `AutoMLConfig` object. Additionally, specify your training data and the type of model, which is `regression` in this case."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.train.automl import AutoMLConfig\n",
        "\n",
        "automl_config = AutoMLConfig(task='regression',\n",
        "                             debug_log='automated_ml_errors.log',\n",
        "                             X=x_train.values,\n",
        "                             y=y_train.values.flatten(),\n",
        "                             **automl_settings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Automated machine learning pre-processing steps (feature normalization, handling missing data, converting text to numeric, etc.) become part of the underlying model. When using the model for predictions, the same pre-processing steps applied during training are applied to your input data automatically."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Train the automatic regression model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create an experiment object in your workspace. An experiment acts as a container for your individual runs. Pass the defined `automl_config` object to the experiment, and set the output to `True` to view progress during the run. \n",
        "\n",
        "After starting the experiment, the output shown updates live as the experiment runs. For each iteration, you see the model type, the run duration, and the training accuracy. The field `BEST` tracks the best running training score based on your metric type."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.experiment import Experiment\n",
        "experiment = Experiment(ws, \"taxi-experiment\")\n",
        "local_run = experiment.submit(automl_config, show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Explore the results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Explore the results of automatic training with a [Jupyter widget](https://docs.microsoft.com/python/api/azureml-widgets/azureml.widgets?view=azure-ml-py). The widget allows you to see a graph and table of all individual run iterations, along with training accuracy metrics and metadata. Additionally, you can filter on different accuracy metrics than your primary metric with the dropdown selector."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(local_run).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieve the best model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Select the best model from your iterations. The `get_output` function returns the best run and the fitted model for the last fit invocation. By using the overloads on `get_output`, you can retrieve the best run and fitted model for any logged metric or a particular iteration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run, fitted_model = local_run.get_output()\n",
        "print(best_run)\n",
        "print(fitted_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Test the best model accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use the best model to run predictions on the test data set to predict taxi fares. The function `predict` uses the best model and predicts the values of y, **trip cost**, from the `x_test` data set. Print the first 10 predicted cost values from `y_predict`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "y_predict = fitted_model.predict(x_test.values)\n",
        "print(y_predict[:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Calculate the `root mean squared error` of the results. Convert the `y_test` dataframe to a list to compare to the predicted values. The function `mean_squared_error` takes two arrays of values and calculates the average squared error between them. Taking the square root of the result gives an error in the same units as the y variable, **cost**. It indicates roughly how far the taxi fare predictions are from the actual fares."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.metrics import mean_squared_error\n",
        "from math import sqrt\n",
        "\n",
        "y_actual = y_test.values.flatten().tolist()\n",
        "rmse = sqrt(mean_squared_error(y_actual, y_predict))\n",
        "rmse"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run the following code to calculate mean absolute percent error (MAPE) by using the full `y_actual` and `y_predict` data sets. This metric calculates an absolute difference between each predicted and actual value and sums all the differences. Then it expresses that sum as a percent of the total of the actual values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "sum_actuals = sum_errors = 0\n",
        "\n",
        "for actual_val, predict_val in zip(y_actual, y_predict):\n",
        "    abs_error = actual_val - predict_val\n",
        "    if abs_error < 0:\n",
        "        abs_error = abs_error * -1\n",
        "\n",
        "    sum_errors = sum_errors + abs_error\n",
        "    sum_actuals = sum_actuals + actual_val\n",
        "\n",
        "mean_abs_percent_error = sum_errors / sum_actuals\n",
        "print(\"Model MAPE:\")\n",
        "print(mean_abs_percent_error)\n",
        "print()\n",
        "print(\"Model Accuracy:\")\n",
        "print(1 - mean_abs_percent_error)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "From the two prediction accuracy metrics, you see that the model is fairly good at predicting taxi fares from the data set's features, typically within +- $4.00, and approximately 15% error. \n",
        "\n",
        "The traditional machine learning model development process is highly resource-intensive, and requires significant domain knowledge and time investment to run and compare the results of dozens of models. Using automated machine learning is a great way to rapidly test many different models for your scenario."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Clean up resources"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Do not complete this section if you plan on running other Azure Machine Learning service tutorials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Stop the notebook VM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you used a cloud notebook server, stop the VM when you are not using it to reduce cost."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "1. In your workspace, select **Notebook VMs**.\n",
        "1. From the list, select the VM.\n",
        "1. Select **Stop**.\n",
        "1. When you're ready to use the server again, select **Start**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Delete everything"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you don't plan to use the resources you created, delete them, so you don't incur any charges."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "1. In the Azure portal, select **Resource groups** on the far left.\n",
        "1. From the list, select the resource group you created.\n",
        "1. Select **Delete resource group**.\n",
        "1. Enter the resource group name. Then select **Delete**.\n",
        "\n",
        "You can also keep the resource group but delete a single workspace. Display the workspace properties and select **Delete**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this automated machine learning tutorial, you did the following tasks:\n",
        "\n",
        "> * Configured a workspace and prepared data for an experiment.\n",
        "> * Trained by using an automated regression model locally with custom parameters.\n",
        "> * Explored and reviewed training results.\n",
        "\n",
        "[Deploy your model](https://docs.microsoft.com/azure/machine-learning/service/tutorial-deploy-models-with-aml) with Azure Machine Learning service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "jeffshep"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "msauthor": "trbye"
  },
  "nbformat": 4,
  "nbformat_minor": 2
}