{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Automated Machine Learning\n",
        "**BikeShare Demand Forecasting**\n",
        "\n",
        "## Contents\n",
        "1. [Introduction](#Introduction)\n",
        "1. [Setup](#Setup)\n",
        "1. [Data](#Data)\n",
        "1. [Train](#Train)\n",
        "1. [Evaluate](#Evaluate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Introduction\n",
        "In this example, we show how AutoML can be used for bike share forecasting.\n",
        "\n",
        "The purpose is to demonstrate how to take advantage of the built-in holiday featurization, access the feature names, and further demonstrate how to work with the `forecast` function. Please also look at the additional forecasting notebooks, which document lagging, rolling windows, forecast quantiles, other ways to use the forecast function, and forecaster deployment.\n",
        "\n",
        "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
        "\n",
        "In this notebook you would see\n",
        "1. Creating an Experiment in an existing Workspace\n",
        "2. Instantiating AutoMLConfig with new task type \"forecasting\" for timeseries data training, and other timeseries related settings: for this dataset we use the basic one: \"time_column_name\" \n",
        "3. Training the Model using local compute\n",
        "4. Exploring the results\n",
        "5. Viewing the engineered names for featurized data and featurization summary for all raw features\n",
        "6. Testing the fitted model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import azureml.core\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import logging\n",
        "import warnings\n",
        "# Squash warning messages for cleaner output in the notebook\n",
        "warnings.showwarning = lambda *args, **kwargs: None\n",
        "\n",
        "\n",
        "from azureml.core.workspace import Workspace\n",
        "from azureml.core.experiment import Experiment\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "from matplotlib import pyplot as plt\n",
        "from sklearn.metrics import mean_absolute_error, mean_squared_error"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As part of the setup you have already created a <b>Workspace</b>. For AutoML you would need to create an <b>Experiment</b>. An <b>Experiment</b> is a named object in a <b>Workspace</b>, which is used to run experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ws = Workspace.from_config()\n",
        "\n",
        "# choose a name for the run history container in the workspace\n",
        "experiment_name = 'automl-bikeshareforecasting'\n",
        "# project folder\n",
        "project_folder = './sample_projects/automl-local-bikeshareforecasting'\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "\n",
        "output = {}\n",
        "output['SDK version'] = azureml.core.VERSION\n",
        "output['Subscription ID'] = ws.subscription_id\n",
        "output['Workspace'] = ws.name\n",
        "output['Resource Group'] = ws.resource_group\n",
        "output['Location'] = ws.location\n",
        "output['Project Directory'] = project_folder\n",
        "output['Run History Name'] = experiment_name\n",
        "pd.set_option('display.max_colwidth', -1)\n",
        "outputDf = pd.DataFrame(data = output, index = [''])\n",
        "outputDf.T"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Data\n",
        "Read bike share demand data from file, and preview data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "data = pd.read_csv('bike-no.csv', parse_dates=['date'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's set up what we know abou the dataset. \n",
        "\n",
        "**Target column** is what we want to forecast.\n",
        "\n",
        "**Time column** is the time axis along which to predict.\n",
        "\n",
        "**Grain** is another word for an individual time series in your dataset. Grains are identified by values of the columns listed `grain_column_names`, for example \"store\" and \"item\" if your data has multiple time series of sales, one series for each combination of store and item sold.\n",
        "\n",
        "This dataset has only one time series. Please see the [orange juice notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales) for an example of a multi-time series dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "target_column_name = 'cnt'\n",
        "time_column_name = 'date'\n",
        "grain_column_names = []"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Split the data\n",
        "\n",
        "The first split we make is into train and test sets. Note we are splitting on time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "train = data[data[time_column_name] < '2012-09-01']\n",
        "test = data[data[time_column_name] >= '2012-09-01']\n",
        "\n",
        "X_train = train.copy()\n",
        "y_train = X_train.pop(target_column_name).values\n",
        "\n",
        "X_test = test.copy()\n",
        "y_test = X_test.pop(target_column_name).values\n",
        "\n",
        "print(X_train.shape)\n",
        "print(y_train.shape)\n",
        "print(X_test.shape)\n",
        "print(y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setting forecaster maximum horizon \n",
        "\n",
        "Assuming your test data forms a full and regular time series(regular time intervals and no holes), \n",
        "the maximum horizon you will need to forecast is the length of the longest grain in your test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if len(grain_column_names) == 0:\n",
        "    max_horizon = len(X_test)\n",
        "else:\n",
        "    max_horizon = X_test.groupby(grain_column_names)[time_column_name].count().max()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Train\n",
        "\n",
        "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
        "\n",
        "|Property|Description|\n",
        "|-|-|\n",
        "|**task**|forecasting|\n",
        "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>\n",
        "|**iterations**|Number of iterations. In each iteration, Auto ML trains a specific pipeline on the given data|\n",
        "|**iteration_timeout_minutes**|Time limit in minutes for each iteration.|\n",
        "|**X**|(sparse) array-like, shape = [n_samples, n_features]|\n",
        "|**y**|(sparse) array-like, shape = [n_samples, ], targets values.|\n",
        "|**n_cross_validations**|Number of cross validation splits.|\n",
        "|**country_or_region**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
        "|**path**|Relative path to the project folder.  AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "time_column_name = 'date'\n",
        "automl_settings = {\n",
        "    \"time_column_name\": time_column_name,\n",
        "    # these columns are a breakdown of the total and therefore a leak\n",
        "    \"drop_column_names\": ['casual', 'registered'],\n",
        "    # knowing the country/region allows Automated ML to bring in holidays\n",
        "    \"country_or_region\" : 'US',\n",
        "    \"max_horizon\" : max_horizon,\n",
        "    \"target_lags\": 1    \n",
        "}\n",
        "\n",
        "automl_config = AutoMLConfig(task = 'forecasting',                             \n",
        "                             primary_metric='normalized_root_mean_squared_error',\n",
        "                             iterations = 10,\n",
        "                             iteration_timeout_minutes = 5,\n",
        "                             X = X_train,\n",
        "                             y = y_train,\n",
        "                             n_cross_validations = 3,                             \n",
        "                             path=project_folder,\n",
        "                             verbosity = logging.INFO,\n",
        "                            **automl_settings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will now run the experiment, starting with 10 iterations of model search. Experiment can be continued for more iterations if the results are not yet good. You will see the currently running iterations printing to the console."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "local_run = experiment.submit(automl_config, show_output=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Displaying the run objects gives you links to the visual tools in the Azure Portal. Go try them!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "local_run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieve the Best Model\n",
        "Below we select the best pipeline from our iterations. The get_output method on automl_classifier returns the best run and the fitted model for the last fit invocation. There are overloads on get_output that allow you to retrieve the best run and fitted model for any logged metric or a particular iteration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_run, fitted_model = local_run.get_output()\n",
        "fitted_model.steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### View the engineered names for featurized data\n",
        "\n",
        "You can accees the engineered feature names generated in time-series featurization. Note that a number of named holiday periods are represented. We recommend that you have at least one year of data when using this feature to ensure that all yearly holidays are captured in the training featurization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fitted_model.named_steps['timeseriestransformer'].get_engineered_feature_names()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### View the featurization summary\n",
        "\n",
        "You can also see what featurization steps were performed on different raw features in the user data. For each raw feature in the user data, the following information is displayed:\n",
        "\n",
        "- Raw feature name\n",
        "- Number of engineered features formed out of this raw feature\n",
        "- Type detected\n",
        "- If feature was dropped\n",
        "- List of feature transformations for the raw feature"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fitted_model.named_steps['timeseriestransformer'].get_featurization_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Test the Best Fitted Model\n",
        "\n",
        "Predict on training and test set, and calculate residual values.\n",
        "\n",
        "We always score on the original dataset whose schema matches the scheme of the training dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "X_test.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "y_query = y_test.copy().astype(np.float)\n",
        "y_query.fill(np.NaN)\n",
        "y_fcst, X_trans = fitted_model.forecast(X_test, y_query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "It is a good practice to always align the output explicitly to the input, as the count and order of the rows may have changed during transformations that span multiple rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def align_outputs(y_predicted, X_trans, X_test, y_test, predicted_column_name = 'predicted'):\n",
        "    \"\"\"\n",
        "    Demonstrates how to get the output aligned to the inputs\n",
        "    using pandas indexes. Helps understand what happened if\n",
        "    the output's shape differs from the input shape, or if\n",
        "    the data got re-sorted by time and grain during forecasting.\n",
        "    \n",
        "    Typical causes of misalignment are:\n",
        "    * we predicted some periods that were missing in actuals -> drop from eval\n",
        "    * model was asked to predict past max_horizon -> increase max horizon\n",
        "    * data at start of X_test was needed for lags -> provide previous periods\n",
        "    \"\"\"\n",
        "    df_fcst = pd.DataFrame({predicted_column_name : y_predicted})\n",
        "    # y and X outputs are aligned by forecast() function contract\n",
        "    df_fcst.index = X_trans.index\n",
        "    \n",
        "    # align original X_test to y_test    \n",
        "    X_test_full = X_test.copy()\n",
        "    X_test_full[target_column_name] = y_test\n",
        "\n",
        "    # X_test_full's index does not include origin, so reset for merge\n",
        "    df_fcst.reset_index(inplace=True)\n",
        "    X_test_full = X_test_full.reset_index().drop(columns='index')\n",
        "    together = df_fcst.merge(X_test_full, how='right')\n",
        "    \n",
        "    # drop rows where prediction or actuals are nan \n",
        "    # happens because of missing actuals \n",
        "    # or at edges of time due to lags/rolling windows\n",
        "    clean = together[together[[target_column_name, predicted_column_name]].notnull().all(axis=1)]\n",
        "    return(clean)\n",
        "\n",
        "df_all = align_outputs(y_fcst, X_trans, X_test, y_test)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def MAPE(actual, pred):\n",
        "    \"\"\"\n",
        "    Calculate mean absolute percentage error.\n",
        "    Remove NA and values where actual is close to zero\n",
        "    \"\"\"\n",
        "    not_na = ~(np.isnan(actual) | np.isnan(pred))\n",
        "    not_zero = ~np.isclose(actual, 0.0)\n",
        "    actual_safe = actual[not_na & not_zero]\n",
        "    pred_safe = pred[not_na & not_zero]\n",
        "    APE = 100*np.abs((actual_safe - pred_safe)/actual_safe)\n",
        "    return np.mean(APE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"Simple forecasting model\")\n",
        "rmse = np.sqrt(mean_squared_error(df_all[target_column_name], df_all['predicted']))\n",
        "print(\"[Test Data] \\nRoot Mean squared error: %.2f\" % rmse)\n",
        "mae = mean_absolute_error(df_all[target_column_name], df_all['predicted'])\n",
        "print('mean_absolute_error score: %.2f' % mae)\n",
        "print('MAPE: %.2f' % MAPE(df_all[target_column_name], df_all['predicted']))\n",
        "\n",
        "# Plot outputs\n",
        "%matplotlib notebook\n",
        "test_pred = plt.scatter(df_all[target_column_name], df_all['predicted'], color='b')\n",
        "test_test = plt.scatter(y_test, y_test, color='g')\n",
        "plt.legend((test_pred, test_test), ('prediction', 'truth'), loc='upper left', fontsize=8)\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "xiaga@microsoft.com, tosingli@microsoft.com"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}