MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/18b.auto-ml-forecasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML 18B: Orange Juice Sales Forecasting\n",
    "\n",
    "In this example, we use AutoML to find and tune a time-series forecasting model.\n",
    "\n",
    "Make sure you have executed the [configuration notebook](00.configuration.ipynb) before running this notebook.\n",
    "\n",
    "In this notebook, you will:\n",
    "1. Create an Experiment in an existing Workspace\n",
    "2. Instantiate an AutoMLConfig \n",
    "3. Find and train a forecasting model using local compute\n",
    "4. Evaluate the performance of the model\n",
    "\n",
    "## Sample Data\n",
    "The examples in the follow code samples use the [University of Chicago's Dominick's Finer Foods dataset](https://research.chicagobooth.edu/kilts/marketing-databases/dominicks) to forecast orange juice sales. Dominick's was a grocery chain in the Chicago metropolitan area."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Experiment\n",
    "\n",
    "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment is a named object in a Workspace which represents a predictive task, the output of which is a trained model and a set of evaluation metrics for the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import logging\n",
    "\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.train.automl.run import AutoMLRun\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# choose a name for the run history container in the workspace\n",
    "experiment_name = 'automl-ojsalesforecasting'\n",
    "# project folder\n",
    "project_folder = './sample_projects/automl-local-ojsalesforecasting'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['SDK version'] = azureml.core.VERSION\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Project Directory'] = project_folder\n",
    "output['Run History Name'] = experiment_name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "pd.DataFrame(data=output, index=['']).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Data\n",
    "You are now ready to load the historical orange juice sales data. We will load the CSV file into a plain pandas DataFrame; the time column in the CSV is called _WeekStarting_, so it will be specially parsed into the datetime type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_column_name = 'WeekStarting'\n",
    "data = pd.read_csv(\"dominicks_OJ.csv\", parse_dates=[time_column_name])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row in the DataFrame holds a quantity of weekly sales for an OJ brand at a single store. The data also includes the sales price, a flag indicating if the OJ brand was advertised in the store that week, and some customer demographic information based on the store location. For historical reasons, the data also include the logarithm of the sales quantity. The Dominick's grocery data is commonly used to illustrate econometric modeling techniques where logarithms of quantities are generally preferred.    \n",
    "\n",
    "The task is now to build a time-series model for the _Quantity_ column. It is important to note that this dataset is comprised of many individual time-series - one for each unique combination of _Store_ and _Brand_. To distinguish the individual time-series, we thus define the **grain** - the columns whose values determine the boundaries between time-series: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grain_column_names = ['Store', 'Brand']\n",
    "nseries = data.groupby(grain_column_names).ngroups\n",
    "print('Data contains {0} individual time-series.'.format(nseries))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Splitting\n",
    "For the purposes of demonstration and later forecast evaluation, we now split the data into a training and a testing set. The test set will contain the final 20 weeks of observed sales for each time-series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ntest_periods = 20\n",
    "\n",
    "def split_last_n_by_grain(df, n):\n",
    "    \"\"\"\n",
    "    Group df by grain and split on last n rows for each group\n",
    "    \"\"\"\n",
    "    df_grouped = (df.sort_values(time_column_name) # Sort by ascending time\n",
    "                  .groupby(grain_column_names, group_keys=False))\n",
    "    df_head = df_grouped.apply(lambda dfg: dfg.iloc[:-n])\n",
    "    df_tail = df_grouped.apply(lambda dfg: dfg.iloc[-n:])\n",
    "    return df_head, df_tail\n",
    "\n",
    "X_train, X_test = split_last_n_by_grain(data, ntest_periods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "For forecasting tasks, AutoML uses pre-processing and estimation steps that are specific to time-series. AutoML will undertake the following pre-processing steps:\n",
    "* Detect time-series sample frequency (e.g. hourly, daily, weekly) and create new records for absent time points to make the series regular. A regular time series has a well-defined frequency and has a value at every sample point in a contiguous time span \n",
    "* Impute missing values in the target (via forward-fill) and feature columns (using median column values) \n",
    "* Create grain-based features to enable fixed effects across different series\n",
    "* Create time-based features to assist in learning seasonal patterns\n",
    "* Encode categorical variables to numeric quantities\n",
    "\n",
    "AutoML will currently train a single, regression-type model across **all** time-series in a given training set. This allows the model to generalize across related series.\n",
    "\n",
    "You are almost ready to start an AutoML training job. We will first need to create a validation set from the existing training set (i.e. for hyper-parameter tuning): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nvalidation_periods = 20\n",
    "X_train, X_validate = split_last_n_by_grain(X_train, nvalidation_periods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to separate the target column from the rest of the DataFrame: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = 'Quantity'\n",
    "y_train = X_train.pop(target_column_name).values\n",
    "y_validate = X_validate.pop(target_column_name).values "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an AutoMLConfig\n",
    "\n",
    "The AutoMLConfig object defines the settings and data for an AutoML training job. Here, we set necessary inputs like the task type, the number of AutoML iterations to try, and the training and validation data. \n",
    "\n",
    "For forecasting tasks, there are some additional parameters that can be set: the name of the input data column, holding the date/time and the grain column names. A time column is required for forecasting, while the grain is optional. If a grain is not given, the forecaster assumes that the whole dataset is a single time-series. We also pass a list of columns to drop prior to modeling. The _logQuantity_ column is completely correlated with the target quantity, so it must be removed to prevent a target leak. \n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**task**|forecasting|\n",
    "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>\n",
    "|**iterations**|Number of iterations. In each iteration, Auto ML trains a specific pipeline on the given data|\n",
    "|**X**|Training matrix of features, shape = [n_training_samples, n_features]|\n",
    "|**y**|Target values, shape = [n_training_samples, ]|\n",
    "|**X_valid**|Validation matrix of features, shape = [n_validation_samples, n_features]|\n",
    "|**y_valid**|Target values for validation, shape = [n_validation_samples, ]\n",
    "|**enable_ensembling**|Allow AutoML to create ensembles of the best performing models\n",
    "|**debug_log**|Log file path for writing debugging information\n",
    "|**path**|Relative path to the project folder.  AutoML stores configuration files for the experiment under this folder. You can specify a new empty folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    'time_column_name': time_column_name,\n",
    "    'grain_column_names': grain_column_names,\n",
    "    'drop_column_names': ['logQuantity']\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(task='forecasting',\n",
    "                             debug_log='automl_oj_sales_errors.log',\n",
    "                             primary_metric='normalized_root_mean_squared_error',\n",
    "                             iterations=10,\n",
    "                             X=X_train,\n",
    "                             y=y_train,\n",
    "                             X_valid=X_validate,\n",
    "                             y_valid=y_validate,\n",
    "                             enable_ensembling=False,\n",
    "                             path=project_folder,\n",
    "                             verbosity=logging.INFO,\n",
    "                            **automl_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model\n",
    "\n",
    "You can now submit a new training run. For local runs, the execution is synchronous. Depending on the data and number of iterations this operation may take several minutes.\n",
    "Information from each iteration will be printed to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_run = experiment.submit(automl_config, show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model\n",
    "Each run within an Experiment stores serialized (i.e. pickled) pipelines from the AutoML iterations. We can now retrieve the pipeline with the best performance on the validation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_pipeline = local_run.get_output()\n",
    "fitted_pipeline.steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Predictions from the Best Fitted Model\n",
    "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. First, we remove the target values from the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = X_test.pop(target_column_name).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To produce predictions on the test set, we need to know the feature values at all dates in the test set. This requirement is somewhat reasonable for the OJ sales data since the features mainly consist of price, which is usually set in advance, and customer demographics which are approximately constant for each store over the 20 week forecast horizon in the testing data. \n",
    "\n",
    "The target predictions can be retrieved by calling the `predict` method on the best model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = fitted_pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate evaluation metrics for the prediction\n",
    "To evaluate the accuracy of the forecast, we'll compare against the actual sales quantities for some select metrics, included the mean absolute percentage error (MAPE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def MAPE(actual, pred):\n",
    "    \"\"\"\n",
    "    Calculate mean absolute percentage error.\n",
    "    Remove NA and values where actual is close to zero\n",
    "    \"\"\"\n",
    "    not_na = ~(np.isnan(actual) | np.isnan(pred))\n",
    "    not_zero = ~np.isclose(actual, 0.0)\n",
    "    actual_safe = actual[not_na & not_zero]\n",
    "    pred_safe = pred[not_na & not_zero]\n",
    "    APE = 100*np.abs((actual_safe - pred_safe)/actual_safe)\n",
    "    return np.mean(APE)\n",
    "\n",
    "print(\"[Test Data] \\nRoot Mean squared error: %.2f\" % np.sqrt(mean_squared_error(y_test, y_pred)))\n",
    "print('mean_absolute_error score: %.2f' % mean_absolute_error(y_test, y_pred))\n",
    "print('MAPE: %.2f' % MAPE(y_test, y_pred))"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "erwright"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}