MachineLearningNotebooks/how-to-use-azureml/explain-model/azure-integration/scoring-time/train-explain-model-keras-locally-and-deploy.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.\n",
        "\n",
        "Licensed under the MIT License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/explain-model/azure-integration/scoring-time/train-explain-model-locally-and-deploy.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Train and explain keras model locally and deploy model with scoring explainer\n",
        "\n",
        "\n",
        "_**This notebook illustrates how to use the Azure Machine Learning Interpretability SDK to deploy a locally-trained keras model and its corresponding deep scoring explainer to Azure Container Instances (ACI) as a web service.**_\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Problem: IBM employee attrition classification with keras (train and explain a model locally and use Azure Container Instances (ACI) for deploying your model and its corresponding deep scoring explainer as a web service.)\n",
        "\n",
        "---\n",
        "\n",
        "## Table of Contents\n",
        "\n",
        "1. [Introduction](#Introduction)\n",
        "1. [Setup](#Setup)\n",
        "1. [Run model explainer locally at training time](#Explain)\n",
        "    1. Apply feature transformations\n",
        "    1. Train a binary classification keras model\n",
        "    1. Explain the model on raw features\n",
        "        1. Generate global explanations\n",
        "        1. Generate local explanations\n",
        "1. [Visualize explanations](#Visualize)\n",
        "1. [Deploy keras model and scoring explainer](#Deploy)\n",
        "1. [Next steps](#Next)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Introduction\n",
        "\n",
        "\n",
        "This notebook showcases how to train and explain a keras classification model locally, and deploy the trained model and its corresponding DeepExplainer to Azure Container Instances (ACI).\n",
        "It demonstrates the API calls that you need to make to submit a run for training and explaining a keras model to AMLCompute, download the compute explanations remotely, and visualizing the global and local explanations via a visualization dashboard that provides an interactive way of discovering patterns in model predictions and downloaded explanations. It also demonstrates how to use Azure Machine Learning MLOps capabilities to deploy your keras model and its corresponding DeepExplainer.\n",
        "\n",
        "We will showcase one of the tabular data explainers, DeepExplainer (SHAP), following these steps:\n",
        "1.\tDevelop a machine learning script in Python which involves the training script and the explanation script.\n",
        "2.\tRun the script locally.\n",
        "3.\tUse the interpretability toolkit\u00e2\u20ac\u2122s visualization dashboard to visualize predictions and their explanation. If the metrics and explanations don't indicate a desired outcome, loop back to step 1 and iterate on your scripts.\n",
        "5.\tAfter a satisfactory run is found, create a Deep Scoring Explainer and register the persisted model and its corresponding DeepExplainer in the model registry.\n",
        "6.\tDevelop a scoring script.\n",
        "7.\tCreate an image and register it in the image registry.\n",
        "8.\tDeploy the image as a web service in Azure.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n",
        "Make sure you go through the [configuration notebook](../../../../configuration.ipynb) first if you haven't."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Check core SDK version number\n",
        "import azureml.core\n",
        "\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Initialize a Workspace\n",
        "\n",
        "Initialize a workspace object from persisted configuration"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "create workspace"
        ]
      },
      "outputs": [],
      "source": [
        "from azureml.core import Workspace\n",
        "\n",
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Explain\n",
        "Create An Experiment: **Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "experiment_name = 'explain_model_at_scoring_time'\n",
        "experiment = Experiment(workspace=ws, name=experiment_name)\n",
        "run = experiment.start_logging()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# get IBM attrition data\n",
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "outdirname = 'dataset.6.21.19'\n",
        "try:\n",
        "    from urllib import urlretrieve\n",
        "except ImportError:\n",
        "    from urllib.request import urlretrieve\n",
        "import zipfile\n",
        "zipfilename = outdirname + '.zip'\n",
        "urlretrieve('https://publictestdatasets.blob.core.windows.net/data/' + zipfilename, zipfilename)\n",
        "with zipfile.ZipFile(zipfilename, 'r') as unzip:\n",
        "    unzip.extractall('.')\n",
        "attritionData = pd.read_csv('./WA_Fn-UseC_-HR-Employee-Attrition.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.externals import joblib\n",
        "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn_pandas import DataFrameMapper\n",
        "\n",
        "os.makedirs('./outputs', exist_ok=True)\n",
        "\n",
        "# Dropping Employee count as all values are 1 and hence attrition is independent of this feature\n",
        "attritionData = attritionData.drop(['EmployeeCount'], axis=1)\n",
        "# Dropping Employee Number since it is merely an identifier\n",
        "attritionData = attritionData.drop(['EmployeeNumber'], axis=1)\n",
        "attritionData = attritionData.drop(['Over18'], axis=1)\n",
        "# Since all values are 80\n",
        "attritionData = attritionData.drop(['StandardHours'], axis=1)\n",
        "\n",
        "# Converting target variables from string to numerical values\n",
        "target_map = {'Yes': 1, 'No': 0}\n",
        "attritionData[\"Attrition_numerical\"] = attritionData[\"Attrition\"].apply(lambda x: target_map[x])\n",
        "target = attritionData[\"Attrition_numerical\"]\n",
        "\n",
        "attritionXData = attritionData.drop(['Attrition_numerical', 'Attrition'], axis=1)\n",
        "\n",
        "# Creating dummy columns for each categorical feature\n",
        "categorical = []\n",
        "for col, value in attritionXData.iteritems():\n",
        "    if value.dtype == 'object':\n",
        "        categorical.append(col)\n",
        "\n",
        "# Store the numerical columns in a list numerical\n",
        "numerical = attritionXData.columns.difference(categorical)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.compose import ColumnTransformer\n",
        "\n",
        "# We create the preprocessing pipelines for both numeric and categorical data.\n",
        "numeric_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='median')),\n",
        "    ('scaler', StandardScaler())])\n",
        "\n",
        "categorical_transformer = Pipeline(steps=[\n",
        "    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n",
        "    ('onehot', OneHotEncoder(handle_unknown='ignore'))])\n",
        "\n",
        "preprocess = ColumnTransformer(\n",
        "    transformers=[\n",
        "        ('num', numeric_transformer, numerical),\n",
        "        ('cat', categorical_transformer, categorical)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.pipeline import make_pipeline\n",
        "pipeline = make_pipeline(preprocess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(attritionXData, \n",
        "                                                    target, \n",
        "                                                    test_size=0.2,\n",
        "                                                    random_state=0,\n",
        "                                                    stratify=target)\n",
        "\n",
        "X_train_t = pipeline.fit_transform(X_train)\n",
        "X_test_t = pipeline.transform(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# check tensorflow version\n",
        "import tensorflow as tf\n",
        "from distutils.version import StrictVersion\n",
        "\n",
        "print(tf.__version__)\n",
        "# Append classifier to preprocessing pipeline.\n",
        "# Now we have a full prediction pipeline.\n",
        "\n",
        "\n",
        "network = tf.keras.models.Sequential()\n",
        "network.add(tf.keras.layers.Dense(units=16, activation='relu', input_shape=(X_train_t.shape[1],)))\n",
        "network.add(tf.keras.layers.Dense(units=16, activation='relu'))\n",
        "network.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))\n",
        "\n",
        "# Compile neural network\n",
        "network.compile(loss='binary_crossentropy', # Cross-entropy\n",
        "                optimizer='rmsprop', # Root Mean Square Propagation\n",
        "                metrics=['accuracy']) # Accuracy performance metric\n",
        "\n",
        "# Train neural network\n",
        "history = network.fit(X_train_t, # Features\n",
        "                      y_train, # Target vector\n",
        "                      epochs=20, # Number of epochs\n",
        "                      verbose=1, # Print description after each epoch\n",
        "                      batch_size=100, # Number of observations per batch\n",
        "                      validation_data=(X_test_t, y_test)) # Data for evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# You can run the DeepExplainer directly, or run the TabularExplainer which will choose the most appropriate explainer\n",
        "from interpret.ext.greybox import DeepExplainer\n",
        "explainer = DeepExplainer(network,\n",
        "                          X_train,\n",
        "                          features=X_train.columns,\n",
        "                          classes=[\"STAYING\", \"LEAVING\"], \n",
        "                          transformations=preprocess,\n",
        "                          model_task=\"classification\",\n",
        "                          is_classifier=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Save featurization prior to keras model in the outputs folder so it automatically get uploaded\n",
        "# We cannot save Keras with the pipeline due to known issues with pickling Keras models\n",
        "featurize_file_name = 'featurize.pkl'\n",
        "\n",
        "with open(featurize_file_name, 'wb') as file:\n",
        "    joblib.dump(value=preprocess, filename=os.path.join('./outputs/', featurize_file_name))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Save keras model to disk\n",
        "keras_model_file_name = 'keras_model.pkl'\n",
        "network.save(os.path.join('./outputs/', keras_model_file_name))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Explain overall model predictions (global explanation)\n",
        "# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data\n",
        "# x_train can be passed as well, but with more examples explanations it will\n",
        "# take longer although they may be more accurate\n",
        "global_explanation = explainer.explain_global(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.interpret.scoring.scoring_explainer import DeepScoringExplainer, save\n",
        "from azureml.interpret.model.serialize import KerasSerializer\n",
        "# ScoringExplainer with custom keras serializer\n",
        "scoring_explainer = DeepScoringExplainer(explainer, serializer=KerasSerializer())\n",
        "# Pickle scoring explainer locally\n",
        "save(scoring_explainer, exist_ok=True)\n",
        "\n",
        "# Register featurization\n",
        "run.upload_file(featurize_file_name, os.path.join('./outputs/', featurize_file_name))\n",
        "featurize_model = run.register_model(model_name='featurize',\n",
        "                                     model_path=featurize_file_name)\n",
        "\n",
        "# Register keras model\n",
        "run.upload_file(keras_model_file_name, os.path.join('./outputs/', keras_model_file_name))\n",
        "keras_model = run.register_model(model_name='keras_model',\n",
        "                                 model_path=keras_model_file_name)\n",
        "\n",
        "# Register scoring explainer\n",
        "run.upload_file('IBM_attrition_explainer.pkl', 'scoring_explainer.pkl')\n",
        "scoring_explainer_model = run.register_model(model_name='IBM_attrition_explainer', model_path='IBM_attrition_explainer.pkl')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Use helper utility to wrap keras model in scikit-learn style API for visualization dashboard\n",
        "from interpret_community.common.model_wrapper import wrap_model\n",
        "from interpret_community.dataset.dataset_wrapper import DatasetWrapper\n",
        "wrapped_model, ml_domain = wrap_model(network, DatasetWrapper(X_test_t), \"classification\")\n",
        "wrapped_model.fit = network.fit\n",
        "from sklearn.pipeline import Pipeline\n",
        "dashboard_pipeline = Pipeline(steps=[('preprocess', preprocess), ('network', wrapped_model)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Visualize\n",
        "Visualize the explanations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from interpret_community.widget import ExplanationDashboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ExplanationDashboard(global_explanation, dashboard_pipeline, datasetX=X_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy \n",
        "\n",
        "Deploy Model and ScoringExplainer.\n",
        "\n",
        "Please note that you must indicate azureml-defaults with verion >= 1.0.45 as a pip dependency, because it contains the functionality needed to host the model as a web service."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.conda_dependencies import CondaDependencies \n",
        "\n",
        "# azureml-defaults is required to host the model as a web service.\n",
        "azureml_pip_packages = [\n",
        "    'azureml-defaults', 'azureml-contrib-interpret', 'azureml-core', 'azureml-telemetry',\n",
        "    'azureml-interpret'\n",
        "]\n",
        "# Note: this is to pin the scikit-learn and pandas versions to be same as notebook.\n",
        "# In production scenario user would choose their dependencies\n",
        "import pkg_resources\n",
        "available_packages = pkg_resources.working_set\n",
        "sklearn_ver = None\n",
        "pandas_ver = None\n",
        "for dist in available_packages:\n",
        "    if dist.key == 'scikit-learn':\n",
        "        sklearn_ver = dist.version\n",
        "    elif dist.key == 'pandas':\n",
        "        pandas_ver = dist.version\n",
        "sklearn_dep = 'scikit-learn'\n",
        "pandas_dep = 'pandas'\n",
        "if sklearn_ver:\n",
        "    sklearn_dep = 'scikit-learn=={}'.format(sklearn_ver)\n",
        "if pandas_ver:\n",
        "    pandas_dep = 'pandas=={}'.format(pandas_ver)\n",
        "# specify CondaDependencies obj\n",
        "myenv = CondaDependencies.create(conda_packages=[sklearn_dep, pandas_dep],\n",
        "                                 pip_packages=['sklearn-pandas', 'pyyaml', 'tensorflow<2.0', 'keras==2.3.1'] + azureml_pip_packages)\n",
        "\n",
        "with open(\"myenv.yml\",\"w\") as f:\n",
        "    f.write(myenv.serialize_to_string())\n",
        "\n",
        "with open(\"myenv.yml\",\"r\") as f:\n",
        "    print(f.read())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.model import Model\n",
        "# retrieve scoring explainer for deployment\n",
        "scoring_explainer_model = Model(ws, 'IBM_attrition_explainer')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from azureml.core.webservice import Webservice\n",
        "from azureml.core.model import InferenceConfig\n",
        "from azureml.core.webservice import AciWebservice\n",
        "from azureml.core.model import Model\n",
        "from azureml.core.environment import Environment\n",
        "\n",
        "\n",
        "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1,\n",
        "                                               memory_gb=1,\n",
        "                                               tags={\"data\": \"IBM_Attrition\",\n",
        "                                                     \"method\" : \"local_explanation\"},\n",
        "                                               description='Get local explanations for IBM Employee Attrition data')\n",
        "\n",
        "myenv = Environment.from_conda_specification(name=\"myenv\", file_path=\"myenv.yml\")\n",
        "inference_config = InferenceConfig(entry_script=\"score_local_explain_keras.py\", environment=myenv)\n",
        "\n",
        "# Use configs and models generated above\n",
        "service = Model.deploy(ws, 'model-scoring-keras-deploy-local', [scoring_explainer_model, featurize_model, keras_model], inference_config, aciconfig)\n",
        "service.wait_for_deployment(show_output=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(service.get_logs())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import requests\n",
        "import json\n",
        "\n",
        "# Create data to test service with\n",
        "sample_data = '{\"Age\":{\"899\":49},\"BusinessTravel\":{\"899\":\"Travel_Rarely\"},\"DailyRate\":{\"899\":1098},\"Department\":{\"899\":\"Research & Development\"},\"DistanceFromHome\":{\"899\":4},\"Education\":{\"899\":2},\"EducationField\":{\"899\":\"Medical\"},\"EnvironmentSatisfaction\":{\"899\":1},\"Gender\":{\"899\":\"Male\"},\"HourlyRate\":{\"899\":85},\"JobInvolvement\":{\"899\":2},\"JobLevel\":{\"899\":5},\"JobRole\":{\"899\":\"Manager\"},\"JobSatisfaction\":{\"899\":3},\"MaritalStatus\":{\"899\":\"Married\"},\"MonthlyIncome\":{\"899\":18711},\"MonthlyRate\":{\"899\":12124},\"NumCompaniesWorked\":{\"899\":2},\"OverTime\":{\"899\":\"No\"},\"PercentSalaryHike\":{\"899\":13},\"PerformanceRating\":{\"899\":3},\"RelationshipSatisfaction\":{\"899\":3},\"StockOptionLevel\":{\"899\":1},\"TotalWorkingYears\":{\"899\":23},\"TrainingTimesLastYear\":{\"899\":2},\"WorkLifeBalance\":{\"899\":4},\"YearsAtCompany\":{\"899\":1},\"YearsInCurrentRole\":{\"899\":0},\"YearsSinceLastPromotion\":{\"899\":0},\"YearsWithCurrManager\":{\"899\":0}}'\n",
        "\n",
        "headers = {'Content-Type':'application/json'}\n",
        "\n",
        "# send request to service\n",
        "resp = requests.post(service.scoring_uri, sample_data, headers=headers)\n",
        "\n",
        "print(\"POST to url\", service.scoring_uri)\n",
        "# can covert back to Python objects from json string if desired\n",
        "print(\"prediction:\", resp.text)\n",
        "result = json.loads(resp.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#plot the feature importance for the prediction\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt; plt.rcdefaults()\n",
        "\n",
        "labels = json.loads(sample_data)\n",
        "labels = labels.keys()\n",
        "objects = labels\n",
        "y_pos = np.arange(len(objects))\n",
        "performance = result[\"local_importance_values\"][0][0]\n",
        "\n",
        "plt.bar(y_pos, performance, align='center', alpha=0.5)\n",
        "plt.xticks(y_pos, objects)\n",
        "locs, labels = plt.xticks()\n",
        "plt.setp(labels, rotation=90)\n",
        "plt.ylabel('Feature impact - leaving vs not leaving')\n",
        "plt.title('Local feature importance for prediction')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "service.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Next\n",
        "Learn about other use cases of the explain package on a:\n",
        "1. [Training time: regression problem](https://github.com/interpretml/interpret-community/blob/master/notebooks/explain-regression-local.ipynb)       \n",
        "1. [Training time: binary classification problem](https://github.com/interpretml/interpret-community/blob/master/notebooks/explain-binary-classification-local.ipynb)\n",
        "1. [Training time: multiclass classification problem](https://github.com/interpretml/interpret-community/blob/master/notebooks/explain-multiclass-classification-local.ipynb)\n",
        "1. Explain models with engineered features:\n",
        "    1. [Simple feature transformations](https://github.com/interpretml/interpret-community/blob/master/notebooks/simple-feature-transformations-explain-local.ipynb)\n",
        "    1. [Advanced feature transformations](https://github.com/interpretml/interpret-community/blob/master/notebooks/advanced-feature-transformations-explain-local.ipynb)\n",
        "1. [Save model explanations via Azure Machine Learning Run History](../run-history/save-retrieve-explanations-run-history.ipynb)\n",
        "1. [Run explainers remotely on Azure Machine Learning Compute (AMLCompute)](../remote-explanation/explain-model-on-amlcompute.ipynb)\n",
        "1. [Inferencing time: deploy a remotely-trained model and explainer](./train-explain-model-on-amlcompute-and-deploy.ipynb)\n",
        "1. [Inferencing time: deploy a locally-trained model and explainer](./train-explain-model-locally-and-deploy.ipynb)"
      ]
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "mesameki"
      }
    ],
    "kernelspec": {
      "display_name": "Python 3.6",
      "language": "python",
      "name": "python36"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}