mirror of
https://github.com/Azure/MachineLearningNotebooks.git
synced 2025-12-22 18:42:41 -05:00
641 lines
22 KiB
Plaintext
641 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
|
"\n",
|
|
"Licensed under the MIT License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 04. Train in a remote Linux VM\n",
|
|
"* Create Workspace\n",
|
|
"* Create `train.py` file\n",
|
|
"* Create and Attach a Remote VM (eg. DSVM) as compute resource.\n",
|
|
"* Upoad data files into default datastore\n",
|
|
"* Configure & execute a run in a few different ways\n",
|
|
" - Use system-built conda\n",
|
|
" - Use existing Python environment\n",
|
|
" - Use Docker \n",
|
|
"* Find the best model in the run"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisites\n",
|
|
"Make sure you go through the [configuration notebook](../../../configuration.ipynb) first if you haven't."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Check core SDK version number\n",
|
|
"import azureml.core\n",
|
|
"\n",
|
|
"print(\"SDK version:\", azureml.core.VERSION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Initialize Workspace\n",
|
|
"\n",
|
|
"Initialize a workspace object from persisted configuration."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core import Workspace\n",
|
|
"\n",
|
|
"ws = Workspace.from_config()\n",
|
|
"print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep='\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Create Experiment\n",
|
|
"\n",
|
|
"**Experiment** is a logical container in an Azure ML Workspace. It hosts run records which can include run metrics and output artifacts from your experiments."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"experiment_name = 'train-on-remote-vm'\n",
|
|
"\n",
|
|
"from azureml.core import Experiment\n",
|
|
"exp = Experiment(workspace=ws, name=experiment_name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's also create a local folder to hold the training script."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"script_folder = './vm-run'\n",
|
|
"os.makedirs(script_folder, exist_ok=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Upload data files into datastore\n",
|
|
"Every workspace comes with a default [datastore](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-access-data) (and you can register more) which is backed by the Azure blob storage account associated with the workspace. We can use it to transfer data from local to the cloud, and access it from the compute target."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# get the default datastore\n",
|
|
"ds = ws.get_default_datastore()\n",
|
|
"print(ds.name, ds.datastore_type, ds.account_name, ds.container_name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Load diabetes data from `scikit-learn` and save it as 2 local files."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.datasets import load_diabetes\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"training_data = load_diabetes()\n",
|
|
"np.save(file='./features.npy', arr=training_data['data'])\n",
|
|
"np.save(file='./labels.npy', arr=training_data['target'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now let's upload the 2 files into the default datastore under a path named `diabetes`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"ds.upload_files(['./features.npy', './labels.npy'], target_path='diabetes', overwrite=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## View `train.py`\n",
|
|
"\n",
|
|
"For convenience, we created a training script for you. It is printed below as a text, but you can also run `%pfile ./train.py` in a cell to show the file. Please pay special attention on how we are loading the features and labels from files in the `data_folder` path, which is passed in as an argument of the training script (shown later)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# copy train.py into the script folder\n",
|
|
"import shutil\n",
|
|
"shutil.copy('./train.py', os.path.join(script_folder, 'train.py'))\n",
|
|
"\n",
|
|
"with open(os.path.join(script_folder, './train.py'), 'r') as training_script:\n",
|
|
" print(training_script.read())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Create and Attach a DSVM as a compute target\n",
|
|
"\n",
|
|
"**Note**: To streamline the compute that Azure Machine Learning creates, we are making updates to support creating only single to multi-node `AmlCompute`. The DSVM can be created using the below single line command and then attached(like any VM) using the sample code below. Also note, that we only support Linux VMs for remote execution from AML and the commands below will spin a Linux VM only.\n",
|
|
"\n",
|
|
"```shell\n",
|
|
"# create a DSVM in your resource group\n",
|
|
"# note you need to be at least a contributor to the resource group in order to execute this command successfully\n",
|
|
"(myenv) $ az vm create --resource-group <resource_group_name> --name <some_vm_name> --image microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:latest --admin-username <username> --admin-password <password> --generate-ssh-keys --authentication-type password\n",
|
|
"```\n",
|
|
"\n",
|
|
"**Note**: You can also use [this url](https://portal.azure.com/#create/microsoft-dsvm.linux-data-science-vm-ubuntulinuxdsvmubuntu) to create the VM using the Azure Portal\n",
|
|
"\n",
|
|
"**Note**: By default SSH runs on port 22 and you don't need to specify it. But if for security reasons you switch to a different port (such as 5022), you can specify the port number in the provisioning configuration object."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core.compute import ComputeTarget, RemoteCompute\n",
|
|
"from azureml.core.compute_target import ComputeTargetException\n",
|
|
"\n",
|
|
"username = os.getenv('AZUREML_DSVM_USERNAME', default='<my_username>')\n",
|
|
"address = os.getenv('AZUREML_DSVM_ADDRESS', default='<ip_address_or_fqdn>')\n",
|
|
"\n",
|
|
"compute_target_name = 'cpudsvm'\n",
|
|
"# if you want to connect using SSH key instead of username/password you can provide parameters private_key_file and private_key_passphrase \n",
|
|
"try:\n",
|
|
" attached_dsvm_compute = RemoteCompute(workspace=ws, name=compute_target_name)\n",
|
|
" print('found existing:', attached_dsvm_compute.name)\n",
|
|
"except ComputeTargetException:\n",
|
|
" attach_config = RemoteCompute.attach_configuration(address=address,\n",
|
|
" ssh_port=22,\n",
|
|
" username=username,\n",
|
|
" private_key_file='./.ssh/id_rsa')\n",
|
|
" attached_dsvm_compute = ComputeTarget.attach(workspace=ws,\n",
|
|
" name=compute_target_name,\n",
|
|
" attach_configuration=attach_config)\n",
|
|
" attached_dsvm_compute.wait_for_completion(show_output=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Configure & Run\n",
|
|
"First let's create a `DataReferenceConfiguration` object to inform the system what data folder to download to the compute target."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core.runconfig import DataReferenceConfiguration\n",
|
|
"dr = DataReferenceConfiguration(datastore_name=ds.name, \n",
|
|
" path_on_datastore='diabetes', \n",
|
|
" mode='download', # download files from datastore to compute target\n",
|
|
" overwrite=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we can try a few different ways to run the training script in the VM."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Conda run\n",
|
|
"You can ask the system to build a conda environment based on your dependency specification, and submit your script to run there. Once the environment is built, and if you don't change your dependencies, it will be reused in subsequent runs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core.runconfig import RunConfiguration\n",
|
|
"from azureml.core.conda_dependencies import CondaDependencies\n",
|
|
"\n",
|
|
"# create a new RunConfig object\n",
|
|
"conda_run_config = RunConfiguration(framework=\"python\")\n",
|
|
"\n",
|
|
"# Set compute target to the Linux DSVM\n",
|
|
"conda_run_config.target = attached_dsvm_compute.name\n",
|
|
"\n",
|
|
"# set the data reference of the run configuration\n",
|
|
"conda_run_config.data_references = {ds.name: dr}\n",
|
|
"\n",
|
|
"# specify CondaDependencies obj\n",
|
|
"conda_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core import ScriptRunConfig\n",
|
|
"\n",
|
|
"src = ScriptRunConfig(source_directory=script_folder, \n",
|
|
" script='train.py', \n",
|
|
" run_config=conda_run_config, \n",
|
|
" # pass the datastore reference as a parameter to the training script\n",
|
|
" arguments=['--data-folder', str(ds.as_download())] \n",
|
|
" ) \n",
|
|
"run = exp.submit(config=src)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note: if you need to cancel a run, you can follow [these instructions](https://aka.ms/aml-docs-cancel-run)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run.wait_for_completion(show_output=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Show the run object. You can navigate to the Azure portal to see detailed information about the run."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Native VM run\n",
|
|
"You can also configure to use an exiting Python environment in the VM to execute the script without asking the system to create a conda environment for you."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# create a new RunConfig object\n",
|
|
"vm_run_config = RunConfiguration(framework=\"python\")\n",
|
|
"\n",
|
|
"# Set compute target to the Linux DSVM\n",
|
|
"vm_run_config.target = attached_dsvm_compute.name\n",
|
|
"\n",
|
|
"# set the data reference of the run coonfiguration\n",
|
|
"conda_run_config.data_references = {ds.name: dr}\n",
|
|
"\n",
|
|
"# Let system know that you will configure the Python environment yourself.\n",
|
|
"vm_run_config.environment.python.user_managed_dependencies = True"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The below run will likely fail because `train.py` needs dependency `azureml`, `scikit-learn` and others, which are not found in that Python environment. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"src = ScriptRunConfig(source_directory=script_folder, \n",
|
|
" script='train.py', \n",
|
|
" run_config=vm_run_config,\n",
|
|
" # pass the datastore reference as a parameter to the training script\n",
|
|
" arguments=['--data-folder', str(ds.as_download())])\n",
|
|
"run = exp.submit(config=src)\n",
|
|
"run.wait_for_completion(show_output=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can choose to SSH into the VM and install Azure ML SDK, and any other missing dependencies, in that Python environment. For demonstration purposes, we simply are going to use another script `train2.py` that doesn't have azureml dependencies, and submit it instead."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# copy train2.py into the script folder\n",
|
|
"shutil.copy('./train2.py', os.path.join(script_folder, 'train2.py'))\n",
|
|
"\n",
|
|
"with open(os.path.join(script_folder, './train2.py'), 'r') as training_script:\n",
|
|
" print(training_script.read())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now let's try again. And this time it should work fine."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"src = ScriptRunConfig(source_directory=script_folder, \n",
|
|
" script='train2.py', \n",
|
|
" run_config=vm_run_config)\n",
|
|
"run = exp.submit(config=src)\n",
|
|
"run.wait_for_completion(show_output=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note even in this case you get a run record with some basic statistics."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Configure a Docker run with new conda environment on the VM\n",
|
|
"You can execute in a Docker container in the VM. If you choose this option, the system will pull down a base Docker image, build a new conda environment in it if you ask for (you can also skip this if you are using a customer Docker image when a preconfigured Python environment), start a container, and run your script in there. This image is also uploaded into your ACR (Azure Container Registry) assoicated with your workspace, an reused if your dependencies don't change in the subsequent runs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Load the \"cpu-dsvm.runconfig\" file (created by the above attach operation) in memory\n",
|
|
"docker_run_config = RunConfiguration(framework=\"python\")\n",
|
|
"\n",
|
|
"# Set compute target to the Linux DSVM\n",
|
|
"docker_run_config.target = attached_dsvm_compute.name\n",
|
|
"\n",
|
|
"# Use Docker in the remote VM\n",
|
|
"docker_run_config.environment.docker.enabled = True\n",
|
|
"\n",
|
|
"# Use CPU base image from DockerHub\n",
|
|
"docker_run_config.environment.docker.base_image = azureml.core.runconfig.DEFAULT_CPU_IMAGE\n",
|
|
"print('Base Docker image is:', docker_run_config.environment.docker.base_image)\n",
|
|
"\n",
|
|
"# set the data reference of the run coonfiguration\n",
|
|
"docker_run_config.data_references = {ds.name: dr}\n",
|
|
"\n",
|
|
"# specify CondaDependencies obj\n",
|
|
"docker_run_config.environment.python.conda_dependencies = CondaDependencies.create(conda_packages=['scikit-learn'])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Submit the Experiment\n",
|
|
"Submit script to run in the Docker image in the remote VM. If you run this for the first time, the system will download the base image, layer in packages specified in the `conda_dependencies.yml` file on top of the base image, create a container and then execute the script in the container."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"src = ScriptRunConfig(source_directory=script_folder, \n",
|
|
" script='train.py', \n",
|
|
" run_config=docker_run_config,\n",
|
|
" # pass the datastore reference as a parameter to the training script\n",
|
|
" arguments=['--data-folder', str(ds.as_download())])\n",
|
|
"run = exp.submit(config=src)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run.wait_for_completion(show_output=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Use a custom Docker image instead\n",
|
|
"\n",
|
|
"You can also specify a custom Docker image if you don't want to use the default image provided by Azure ML.\n",
|
|
"\n",
|
|
"```python\n",
|
|
"# use an image available in Docker Hub without authentication\n",
|
|
"run_config_docker.environment.docker.base_image = \"continuumio/miniconda3\"\n",
|
|
"\n",
|
|
"# or, use an image available in a private Azure Container Registry\n",
|
|
"run_config_docker.environment.docker.base_image = \"mycustomimage:1.0\"\n",
|
|
"run_config_docker.environment.docker.base_image_registry.address = \"myregistry.azurecr.io\"\n",
|
|
"run_config_docker.environment.docker.base_image_registry.username = \"username\"\n",
|
|
"run_config_docker.environment.docker.base_image_registry.password = \"password\"\n",
|
|
"```\n",
|
|
"\n",
|
|
"When you are using a custom Docker image, you might already have your environment setup properly in a Python environment in the Docker image. In that case, you can skip specifying conda dependencies, and just use `user_managed_dependencies` option instead:\n",
|
|
"```python\n",
|
|
"run_config_docker.environment.python.user_managed_dependencies = True\n",
|
|
"# path to the Python environment in the custom Docker image\n",
|
|
"run_config.environment.python.interpreter_path = '/opt/conda/bin/python'\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### View run history details"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Find the best model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we have tried various execution modes, we can find the best model from the last run."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# get all metris logged in the run\n",
|
|
"run.get_metrics()\n",
|
|
"metrics = run.get_metrics()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# find the index where MSE is the smallest\n",
|
|
"indices = list(range(0, len(metrics['mse'])))\n",
|
|
"min_mse_index = min(indices, key=lambda x: metrics['mse'][x])\n",
|
|
"\n",
|
|
"print('When alpha is {1:0.2f}, we have min MSE {0:0.2f}.'.format(\n",
|
|
" metrics['mse'][min_mse_index], \n",
|
|
" metrics['alpha'][min_mse_index]\n",
|
|
"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Clean up compute resource\n",
|
|
"\n",
|
|
"Use ```detach()``` to detach an existing DSVM from Workspace without deleting it. Use ```delete()``` if you created a new ```DsvmCompute``` and want to delete it."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# dsvm_compute.detach()\n",
|
|
"# dsvm_compute.delete()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"authors": [
|
|
{
|
|
"name": "roastala"
|
|
}
|
|
],
|
|
"kernelspec": {
|
|
"display_name": "Python 3.6",
|
|
"language": "python",
|
|
"name": "python36"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.6.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
} |