mirror of
https://github.com/Azure/MachineLearningNotebooks.git
synced 2025-12-19 17:17:04 -05:00
611 lines
25 KiB
Plaintext
611 lines
25 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Copyright (c) Microsoft Corporation. All rights reserved.\n",
|
|
"\n",
|
|
"Licensed under the MIT License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Reinforcement Learning in Azure Machine Learning - Pong problem\n",
|
|
"Reinforcement Learning in Azure Machine Learning is a managed service for running distributed reinforcement learning training and simulation using the open source Ray framework.\n",
|
|
"This example uses Ray RLlib to train a Pong playing agent on a multi-node cluster.\n",
|
|
"\n",
|
|
"## Pong problem\n",
|
|
"[Pong](https://en.wikipedia.org/wiki/Pong) is a two-dimensional sports game that simulates table tennis. The player controls an in-game paddle by moving it vertically across the left or right side of the screen. They can compete against another player controlling a second paddle on the opposing side. Players use the paddles to hit a ball back and forth."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<table style=\"width:50%\">\n",
|
|
" <tr>\n",
|
|
" <th style=\"text-align: center;\"><img src=\"./images/pong.gif\" alt=\"Pong image\" align=\"middle\" margin-left=\"auto\" margin-right=\"auto\"/></th>\n",
|
|
" </tr>\n",
|
|
" <tr style=\"text-align: center;\">\n",
|
|
" <th>Fig 1. Pong game animation (from <a href=\"https://towardsdatascience.com/intro-to-reinforcement-learning-pong-92a94aa0f84d\">towardsdatascience.com</a>).</th>\n",
|
|
" </tr>\n",
|
|
"</table>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The goal here is to train an agent to win an episode of Pong game against opponent with the score of at least 18 points. An episode in Pong runs until one of the players reaches a score of 21. Episodes are a terminology that is used across all the [OpenAI gym](https://gym.openai.com/envs/Pong-v0/) environments that contains a strictly defined task.\n",
|
|
"\n",
|
|
"Training a Pong agent is a compute-intensive task and this example demonstrates the use of Reinforcement Learning in Azure Machine Learning service to train an agent faster in a distributed, parallel environment. You'll learn more about using the head and the worker compute targets to train an agent in this notebook below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisite\n",
|
|
"\n",
|
|
"The user should have completed the [Reinforcement Learning in Azure Machine Learning - Setting Up Development Environment](../setup/devenv_setup.ipynb) to setup a virtual network. This virtual network will be used here for head and worker compute targets. It is highly recommended that the user should go through the [Reinforcement Learning in Azure Machine Learning - Cartpole Problem on Single Compute](../cartpole-on-single-compute/cartpole_sc.ipynb) to understand the basics of Reinforcement Learning in Azure Machine Learning and Ray RLlib used in this notebook."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Set up Development Environment\n",
|
|
"The following subsections show typical steps to setup your development environment. Setup includes:\n",
|
|
"\n",
|
|
"* Connecting to a workspace to enable communication between your local machine and remote resources\n",
|
|
"* Creating an experiment to track all your runs\n",
|
|
"* Creating remote head and worker compute target on a virtual network to use for training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Azure Machine Learning SDK\n",
|
|
"Display the Azure Machine Learning SDK version."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Azure Machine Learning core imports\n",
|
|
"import azureml.core\n",
|
|
"\n",
|
|
"# Check core SDK version number\n",
|
|
"print(\"Azure Machine Learning SDK Version: \", azureml.core.VERSION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Get Azure Machine Learning workspace\n",
|
|
"Get a reference to an existing Azure Machine Learning workspace."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core import Workspace\n",
|
|
"\n",
|
|
"ws = Workspace.from_config()\n",
|
|
"print(ws.name, ws.location, ws.resource_group, sep = ' | ')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create Azure Machine Learning experiment\n",
|
|
"Create an experiment to track the runs in your workspace."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core.experiment import Experiment\n",
|
|
"\n",
|
|
"# Experiment name\n",
|
|
"experiment_name = 'rllib-pong-multi-node'\n",
|
|
"exp = Experiment(workspace=ws, name=experiment_name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Specify the name of your virtual network\n",
|
|
"\n",
|
|
"The resource group you use must contain a virtual network. Specify the name of the virtual network here created in the [Azure Machine Learning Reinforcement Learning Sample - Setting Up Development Environment](../setup/devenv_setup.ipynb)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Virtual network name\n",
|
|
"vnet_name = 'your_vnet'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create head compute target\n",
|
|
"\n",
|
|
"In this example, we show how to set up separate compute targets for the Ray head and Ray worker nodes. First we define the head cluster with GPU for the Ray head node. One CPU of the head node will be used for the Ray head process and the rest of the CPUs will be used by the Ray worker processes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.core.compute import AmlCompute, ComputeTarget\n",
|
|
"\n",
|
|
"# Choose a name for the Ray head cluster\n",
|
|
"head_compute_name = 'head-gpu'\n",
|
|
"head_compute_min_nodes = 0\n",
|
|
"head_compute_max_nodes = 2\n",
|
|
"\n",
|
|
"# This example uses GPU VM. For using CPU VM, set SKU to STANDARD_D2_V2\n",
|
|
"head_vm_size = 'STANDARD_NC6'\n",
|
|
"\n",
|
|
"if head_compute_name in ws.compute_targets:\n",
|
|
" head_compute_target = ws.compute_targets[head_compute_name]\n",
|
|
" if head_compute_target and type(head_compute_target) is AmlCompute:\n",
|
|
" if head_compute_target.provisioning_state == 'Succeeded':\n",
|
|
" print('found head compute target. just use it', head_compute_name)\n",
|
|
" else: \n",
|
|
" raise Exception(\n",
|
|
" 'found head compute target but it is in state', head_compute_target.provisioning_state)\n",
|
|
"else:\n",
|
|
" print('creating a new head compute target...')\n",
|
|
" provisioning_config = AmlCompute.provisioning_configuration(\n",
|
|
" vm_size=head_vm_size,\n",
|
|
" min_nodes=head_compute_min_nodes, \n",
|
|
" max_nodes=head_compute_max_nodes,\n",
|
|
" vnet_resourcegroup_name=ws.resource_group,\n",
|
|
" vnet_name=vnet_name,\n",
|
|
" subnet_name='default')\n",
|
|
"\n",
|
|
" # Create the cluster\n",
|
|
" head_compute_target = ComputeTarget.create(ws, head_compute_name, provisioning_config)\n",
|
|
" \n",
|
|
" # Can poll for a minimum number of nodes and for a specific timeout. \n",
|
|
" # If no min node count is provided it will use the scale settings for the cluster\n",
|
|
" head_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
|
|
" \n",
|
|
" # For a more detailed view of current AmlCompute status, use get_status()\n",
|
|
" print(head_compute_target.get_status().serialize())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create worker compute target\n",
|
|
"\n",
|
|
"Now we create a compute target with CPUs for the additional Ray worker nodes. CPUs in these worker nodes are used by Ray worker processes. Each Ray worker node, depending on the CPUs on the node, may have multiple Ray worker processes. There can be multiple worker tasks on each worker process (core)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Choose a name for your Ray worker compute target\n",
|
|
"worker_compute_name = 'worker-cpu'\n",
|
|
"worker_compute_min_nodes = 0 \n",
|
|
"worker_compute_max_nodes = 4\n",
|
|
"\n",
|
|
"# This example uses CPU VM. For using GPU VM, set SKU to STANDARD_NC6\n",
|
|
"worker_vm_size = 'STANDARD_D2_V2'\n",
|
|
"\n",
|
|
"# Create the compute target if it hasn't been created already\n",
|
|
"if worker_compute_name in ws.compute_targets:\n",
|
|
" worker_compute_target = ws.compute_targets[worker_compute_name]\n",
|
|
" if worker_compute_target and type(worker_compute_target) is AmlCompute:\n",
|
|
" if worker_compute_target.provisioning_state == 'Succeeded':\n",
|
|
" print('found worker compute target. just use it', worker_compute_name)\n",
|
|
" else: \n",
|
|
" raise Exception(\n",
|
|
" 'found worker compute target but it is in state', head_compute_target.provisioning_state)\n",
|
|
"else:\n",
|
|
" print('creating a new worker compute target...')\n",
|
|
" provisioning_config = AmlCompute.provisioning_configuration(\n",
|
|
" vm_size=worker_vm_size,\n",
|
|
" min_nodes=worker_compute_min_nodes,\n",
|
|
" max_nodes=worker_compute_max_nodes,\n",
|
|
" vnet_resourcegroup_name=ws.resource_group,\n",
|
|
" vnet_name=vnet_name,\n",
|
|
" subnet_name='default')\n",
|
|
"\n",
|
|
" # Create the compute target\n",
|
|
" worker_compute_target = ComputeTarget.create(ws, worker_compute_name, provisioning_config)\n",
|
|
" \n",
|
|
" # Can poll for a minimum number of nodes and for a specific timeout. \n",
|
|
" # If no min node count is provided it will use the scale settings for the cluster\n",
|
|
" worker_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
|
|
" \n",
|
|
" # For a more detailed view of current AmlCompute status, use get_status()\n",
|
|
" print(worker_compute_target.get_status().serialize())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Train Pong Agent\n",
|
|
"To facilitate reinforcement learning, Azure Machine Learning Python SDK provides a high level abstraction, the _ReinforcementLearningEstimator_ class, which allows users to easily construct reinforcement learning run configurations for the underlying reinforcement learning framework. Reinforcement Learning in Azure Machine Learning supports the open source [Ray framework](https://ray.io/) and its highly customizable [RLLib](https://ray.readthedocs.io/en/latest/rllib.html#rllib-scalable-reinforcement-learning). In this section we show how to use _ReinforcementLearningEstimator_ and Ray/RLLib framework to train a Pong playing agent.\n",
|
|
"\n",
|
|
"\n",
|
|
"### Define worker configuration\n",
|
|
"Define a `WorkerConfiguration` using your worker compute target. We specify the number of nodes in the worker compute target to be used for training and additional PIP packages to install on those nodes as a part of setup.\n",
|
|
"In this case, we define the PIP packages as dependencies for both head and worker nodes. With this setup, the game simulations will run directly on the worker compute nodes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.contrib.train.rl import WorkerConfiguration\n",
|
|
"\n",
|
|
"# Pip packages we will use for both head and worker\n",
|
|
"pip_packages=[\"ray[rllib]==0.8.3\"] # Latest version of Ray has fixes for isses related to object transfers\n",
|
|
"\n",
|
|
"# Specify the Ray worker configuration\n",
|
|
"worker_conf = WorkerConfiguration(\n",
|
|
" \n",
|
|
" # Azure Machine Learning compute target to run Ray workers\n",
|
|
" compute_target=worker_compute_target, \n",
|
|
" \n",
|
|
" # Number of worker nodes\n",
|
|
" node_count=4,\n",
|
|
" \n",
|
|
" # GPU\n",
|
|
" use_gpu=False, \n",
|
|
" \n",
|
|
" # PIP packages to use\n",
|
|
" pip_packages=pip_packages\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create reinforcement learning estimator\n",
|
|
"\n",
|
|
"The `ReinforcementLearningEstimator` is used to submit a job to Azure Machine Learning to start the Ray experiment run. We define the training script parameters here that will be passed to the estimator. \n",
|
|
"\n",
|
|
"We specify `episode_reward_mean` to 18 as we want to stop the training as soon as the trained agent reaches an average win margin of at least 18 point over opponent over all episodes in the training epoch.\n",
|
|
"Number of Ray worker processes are defined by parameter `num_workers`. We set it to 13 as we have 13 CPUs available in our compute targets. Multiple Ray worker processes parallelizes agent training and helps in achieving our goal faster. \n",
|
|
"\n",
|
|
"```\n",
|
|
"Number of CPUs in head_compute_target = 6 CPUs in 1 node = 6\n",
|
|
"Number of CPUs in worker_compute_target = 2 CPUs in each of 4 nodes = 8\n",
|
|
"Number of CPUs available = (Number of CPUs in head_compute_target) + (Number of CPUs in worker_compute_target) - (1 CPU for head node) = 6 + 8 - 1 = 13\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.contrib.train.rl import ReinforcementLearningEstimator, Ray\n",
|
|
"\n",
|
|
"training_algorithm = \"IMPALA\"\n",
|
|
"rl_environment = \"PongNoFrameskip-v4\"\n",
|
|
"\n",
|
|
"# Training script parameters\n",
|
|
"script_params = {\n",
|
|
" \n",
|
|
" # Training algorithm, IMPALA in this case\n",
|
|
" \"--run\": training_algorithm,\n",
|
|
" \n",
|
|
" # Environment, Pong in this case\n",
|
|
" \"--env\": rl_environment,\n",
|
|
" \n",
|
|
" # Add additional single quotes at the both ends of string values as we have spaces in the \n",
|
|
" # string parameters, outermost quotes are not passed to scripts as they are not actually part of string\n",
|
|
" # Number of GPUs\n",
|
|
" # Number of ray workers\n",
|
|
" \"--config\": '\\'{\"num_gpus\": 1, \"num_workers\": 13}\\'',\n",
|
|
" \n",
|
|
" # Target episode reward mean to stop the training\n",
|
|
" # Total training time in seconds\n",
|
|
" \"--stop\": '\\'{\"episode_reward_mean\": 18, \"time_total_s\": 3600}\\'',\n",
|
|
"}\n",
|
|
"\n",
|
|
"# Reinforcement learning estimator\n",
|
|
"rl_estimator = ReinforcementLearningEstimator(\n",
|
|
" \n",
|
|
" # Location of source files\n",
|
|
" source_directory='files',\n",
|
|
" \n",
|
|
" # Python script file\n",
|
|
" entry_script=\"pong_rllib.py\",\n",
|
|
" \n",
|
|
" # Parameters to pass to the script file\n",
|
|
" # Defined above.\n",
|
|
" script_params=script_params,\n",
|
|
" \n",
|
|
" # The Azure Machine Learning compute target set up for Ray head nodes\n",
|
|
" compute_target=head_compute_target,\n",
|
|
" \n",
|
|
" # Pip packages\n",
|
|
" pip_packages=pip_packages,\n",
|
|
" \n",
|
|
" # GPU usage\n",
|
|
" use_gpu=True,\n",
|
|
" \n",
|
|
" # Reinforcement learning framework. Currently must be Ray.\n",
|
|
" rl_framework=Ray(),\n",
|
|
" \n",
|
|
" # Ray worker configuration defined above.\n",
|
|
" worker_configuration=worker_conf,\n",
|
|
" \n",
|
|
" # How long to wait for whole cluster to start\n",
|
|
" cluster_coordination_timeout_seconds=3600,\n",
|
|
" \n",
|
|
" # Maximum time for the whole Ray job to run\n",
|
|
" # This will cut off the run after an hour\n",
|
|
" max_run_duration_seconds=3600,\n",
|
|
" \n",
|
|
" # Allow the docker container Ray runs in to make full use\n",
|
|
" # of the shared memory available from the host OS.\n",
|
|
" shm_size=24*1024*1024*1024\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Training script\n",
|
|
"As recommended in [RLlib](https://ray.readthedocs.io/en/latest/rllib.html) documentations, we use Ray [Tune](https://ray.readthedocs.io/en/latest/tune.html) API to run the training algorithm. All the RLlib built-in trainers are compatible with the Tune API. Here we use tune.run() to execute a built-in training algorithm. For convenience, down below you can see part of the entry script where we make this call.\n",
|
|
"\n",
|
|
"```python\n",
|
|
" tune.run(\n",
|
|
" run_or_experiment=args.run,\n",
|
|
" config={\n",
|
|
" \"env\": args.env,\n",
|
|
" \"num_gpus\": args.config[\"num_gpus\"],\n",
|
|
" \"num_workers\": args.config[\"num_workers\"],\n",
|
|
" \"callbacks\": {\"on_train_result\": callbacks.on_train_result},\n",
|
|
" \"sample_batch_size\": 50,\n",
|
|
" \"train_batch_size\": 1000,\n",
|
|
" \"num_sgd_iter\": 2,\n",
|
|
" \"num_data_loader_buffers\": 2,\n",
|
|
" \"model\": {\"dim\": 42},\n",
|
|
" },\n",
|
|
" stop=args.stop,\n",
|
|
" local_dir='./logs')\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Submit the estimator to start a run\n",
|
|
"Now we use the rl_estimator configured above to submit a run."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run = exp.submit(config=rl_estimator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Monitor the run\n",
|
|
"\n",
|
|
"Azure Machine Learning provides a Jupyter widget to show the status of an experiment run. You could use this widget to monitor the status of the runs. The widget shows the list of two child runs, one for head compute target run and one for worker compute target run. You can click on the link under **Status** to see the details of the child run. It will also show the metrics being logged."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azureml.widgets import RunDetails\n",
|
|
"\n",
|
|
"RunDetails(run).show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Stop the run\n",
|
|
"\n",
|
|
"To stop the run, call `run.cancel()`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Uncomment line below to cancel the run\n",
|
|
"# run.cancel()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Wait for completion\n",
|
|
"Wait for the run to complete before proceeding. If you want to stop the run, you may skip this and move to next section below. \n",
|
|
"\n",
|
|
"**Note: The run may take anywhere from 30 minutes to 45 minutes to complete.**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"run.wait_for_completion()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Performance of the agent during training\n",
|
|
"\n",
|
|
"Let's get the reward metrics for the training run agent and observe how the agent's rewards improved over the training iterations and how the agent learns to win the Pong game. \n",
|
|
"\n",
|
|
"Collect the episode reward metrics from the worker run's metrics. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Get all child runs\n",
|
|
"child_runs = list(run.get_children(_rehydrate_runs=False))\n",
|
|
"\n",
|
|
"# Get the reward metrics from worker run\n",
|
|
"if child_runs[0].id.endswith(\"_worker\"):\n",
|
|
" episode_reward_mean = child_runs[0].get_metrics(name='episode_reward_mean')\n",
|
|
"else:\n",
|
|
" episode_reward_mean = child_runs[1].get_metrics(name='episode_reward_mean')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Plot the reward metrics. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"plt.plot(episode_reward_mean['episode_reward_mean'])\n",
|
|
"plt.xlabel('training_iteration')\n",
|
|
"plt.ylabel('episode_reward_mean')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We observe that during the training over multiple episodes, the agent learns to win the Pong game against opponent with our target of 18 points in each episode of 21 points.\n",
|
|
"**Congratulations!! You have trained your Pong agent to win a game.**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Cleaning up\n",
|
|
"For your convenience, below you can find code snippets to clean up any resources created as part of this tutorial that you don't wish to retain."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# To archive the created experiment:\n",
|
|
"#experiment.archive()\n",
|
|
"\n",
|
|
"# To delete the compute targets:\n",
|
|
"#head_compute_target.delete()\n",
|
|
"#worker_compute_target.delete()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Next\n",
|
|
"In this example, you learned how to solve distributed reinforcement learning training problems using head and worker compute targets. This was an introductory tutorial on Reinforement Learning in Azure Machine Learning service offering. We would love to hear your feedback to build the features you need!"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"authors": [
|
|
{
|
|
"name": "vineetg"
|
|
}
|
|
],
|
|
"kernelspec": {
|
|
"display_name": "Python 3.6",
|
|
"language": "python",
|
|
"name": "python36"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.6.9"
|
|
},
|
|
"notice": "Copyright (c) Microsoft Corporation. All rights reserved.\u00e2\u20ac\u00afLicensed under the MIT License.\u00e2\u20ac\u00af "
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
} |