Add articles about testing and troubleshooting webhooks (#42656)

Co-authored-by: Kevin Heis <heiskr@users.noreply.github.com>
Co-authored-by: Nathan Stocks <cleancut@github.com>
Co-authored-by: Jess Hosman <1183847+jhosman@users.noreply.github.com>

content/webhooks/index.md

@@ -13,6 +13,7 @@ featuredLinks:
   popular:
     - /webhooks-and-events/webhooks/webhook-events-and-payloads
     - /webhooks/using-webhooks/best-practices-for-using-webhooks
+    - /webhooks/testing-and-troubleshooting-webhooks/troubleshooting-webhooks
   guideCards:
     - /webhooks/using-webhooks/handling-webhook-deliveries
     - /webhooks/using-webhooks/handling-failed-webhook-deliveries

content/webhooks/testing-and-troubleshooting-webhooks/index.md

@@ -12,6 +12,7 @@ children:
   - /viewing-webhook-deliveries
   - /redelivering-webhooks
   - /testing-webhooks
+  - /troubleshooting-webhooks
   - /receiving-webhooks-with-the-github-cli
 ---
 

content/webhooks/testing-and-troubleshooting-webhooks/testing-webhooks.md

@@ -1,6 +1,6 @@
 ---
 title: Testing webhooks
-intro: 'Review your webhook deliveries on {% data variables.product.prodname_dotcom %}, including the HTTP Request and payload as well as the response.'
+intro: 'Learn how to test your webhooks and your code that handles webhook deliveries.'
 redirect_from:
   - /webhooks/testing
   - /developers/webhooks-and-events/testing-webhooks
@@ -17,35 +17,112 @@ versions:
 topics:
   - Webhooks
 ---
-Now that you've [configured your local server](/webhooks-and-events/webhooks/configuring-your-server-to-receive-payloads), you might
-be interested in pushing your code to the limits. To that end, GitHub's webhooks
-view provides some tooling for testing your deployed payloads.
 
-## Listing recent deliveries
+## About testing webhooks
 
-Every webhook has its own "Recent Deliveries" section, which lists, at a glance whether a delivery was successful (green check) or failed (red x). You can also identify when each delivery was attempted.
+You can test webhook delivery. This will let you verify that {% data variables.product.company_short %} sends a webhook delivery in response to an event that you expect to trigger a webhook delivery.
 
-{% data variables.product.product_name %} keeps a log of each webhook delivery for {% ifversion fpt or ghec %} 30 {% else %} 8 {% endif %} days.
+You can also test your code that handles webhook deliveries by using your computer or codespace as a local server and forwarding webhook deliveries to your local server. This will let you develop and debug your code without deploying your code to your production server.
 
-![Screenshot of the "Recent Deliveries" tab on the "Manage webhook" page.](/assets/images/help/webhooks/webhooks-recent-deliveries.png)
+## Testing webhook delivery
 
-## Digging into results
+You can trigger a webhook event and verify that {% data variables.product.company_short %} sent a webhook delivery.
 
-By expanding an individual delivery, you'll be able to witness _precisely_
+1. Trigger your webhook. For example, if you are testing a repository webhook that is subscribed to the `issues` event, open an issue in the repository where the webhook is configured.
-what information GitHub is attempting to send to your server. This includes
-both the HTTP Request and Response.
 
-### Request
+   You can also redeliver a previous webhook delivery. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks)."
 
-The webhook delivery view provides information on which Headers were sent by GitHub.
+1. Check {% data variables.product.company_short %} to verify that a webhook delivery was sent. For information about how to do this for each webhook type, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/viewing-webhook-deliveries)."
-It also includes details about the JSON payload.
 
-![Screenshot of the "Request" tab of a webhook delivery, including the "Headers" and "Payload" sections.](/assets/images/help/webhooks/payload-request-tab.png)
+If a webhook delivery was not sent, or if a webhook delivery was sent but {% data variables.product.company_short %} indicates that the delivery failed, refer to the troubleshooting guide to help diagnose the problem. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/troubleshooting-webhooks)."
 
-### Response
+## Testing webhook code locally
 
-The response tab lists how your server replied once it received the payload from
+In order to test your webhook code locally on your computer or codespace, you can use a webhook proxy URL to forward webhooks from {% data variables.product.company_short %} to your computer or codespace. You can use your computer or codespace as a local server to receive these forwarded webhooks.
-GitHub. This includes the status code, the headers, and any additional data
-within the response body.
 
-![Screenshot of the "Response" tab of a webhook delivery, including the "Headers" and "Body" sections.](/assets/images/help/webhooks/payload-response-tab.png)
+The following sections demonstrate how to use smee.io to provide a webhook proxy URL and forward webhooks.
+
+For specific examples of code and testing steps, see "[AUTOTITLE](/webhooks/using-webhooks/handling-webhook-deliveries)."
+
+### Get a webhook proxy URL
+
+1. In your browser, navigate to https://smee.io/.
+1. Click **Start a new channel**.
+1. Copy the full URL under "Webhook Proxy URL". You will use this URL in the following setup steps.
+
+### Configure a webhook to use the webhook proxy URL
+
+Configure your webhook to use the webhook proxy URL from above. For more information, see "[AUTOTITLE](/webhooks/using-webhooks/creating-webhooks)" and "[AUTOTITLE](/webhooks/using-webhooks/editing-webhooks)."
+
+Now, {% data variables.product.company_short %} will send webhook deliveries to that URL.
+
+### Start a local server
+
+On your computer or codespace, start a local server. The way that you do this depends on how your code to receive webhooks is written. For examples, see "[AUTOTITLE](/webhooks/using-webhooks/handling-webhook-deliveries)."
+
+You should make sure that your code can run locally. For example, if your code relies on environment variables on your server in production, you should make sure that the environment variables are also available on your local server.
+
+You may also find it useful to add log statements so that you can verify that steps of your code executed as expected.
+
+Keep your local server running while you test out your webhook.
+
+### Forward webhooks
+
+1. If you don't already have [smee-client](https://www.npmjs.com/package/smee-client) installed, run the following command in your terminal:
+
+   ```shell copy
+   npm install --global smee-client
+   ```
+
+1. To receive forwarded webhooks from smee.io, run the following command in your terminal. Replace `WEBHOOK_PROXY_URL` with your webhook proxy URL from earlier. Replace `PATH` with the path or route that your server will handle. Replace `PORT` with the port where your local server is listening.
+
+   ```shell copy
+   smee --url WEBHOOK_PROXY_URL --path /PATH --port PORT
+   ```
+
+   You should see output that looks like this, with the `WEBHOOK_PROXY_URL`, `PORT`, and `PATH` placeholders replaced with the values you specified:
+
+   ```shell copy
+   Forwarding WEBHOOK_PROXY_URL to http://127.0.0.1:PORT/PATH
+   Connected WEBHOOK_PROXY_URL
+   ```
+
+   Now, when your webhook proxy URL (smee.io URL) receives a webhook delivery from {% data variables.product.company_short %}, smee will forward the webhook delivery to your local server.
+
+1. Keep this running while you test out your webhook. When you want to stop forwarding webhooks, enter <kbd>Ctrl</kbd>+<kbd>C</kbd>.
+
+At this point, you should have both your local server running and the smee forwarding running.
+
+### Trigger a webhook delivery
+
+Trigger your webhook. For example, if you are testing a repository webhook that is subscribed to the `issues` event, open an issue in the repository where the webhook is configured.
+
+You can also redeliver a previous webhook delivery. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks)."
+
+### Verify delivery
+
+You can verify that {% data variables.product.company_short %} sent a webhook delivery, that smee received and forwarded the delivery, and that your local server processed the webhook delivery.
+
+#### Verify that {% data variables.product.company_short %} sent a delivery
+
+Check {% data variables.product.company_short %} to verify that a webhook delivery was sent. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/viewing-webhook-deliveries)."
+
+If a webhook delivery was not sent, or if a webhook delivery was sent but {% data variables.product.company_short %} indicates that the delivery failed, refer to the troubleshooting guide to help diagnose the problem. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/troubleshooting-webhooks)."
+
+#### Verify that smee received your webhook delivery
+
+Navigate to your webhook proxy URL on smee.io. You should see an event that corresponds to the event that you triggered or redelivered. This indicates that {% data variables.product.company_short %} successfully sent a webhook delivery to the payload URL that you specified.
+
+If you don't see your webhook delivery on smee.io, verify that your webhook is using your webhook proxy URL (smee.io URL).
+
+#### Verify that smee forwarded your webhook delivery
+
+In the terminal window where you ran `smee --url WEBHOOK_PROXY_URL --path /PATH --port PORT`, you should see something like `POST http://127.0.0.1:3000/webhook - 202`. This indicates that smee successfully forwarded your webhook to your local server.
+
+If you don't see this, make sure that both the smee client and your local server are running. You should have these processes running in two separate terminal windows.
+
+You should also check for errors in the terminal windows where you are running the smee client and your local server. The specific errors depend on how your code to receive webhooks is written. For examples, see "[AUTOTITLE](/webhooks/using-webhooks/handling-webhook-deliveries)."
+
+#### Verify that your local server processed the webhook delivery
+
+At this point, you have verified that {% data variables.product.company_short %} sent a webhook delivery and that smee forwarded the delivery to your local server. Now, you should verify that your code processed the webhook delivery as expected. The way that you do this depends on how your code to receive webhooks is written. For examples, see "[AUTOTITLE](/webhooks/using-webhooks/handling-webhook-deliveries)."

content/webhooks/testing-and-troubleshooting-webhooks/troubleshooting-webhooks.md

@@ -0,0 +1,90 @@
+---
+title: Troubleshooting webhooks
+intro: 'Learn how to diagnose and resolve common errors for webhooks.'
+versions:
+  fpt: '*'
+  ghes: '*'
+  ghae: '*'
+  ghec: '*'
+topics:
+  - Webhooks
+---
+
+## Missing webhook deliveries
+
+If you are not receiving the webhook deliveries that you expect, you should identify the point at which the delivery is missing.
+
+1. Trigger an event that you expect to result in a webhook delivery. For example, if your webhook is a repository webhook that is subscribed to the `issues` event, you can open an issue on that repository.
+1. Look at the recent deliveries log for your webhook. For information about how to do this for each webhook type, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/viewing-webhook-deliveries)."
+
+   If the recent deliveries log does not include a delivery that corresponds to the webhook event that you triggered in the previous step, then {% data variables.product.company_short %} did not attempt a delivery. To identify the cause:
+
+   1. Wait a few minutes, and then check again. Webhook deliveries can take a few minutes to appear.
+   1. Make sure that you triggered an event in the location where your webhook is configured. For example, if your webhook is a repository webhook, make sure that you triggered the event in the same repository where your webhook is configured.
+   1. Make sure that your webhook is subscribed to the event that you triggered. For example, if you expect a webhook delivery when you open an issue, make sure your webhook is subscribed to the `issues` event.
+   1. Make sure that your webhook is active. For more information, see "[AUTOTITLE](/webhooks/using-webhooks/disabling-webhooks)."
+   1. Make sure that your webhook is not impacted by {% data variables.product.prodname_oauth_app %} access restrictions. If your webhook was created by an {% data variables.product.prodname_oauth_app %} on behalf of a user who authorized the {% data variables.product.prodname_oauth_app %}, the webhook will be automatically disabled if it is an organization or repository webhook for an organization that has restricted access by the {% data variables.product.prodname_oauth_app %}. For more information, see {% ifversion ghec or fpt %}"[AUTOTITLE](/organizations/managing-oauth-access-to-your-organizations-data/about-oauth-app-access-restrictions)."{% else %}"[AUTOTITLE](/free-pro-team@latest/organizations/managing-oauth-access-to-your-organizations-data/about-oauth-app-access-restrictions)" in the {% data variables.product.prodname_free_user %} documentation.{% endif %}
+
+   1. Check whether your event may have hit a documented limit. For example, if you push more than three tags at once, the `push` event will not be triggered for that push. For more information about documented limits for each event, see "[AUTOTITLE](/webhooks/webhook-events-and-payloads)."
+   1. Check the status of webhooks at [githubstatus.com](https://www.githubstatus.com/).
+
+   If the recent deliveries log indicates that there was an error with the delivery, then {% data variables.product.company_short %} attempted delivery but the delivery was unsuccessful. This is typically due to a problem with your server. You can refer to the sections below to help resolve the specific error.
+
+1. Look at the logs for your server. The information in the logs depends on the code that your server runs to handle webhook deliveries. To help you diagnose problems on your server, you may want to add additional log statements to your code.
+
+## Cannot have more than {% ifversion ghes or ghae %}250{% else %}20{% endif %} webhooks
+
+You can create up to {% ifversion ghes or ghae %}250{% else %}20{% endif %} {% ifversion ghec or ghes or ghae %} repository, organization, or global {% else %} repository or organization {% endif %}webhooks for each event type. If you attempt to create more, you will receive an error stating that you cannot have more than {% ifversion ghes or ghae %}250{% else %}20{% endif %} webhooks.
+
+If you require more than {% ifversion ghes or ghae %}250{% else %}20{% endif %} webhooks, you can run a proxy that receives webhooks from {% data variables.product.company_short %} and forwards them to an unlimited number of destination URLs.
+
+## URL host localhost is not supported
+
+You cannot use `localhost` or `127.0.0.1` as a webhook URL.
+
+If you want to deliver webhooks to your local server for testing, you can use a webhook forwarding service like smee.io. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/testing-webhooks)."
+
+## Failed to connect to host
+
+The `failed to connect to host` error occurs when {% data variables.product.company_short %} attempts a webhook delivery but could not resolve the webhook's URL to an IP address.
+
+To check whether a host name resolves to an IP address, you can use `nslookup`. For example, if your payload URL is `https://octodex.github.com/webhooks`, you can run `nslookup octodex.github.com`. If the host name could not be resolved to an IP address, the nslookup command will indicate that the server can't find the host name.
+
+## Failed to connect to network
+
+The `failed to connect to network` error indicates that your server refused the connection when {% data variables.product.company_short %} attempted to deliver a webhook.
+
+You should make sure that your server allows connections from {% data variables.product.company_short %}'s IP addresses. You can use the `GET /meta` endpoint to find the current list of {% data variables.product.company_short %}'s IP addresses. For more information, see "[AUTOTITLE](/rest/meta/meta#get-github-meta-information)." {% data variables.product.company_short %} occasionally makes changes to its IP addresses, so you should update your IP allow list periodically.
+
+## Timed out
+
+The `timed out` error indicates that {% data variables.product.company_short %} did not receive a response from your server within {% ifversion fpt or ghec %}10{% else %}30{% endif %} seconds of delivering a webhook.
+
+Your server should respond with a 2XX response within {% ifversion fpt or ghec %}10{% else %}30{% endif %} seconds of receiving a webhook delivery. If your server takes longer than that to respond, then {% data variables.product.company_short %} terminates the connection and considers the delivery a failure.
+
+In order to respond in a timely manner, you may want to set up a queue to process webhook payloads asynchronously. Your server can respond when it receives the webhook, and then process the payload in the background without blocking future webhook deliveries. For example, you can use services like [Hookdeck](https://hookdeck.com) or libraries like [Resque](https://github.com/resque/resque/) (Ruby), [RQ](http://python-rq.org/) (Python), or [RabbitMQ](http://www.rabbitmq.com/).
+
+## Peer certificate cannot be authenticated with given CA certificates
+
+This error indicates that there is a problem related to your server's certificates. The most common problems are:
+
+- Your server is using a self-signed certificate.
+- Your server is not sending the full certificate chain when the connection is established.
+
+To help diagnose the problem, you can use the [SSL server test](https://www.ssllabs.com/ssltest/analyze.html) from SSL Labs. This service can only work with the default port for HTTPS (port 443) and can only work with servers that are accessible from the Internet.
+
+You can also use `openssl` to help diagnose the problem. To do so, run `openssl s_client -connect HOST:PORT` in a terminal. Replace `HOST` with your server's host name and `PORT` with the port. For example, `openssl s_client -connect example.com:443`. To identify problems, look for `verify error` in the output.
+
+## Invalid HTTP response
+
+The `invalid HTTP response` error occurs when your server returns a 4xx or 5xx status in response to a webhook delivery from {% data variables.product.company_short %}.
+
+You should configure your server to return a 2xx status. If your server returns a 4xx or 5xx status, {% data variables.product.company_short %} will record the delivery as a failure.
+
+## Webhooks deliveries are out of order
+
+{% data variables.product.company_short %} may deliver webhooks in a different order than the order in which the events took place. If you need to know when the event occurred relative to another event, you should use the timestamps that are included in the delivery payload.
+
+## Webhook deliveries are not immediate
+
+Webhook deliveries can take a few minutes to be delivered and to appear in the recent deliveries log. Before concluding that your webhook delivery failed, wait a few minutes and then check again.

content/webhooks/using-webhooks/handling-failed-webhook-deliveries.md

@@ -30,4 +30,4 @@ You can also write a script that checks for failed deliveries and attempts to re
 - Look at the fetched data to see if any deliveries failed. The data for a failed delivery will have a `status` value that is not `OK`.
 - Use the {% data variables.product.company_short %} REST API to redeliver any deliveries that failed. For more information, see "[AUTOTITLE](/rest/webhooks/repo-deliveries#redeliver-a-delivery-for-a-repository-webhook)," "[AUTOTITLE](/rest/orgs/webhooks#redeliver-a-delivery-for-an-organization-webhook)," and "[AUTOTITLE](/rest/apps/webhooks#redeliver-a-delivery-for-an-app-webhook)."
 
-If a webhook delivery fails repeatedly, you should investigate the cause. Each failed delivery will give a reason for failure. For example, if the delivery failure indicates that {% data variables.product.company_short %} couldn't connect to the host, you should verify that the domain portion of the webhook URL that you specified resolves to an IP address.
+If a webhook delivery fails repeatedly, you should investigate the cause. Each failed delivery will give a reason for failure. For more information, see "[AUTOTITLE](/webhooks/testing-and-troubleshooting-webhooks/troubleshooting-webhooks)."