airbyte/docs/connector-development/config-based/understanding-the-yaml-file/error-handling.md

# Error handling

By default, only server errors (HTTP 5XX) and too many requests (HTTP 429) will be retried up to 5 times with exponential backoff.
Other HTTP errors will result in a failed read.

Other behaviors can be configured through the `Requester`'s `error_handler` field.

Schema:

```yaml
ErrorHandler:
  type: object
  description: "Error handler"
  anyOf:
    - "$ref": "#/definitions/DefaultErrorHandler"
    - "$ref": "#/definitions/CompositeErrorHandler"
```

## Default error handler

Schema:

```yaml
DefaultErrorHandler:
  type: object
  required:
    - max_retries
  additionalProperties: true
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    response_filters:
      type: array
      items:
        "$ref": "#/definitions/HttpResponseFilter"
    max_retries:
      type: integer
      default: 5
    backoff_strategies:
      type: array
      items:
        "$ref": "#/definitions/BackoffStrategy"
      default: []
```

## Defining errors

### From status code

Response filters can be used to define how to handle requests resulting in responses with a specific HTTP status code.
For instance, this example will configure the handler to also retry responses with 404 error:

Schema:

```yaml
HttpResponseFilter:
  type: object
  required:
    - action
  additionalProperties: true
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    action:
      "$ref": "#/definitions/ResponseAction"
    http_codes:
      type: array
      items:
        type: integer
      default: []
    error_message_contains:
      type: string
    predicate:
      type: string
ResponseAction:
  type: string
  enum:
    - SUCCESS
    - FAIL
    - IGNORE
    - RETRY
```

Example:

```yaml
requester:
  <...>
  error_handler:
    response_filters:
        - http_codes: [ 404 ]
          action: RETRY
```

Response filters can be used to specify HTTP errors to ignore.
For instance, this example will configure the handler to ignore responses with 404 error:

```yaml
requester:
  <...>
  error_handler:
    response_filters:
        - http_codes: [ 404 ]
          action: IGNORE
```

### From error message

Errors can also be defined by parsing the error message.
For instance, this error handler will ignore responses if the error message contains the string "ignorethisresponse"

```yaml
requester:
  <...>
  error_handler:
    response_filters:
        - error_message_contains: "ignorethisresponse"
          action: IGNORE
```

This can also be done through a more generic string interpolation strategy with the following parameters:

- response: the decoded response

This example ignores errors where the response contains a "code" field:

```yaml
requester:
  <...>
  error_handler:
    response_filters:
        - predicate: "{{ 'code' in response }}"
          action: IGNORE
```

The error handler can have multiple response filters.
The following example is configured to ignore 404 errors, and retry 429 errors:

```yaml
requester:
  <...>
  error_handler:
    response_filters:
        - http_codes: [ 404 ]
          action: IGNORE
        - http_codes: [ 429 ]
          action: RETRY
```

## Backoff Strategies

The error handler supports a few backoff strategies, which are described in the following sections.

Schema:

```yaml
BackoffStrategy:
  type: object
  anyOf:
    - "$ref": "#/definitions/ExponentialBackoffStrategy"
    - "$ref": "#/definitions/ConstantBackoffStrategy"
    - "$ref": "#/definitions/WaitTimeFromHeader"
    - "$ref": "#/definitions/WaitUntilTimeFromHeader"
```

### Exponential backoff

This is the default backoff strategy. The requester will backoff with an exponential backoff interval

Schema:

```yaml
ExponentialBackoffStrategy:
  type: object
  additionalProperties: true
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    factor:
      type: integer
      default: 5
```

### Constant Backoff

When using the `ConstantBackoffStrategy` strategy, the requester will backoff with a constant interval.

Schema:

```yaml
ConstantBackoffStrategy:
  type: object
  additionalProperties: true
  required:
    - backoff_time_in_seconds
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    backoff_time_in_seconds:
      type: number
```

### Wait time defined in header

When using the `WaitTimeFromHeader`, the requester will backoff by an interval specified in the response header.
In this example, the requester will backoff by the response's "wait_time" header value:

Schema:

```yaml
WaitTimeFromHeader:
  type: object
  additionalProperties: true
  required:
    - header
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    header:
      type: string
    regex:
      type: string
```

Example:

```yaml
requester:
  <...>
  error_handler:
    <...>
    backoff_strategies:
        - type: "WaitTimeFromHeader"
          header: "wait_time"
```

Optionally, a regular expression can be configured to extract the wait time from the header value.

Example:

```yaml
requester:
  <...>
  error_handler:
    <...>
    backoff_strategies:
        - type: "WaitTimeFromHeader"
          header: "wait_time"
          regex: "[-+]?\d+"
```

### Wait until time defined in header

When using the `WaitUntilTimeFromHeader` backoff strategy, the requester will backoff until the time specified in the response header.
In this example, the requester will wait until the time specified in the "wait_until" header value:

Schema:

```yaml
WaitUntilTimeFromHeader:
  type: object
  additionalProperties: true
  required:
    - header
  properties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    header:
      type: string
    regex:
      type: string
    min_wait:
      type: number
```

Example:

```yaml
requester:
  <...>
  error_handler:
    <...>
    backoff_strategies:
        - type: "WaitUntilTimeFromHeader"
          header: "wait_until"
          regex: "[-+]?\d+"
          min_wait: 5
```

The strategy accepts an optional regular expression to extract the time from the header value, and a minimum time to wait.

## Advanced error handling

The error handler can have multiple backoff strategies, allowing it to fallback if a strategy cannot be evaluated.
For instance, the following defines an error handler that will read the backoff time from a header, and default to a constant backoff if the wait time could not be extracted from the response:

Example:

```yaml
requester:
  <...>
  error_handler:
    <...>
    backoff_strategies:
        - type: "WaitTimeFromHeader"
          header: "wait_time"
            - type: "ConstantBackoff"
              backoff_time_in_seconds: 5
```

The `requester` can be configured to use a `CompositeErrorHandler`, which sequentially iterates over a list of error handlers, enabling different retry mechanisms for different types of errors.

In this example, a constant backoff of 5 seconds, will be applied if the response contains a "code" field, and an exponential backoff will be applied if the error code is 403:

Schema:

```yaml
CompositeErrorHandler:
  type: object
  required:
    - error_handlers
  additionalProperties:
    "$parameters":
      "$ref": "#/definitions/$parameters"
    error_handlers:
      type: array
      items:
        "$ref": "#/definitions/ErrorHandler"
```

Example:

```yaml
requester:
  <...>
  error_handler:
    type: "CompositeErrorHandler"
    error_handlers:
      - response_filters:
          - predicate: "{{ 'code' in response }}"
            action: RETRY
        backoff_strategies:
          - type: "ConstantBackoffStrategy"
            backoff_time_in_seconds: 5
      - response_filters:
          - http_codes: [ 403 ]
            action: RETRY
        backoff_strategies:
          - type: "ExponentialBackoffStrategy"
```

## More readings

- [Requester](./requester.md)