* [ISSUE #19410] remove request_options_provider from the … (#21403) * [ISSUE #19410] (incomplete) remove request_options_provider from the manifest * [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well * [ISSUE #19410] update source-monday * [ISSUE #19410] code review * [ISSUE #19410] formatting files * [Low-Code CDK] Replace the $options keyword with $parameters (#21632) * refactor flows and tests to use parameters instead of options * update documentation to reflect the change from options to parameters * create migration script to replace options with parameters in existing manifests * update template to use parameters instead of options * fix tests after rebasing from the branch * address pr feedback and extra uses of options that I missed * additional changes needed after rebasing from master * migrate low-code connectors to use parameters instead of options * 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434) * [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832) * fully deprecate JsonSchema in favor of JsonFileSchemaLoader * remove usage in the legacy registry * Update migration scripts according to manifest file rename (#21920) * Issue 21866 remove legacy factory and validation flow (#21878) * [ISSUE #21866] clean ManifestDeclarativeSource validation * [ISSUE #21866] remove dataclasses-jsonschema * [ISSUE #21866] code review * [ISSUE-21866] flake8 * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] fix manifests * [ISSUE #21926] setup server to allow for local tests (#21974) * [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120) * Issue #21576 rename dpathextractor fieldpointer (#21990) * [ISSUE #21926] setup server to allow for local tests * [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path * [ISSUE #21576] migration script * [ISSUE #21576] update source-monday and source-pocket as well * [ISSUE #21576] migration (#21997) * [ISSUE #21576] code review * Remove checkpoint_interval from source-prestashop manifest (#22141) * replacing options with parameters for a few connectors I missed or were newly added * [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294) * update schema to derive cursor_field from a stream slicer if it exists * remove usage of stream_cursor_field on simple connector use cases * fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers * fix documentation to replace references to stream_cursor_field * Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891) * fix eslint issues for webapp (#22462) * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375) * bump connector builder server to latest CDK version * fix breaking CDK changes in connector builder FE * [Low-Code CDK] Separate request path from RequestOption component (#22398) * split apart path from RequestOption and fix usages and cleanup the code * replace usage of path with RequestPath and get rid of default to RequestOption * fix bug where stream_slice_field was used in outbound request instead of request_option field_name * organize yaml schema names and update documentation for RequestOption and RequestPath * clean up tests * regenerate models * [ISSUE #19961] refactor stream slices (#22225) * [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet * [ISSUE #19961] fixing Google PageSpeed Insights * move incremental_sync field to the stream level and perform merging into one stream slicer at that level * add tests to merging incremental and iterable into cartesian * rewrite documentation to separate incremental sync and iterator concepts * update documentation to use partition router and revise the tutorial to reflect the new changes to the components * [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670) * [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] clean faulty merge * [ISSUE #19961] rename DatetimeStreamSlicer (#22617) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename DatetimeStreamSlicer * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * update docs * [ISSUE #19961] clean unit tests files * [ISSUE #19961] code review --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379) * Fix a bug where child components of a custom component cannot receive fields from other components * add tests, documentation and commenting * fix test from merge * add better error message for nested initialization failures * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880) * restrict name to stream level * remove checkpoint interval * adjust logic for new request options * refactor slicers * wording * review comments * make oldest supported version explicit * separate the frontend and connector builder changes from the low-code to beta release * [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123) * consolidate all the changes into a new PR after I messed up the merge on the side branch * add set to allow this to be called externally if necessary later * remove last few extra fields i found and fix docs links * fix docs one more time --------- Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> Co-authored-by: maxi297 <maxime@airbyte.io> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Joe Reuter <joe@airbyte.io>
172 lines
4.9 KiB
Markdown
172 lines
4.9 KiB
Markdown
# Request Options
|
|
|
|
The primary way to set request parameters and headers is to define them as key-value pairs using a `RequestOptionsProvider`.
|
|
Other components, such as an `Authenticator` can also set additional request params or headers as needed.
|
|
|
|
Additionally, some stateful components use a `RequestOption` to configure the options and update the value. Example of such components are [Paginators](./pagination.md) and [DatetimeBasedCursors](./incremental-syncs.md#DatetimeBasedCursor).
|
|
|
|
## Request Options Provider
|
|
|
|
The primary way to set request options is through the `Requester`'s `RequestOptionsProvider`.
|
|
The options can be configured as key value pairs:
|
|
|
|
Schema:
|
|
|
|
```yaml
|
|
RequestOptionsProvider:
|
|
type: object
|
|
anyOf:
|
|
- "$ref": "#/definitions/InterpolatedRequestOptionsProvider"
|
|
InterpolatedRequestOptionsProvider:
|
|
type: object
|
|
additionalProperties: true
|
|
properties:
|
|
"$parameters":
|
|
"$ref": "#/definitions/$parameters"
|
|
request_parameters:
|
|
"$ref": "#/definitions/RequestInput"
|
|
request_headers:
|
|
"$ref": "#/definitions/RequestInput"
|
|
request_body_data:
|
|
"$ref": "#/definitions/RequestInput"
|
|
request_body_json:
|
|
"$ref": "#/definitions/RequestInput"
|
|
```
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
requester:
|
|
type: HttpRequester
|
|
url_base: "https://api.exchangeratesapi.io/v1/"
|
|
http_method: "GET"
|
|
request_options_provider:
|
|
request_parameters:
|
|
k1: v1
|
|
k2: v2
|
|
request_headers:
|
|
header_key1: header_value1
|
|
header_key2: header_value2
|
|
```
|
|
|
|
It is also possible to configure add a json-encoded body to outgoing requests.
|
|
|
|
```yaml
|
|
requester:
|
|
type: HttpRequester
|
|
url_base: "https://api.exchangeratesapi.io/v1/"
|
|
http_method: "GET"
|
|
request_options_provider:
|
|
request_body_json:
|
|
key: value
|
|
```
|
|
|
|
### Request Options
|
|
|
|
Some components can add request options to the requests sent to the API endpoint.
|
|
|
|
Schema:
|
|
|
|
```yaml
|
|
RequestOption:
|
|
description: A component that specifies the key field and where in the request a component's value should be inserted into.
|
|
type: object
|
|
required:
|
|
- type
|
|
- field_name
|
|
- inject_into
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RequestOption]
|
|
field_name:
|
|
type: string
|
|
inject_into:
|
|
enum:
|
|
- request_parameter
|
|
- header
|
|
- body_data
|
|
- body_json
|
|
```
|
|
|
|
### Request Path
|
|
|
|
As an alternative to adding various options to the request being sent, some components can be configured to
|
|
modify the HTTP path of the API endpoint being accessed.
|
|
|
|
Schema:
|
|
|
|
```yaml
|
|
RequestPath:
|
|
description: A component that specifies where in the request path a component's value should be inserted into.
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RequestPath]
|
|
```
|
|
|
|
## Authenticators
|
|
|
|
It is also possible for authenticators to set request parameters or headers as needed.
|
|
For instance, the `BearerAuthenticator` will always set the `Authorization` header.
|
|
|
|
More details on the various authenticators can be found in the [authentication section](authentication.md).
|
|
|
|
## Paginators
|
|
|
|
The `DefaultPaginator` can optionally set request options through the `page_size_option` and the `page_token_option`.
|
|
The respective values can be set on the outgoing HTTP requests by specifying where it should be injected.
|
|
|
|
The following example will set the "page" request parameter value to the page to fetch, and the "page_size" request parameter to 5:
|
|
|
|
```yaml
|
|
paginator:
|
|
type: "DefaultPaginator"
|
|
page_size_option:
|
|
type: "RequestOption"
|
|
inject_into: request_parameter
|
|
field_name: page_size
|
|
pagination_strategy:
|
|
type: "PageIncrement"
|
|
page_size: 5
|
|
page_token:
|
|
type: "RequestOption"
|
|
inject_into: "request_parameter"
|
|
field_name: "page"
|
|
```
|
|
|
|
More details on paginators can be found in the [pagination section](./pagination.md).
|
|
|
|
## Incremental syncs
|
|
|
|
The `DatetimeBasedCursor` can optionally set request options through the `start_time_option` and `end_time_option` fields.
|
|
The respective values can be set on the outgoing HTTP requests by specifying where it should be injected.
|
|
|
|
The following example will set the "created[gte]" request parameter value to the start of the time window, and "created[lte]" to the end of the time window.
|
|
|
|
```yaml
|
|
incremental_sync:
|
|
type: DatetimeBasedCursor
|
|
start_datetime: "2021-02-01T00:00:00.000000+0000",
|
|
end_datetime: "2021-03-01T00:00:00.000000+0000",
|
|
step: "P1D"
|
|
start_time_option:
|
|
type: "RequestOption"
|
|
field_name: "created[gte]"
|
|
inject_into: "request_parameter"
|
|
end_time_option:
|
|
type: "RequestOption"
|
|
field_name: "created[lte]"
|
|
inject_into: "request_parameter"
|
|
```
|
|
|
|
More details on incremental syncs can be found in the [incremental syncs section](./incremental-syncs.md).
|
|
|
|
## More readings
|
|
|
|
- [Requester](./requester.md)
|
|
- [Pagination](./pagination.md)
|
|
- [Incremental Syncs](./incremental-syncs.md) |