288c3cabad
* 5-step tutorial * move * tiny bit of editing * Update tutorial * update docs * reset * move files * record selector, request options, and more links * update * update * connector definition * link * links * update example * footnote * typo * document string interpolation * note on string interpolation * update * fix code sample * fix * update sample * fix * use the actual config * Update as per comments * write as yaml * typo * Clarify options overloading * clarify that docker must be running * remove extra footnote * use venv directly * Apply suggestions from code review Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * signup instructions * update * clarify that both dot and bracket notations are interchangeable * Clarify how check works * create spec and config before updating connector definition * clarify what now_local() is * rename to yaml structure * Go through tutorial and update end of section code samples * fix link * update * update code samples * Update code samples * Update to bracket notation * remove superfluous comments * Update docs/connector-development/config-based/tutorial/2-install-dependencies.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/4-reading-data.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * fix path * update * motivation blurp * warning * warning * fix code block * update code samples * update code sample * update code samples * small updates * update yaml structure * custom class example * language annotations * update warning * Update tutorial to use dpath extractor * Update record selector docs * unit test * link to contributing * tiny update * $ in front of commands * $ in front of commands * More readings * link to existing config-based connectors * index * update * delete broken link * supported features * update * Add some links * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/record-selector.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * mention the unit * headers * remove mentions of interpolating on stream slice, etc. * update * exclude config-based docs Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
2.7 KiB
Request Options
There are a few ways to set request parameters, headers, and body on ongoing HTTP requests.
Request Options Provider
The primary way to set request options is through the Requester's RequestOptionsProvider.
The options can be configured as key value pairs:
requester:
type: HttpRequester
name: "{{ options['name'] }}"
url_base: "https://api.exchangeratesapi.io/v1/"
http_method: "GET"
request_options_provider:
request_parameters:
k1: v1
k2: v2
request_headers:
header_key1: header_value1
header_key2: header_value2
It is also possible to configure add a json-encoded body to outgoing requests.
requester:
type: HttpRequester
name: "{{ options['name'] }}"
url_base: "https://api.exchangeratesapi.io/v1/"
http_method: "GET"
request_options_provider:
request_body_json:
key: value
Authenticators
It is also possible for authenticators to set request parameters or headers as needed.
For instance, the BearerAuthenticator will always set the Authorization header.
More details on the various authenticators can be found in the authentication section
Paginators
The LimitPaginator can optionally set request options through the limit_option and the page_token_option.
The respective values can be set on the outgoing HTTP requests by specifying where it should be injected.
The following example will set the "page" request parameter value to the page to fetch, and the "page_size" request parameter to 5:
paginator:
type: "LimitPaginator"
page_size: 5
limit_option:
option_type: request_parameter
field_name: page_size
pagination_strategy:
type: "PageIncrement"
page_token:
option_type: "request_parameter"
field_name: "page"
More details on paginators can be found in the pagination section
Stream slicers
The DatetimeStreamSlicer can optionally set request options through the start_time_option and end_time_option fields.
The respective values can be set on the outgoing HTTP requests by specifying where it should be injected.
The following example will set the "created[gte]" request parameter value to the start of the time window, and "created[lte]" to the end of the time window.
stream_slicer:
start_datetime: "2021-02-01T00:00:00.000000+0000",
end_datetime: "2021-03-01T00:00:00.000000+0000",
step: "1d"
start_time_option:
field_name: "created[gte]"
inject_into: "request_parameter"
end_time_option:
field_name: "created[lte]"
inject_into: "request_parameter"
More details on the stream slicers can be found in the stream-slicers section