* [ISSUE #19410] remove request_options_provider from the … (#21403) * [ISSUE #19410] (incomplete) remove request_options_provider from the manifest * [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well * [ISSUE #19410] update source-monday * [ISSUE #19410] code review * [ISSUE #19410] formatting files * [Low-Code CDK] Replace the $options keyword with $parameters (#21632) * refactor flows and tests to use parameters instead of options * update documentation to reflect the change from options to parameters * create migration script to replace options with parameters in existing manifests * update template to use parameters instead of options * fix tests after rebasing from the branch * address pr feedback and extra uses of options that I missed * additional changes needed after rebasing from master * migrate low-code connectors to use parameters instead of options * 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434) * [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832) * fully deprecate JsonSchema in favor of JsonFileSchemaLoader * remove usage in the legacy registry * Update migration scripts according to manifest file rename (#21920) * Issue 21866 remove legacy factory and validation flow (#21878) * [ISSUE #21866] clean ManifestDeclarativeSource validation * [ISSUE #21866] remove dataclasses-jsonschema * [ISSUE #21866] code review * [ISSUE-21866] flake8 * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] remove DefaultPaginator.url_base (#21823) * [ISSUE #21559] remove DefaultPaginator.url_base * [ISSUE #21559] code review * [ISSUE #21559] update migration script * [ISSUE #21559] code review * [ISSUE #21559] update documentation * [ISSUE #21559] run migration (#21824) * [ISSUE #21559] fix manifests * [ISSUE #21926] setup server to allow for local tests (#21974) * [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120) * Issue #21576 rename dpathextractor fieldpointer (#21990) * [ISSUE #21926] setup server to allow for local tests * [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path * [ISSUE #21576] migration script * [ISSUE #21576] update source-monday and source-pocket as well * [ISSUE #21576] migration (#21997) * [ISSUE #21576] code review * Remove checkpoint_interval from source-prestashop manifest (#22141) * replacing options with parameters for a few connectors I missed or were newly added * [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294) * update schema to derive cursor_field from a stream slicer if it exists * remove usage of stream_cursor_field on simple connector use cases * fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers * fix documentation to replace references to stream_cursor_field * Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891) * fix eslint issues for webapp (#22462) * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375) * bump connector builder server to latest CDK version * fix breaking CDK changes in connector builder FE * [Low-Code CDK] Separate request path from RequestOption component (#22398) * split apart path from RequestOption and fix usages and cleanup the code * replace usage of path with RequestPath and get rid of default to RequestOption * fix bug where stream_slice_field was used in outbound request instead of request_option field_name * organize yaml schema names and update documentation for RequestOption and RequestPath * clean up tests * regenerate models * [ISSUE #19961] refactor stream slices (#22225) * [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet * [ISSUE #19961] fixing Google PageSpeed Insights * move incremental_sync field to the stream level and perform merging into one stream slicer at that level * add tests to merging incremental and iterable into cartesian * rewrite documentation to separate incremental sync and iterator concepts * update documentation to use partition router and revise the tutorial to reflect the new changes to the components * [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670) * [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] clean faulty merge * [ISSUE #19961] rename DatetimeStreamSlicer (#22617) * [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well) * [ISSUE 19961] rename CustomStreamSlicer (#22598) * [ISSUE 19961] rename CustomStreamSlicer * [ISSUE #19961] code review CustomStreamSlicer * [ISSUE #19961] fix source_square incremental sync * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591) * [ISSUE #19961] rename SingleSlice to SinglePartitionRouter * remove SinglePartitionRouter from the schema --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * [ISSUE #19961] rename DatetimeStreamSlicer * [ISSUE #19961] rename SubstreamSlicer to SubstreamPartitionRouter (#22596) * [ISSUE #19961] TMP rename SubstreamSlicer to SubstreamPartitionRouter * [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end * [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593) --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> * update docs * [ISSUE #19961] clean unit tests files * [ISSUE #19961] code review --------- Co-authored-by: brianjlai <brian.lai@airbyte.io> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379) * Fix a bug where child components of a custom component cannot receive fields from other components * add tests, documentation and commenting * fix test from merge * add better error message for nested initialization failures * 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880) * restrict name to stream level * remove checkpoint interval * adjust logic for new request options * refactor slicers * wording * review comments * make oldest supported version explicit * separate the frontend and connector builder changes from the low-code to beta release * [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123) * consolidate all the changes into a new PR after I messed up the merge on the side branch * add set to allow this to be called externally if necessary later * remove last few extra fields i found and fix docs links * fix docs one more time --------- Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com> Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com> Co-authored-by: maxi297 <maxime@airbyte.io> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Joe Reuter <joe@airbyte.io>
1101 lines
31 KiB
YAML
1101 lines
31 KiB
YAML
"$schema": http://json-schema.org/draft-07/schema#
|
|
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml
|
|
title: DeclarativeSource
|
|
type: object
|
|
description: An API source that extracts data according to its declarative components
|
|
version: 1.0.0
|
|
required:
|
|
- type
|
|
- check
|
|
- streams
|
|
- version
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DeclarativeSource]
|
|
check:
|
|
"$ref": "#/definitions/CheckStream"
|
|
streams:
|
|
type: array
|
|
items:
|
|
"$ref": "#/definitions/DeclarativeStream"
|
|
version:
|
|
type: string
|
|
schemas:
|
|
"$ref": "#/definitions/Schemas"
|
|
definitions:
|
|
type: object
|
|
spec:
|
|
"$ref": "#/definitions/Spec"
|
|
additionalProperties: false
|
|
definitions:
|
|
AddedFieldDefinition:
|
|
description: Defines the field to add on a record
|
|
type: object
|
|
required:
|
|
- type
|
|
- path
|
|
- value
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [AddedFieldDefinition]
|
|
path:
|
|
type: array
|
|
items:
|
|
type: string
|
|
value:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
AddFields:
|
|
description: Transformation which adds field to an output record. The path of the added field can be nested.
|
|
type: object
|
|
required:
|
|
- type
|
|
- fields
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [AddFields]
|
|
fields:
|
|
type: array
|
|
items:
|
|
- "$ref": "#/definitions/AddedFieldDefinition"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
ApiKeyAuthenticator:
|
|
description: Authenticator for requests requiring an api token
|
|
type: object
|
|
required:
|
|
- type
|
|
- api_token
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [ApiKeyAuthenticator]
|
|
api_token:
|
|
type: string
|
|
header:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
BasicHttpAuthenticator:
|
|
description: Authenticator for requests authenticated with a username and optional password
|
|
type: object
|
|
required:
|
|
- type
|
|
- username
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [BasicHttpAuthenticator]
|
|
username:
|
|
description: The username that will be combined with the password, base64 encoded and used to make requests
|
|
type: string
|
|
password:
|
|
description: The password that will be combined with the username, base64 encoded and used to make requests
|
|
type: string
|
|
default: ""
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
BearerAuthenticator:
|
|
description: Authenticator for requests authenticated with a Bearer token
|
|
type: object
|
|
required:
|
|
- type
|
|
- api_token
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [BearerAuthenticator]
|
|
api_token:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CheckStream:
|
|
description: Checks the connections by trying to read records from one or many of the streams selected by the developer
|
|
type: object
|
|
required:
|
|
- type
|
|
- stream_names
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CheckStream]
|
|
stream_names:
|
|
type: array
|
|
items:
|
|
type: string
|
|
CompositeErrorHandler:
|
|
description: Error handler that sequentially iterates over a list of error handlers
|
|
type: object
|
|
required:
|
|
- type
|
|
- error_handlers
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CompositeErrorHandler]
|
|
error_handlers:
|
|
type: array
|
|
items:
|
|
anyOf:
|
|
- "$ref": "#/definitions/CompositeErrorHandler"
|
|
- "$ref": "#/definitions/DefaultErrorHandler"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
ConstantBackoffStrategy:
|
|
description: Backoff strategy with a constant backoff interval
|
|
type: object
|
|
required:
|
|
- type
|
|
- backoff_time_in_seconds
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [ConstantBackoffStrategy]
|
|
backoff_time_in_seconds:
|
|
anyOf:
|
|
- type: number
|
|
- type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CursorPagination:
|
|
description: Pagination strategy that evaluates an interpolated string to define the next page token
|
|
type: object
|
|
required:
|
|
- type
|
|
- cursor_value
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CursorPagination]
|
|
cursor_value:
|
|
type: string
|
|
page_size:
|
|
type: integer
|
|
stop_condition:
|
|
type: string
|
|
decoder:
|
|
"$ref": "#/definitions/JsonDecoder"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomAuthenticator:
|
|
description: Authenticator component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomAuthenticator]
|
|
class_name:
|
|
type: string
|
|
additionalProperties: true
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomBackoffStrategy:
|
|
description: Backoff strategy component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomBackoffStrategy]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomErrorHandler:
|
|
description: Error handler component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomErrorHandler]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomIncrementalSync:
|
|
description: Incremental component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
- cursor_field
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomIncrementalSync]
|
|
class_name:
|
|
description: The class that will be implementing the custom incremental sync. The format is `source_<name>.<package>.<class_name>`
|
|
type: string
|
|
additionalProperties: true
|
|
cursor_field:
|
|
description: The location of the value on a record that will be used as a bookmark during sync
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomPaginationStrategy:
|
|
description: Pagination strategy component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomPaginationStrategy]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomRecordExtractor:
|
|
description: Record extractor component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomRecordExtractor]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomRequester:
|
|
description: Requester component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomRequester]
|
|
class_name:
|
|
type: string
|
|
additionalProperties: true
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomRetriever:
|
|
description: Retriever component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomRetriever]
|
|
class_name:
|
|
type: string
|
|
additionalProperties: true
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomPartitionRouter:
|
|
description: Partition router component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomPartitionRouter]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
CustomTransformation:
|
|
description: Transformation component whose behavior is derived from a custom code implementation of the connector
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- class_name
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [CustomTransformation]
|
|
class_name:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
DatetimeBasedCursor:
|
|
description: Cursor to provide incremental capabilities over datetime
|
|
type: object
|
|
required:
|
|
- type
|
|
- cursor_field
|
|
- end_datetime
|
|
- datetime_format
|
|
- cursor_granularity
|
|
- start_datetime
|
|
- step
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DatetimeBasedCursor]
|
|
cursor_field:
|
|
description: The location of the value on a record that will be used as a bookmark during sync
|
|
type: string
|
|
datetime_format:
|
|
description: The format of the datetime
|
|
type: string
|
|
cursor_granularity:
|
|
description: Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one
|
|
type: string
|
|
end_datetime:
|
|
description: The datetime that determines the last record that should be synced
|
|
anyOf:
|
|
- type: string
|
|
- "$ref": "#/definitions/MinMaxDatetime"
|
|
start_datetime:
|
|
description: The datetime that determines the earliest record that should be synced
|
|
anyOf:
|
|
- type: string
|
|
- "$ref": "#/definitions/MinMaxDatetime"
|
|
step:
|
|
description: The size of the time window (ISO8601 duration)
|
|
type: string
|
|
end_time_option:
|
|
description: Request option for end time
|
|
"$ref": "#/definitions/RequestOption"
|
|
lookback_window:
|
|
description: How many days before start_datetime to read data for (ISO8601 duration)
|
|
type: string
|
|
partition_field_end:
|
|
description: Partition start time field
|
|
type: string
|
|
partition_field_start:
|
|
description: Partition end time field
|
|
type: string
|
|
start_time_option:
|
|
description: Request option for start time
|
|
"$ref": "#/definitions/RequestOption"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
OAuthAuthenticator:
|
|
description: Authenticator for requests using OAuth 2.0 authentication
|
|
type: object
|
|
required:
|
|
- type
|
|
- client_id
|
|
- client_secret
|
|
- refresh_token
|
|
- token_refresh_endpoint
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [OAuthAuthenticator]
|
|
client_id:
|
|
type: string
|
|
client_secret:
|
|
type: string
|
|
refresh_token:
|
|
type: string
|
|
token_refresh_endpoint:
|
|
type: string
|
|
access_token_name:
|
|
type: string
|
|
default: "access_token"
|
|
expires_in_name:
|
|
type: string
|
|
default: "expires_in"
|
|
grant_type:
|
|
type: string
|
|
default: "refresh_token"
|
|
refresh_request_body:
|
|
type: object
|
|
additionalProperties: true
|
|
scopes:
|
|
type: array
|
|
items:
|
|
type: string
|
|
token_expiry_date:
|
|
type: string
|
|
token_expiry_date_format:
|
|
description: The format of the datetime; provide it if expires_in is returned in datetime instead of seconds
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
DeclarativeStream:
|
|
description: A stream whose behavior is described by a set of declarative low code components
|
|
type: object
|
|
additionalProperties: true
|
|
required:
|
|
- type
|
|
- retriever
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DeclarativeStream]
|
|
retriever:
|
|
definition: Component used to coordinate how records are extracted across stream slices and request pages
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomRetriever"
|
|
- "$ref": "#/definitions/SimpleRetriever"
|
|
incremental_sync:
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomIncrementalSync"
|
|
- "$ref": "#/definitions/DatetimeBasedCursor"
|
|
name:
|
|
definition: The stream name
|
|
type: string
|
|
default: ""
|
|
primary_key:
|
|
definition: The primary key of the stream
|
|
"$ref": "#/definitions/PrimaryKey"
|
|
default: ""
|
|
schema_loader:
|
|
definition: The schema loader used to retrieve the schema for the current stream
|
|
anyOf:
|
|
- "$ref": "#/definitions/InlineSchemaLoader"
|
|
- "$ref": "#/definitions/JsonFileSchemaLoader"
|
|
transformations:
|
|
definition: A list of transformations to be applied to each output record in the
|
|
type: array
|
|
items:
|
|
anyOf:
|
|
- "$ref": "#/definitions/AddFields"
|
|
- "$ref": "#/definitions/CustomTransformation"
|
|
- "$ref": "#/definitions/RemoveFields"
|
|
$parameters:
|
|
type: object
|
|
additional_properties: true
|
|
DefaultErrorHandler:
|
|
description: The default error handler. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DefaultErrorHandler]
|
|
backoff_strategies:
|
|
type: array
|
|
items:
|
|
anyOf:
|
|
- "$ref": "#/definitions/ConstantBackoffStrategy"
|
|
- "$ref": "#/definitions/CustomBackoffStrategy"
|
|
- "$ref": "#/definitions/ExponentialBackoffStrategy"
|
|
- "$ref": "#/definitions/WaitTimeFromHeader"
|
|
- "$ref": "#/definitions/WaitUntilTimeFromHeader"
|
|
max_retries:
|
|
type: integer
|
|
default: 5
|
|
response_filters:
|
|
type: array
|
|
items:
|
|
"$ref": "#/definitions/HttpResponseFilter"
|
|
$parameters:
|
|
type: object
|
|
additional_properties: true
|
|
DefaultPaginator:
|
|
description: Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token
|
|
type: object
|
|
required:
|
|
- type
|
|
- pagination_strategy
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DefaultPaginator]
|
|
pagination_strategy:
|
|
anyOf:
|
|
- "$ref": "#/definitions/CursorPagination"
|
|
- "$ref": "#/definitions/CustomPaginationStrategy"
|
|
- "$ref": "#/definitions/OffsetIncrement"
|
|
- "$ref": "#/definitions/PageIncrement"
|
|
decoder:
|
|
"$ref": "#/definitions/JsonDecoder"
|
|
page_size_option:
|
|
"$ref": "#/definitions/RequestOption"
|
|
page_token_option:
|
|
anyOf:
|
|
- "$ref": "#/definitions/RequestOption"
|
|
- "$ref": "#/definitions/RequestPath"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
DpathExtractor:
|
|
description: Record extractor that searches a decoded response over a path defined as an array of fields
|
|
type: object
|
|
required:
|
|
- type
|
|
- field_path
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [DpathExtractor]
|
|
field_path:
|
|
type: array
|
|
items:
|
|
- type: string
|
|
decoder:
|
|
"$ref": "#/definitions/JsonDecoder"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
ExponentialBackoffStrategy:
|
|
description: Backoff strategy with an exponential backoff interval
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [ExponentialBackoffStrategy]
|
|
factor:
|
|
anyOf:
|
|
- type: number
|
|
- type: string
|
|
default: 5
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
HttpRequester:
|
|
description: Default implementation of a requester
|
|
type: object
|
|
required:
|
|
- type
|
|
- path
|
|
- url_base
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [HttpRequester]
|
|
path:
|
|
description: The specific endpoint to fetch data from for a resource.
|
|
type: string
|
|
url_base:
|
|
description: The root of the API source.
|
|
type: string
|
|
authenticator:
|
|
description: Authenticator component that defines how to authenticate to the source.
|
|
anyOf:
|
|
- "$ref": "#/definitions/ApiKeyAuthenticator"
|
|
- "$ref": "#/definitions/BasicHttpAuthenticator"
|
|
- "$ref": "#/definitions/BearerAuthenticator"
|
|
- "$ref": "#/definitions/CustomAuthenticator"
|
|
- "$ref": "#/definitions/OAuthAuthenticator"
|
|
- "$ref": "#/definitions/NoAuth"
|
|
- "$ref": "#/definitions/SessionTokenAuthenticator"
|
|
error_handler:
|
|
description: Error handler component that defines how to handle errors.
|
|
anyOf:
|
|
- "$ref": "#/definitions/DefaultErrorHandler"
|
|
- "$ref": "#/definitions/CustomErrorHandler"
|
|
- "$ref": "#/definitions/CompositeErrorHandler"
|
|
http_method:
|
|
description: The HTTP method used to fetch data from the source (can be GET or POST).
|
|
anyOf:
|
|
- type: string
|
|
- type: string
|
|
enum:
|
|
- GET
|
|
- POST
|
|
default: GET
|
|
request_body_data:
|
|
description: Specifies how to populate the body of the request with a non-JSON payload. If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form.
|
|
anyOf:
|
|
- type: string
|
|
- type: object
|
|
additionalProperties:
|
|
type: string
|
|
request_body_json:
|
|
description: Specifies how to populate the body of the request with a JSON payload.
|
|
anyOf:
|
|
- type: string
|
|
- type: object
|
|
additionalProperties:
|
|
type: string
|
|
request_headers:
|
|
description: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
|
|
anyOf:
|
|
- type: string
|
|
- type: object
|
|
additionalProperties:
|
|
type: string
|
|
request_parameters:
|
|
description: Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
|
|
anyOf:
|
|
- type: string
|
|
- type: object
|
|
additionalProperties:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
HttpResponseFilter:
|
|
description: A filter that is used to select on properties of the HTTP response received
|
|
type: object
|
|
required:
|
|
- type
|
|
- action
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [HttpResponseFilter]
|
|
action:
|
|
type: string
|
|
enum:
|
|
- SUCCESS
|
|
- FAIL
|
|
- RETRY
|
|
- IGNORE
|
|
error_message:
|
|
type: string
|
|
error_message_contains:
|
|
type: string
|
|
http_codes:
|
|
type: array
|
|
items:
|
|
type: integer
|
|
uniqueItems: true
|
|
predicate:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
InlineSchemaLoader:
|
|
description: Loads a schema that is defined directly in the manifest file
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [InlineSchemaLoader]
|
|
schema:
|
|
type: object
|
|
JsonFileSchemaLoader:
|
|
description: Loads the schema from a json file
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [JsonFileSchemaLoader]
|
|
file_path:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
JsonDecoder:
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [JsonDecoder]
|
|
ListPartitionRouter:
|
|
description: Partition router that is used to retrieve records that have been partitioned according to a list of values
|
|
type: object
|
|
required:
|
|
- type
|
|
- cursor_field
|
|
- values
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [ListPartitionRouter]
|
|
cursor_field:
|
|
type: string
|
|
values:
|
|
anyOf:
|
|
- type: string
|
|
- type: array
|
|
items:
|
|
type: string
|
|
request_option:
|
|
"$ref": "#/definitions/RequestOption"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
MinMaxDatetime:
|
|
description: Compares the provided date against optional minimum or maximum times. The max_datetime serves as the ceiling and will be returned when datetime exceeds it. The min_datetime serves as the floor
|
|
type: object
|
|
required:
|
|
- type
|
|
- datetime
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [MinMaxDatetime]
|
|
datetime:
|
|
type: string
|
|
datetime_format:
|
|
type: string
|
|
default: ""
|
|
max_datetime:
|
|
type: string
|
|
min_datetime:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
NoAuth:
|
|
description: Authenticator for requests requiring no authentication
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [NoAuth]
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
NoPagination:
|
|
description: Pagination implementation that never returns a next page
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [NoPagination]
|
|
OffsetIncrement:
|
|
description: Pagination strategy that returns the number of records reads so far and returns it as the next page token
|
|
type: object
|
|
required:
|
|
- type
|
|
- page_size
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [OffsetIncrement]
|
|
page_size:
|
|
description: The number of records to request
|
|
anyOf:
|
|
- type: integer
|
|
- type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
PageIncrement:
|
|
description: Pagination strategy that returns the number of pages reads so far and returns it as the next page token
|
|
type: object
|
|
required:
|
|
- type
|
|
- page_size
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [PageIncrement]
|
|
page_size:
|
|
description: The number of records to request
|
|
type: integer
|
|
start_from_page:
|
|
type: integer
|
|
default: 0
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
ParentStreamConfig:
|
|
description: Describes how to create a stream slice from a parent stream
|
|
type: object
|
|
required:
|
|
- type
|
|
- parent_key
|
|
- partition_field
|
|
- stream
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [ParentStreamConfig]
|
|
parent_key:
|
|
type: string
|
|
stream:
|
|
"$ref": "#/definitions/DeclarativeStream"
|
|
partition_field:
|
|
type: string
|
|
request_option:
|
|
"$ref": "#/definitions/RequestOption"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
PrimaryKey:
|
|
description: The stream field to be used to distinguish unique rows
|
|
anyOf:
|
|
- type: string
|
|
- type: array
|
|
items:
|
|
type: string
|
|
- type: array
|
|
items:
|
|
type: array
|
|
items:
|
|
type: string
|
|
default: ""
|
|
RecordFilter:
|
|
description: Filter applied on a list of Records
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RecordFilter]
|
|
condition:
|
|
description: The predicate to filter a record. Records will be removed if evaluated to False
|
|
type: string
|
|
default: ""
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
RecordSelector:
|
|
description: Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
|
|
type: object
|
|
required:
|
|
- type
|
|
- extractor
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RecordSelector]
|
|
extractor:
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomRecordExtractor"
|
|
- "$ref": "#/definitions/DpathExtractor"
|
|
record_filter:
|
|
"$ref": "#/definitions/RecordFilter"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
RemoveFields:
|
|
description: A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.
|
|
type: object
|
|
required:
|
|
- type
|
|
- field_pointers
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RemoveFields]
|
|
field_pointers:
|
|
type: array
|
|
items:
|
|
items:
|
|
type: string
|
|
RequestPath:
|
|
description: A component that specifies where in the request path a component's value should be inserted into.
|
|
type: object
|
|
required:
|
|
- type
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RequestPath]
|
|
RequestOption:
|
|
description: A component that specifies the key field and where in the request a component's value should be inserted into.
|
|
type: object
|
|
required:
|
|
- type
|
|
- field_name
|
|
- inject_into
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [RequestOption]
|
|
field_name:
|
|
type: string
|
|
inject_into:
|
|
enum:
|
|
- request_parameter
|
|
- header
|
|
- body_data
|
|
- body_json
|
|
Schemas:
|
|
description: The stream schemas representing the shape of the data emitted by the stream
|
|
type: object
|
|
additionalProperties: true
|
|
SessionTokenAuthenticator:
|
|
description: Authenticator for requests based on session tokens. A session token is a random value generated by a server to identify a specific user for the duration of one interaction session
|
|
type: object
|
|
required:
|
|
- type
|
|
- api_url
|
|
- header
|
|
- login_url
|
|
- session_token
|
|
- session_token_response_key
|
|
- username
|
|
- validate_session_url
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [SessionTokenAuthenticator]
|
|
api_url:
|
|
type: string
|
|
header:
|
|
type: string
|
|
login_url:
|
|
type: string
|
|
session_token:
|
|
type: string
|
|
session_token_response_key:
|
|
type: string
|
|
username:
|
|
type: string
|
|
validate_session_url:
|
|
type: string
|
|
password:
|
|
type: string
|
|
default: ""
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
SimpleRetriever:
|
|
description: Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.
|
|
type: object
|
|
required:
|
|
- type
|
|
- record_selector
|
|
- requester
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [SimpleRetriever]
|
|
record_selector:
|
|
description: Component that describes how to extract records from a HTTP response.
|
|
"$ref": "#/definitions/RecordSelector"
|
|
requester:
|
|
description: Requester component that describes how to prepare HTTP requests to send to the source API.
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomRequester"
|
|
- "$ref": "#/definitions/HttpRequester"
|
|
paginator:
|
|
description: Paginator component that describes how to navigate through the API's pages.
|
|
anyOf:
|
|
- "$ref": "#/definitions/DefaultPaginator"
|
|
- "$ref": "#/definitions/NoPagination"
|
|
partition_router:
|
|
default: []
|
|
description: StreamSlicer component that describes how to partition the stream, enabling incremental syncs and checkpointing.
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomPartitionRouter"
|
|
- "$ref": "#/definitions/ListPartitionRouter"
|
|
- "$ref": "#/definitions/SubstreamPartitionRouter"
|
|
- type: array
|
|
items:
|
|
anyOf:
|
|
- "$ref": "#/definitions/CustomPartitionRouter"
|
|
- "$ref": "#/definitions/ListPartitionRouter"
|
|
- "$ref": "#/definitions/SubstreamPartitionRouter"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
Spec:
|
|
description: A connection specification made up of information about the connector and how it can be configured
|
|
type: object
|
|
required:
|
|
- type
|
|
- connection_specification
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [Spec]
|
|
connection_specification:
|
|
type: object
|
|
additionalProperties: true
|
|
documentation_url:
|
|
type: string
|
|
SubstreamPartitionRouter:
|
|
description: Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams
|
|
type: object
|
|
required:
|
|
- type
|
|
- parent_stream_configs
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [SubstreamPartitionRouter]
|
|
parent_stream_configs:
|
|
type: array
|
|
items:
|
|
"$ref": "#/definitions/ParentStreamConfig"
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
WaitTimeFromHeader:
|
|
description: Extract wait time from http header
|
|
type: object
|
|
required:
|
|
- type
|
|
- header
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [WaitTimeFromHeader]
|
|
header:
|
|
type: string
|
|
regex:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|
|
WaitUntilTimeFromHeader:
|
|
description: Extract time at which we can retry the request from response header and wait for the difference between now and that time
|
|
type: object
|
|
required:
|
|
- type
|
|
- header
|
|
properties:
|
|
type:
|
|
type: string
|
|
enum: [WaitUntilTimeFromHeader]
|
|
header:
|
|
type: string
|
|
min_wait:
|
|
anyOf:
|
|
- type: number
|
|
- type: string
|
|
regex:
|
|
type: string
|
|
$parameters:
|
|
type: object
|
|
additionalProperties: true
|