1
0
mirror of synced 2026-01-23 22:01:53 -05:00
Files
airbyte/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml
Brian Lai 1c482ee339 Low code cdk to beta (#22853)
* [ISSUE #19410] remove request_options_provider from the … (#21403)

* [ISSUE #19410] (incomplete) remove request_options_provider from the manifest

* [ISSUE #19410] (incomplete) incomplete cleanup config_component_schema.json as well

* [ISSUE #19410] update source-monday

* [ISSUE #19410] code review

* [ISSUE #19410] formatting files

* [Low-Code CDK] Replace the $options keyword with $parameters (#21632)

* refactor flows and tests to use parameters instead of options

* update documentation to reflect the change from options to parameters

* create migration script to replace options with parameters in existing manifests

* update template to use parameters instead of options

* fix tests after rebasing from the branch

* address pr feedback and extra uses of options that I missed

* additional changes needed after rebasing from master

* migrate low-code connectors to use parameters instead of options

* 🚨🚨 [Low Code CDK] Update `*ref` format to `#/` (#21434)

* [Low-Code CDK] Remove JsonSchema type in favor of JsonSchemaFileLoader (#21832)

* fully deprecate JsonSchema in favor of JsonFileSchemaLoader

* remove usage in the legacy registry

* Update migration scripts according to manifest file rename (#21920)

* Issue 21866 remove legacy factory and validation flow (#21878)

* [ISSUE #21866] clean ManifestDeclarativeSource validation

* [ISSUE #21866] remove dataclasses-jsonschema

* [ISSUE #21866] code review

* [ISSUE-21866] flake8

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] remove DefaultPaginator.url_base (#21823)

* [ISSUE #21559] remove DefaultPaginator.url_base

* [ISSUE #21559] code review

* [ISSUE #21559] update migration script

* [ISSUE #21559] code review

* [ISSUE #21559] update documentation

* [ISSUE #21559] run migration (#21824)

* [ISSUE #21559] fix manifests

* [ISSUE #21926] setup server to allow for local tests (#21974)

* [Low Code CDK] remove checkpoint_interval from DeclarativeStream component (#22120)

* Issue #21576 rename dpathextractor fieldpointer (#21990)

* [ISSUE #21926] setup server to allow for local tests

* [ISSUE #21576] Rename DpathExtractor.field_pointer to field_path

* [ISSUE #21576] migration script

* [ISSUE #21576] update source-monday and source-pocket as well

* [ISSUE #21576] migration (#21997)

* [ISSUE #21576] code review

* Remove checkpoint_interval from source-prestashop manifest (#22141)

* replacing options with parameters for a few connectors I missed or were newly added

* [Low-Code CDK] Rremove stream_cursor_field from stream and derive it from stream_slicer (#22294)

* update schema to derive cursor_field from a stream slicer if it exists

* remove usage of stream_cursor_field on simple connector use cases

* fixing some of the more complex usage of stream_cursor_field that rely on cartesian product stream slicers

* fix documentation to replace references to stream_cursor_field

* Low Code CDK: Remove `name` and `primary_key` from non-DeclarativeStream components (#21891)

* fix eslint issues for webapp (#22462)

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22375)

* bump connector builder server to latest CDK version

* fix breaking CDK changes in connector builder FE

* [Low-Code CDK] Separate request path from RequestOption component (#22398)

* split apart path from RequestOption and fix usages and cleanup the code

* replace usage of path with RequestPath and get rid of default to RequestOption

* fix bug where stream_slice_field was used in outbound request instead of request_option field_name

* organize yaml schema names and update documentation for RequestOption and RequestPath

* clean up tests

* regenerate models

* [ISSUE #19961] refactor stream slices (#22225)

* [ISSUE #19961] add 'incremental' and partially remove CartesianProductStreamSlicer - Google PageSpeed Insights not working yet

* [ISSUE #19961] fixing Google PageSpeed Insights

* move incremental_sync field to the stream level and perform merging into one stream slicer at that level

* add tests to merging incremental and iterable into cartesian

* rewrite documentation to separate incremental sync and iterator concepts

* update documentation to use partition router and revise the tutorial to reflect the new changes to the components

* [ISSUE #19961] update code to newest CDK version and clean autogenerated files (#22670)

* [ISSUE #19961] rename stream_slicer to partition_router and update ma… (#22590)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] clean faulty merge

* [ISSUE #19961] rename DatetimeStreamSlicer (#22617)

* [ISSUE #19961] rename stream_slicer to partition_router and update manifests (for incremental_sync as well)

* [ISSUE 19961] rename CustomStreamSlicer (#22598)

* [ISSUE 19961] rename CustomStreamSlicer

* [ISSUE #19961] code review CustomStreamSlicer

* [ISSUE #19961] fix source_square incremental sync

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter (#22591)

* [ISSUE #19961] rename SingleSlice to SinglePartitionRouter

* remove SinglePartitionRouter from the schema

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* [ISSUE #19961] rename DatetimeStreamSlicer

* [ISSUE #19961] rename SubstreamSlicer  to SubstreamPartitionRouter (#22596)

* [ISSUE #19961] TMP rename SubstreamSlicer  to SubstreamPartitionRouter

* [ISSUE #19961] revert DatetimeStreamSlicer.stream_state_field_start and DatetimeStreamSlicer.stream_state_field_end

* [ISSUE #19961] rename ListStreamSlicer to ListPartitionRouter (#22593)

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/yaml-overview.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/partition-router.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* Update docs/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>

* update docs

* [ISSUE #19961] clean unit tests files

* [ISSUE #19961] code review

---------

Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

* [Low-Code CDK] Allow for children of custom components to specify parameters that are normally derived (#22379)

* Fix a bug where child components of a custom component cannot receive fields from other components

* add tests, documentation and commenting

* fix test from merge

* add better error message for nested initialization failures

* 🪟 🔧 Connector Builder frontend fixes for low_code_cdk_to_beta (#22880)

* restrict name to stream level

* remove checkpoint interval

* adjust logic for new request options

* refactor slicers

* wording

* review comments

* make oldest supported version explicit

* separate the frontend and connector builder changes from the low-code to beta release

* [Low-Code CDK] Add script to run low code unit tests and address issues with a few connectors (#23123)

* consolidate all the changes into a new PR after I messed up the merge on the side branch

* add set to allow this to be called externally if necessary later

* remove last few extra fields i found and fix docs links

* fix docs one more time

---------

Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: Joe Reuter <joe@airbyte.io>
2023-02-15 21:46:49 -05:00

1101 lines
31 KiB
YAML

"$schema": http://json-schema.org/draft-07/schema#
"$id": https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml
title: DeclarativeSource
type: object
description: An API source that extracts data according to its declarative components
version: 1.0.0
required:
- type
- check
- streams
- version
properties:
type:
type: string
enum: [DeclarativeSource]
check:
"$ref": "#/definitions/CheckStream"
streams:
type: array
items:
"$ref": "#/definitions/DeclarativeStream"
version:
type: string
schemas:
"$ref": "#/definitions/Schemas"
definitions:
type: object
spec:
"$ref": "#/definitions/Spec"
additionalProperties: false
definitions:
AddedFieldDefinition:
description: Defines the field to add on a record
type: object
required:
- type
- path
- value
properties:
type:
type: string
enum: [AddedFieldDefinition]
path:
type: array
items:
type: string
value:
type: string
$parameters:
type: object
additionalProperties: true
AddFields:
description: Transformation which adds field to an output record. The path of the added field can be nested.
type: object
required:
- type
- fields
properties:
type:
type: string
enum: [AddFields]
fields:
type: array
items:
- "$ref": "#/definitions/AddedFieldDefinition"
$parameters:
type: object
additionalProperties: true
ApiKeyAuthenticator:
description: Authenticator for requests requiring an api token
type: object
required:
- type
- api_token
properties:
type:
type: string
enum: [ApiKeyAuthenticator]
api_token:
type: string
header:
type: string
$parameters:
type: object
additionalProperties: true
BasicHttpAuthenticator:
description: Authenticator for requests authenticated with a username and optional password
type: object
required:
- type
- username
properties:
type:
type: string
enum: [BasicHttpAuthenticator]
username:
description: The username that will be combined with the password, base64 encoded and used to make requests
type: string
password:
description: The password that will be combined with the username, base64 encoded and used to make requests
type: string
default: ""
$parameters:
type: object
additionalProperties: true
BearerAuthenticator:
description: Authenticator for requests authenticated with a Bearer token
type: object
required:
- type
- api_token
properties:
type:
type: string
enum: [BearerAuthenticator]
api_token:
type: string
$parameters:
type: object
additionalProperties: true
CheckStream:
description: Checks the connections by trying to read records from one or many of the streams selected by the developer
type: object
required:
- type
- stream_names
properties:
type:
type: string
enum: [CheckStream]
stream_names:
type: array
items:
type: string
CompositeErrorHandler:
description: Error handler that sequentially iterates over a list of error handlers
type: object
required:
- type
- error_handlers
properties:
type:
type: string
enum: [CompositeErrorHandler]
error_handlers:
type: array
items:
anyOf:
- "$ref": "#/definitions/CompositeErrorHandler"
- "$ref": "#/definitions/DefaultErrorHandler"
$parameters:
type: object
additionalProperties: true
ConstantBackoffStrategy:
description: Backoff strategy with a constant backoff interval
type: object
required:
- type
- backoff_time_in_seconds
properties:
type:
type: string
enum: [ConstantBackoffStrategy]
backoff_time_in_seconds:
anyOf:
- type: number
- type: string
$parameters:
type: object
additionalProperties: true
CursorPagination:
description: Pagination strategy that evaluates an interpolated string to define the next page token
type: object
required:
- type
- cursor_value
properties:
type:
type: string
enum: [CursorPagination]
cursor_value:
type: string
page_size:
type: integer
stop_condition:
type: string
decoder:
"$ref": "#/definitions/JsonDecoder"
$parameters:
type: object
additionalProperties: true
CustomAuthenticator:
description: Authenticator component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomAuthenticator]
class_name:
type: string
additionalProperties: true
$parameters:
type: object
additionalProperties: true
CustomBackoffStrategy:
description: Backoff strategy component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomBackoffStrategy]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
CustomErrorHandler:
description: Error handler component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomErrorHandler]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
CustomIncrementalSync:
description: Incremental component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
- cursor_field
properties:
type:
type: string
enum: [CustomIncrementalSync]
class_name:
description: The class that will be implementing the custom incremental sync. The format is `source_<name>.<package>.<class_name>`
type: string
additionalProperties: true
cursor_field:
description: The location of the value on a record that will be used as a bookmark during sync
type: string
$parameters:
type: object
additionalProperties: true
CustomPaginationStrategy:
description: Pagination strategy component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomPaginationStrategy]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
CustomRecordExtractor:
description: Record extractor component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomRecordExtractor]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
CustomRequester:
description: Requester component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomRequester]
class_name:
type: string
additionalProperties: true
$parameters:
type: object
additionalProperties: true
CustomRetriever:
description: Retriever component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomRetriever]
class_name:
type: string
additionalProperties: true
$parameters:
type: object
additionalProperties: true
CustomPartitionRouter:
description: Partition router component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomPartitionRouter]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
CustomTransformation:
description: Transformation component whose behavior is derived from a custom code implementation of the connector
type: object
additionalProperties: true
required:
- type
- class_name
properties:
type:
type: string
enum: [CustomTransformation]
class_name:
type: string
$parameters:
type: object
additionalProperties: true
DatetimeBasedCursor:
description: Cursor to provide incremental capabilities over datetime
type: object
required:
- type
- cursor_field
- end_datetime
- datetime_format
- cursor_granularity
- start_datetime
- step
properties:
type:
type: string
enum: [DatetimeBasedCursor]
cursor_field:
description: The location of the value on a record that will be used as a bookmark during sync
type: string
datetime_format:
description: The format of the datetime
type: string
cursor_granularity:
description: Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one
type: string
end_datetime:
description: The datetime that determines the last record that should be synced
anyOf:
- type: string
- "$ref": "#/definitions/MinMaxDatetime"
start_datetime:
description: The datetime that determines the earliest record that should be synced
anyOf:
- type: string
- "$ref": "#/definitions/MinMaxDatetime"
step:
description: The size of the time window (ISO8601 duration)
type: string
end_time_option:
description: Request option for end time
"$ref": "#/definitions/RequestOption"
lookback_window:
description: How many days before start_datetime to read data for (ISO8601 duration)
type: string
partition_field_end:
description: Partition start time field
type: string
partition_field_start:
description: Partition end time field
type: string
start_time_option:
description: Request option for start time
"$ref": "#/definitions/RequestOption"
$parameters:
type: object
additionalProperties: true
OAuthAuthenticator:
description: Authenticator for requests using OAuth 2.0 authentication
type: object
required:
- type
- client_id
- client_secret
- refresh_token
- token_refresh_endpoint
properties:
type:
type: string
enum: [OAuthAuthenticator]
client_id:
type: string
client_secret:
type: string
refresh_token:
type: string
token_refresh_endpoint:
type: string
access_token_name:
type: string
default: "access_token"
expires_in_name:
type: string
default: "expires_in"
grant_type:
type: string
default: "refresh_token"
refresh_request_body:
type: object
additionalProperties: true
scopes:
type: array
items:
type: string
token_expiry_date:
type: string
token_expiry_date_format:
description: The format of the datetime; provide it if expires_in is returned in datetime instead of seconds
type: string
$parameters:
type: object
additionalProperties: true
DeclarativeStream:
description: A stream whose behavior is described by a set of declarative low code components
type: object
additionalProperties: true
required:
- type
- retriever
properties:
type:
type: string
enum: [DeclarativeStream]
retriever:
definition: Component used to coordinate how records are extracted across stream slices and request pages
anyOf:
- "$ref": "#/definitions/CustomRetriever"
- "$ref": "#/definitions/SimpleRetriever"
incremental_sync:
anyOf:
- "$ref": "#/definitions/CustomIncrementalSync"
- "$ref": "#/definitions/DatetimeBasedCursor"
name:
definition: The stream name
type: string
default: ""
primary_key:
definition: The primary key of the stream
"$ref": "#/definitions/PrimaryKey"
default: ""
schema_loader:
definition: The schema loader used to retrieve the schema for the current stream
anyOf:
- "$ref": "#/definitions/InlineSchemaLoader"
- "$ref": "#/definitions/JsonFileSchemaLoader"
transformations:
definition: A list of transformations to be applied to each output record in the
type: array
items:
anyOf:
- "$ref": "#/definitions/AddFields"
- "$ref": "#/definitions/CustomTransformation"
- "$ref": "#/definitions/RemoveFields"
$parameters:
type: object
additional_properties: true
DefaultErrorHandler:
description: The default error handler. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff
type: object
required:
- type
properties:
type:
type: string
enum: [DefaultErrorHandler]
backoff_strategies:
type: array
items:
anyOf:
- "$ref": "#/definitions/ConstantBackoffStrategy"
- "$ref": "#/definitions/CustomBackoffStrategy"
- "$ref": "#/definitions/ExponentialBackoffStrategy"
- "$ref": "#/definitions/WaitTimeFromHeader"
- "$ref": "#/definitions/WaitUntilTimeFromHeader"
max_retries:
type: integer
default: 5
response_filters:
type: array
items:
"$ref": "#/definitions/HttpResponseFilter"
$parameters:
type: object
additional_properties: true
DefaultPaginator:
description: Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token
type: object
required:
- type
- pagination_strategy
properties:
type:
type: string
enum: [DefaultPaginator]
pagination_strategy:
anyOf:
- "$ref": "#/definitions/CursorPagination"
- "$ref": "#/definitions/CustomPaginationStrategy"
- "$ref": "#/definitions/OffsetIncrement"
- "$ref": "#/definitions/PageIncrement"
decoder:
"$ref": "#/definitions/JsonDecoder"
page_size_option:
"$ref": "#/definitions/RequestOption"
page_token_option:
anyOf:
- "$ref": "#/definitions/RequestOption"
- "$ref": "#/definitions/RequestPath"
$parameters:
type: object
additionalProperties: true
DpathExtractor:
description: Record extractor that searches a decoded response over a path defined as an array of fields
type: object
required:
- type
- field_path
properties:
type:
type: string
enum: [DpathExtractor]
field_path:
type: array
items:
- type: string
decoder:
"$ref": "#/definitions/JsonDecoder"
$parameters:
type: object
additionalProperties: true
ExponentialBackoffStrategy:
description: Backoff strategy with an exponential backoff interval
type: object
required:
- type
properties:
type:
type: string
enum: [ExponentialBackoffStrategy]
factor:
anyOf:
- type: number
- type: string
default: 5
$parameters:
type: object
additionalProperties: true
HttpRequester:
description: Default implementation of a requester
type: object
required:
- type
- path
- url_base
properties:
type:
type: string
enum: [HttpRequester]
path:
description: The specific endpoint to fetch data from for a resource.
type: string
url_base:
description: The root of the API source.
type: string
authenticator:
description: Authenticator component that defines how to authenticate to the source.
anyOf:
- "$ref": "#/definitions/ApiKeyAuthenticator"
- "$ref": "#/definitions/BasicHttpAuthenticator"
- "$ref": "#/definitions/BearerAuthenticator"
- "$ref": "#/definitions/CustomAuthenticator"
- "$ref": "#/definitions/OAuthAuthenticator"
- "$ref": "#/definitions/NoAuth"
- "$ref": "#/definitions/SessionTokenAuthenticator"
error_handler:
description: Error handler component that defines how to handle errors.
anyOf:
- "$ref": "#/definitions/DefaultErrorHandler"
- "$ref": "#/definitions/CustomErrorHandler"
- "$ref": "#/definitions/CompositeErrorHandler"
http_method:
description: The HTTP method used to fetch data from the source (can be GET or POST).
anyOf:
- type: string
- type: string
enum:
- GET
- POST
default: GET
request_body_data:
description: Specifies how to populate the body of the request with a non-JSON payload. If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form.
anyOf:
- type: string
- type: object
additionalProperties:
type: string
request_body_json:
description: Specifies how to populate the body of the request with a JSON payload.
anyOf:
- type: string
- type: object
additionalProperties:
type: string
request_headers:
description: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
anyOf:
- type: string
- type: object
additionalProperties:
type: string
request_parameters:
description: Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
anyOf:
- type: string
- type: object
additionalProperties:
type: string
$parameters:
type: object
additionalProperties: true
HttpResponseFilter:
description: A filter that is used to select on properties of the HTTP response received
type: object
required:
- type
- action
properties:
type:
type: string
enum: [HttpResponseFilter]
action:
type: string
enum:
- SUCCESS
- FAIL
- RETRY
- IGNORE
error_message:
type: string
error_message_contains:
type: string
http_codes:
type: array
items:
type: integer
uniqueItems: true
predicate:
type: string
$parameters:
type: object
additionalProperties: true
InlineSchemaLoader:
description: Loads a schema that is defined directly in the manifest file
type: object
required:
- type
properties:
type:
type: string
enum: [InlineSchemaLoader]
schema:
type: object
JsonFileSchemaLoader:
description: Loads the schema from a json file
type: object
required:
- type
properties:
type:
type: string
enum: [JsonFileSchemaLoader]
file_path:
type: string
$parameters:
type: object
additionalProperties: true
JsonDecoder:
type: object
required:
- type
properties:
type:
type: string
enum: [JsonDecoder]
ListPartitionRouter:
description: Partition router that is used to retrieve records that have been partitioned according to a list of values
type: object
required:
- type
- cursor_field
- values
properties:
type:
type: string
enum: [ListPartitionRouter]
cursor_field:
type: string
values:
anyOf:
- type: string
- type: array
items:
type: string
request_option:
"$ref": "#/definitions/RequestOption"
$parameters:
type: object
additionalProperties: true
MinMaxDatetime:
description: Compares the provided date against optional minimum or maximum times. The max_datetime serves as the ceiling and will be returned when datetime exceeds it. The min_datetime serves as the floor
type: object
required:
- type
- datetime
properties:
type:
type: string
enum: [MinMaxDatetime]
datetime:
type: string
datetime_format:
type: string
default: ""
max_datetime:
type: string
min_datetime:
type: string
$parameters:
type: object
additionalProperties: true
NoAuth:
description: Authenticator for requests requiring no authentication
type: object
required:
- type
properties:
type:
type: string
enum: [NoAuth]
$parameters:
type: object
additionalProperties: true
NoPagination:
description: Pagination implementation that never returns a next page
type: object
required:
- type
properties:
type:
type: string
enum: [NoPagination]
OffsetIncrement:
description: Pagination strategy that returns the number of records reads so far and returns it as the next page token
type: object
required:
- type
- page_size
properties:
type:
type: string
enum: [OffsetIncrement]
page_size:
description: The number of records to request
anyOf:
- type: integer
- type: string
$parameters:
type: object
additionalProperties: true
PageIncrement:
description: Pagination strategy that returns the number of pages reads so far and returns it as the next page token
type: object
required:
- type
- page_size
properties:
type:
type: string
enum: [PageIncrement]
page_size:
description: The number of records to request
type: integer
start_from_page:
type: integer
default: 0
$parameters:
type: object
additionalProperties: true
ParentStreamConfig:
description: Describes how to create a stream slice from a parent stream
type: object
required:
- type
- parent_key
- partition_field
- stream
properties:
type:
type: string
enum: [ParentStreamConfig]
parent_key:
type: string
stream:
"$ref": "#/definitions/DeclarativeStream"
partition_field:
type: string
request_option:
"$ref": "#/definitions/RequestOption"
$parameters:
type: object
additionalProperties: true
PrimaryKey:
description: The stream field to be used to distinguish unique rows
anyOf:
- type: string
- type: array
items:
type: string
- type: array
items:
type: array
items:
type: string
default: ""
RecordFilter:
description: Filter applied on a list of Records
type: object
required:
- type
properties:
type:
type: string
enum: [RecordFilter]
condition:
description: The predicate to filter a record. Records will be removed if evaluated to False
type: string
default: ""
$parameters:
type: object
additionalProperties: true
RecordSelector:
description: Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.
type: object
required:
- type
- extractor
properties:
type:
type: string
enum: [RecordSelector]
extractor:
anyOf:
- "$ref": "#/definitions/CustomRecordExtractor"
- "$ref": "#/definitions/DpathExtractor"
record_filter:
"$ref": "#/definitions/RecordFilter"
$parameters:
type: object
additionalProperties: true
RemoveFields:
description: A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.
type: object
required:
- type
- field_pointers
properties:
type:
type: string
enum: [RemoveFields]
field_pointers:
type: array
items:
items:
type: string
RequestPath:
description: A component that specifies where in the request path a component's value should be inserted into.
type: object
required:
- type
properties:
type:
type: string
enum: [RequestPath]
RequestOption:
description: A component that specifies the key field and where in the request a component's value should be inserted into.
type: object
required:
- type
- field_name
- inject_into
properties:
type:
type: string
enum: [RequestOption]
field_name:
type: string
inject_into:
enum:
- request_parameter
- header
- body_data
- body_json
Schemas:
description: The stream schemas representing the shape of the data emitted by the stream
type: object
additionalProperties: true
SessionTokenAuthenticator:
description: Authenticator for requests based on session tokens. A session token is a random value generated by a server to identify a specific user for the duration of one interaction session
type: object
required:
- type
- api_url
- header
- login_url
- session_token
- session_token_response_key
- username
- validate_session_url
properties:
type:
type: string
enum: [SessionTokenAuthenticator]
api_url:
type: string
header:
type: string
login_url:
type: string
session_token:
type: string
session_token_response_key:
type: string
username:
type: string
validate_session_url:
type: string
password:
type: string
default: ""
$parameters:
type: object
additionalProperties: true
SimpleRetriever:
description: Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.
type: object
required:
- type
- record_selector
- requester
properties:
type:
type: string
enum: [SimpleRetriever]
record_selector:
description: Component that describes how to extract records from a HTTP response.
"$ref": "#/definitions/RecordSelector"
requester:
description: Requester component that describes how to prepare HTTP requests to send to the source API.
anyOf:
- "$ref": "#/definitions/CustomRequester"
- "$ref": "#/definitions/HttpRequester"
paginator:
description: Paginator component that describes how to navigate through the API's pages.
anyOf:
- "$ref": "#/definitions/DefaultPaginator"
- "$ref": "#/definitions/NoPagination"
partition_router:
default: []
description: StreamSlicer component that describes how to partition the stream, enabling incremental syncs and checkpointing.
anyOf:
- "$ref": "#/definitions/CustomPartitionRouter"
- "$ref": "#/definitions/ListPartitionRouter"
- "$ref": "#/definitions/SubstreamPartitionRouter"
- type: array
items:
anyOf:
- "$ref": "#/definitions/CustomPartitionRouter"
- "$ref": "#/definitions/ListPartitionRouter"
- "$ref": "#/definitions/SubstreamPartitionRouter"
$parameters:
type: object
additionalProperties: true
Spec:
description: A connection specification made up of information about the connector and how it can be configured
type: object
required:
- type
- connection_specification
properties:
type:
type: string
enum: [Spec]
connection_specification:
type: object
additionalProperties: true
documentation_url:
type: string
SubstreamPartitionRouter:
description: Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams
type: object
required:
- type
- parent_stream_configs
properties:
type:
type: string
enum: [SubstreamPartitionRouter]
parent_stream_configs:
type: array
items:
"$ref": "#/definitions/ParentStreamConfig"
$parameters:
type: object
additionalProperties: true
WaitTimeFromHeader:
description: Extract wait time from http header
type: object
required:
- type
- header
properties:
type:
type: string
enum: [WaitTimeFromHeader]
header:
type: string
regex:
type: string
$parameters:
type: object
additionalProperties: true
WaitUntilTimeFromHeader:
description: Extract time at which we can retry the request from response header and wait for the difference between now and that time
type: object
required:
- type
- header
properties:
type:
type: string
enum: [WaitUntilTimeFromHeader]
header:
type: string
min_wait:
anyOf:
- type: number
- type: string
regex:
type: string
$parameters:
type: object
additionalProperties: true