1
0
mirror of synced 2026-01-01 00:02:54 -05:00
Commit Graph

538 Commits

Author SHA1 Message Date
Evan Tahler
69a0f24084 lint CDK files (#20163)
* lint CDK files

* bump to rebuild file

* format again?

* remove double line

* fix spacing in template

* remove trailing space
2022-12-06 16:59:45 -08:00
Maxime Carbonneau-Leclerc
2568d6107c Issue 19733 cdk clarify config error message for config files (#20019)
* [ISSUE-19733] clarify error message when reading config files

* [ISSUE #19733] code review and adding validation for spec file as well

* [ISSUE #19733] updating typing of read_json_file

* [ISSUE #19733] fix flake8 error

* [ISSUE #19733] fix linting error

* [ISSUE #19733] remove breaking change

* [ISSUE #19733] bump airbyte cdk version

* [ISSUE #19733] add test for invalid json file on read_state

* [ISSUE #19733] bump version
2022-12-06 18:55:00 -05:00
Roman Yermilov [GL]
23b35ba27c airbyte-cdk: bump version (#20143) 2022-12-06 20:47:16 +04:00
Roman Yermilov [GL]
bedc3b99a3 airbyte-cdk offset pagination strategy: page_size to be interpolated … (#19646)
* airbyte-cdk offset pagination strategy: page_size to be interpolated string
* airbyte-cdk offset pagination strategy: bump version
2022-12-06 18:48:56 +04:00
Ella Rohm-Ensing
4bef54d1b9 Add README info about using your local CDK locally and in docker (#20082)
* Add README info about installing the local CDK locally and in Docker

* Add '|| true' to validate_dockerignore()

* Update airbyte-cdk/python/README.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-12-05 17:52:35 -05:00
Alexandre Girard
4dcc68ade9 SimpleRetriever raises exception with error message extracted from the response (#20032)
* Raise exception with error message

* read from detail field if no other value was found

* format

* dont test the error message

* format

* error code

* bump
2022-12-02 17:56:23 -08:00
Evan Tahler
1e160b2858 Faker V0.3 - CDK, consistent naming, determinism, display name, and SAT (#19490)
* better comment to show sync types

* WIP - basics in place

* test cleanup

* emit `AirbyteEstimateTraceMessage`

* assert estimates emitted

* unit tests passing

* format

* flake

* check tests

* add newline

* lint

* check in with new file name

* bump acceptance test yml

* acceptance test progress

* deterministic purchases

* SAT making progress

* better disable

* adjust yaml

* change abnormal_state.json format

* per-stream state

* products for abnormal state

* products have state + incremental

* purchases state = total_user_records

* update docs and name

* disable proper version

* typo

* try disable 0.2.1

* kwargs

* v1.0.0

* lint

* backward_compatibility_tests_config

* auto-bump connector version

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-12-01 14:53:38 -08:00
Catherine Noll
44a1440d49 Include http method in SimpleRetriever log message for requests (#19964)
* Add http method to `SimpleRetriever` log message on request

* Update setup.py (bump version to 0.11.2)

* Update CHANGELOG.md
2022-12-01 15:52:20 -05:00
Brian Lai
b1c50b8470 Fix the underlying component schema to use check instead of checker (#19910)
* fix underlying schema to use check instead of checker

* regenerate the component manifest

* regen manifest

* bump patch version
2022-11-30 20:42:48 -05:00
Augustin
3b15f9ebe1 CDK: Emit control message on config mutation (#19428) 2022-11-29 14:51:01 +01:00
Evan Tahler
1598126ebc Add description for AirbyteEstimateTraceMessage types (#19843)
* Add description for `AirbyteEstimateTraceMessage` types

* cleanup

* update python CDK docs too
2022-11-28 12:58:20 -08:00
Denys Davydov
fc52644bac CDK Lowcode: allow parametrizing initial page when using PageIncrement (#19712)
* #CDK Lowcode: allow parametrizing initial page when using PageIncrement

* bump minor version
2022-11-28 18:26:08 +02:00
Serhii Chvaliuk
dc2c61837b CDK: Add jinja macro format_datetime (#19506)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-18 19:18:32 +02:00
Brian Lai
e97ece5532 [low-code cdk] break resolving reference preprocessing into its own class so it can be reused (#19517)
* break resolving reference preprocessing into its own class so it can be reused

* move reference resolution into the ManifestDeclarativeSource and deprecate the parser

* formatting

* last formatting i promise

* rename

* bump version

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-11-17 13:15:28 -08:00
Jimmy Ma
d3f35335da format (#19507) 2022-11-16 13:54:59 -08:00
Serhii Chvaliuk
22d226aabe CDK: Avoid duplicate HTTP query in simple_retriever (#19485)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-16 18:55:29 +02:00
Serhii Chvaliuk
d5b66e9d7b CDK: make default_paginator.page_token_option optional (#19368)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-14 20:16:30 +02:00
Serhii Chvaliuk
5cd88e7c05 CDK: don't filter failed interpolated vars for request_options_provider.request_body_json (#19297)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-11-11 10:44:23 +02:00
JJ Nilbodee
33227f5812 Allows user to specify grant_type in OAuthAuthenticator (#19139)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-11-10 22:22:17 -08:00
Brian Lai
60c008f6cd don't update cursor for log messages and and default schema path coming from connector builder (#19271)
* don't update cursor for log messages and and default schema path coming from connector builder

* replace check for connector_builder module with a basic default file path

* update changelog and patch version

* catch the correct exception when pkgutil can't load json file
2022-11-10 19:23:05 -05:00
Brian Lai
f54f4af43f bump cdk version (#19255) 2022-11-09 17:53:29 -05:00
Alexandre Girard
e029353b9f SimpleRetriever yield request and response as log messages (#18644)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* Yield request and response as log messages

* only emit request and responses if the debug flag is on

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* remove deprecated comment

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* fix

* slightly better test

* typing

* extract method

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Comment

* operate on the message

* Revert "Revert "reset to master""

This reverts commit 5833c84d0a.

* comment

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format

* format

* symlink

* Update setup

* update path

* reset to master

* update

* Add local files

* greenhouse

* format

* symlink

* try reordering

* better error message

* better log message

* reset to master

* Revert "merge for qa"

This reverts commit ad7128f2c5, reversing
changes made to 7196c22a73.

* reset to master

* reset to master

* reset to master

* format

* gradlew format

* right type hints

* reset to master

* reset to master

* gradlew format

* a bunch of small fixes

* Update output format

* fixes from feedback

* fixme comment

* streams cannot return AirbyteRecordMessage

* fix

* format

* only return logs when running on debug mode

* move branching

* update typing

* remove dead code

* fix simpleretriever name

* i think this is better

* log response.text

* debug flag

* comment

* pass config

* comments

* run SATs

* fix most of the unit tests

* fix unit test

* reset to master

* runFromPath

* Revert "runFromPath"

This reverts commit 85979a801a.

* Revert "run SATs"

This reverts commit a8a8a2da95.

* no need to convert to dict

* fix test
2022-11-09 13:26:50 -08:00
Brian Lai
5c9e5d9934 [low-code cdk] decouple parsing the yaml manifest from the declarative source implementation (#19095)
* decouple parsing the yaml manifest from the declarative source implementation

* bump version and changelog
2022-11-08 15:45:01 -05:00
Alexandre Girard
1a608f846a Stream returns AirbyteMessages (#18572)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format
2022-11-07 21:23:11 -08:00
Evan Tahler
e6b06a88ac AirbyteEstimateTraceMessage (#18875)
* `AirbyteEstimateTraceMessage`

* Add PR number

* fix method name

* Lint

* Lint

* fix merge

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

* `EstimateType` sub type in python

* lint

Co-authored-by: Davin Chia <davinchia@gmail.com>
2022-11-07 12:45:39 -08:00
Brian Lai
f9863d6374 [low-code cdk] Allow for spec file to be defined in the yaml manifest instead of an external file (#18411)
* allow for spec to be defined in the source.yaml manifest instead of an external file

* make spec a component within the language to get schema validation and rework the code for better testing

* fix formatting and extra method

* pr feedback and add some more test

* pr feedback

* bump airbyte-cdk version

* bump version

* gradle format

* remove  from manifest spec
2022-11-07 14:44:45 -05:00
Alexandre Girard
3ff3e9aeee Bump CDK version (#19033)
* bump

* bump to 0.6.0
2022-11-07 09:27:09 -08:00
Xabier Lahuerta Vazquez
aa5da75d81 Add support for monthly and yearly incremental updates (#18861)
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-11-07 07:49:57 -08:00
Denys Davydov
8145be55a3 Low-code CDK: safe get response.json (#18931)
* low-code cdk: safe get response.json

* flake fix
2022-11-04 11:44:51 +02:00
Brian Lai
186580a6ee [low-code] replace emptySchemaLoader with DefaultSchemaLoader (#18947)
* replace emptySchemaLoader with DefaultSchemaLoader

* fix test name

* fix test

* add logging for when we default to the empty schema

* increment patch version

* fix formatting

* update changelog
2022-11-03 23:25:01 -04:00
Alexandre Girard
605fb921c4 [low-code]: Evaluate backoff strategies at runtime (#18053)
* pass options to wait time from header

* fix constant backoff

* parameterize test

* fix tests

* missing unit tests

* eval header at runtime

* eval regex at runtime

* evaluate min_wait at runtime

* eval factor at runtime

* missing unit tests

* remove debug print

* rename

* Add tests

* Add tests

* Update docs
2022-11-03 18:50:47 -07:00
Brian Lai
dd7184dbca [low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file (#18532)
* use mockschemaloader as the default and rename JsonSchema to JsonFileSchemaLoader

* rename mock to EmptySchemaLoader

* retain existing JsonSchema while hacktoberfest is still in progress

* bump version and changelog
2022-11-03 12:05:04 -04:00
Alexandre Girard
cd1f492140 🐛 Lowcode: ListStreamSlicer and SubstreamSlicer should get the stream_slice from the arguments (#18574)
* fix list stream slicer

* get the stream slice from the parameters

* delete unused code

* Add comments

* missing test

* newline

* with pytest.raise

* reset to master

* newline

* Update __init__.py

* Update __init__.py

* 🎉 New Source: ConvertKit (#18455)

* Init source omnisend

* Removed unnecessary files

* Init source convertkit

* Added forms, sequences streams

* Added tags stream

* Added subscribers, broadcasts streams

* Added documentation

* Removed unnecessary files

* Updated pull request information in documentation

* Added sample config

* Updated invalid config

* Formatting, removed abnormal state, uncommented full_refresh acceptance test

* Added pagination for subscribers stream

* fix: add source definition for convertkit

* auto-bump connector version

Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* 🎉New Source: Google Webfonts [low-code cdk] (#18496)

* New source: Google webfonts

* chore: Add Docs

* chore: update changelog

* chore: resolved given comments for PR

* chore: unwanted files removed

* fix: generate and add source definitions

Co-authored-by: sajarin <sajarindider@gmail.com>

* Clean up build.gradle. (#18555)

Upstream has fixed this bug so we no longer need to host it internally. Remove this to clean up build.gradle.

* 🎉 New Destination: Typesense (#18349)

* Initial boilerplate

* 🎉 New Destination: Typesense

* remove .java-version

* fix doc

* add typesense to dest def

* add release stage

* add requirement to main

* auto-bump connector version

* add changelog

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* 🎉 New Source: Waiteraid [low-code cdk] (#18165)

* added source waiteraid

* 🎉 New Source: Waiteraid

* add searchBookings stream

* add P/R number

* add SUMMARY entry

* add docs/integrations/README.md entry

* add builds.md entry

* add docs to each endpoint

* fix schema
~

* Update airbyte-integrations/builds.md

added web address

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Delete catalog.json

* Delete TODO.md

* Update spec.yaml

* add waiteraid to source def seed

* auto-bump connector version

Co-authored-by: Sebastian Brickel <sebastianbrickel@Sebastians-MacBook-Air.local>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

* Replace `recipesLink` link with `tutorialsLink` (#18616)

* Protocol Change: `AirbyteControlMessage.ConnectorConfig` (#17907)

* Protocol Change: AirbyteConfigMessage

* update PR link in docs

* Lint

* Update python files

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage`

* AirbyteOrchestratorMessage

* Update docs

* `AirbyteControlConnectorConfigMessage`

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Remove workflow version check (#18613)

* Improved the Oracle cloud deployment guide (#18615)

* Update OCI VM deployment guide

* Update OCI deployment guide

* Update on oci-vm.md

* bump

Co-authored-by: Dainius Salkauskas <dainiuxazz@gmail.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Balasubramanian T K <btk.codedev@gmail.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Cirdes <cirdes@linkana.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: sbrickel-nimble <113671803+sbrickel-nimble@users.noreply.github.com>
Co-authored-by: Sebastian Brickel <sebastianbrickel@Sebastians-MacBook-Air.local>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io>
Co-authored-by: Victor Ikechukwu <vickyikechukwu13@gmail.com>
2022-10-28 13:09:19 -07:00
Evan Tahler
02459e8354 Protocol Change: AirbyteControlMessage.ConnectorConfig (#17907)
* Protocol Change: AirbyteConfigMessage

* update PR link in docs

* Lint

* Update python files

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/understanding-airbyte/airbyte-protocol.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* `AirbyteConfigMessage` -> `AirbyteConnectorConfigMessage`

* AirbyteOrchestratorMessage

* Update docs

* `AirbyteControlConnectorConfigMessage`

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-10-28 11:11:31 -07:00
Evan Tahler
9126dc21e3 Add missing generated python files (#18580) 2022-10-27 18:01:04 -07:00
Alexandre Girard
1ddc8c6185 Lowcode connectors: Fix few issues with stream slicers (#18529)
* Fix a few bugs

* bump
2022-10-27 13:33:13 -07:00
Brian Lai
5222093b54 support custom error messaging for error response + retryable errors (#18204)
* support custom error messaging for error response + retryable errors

* remove changed backoff i was using for testing

* refactor filter to construct response status internally

* pr feedback

* bump version and update changelog
2022-10-26 15:39:36 -04:00
Sherif A. Nada
25e7a37de6 Publish new CDK version with typehints exported (#18398) 2022-10-25 09:02:06 -07:00
Nipunn Koorapati
d77f913373 Export python type information for airbyte-cdk (#18387)
Fixes #18384
2022-10-24 12:20:43 -07:00
Serhii Chvaliuk
69092edf17 requester = MagicMock(use_cache=False) (#18289)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-22 20:47:32 +03:00
Alexandre Girard
76acfb8192 [low-code] Propagate options to InterpolatedRequestInputProvider (#18050)
* properly propagate options

* cleanup

* turn into dataclass

* rename

* no need for deepcopy

* fix test

* bump

* cleaner
2022-10-21 10:16:25 -07:00
Pedro S. Lopez
ad29513a43 CDK: Handle config validation errors as config_error and failed check status (#18214)
* schema helpers raises a config_error

* traced exceptions can be turned into connection status messages

* add tests for schema helpers

* return a failed status message rather than throwing in check command

* remove unused imports

* add comment

* bump version / update changelog
2022-10-20 13:23:06 -04:00
Brian Lai
de24ddced5 make sure stream_slices is an iterator (#18092)
* ensure output from stream_slices() is always an iterator

* bump cdk version

* bump version after merging latest cdk

* bump cdk version post rebase
2022-10-19 15:37:00 -04:00
Tanvesh Takawale
da5aed8a31 CDK: fixed document links regarding airbyte specification (#18169) 2022-10-19 09:50:01 -07:00
Serhii Chvaliuk
258b23cf79 CDK: VCR -> requests_cache + SQLite (#17990)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-19 19:31:28 +03:00
Augustin
b564f3eb78 Protocol: make supported_sync_modes a required not empty list on AirbyteStream (#15591) 2022-10-19 15:22:25 +02:00
Delena Malan
3b12a585ec Docs: Fix backoff stategy docs (#18143)
* Fix backoff strategy docs

* Update config_component_schema.json
2022-10-18 16:28:38 -07:00
Alexander Marquardt
26b6317198 Added new "filters" python file, along with a "hash" filter. This can… (#18000)
* Added new "filters" python file, along with a "hash" filter. This can be extended to include other custom filters in the future.

* Added additional comments

* Moved usage of the hash_obj inside the conditional that confirms it exists

* Moved the hash function call inside a condition to ensure that it exists

* Fixed the application of the salt , so that it does not modify the hash unless it is actually passed in.

* Added unit tests to validate new jinja hash functionality

* Updated unit test to pass numeric value as a float instead of string

* Removed unreferenced import to pytest

* Updated version

* format

* format

* format

* format

* format

Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
2022-10-17 23:19:55 -07:00
Brian Lai
4714a1c004 fix check for streams that do not use a stream slicer (#18080)
* fix check for streams that do not use a stream slicer

* increment version and changelog before publish
2022-10-17 17:11:11 -04:00
Alexandre Girard
0ffd503b21 🐛 [low-code] $options shouldn't overwrite values that are already defined (#18060)
* fix

* Add missing test

* remove prints

* extract to method

* rename

* Add missing test

* rename

* bump
2022-10-17 11:07:04 -07:00