Brian Lai
be01b476ce
Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments ( #13472 )
...
* write out new request options provider and refactor components and parts of the YAML config
* fix formatting
* pr feedback to consolidate body_data_provider to simplify the code
* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Alexandre Girard
34e2dc36c6
[low-code connectors] Cartesian product stream slicer ( #13740 )
...
* list slicer
* Add comment
* product stream slicer
* comment
* rename
* format
* Update comment
* split on 2 lines for readability
* Revert "rename"
This reverts commit e801f2d1f8
2022-06-16 08:48:32 -07:00
Alexandre Girard
2c8fef8b7d
[low-code connectors] Support for list-based stream slicers ( #13668 )
...
* list slicer
* Add comment
* test
* ast eval
* refactor
* fix
* remove print
* rename
* return a generator
* smaller fix and reset to master
* union type hint
* assert
* comment
* format
2022-06-14 17:12:53 -07:00
Alexandre Girard
15fe51b788
[low-code connectors] add a few unit tests ( #13666 )
...
* add a couple of unit tests
* refactor tests
2022-06-13 16:51:36 -07:00
Pedro S. Lopez
4f8f979045
Fix Connectors Base Build: CDK tests fail after dependency update ( #13652 )
2022-06-09 16:05:36 -04:00
Alexandre Girard
57dd7a9d4b
[low-code connectors] use jello instead of jq ( #13570 )
...
* use jello instead of jq
* reset
* rename
* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
7fbe6dab5c
[Low code CDK] add support for iterating over parent stream slices ( #13287 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-02 08:53:04 -07:00
Brian Lai
ccbd7c6508
conditional paginator + interpolated boolean components ( #13389 )
...
* conditional paginator + interpolated boolean components
* fix bug where empty arrays and dicts evaluated to true
* add more falsey values
* adjust falsey values and tweak based on review comments
2022-06-01 20:24:32 -04:00
Alexandre Girard
aa92518721
Low-code connectors: configurable source from yaml ( #13038 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* sengrid low code connector
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* update import
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944snadalive@gmail.com >
* more precise exception
* rename class
* add comment
* Try to install packages to build jq
* isort
* only automake
* Revert "only automake"
This reverts commit c8fe154ffcab5a7afd087c9fb8eb33snadalive@gmail.com >
* fix tests and format
* format
* update
* better error message
* Add jq dependency
* Use request header provider
* rename
* rename field
* remove get_context method
* rename
* add a comment
* format
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2022-06-01 07:31:52 -07:00
Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Marcos Marx
dca2256a7c
Bump 2022 license version ( #13233 )
...
* Bump year in license short to 2022
* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Lai
c4ef7db5eb
[Low Code CDK] add support for string interpolation for request headers ( #13215 )
...
* add support for string interpolation for request headers + small refactor
* fix formatting
* move parameter provider to folder for consistency
2022-05-25 22:15:19 -04:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
Pedro S. Lopez
73c7fad7fc
CDK: emit AirbyteTraceMessage with exception trace information ( #12593 )
2022-05-06 17:16:57 -04:00
Augustin
2d277747be
cdk: remove false error logging in _send ( #11650 )
2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4
Python CDK - Fix BaseBackoffException constructor ( #11653 )
...
* - Fix BaseBackoffException constructor
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab
CDK improve error handling responses ( #11629 )
...
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
d173ce741f
CDK: add support for streams with state attribute ( #9746 )
...
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea
CDK: Fix typing errors ( #9037 )
...
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
Co-authored-by: auganbay <auganenu@gmail.com >
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16
Integrate Sentry for performance and errors tracking. ( #8248 )
...
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com >
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-21 20:02:02 +06:00
Eugene Kulak
d50ae47b49
🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace ( #8704 )
...
* print stacktrace for unhandled exceptions
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09
CDK: Add base pydantic model for connector config and schemas ( #8485 )
...
* add base spec model
* fix usage of state_checkpoint_interval in case it is dynamic
* add schema base models, fix spelling, signatures and polishing
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-08 01:14:59 +02:00
Dmytro
e56e86dd28
Fix schema transformation false warnings. ( #7863 )
2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c
Fix master build ( #7873 )
...
* Check in format diff
* Remove cache http yaml files
* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Dmytro
16b626cc91
Resolve discover catalog ref fields ( #7734 )
2021-11-09 12:42:15 +02:00
Michel Tricot
1773e41e47
Shorten our headers + adds contributors file ( #6478 )
2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea
🏗️ Python CDK: add schema transformer class ( #6139 )
...
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7
correct CI applying gradlew format ( #6390 )
...
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local >
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d
🎉 CDK: Added support for efficient parent/child streams using cache ( #6057 )
...
* Add caching
* Upd cache file handling
* Upd slices, sync mode, docs
* Bump version
* Use SyncMode.full_refresh for parent stream_slices
* Refactor
2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063
Format code ( #6381 )
2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b
🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token ( #6219 )
...
* Add support for headers to OAuth2Authenticator
Send custom headers in `refresh_access_token()`.
* Bump version + update CHANGELOG.md
* Add tests
* Update tests for refresh_access_token()
* Assert that mock_refresh_token_call was called
* Remove init file
2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065
CDK: support nested refs resolving ( #6044 )
...
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1
🎉 CDK: Add requests native authenticator support ( #5731 )
...
* Add requests native auth class
* Update init file.
Update type annotations.
Bump version.
* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.
* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md
* Add requests native authenticator tests
* Add CDK requests native __call__ method tests.
Update CHANGELOG.md
* Add outdated auth deprication messages
* Update requests native auth __call__ method tests
* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c
#5796 silence printing full config when config validation fails ( #5879 )
...
* - #5796 silence printing full config when config validation fails
* fix unit tests after config validation check changes
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com >
2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37
🐛 CDK: fix bug with limit parameter for incremental stream ( #5833 )
...
* CDK: fix bug with limit parameter for incremental stream
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-09 19:55:29 +03:00
Dmytro
7584440515
CDK: private configuration option _limit and _page_size ( #5617 )
...
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
Dmytro
eed2e107fd
Python CDK: fix retry attempts in case of user defined backoff time ( #5707 )
...
* Python CDK: fix retry attempts in case of user defined backoff time
2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355
🎉 Python CDK: Allow to ignore http status errors and override retry parameters ( #5363 )
...
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
2021-08-25 10:31:24 +03:00
Dmytro
b1f2bf5665
4776: Python CDK: Validate input config.py against spec ( #5457 )
...
Python CDK: Validate input config.py against spec
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2
🎉 Source Github: Add MultipleTokenAuthenticator ( #5223 )
...
* Add multiple token authenticator
* Add MultipleTokenAuthenticator
* Upd docs
* Refactor
* Upd docs
* Fix merge typo
* Upd multiple token support: switch to list of tokens
* Upd multiple token support: refactoring
* Update airbyte-integrations/connectors/source-github/source_github/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Update airbyte-integrations/connectors/source-github/source_github/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Cleanup, switch to MultipleTokenAuthenticator completely
* Upd changelog
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2021-08-19 08:41:14 +03:00
Maksym Pavlenok
106c3cd641
🎉 CDK: Allow setting request non-JSON data ( #5161 )
...
* add the function request_body_data
* gradlew format
* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* add test for application/x-www-form-urlencoded
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com >
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2021-08-06 15:32:25 +03:00
Sherif A. Nada
3aebe55cd5
🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests ( #4493 )
2021-07-07 15:17:58 -07:00
Davin Chia
9460cb4422
Update all of Pydantic to 1.6.2 per Dependabot. ( #3408 )
...
* Update all to 1.6.2.
* Publish new airbyte-cdk version.
* Use repr instead of str for exceptions.
* Use rc.
* Edit test.
* Bump for SAT.
* Format.
2021-05-14 16:44:49 +08:00
Charles
0df53170c9
Stop formatting python with spotless ( #3388 )
2021-05-13 17:46:34 -07:00
Charles
a2f29f6024
fix formatting ( #3370 )
2021-05-11 15:46:59 -07:00
Sherif A. Nada
eb812eaecd
Bump CDK consumers ( #3368 )
2021-05-11 15:27:44 -07:00
Sherif A. Nada
535d83e0af
bugfix infinite pagination in CDK ( #3366 )
2021-05-11 14:45:27 -07:00
