Alexandre Girard
d9fa24ffff
Low code connectors: implement components for sendgrid ( #12853 )
...
* checkout from alex/cac
* checkout from alex/cac
* checkout from alex/cac
* Add missing tests
* Add missing files
* Add missing tests
* add missing file
* missing file
* missing file
* rename
* doc
* doc
* remove broken test
* rename
* jinja dependency
* Add comment
* comment
* comment
* pyjq dependency
* rename file
* delete unused file
* Revert "delete unused file"
This reverts commit 758e9393674a79223944758e9393674a79223944#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944c8fe154ffc
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec
🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections ( #13260 )
2022-05-27 22:05:39 +03:00
Marcos Marx
dca2256a7c
Bump 2022 license version ( #13233 )
...
* Bump year in license short to 2022
* remove protocol from cdk
2022-05-26 15:00:42 -03:00
Brian Lai
c4ef7db5eb
[Low Code CDK] add support for string interpolation for request headers ( #13215 )
...
* add support for string interpolation for request headers + small refactor
* fix formatting
* move parameter provider to folder for consistency
2022-05-25 22:15:19 -04:00
Alexandre Girard
3894134d11
Bump year in license short to 2022 ( #13191 )
...
* Bump to 2022
* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8
Low code connectors: core structure ( #12850 )
...
* checkout from alex/cac
* doc
* doc
* remove broken test
* rename
* rename file
* delete unused file
* rename
* abstract property
* isort
* update state
* Update comment
* remove incremental mixin
* delete comment
* update comments
* update comments
* remove no_state
* rename package
* pass parameters through kwargs
* update interface to pass source in interface
* update interface to pass source in interface
* rename to stream_slicer
* Low code connectors: string interpolation with jinja (#12852 )
* checkout from alex/cac
* Add missing tests
* Add missing files
* missing file
* rename
* jinja dependency
* Add comment
* comment
* comment
* Revert "delete unused file"
This reverts commit 758e9393674a79223944
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056
CDK: provide better user-friendly error messages ( #12944 )
...
* initial implementation
* add parse_response_error_message tests
* move error handler to existing try/catch in AbstractSource
* formatting
* var rename
* use isinstance for type checking
* add docstrings
* add more abstract_source and httpstream tests
* fix wrong httperror usage
* more test cases
* bump version, update changelog
2022-05-19 17:28:24 -04:00
Pedro S. Lopez
73c7fad7fc
CDK: emit AirbyteTraceMessage with exception trace information ( #12593 )
2022-05-06 17:16:57 -04:00
Pedro S. Lopez
53799cb9a2
🎉 CDK: support loading spec from yaml file ( #12104 )
...
* support loading spec from yaml file
* formatting
* remove commented code
* update comment
* remove unused file
* raise correct exception types
* bump version, update changelog
2022-04-20 13:18:46 -07:00
Augustin
2d277747be
cdk: remove false error logging in _send ( #11650 )
2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4
Python CDK - Fix BaseBackoffException constructor ( #11653 )
...
* - Fix BaseBackoffException constructor
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com >
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab
CDK improve error handling responses ( #11629 )
...
* add response text and message to backoff error handling
* add response text and message to backoff error handling
* add response text case http error
* change response text to before raise error
* apply suggestions
* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
d173ce741f
CDK: add support for streams with state attribute ( #9746 )
...
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea
CDK: Fix typing errors ( #9037 )
...
* fix typing, drop AirbyteLogger
* format
* bump the version
* use logger instead of fixture logger
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
Co-authored-by: auganbay <auganenu@gmail.com >
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16
Integrate Sentry for performance and errors tracking. ( #8248 )
...
* Integrate Sentry for performance and errors tracking.
* Add sentry sensitive data scrubbing.
* updated cdk version and changelog
* Integrate Sentry for performance and errors tracking
Add `SENTRY_DSN` environment variable
* Integrate Sentry for performance and errors tracking
Add `sentry_sdk` to install requirements
* format cdk
* enable Sentry for google-search-console
* updated connector version
* update spec and source yamls
Co-authored-by: auganbay <auganenu@gmail.com >
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com >
2021-12-21 20:02:02 +06:00
Eugene Kulak
d50ae47b49
🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace ( #8704 )
...
* print stacktrace for unhandled exceptions
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09
CDK: Add base pydantic model for connector config and schemas ( #8485 )
...
* add base spec model
* fix usage of state_checkpoint_interval in case it is dynamic
* add schema base models, fix spelling, signatures and polishing
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com >
2021-12-08 01:14:59 +02:00
Marcos Eliziario Santos
34ff075ba7
flake8 fix 3764 obfuscate secrets logging ( #8583 )
...
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
* Format and flake8
* Fix build error/bump version/format
* Add airbyte prefix on test logger
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-12-07 11:35:50 -03:00
Marcos Eliziario Santos
1d493ec620
CDK: #3764 obfuscate secrets in logging ( #8211 )
...
* Secure logger implementation minus still broken new tests
* Secure logger implementation and unit tests
* code review changes
* filter text on uncaught exceptions
* auto-formatting
* Mistaken change left in code
* filter text on uncaught exceptions
* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter
* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
* Obfuscate Secrets in Logging, code review changes
* Obfuscate Secrets in Logging, code review changes, unit test fixes
* CHANGELOG.md
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-12-07 10:45:22 -03:00
Marcos Eliziario Santos
65d471abf4
Output run time per stream ( #8173 )
...
* Output run time per stream
* Code review changes: Remove multithreaded support.
Port code to cdk
* Formatting
* remove extraneous try block
2021-11-30 10:20:51 -03:00
Dmytro
e56e86dd28
Fix schema transformation false warnings. ( #7863 )
2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c
Fix master build ( #7873 )
...
* Check in format diff
* Remove cache http yaml files
* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Dmytro
16b626cc91
Resolve discover catalog ref fields ( #7734 )
2021-11-09 12:42:15 +02:00
vitaliizazmic
0c7f06018b
🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels
...
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels
* Airbyte CDK native logger #1279 - add test for level types transforming
* CDK standard logger #1279 - test for transform critical level to fatal
2021-10-22 17:14:01 +03:00
vitaliizazmic
d5c04991bc
🎉 CDK: use standard logger with custom handler
...
* CDK - using native logger with custom formatter
* CDK - using native logger(fix exception and add trace)
* CDK - using native logger in AirbyteEntrypoint
* CDK - CHANGELOG.md
* CDK - remove unnecessary imports
* CDK - fixing according to PR review
* CDK native logger #1279 - annotations
* CDK native logger #1279 - fixing according to PR review
* CDK standard logger #1279 - tests
* CDK standard logger #1279 - reformat
* Airbyte CDK #1279 - improve docstrings
* Airbyte CDK #1279 - improve log levels
* Airbyte CDK #1279 - fix init get source name
* Airbyte CDK #1279 - update test licence
* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Michel Tricot
1773e41e47
Shorten our headers + adds contributors file ( #6478 )
2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea
🏗️ Python CDK: add schema transformer class ( #6139 )
...
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7
correct CI applying gradlew format ( #6390 )
...
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local >
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d
🎉 CDK: Added support for efficient parent/child streams using cache ( #6057 )
...
* Add caching
* Upd cache file handling
* Upd slices, sync mode, docs
* Bump version
* Use SyncMode.full_refresh for parent stream_slices
* Refactor
2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063
Format code ( #6381 )
2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b
🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token ( #6219 )
...
* Add support for headers to OAuth2Authenticator
Send custom headers in `refresh_access_token()`.
* Bump version + update CHANGELOG.md
* Add tests
* Update tests for refresh_access_token()
* Assert that mock_refresh_token_call was called
* Remove init file
2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065
CDK: support nested refs resolving ( #6044 )
...
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1
🎉 CDK: Add requests native authenticator support ( #5731 )
...
* Add requests native auth class
* Update init file.
Update type annotations.
Bump version.
* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.
* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md
* Add requests native authenticator tests
* Add CDK requests native __call__ method tests.
Update CHANGELOG.md
* Add outdated auth deprication messages
* Update requests native auth __call__ method tests
* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c
#5796 silence printing full config when config validation fails ( #5879 )
...
* - #5796 silence printing full config when config validation fails
* fix unit tests after config validation check changes
Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com >
2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37
🐛 CDK: fix bug with limit parameter for incremental stream ( #5833 )
...
* CDK: fix bug with limit parameter for incremental stream
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-09-09 19:55:29 +03:00
Dmytro
7584440515
CDK: private configuration option _limit and _page_size ( #5617 )
...
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137
Fix facebook marketing source SAT fail on invalid config file ( #5621 )
...
* Fix facebook marketing source SAT fail on invalid config file
2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd
Python CDK: fix retry attempts in case of user defined backoff time ( #5707 )
...
* Python CDK: fix retry attempts in case of user defined backoff time
2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355
🎉 Python CDK: Allow to ignore http status errors and override retry parameters ( #5363 )
...
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
2021-08-25 10:31:24 +03:00
Dmytro
b1f2bf5665
4776: Python CDK: Validate input config.py against spec ( #5457 )
...
Python CDK: Validate input config.py against spec
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com >
2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2
🎉 Source Github: Add MultipleTokenAuthenticator ( #5223 )
...
* Add multiple token authenticator
* Add MultipleTokenAuthenticator
* Upd docs
* Refactor
* Upd docs
* Fix merge typo
* Upd multiple token support: switch to list of tokens
* Upd multiple token support: refactoring
* Update airbyte-integrations/connectors/source-github/source_github/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Update airbyte-integrations/connectors/source-github/source_github/spec.json
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Cleanup, switch to MultipleTokenAuthenticator completely
* Upd changelog
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c
Populate the PK from the Singer discovery run ( #2713 ) ( #4789 )
...
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
2021-08-09 20:11:22 +03:00
Maksym Pavlenok
106c3cd641
🎉 CDK: Allow setting request non-JSON data ( #5161 )
...
* add the function request_body_data
* gradlew format
* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
* add test for application/x-www-form-urlencoded
Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com >
Co-authored-by: Sherif A. Nada <snadalive@gmail.com >
2021-08-06 15:32:25 +03:00
Sherif A. Nada
cb4fe7254c
CDK: Add initial Destination abstraction and tests ( #4719 )
...
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com >
2021-07-13 16:18:08 -07:00
Sherif A. Nada
3aebe55cd5
🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests ( #4493 )
2021-07-07 15:17:58 -07:00
Davin Chia
9460cb4422
Update all of Pydantic to 1.6.2 per Dependabot. ( #3408 )
...
* Update all to 1.6.2.
* Publish new airbyte-cdk version.
* Use repr instead of str for exceptions.
* Use rc.
* Edit test.
* Bump for SAT.
* Format.
2021-05-14 16:44:49 +08:00
Charles
0df53170c9
Stop formatting python with spotless ( #3388 )
2021-05-13 17:46:34 -07:00
Charles
a2f29f6024
fix formatting ( #3370 )
2021-05-11 15:46:59 -07:00
Sherif A. Nada
eb812eaecd
Bump CDK consumers ( #3368 )
2021-05-11 15:27:44 -07:00
Sherif A. Nada
535d83e0af
bugfix infinite pagination in CDK ( #3366 )
2021-05-11 14:45:27 -07:00
