1
0
mirror of synced 2025-12-30 12:04:43 -05:00
Commit Graph

69 Commits

Author SHA1 Message Date
Serhii Chvaliuk
9a960c4ca6 CDK: TypeTransformer.default_convert catch TypeError (#12948)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
7ea27d0ecd CDK: publish version 0.1.57 (#12938) 2022-05-17 16:33:06 -04:00
Pedro S. Lopez
73c7fad7fc CDK: emit AirbyteTraceMessage with exception trace information (#12593) 2022-05-06 17:16:57 -04:00
Sherif A. Nada
cc46255190 cull base-python and base-python-test (#12596) 2022-05-05 13:48:15 -07:00
Pedro S. Lopez
53799cb9a2 🎉 CDK: support loading spec from yaml file (#12104)
* support loading spec from yaml file

* formatting

* remove commented code

* update comment

* remove unused file

* raise correct exception types

* bump version, update changelog
2022-04-20 13:18:46 -07:00
Pedro S. Lopez
a6a6be2588 allow importing IncrementalMixin from airbyte_cdk.sources.streams (#11858)
* allow importing IncrementalMixin from airbyte_cdk.sources.streams

* format file

* update docs with hint on where to import IncrementalMixin from

* bump cdk version to 0.1.54

* update changelog

* only build python 3.9

* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Augustin
2d277747be cdk: remove false error logging in _send (#11650) 2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4 Python CDK - Fix BaseBackoffException constructor (#11653)
* - Fix BaseBackoffException constructor

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab CDK improve error handling responses (#11629)
* add response text and message to backoff error handling

* add response text and message to backoff error handling

* add response text case http error

* change response text to before raise error

* apply suggestions

* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
vitaliizazmic
936050847f 🐛 CDK: fixing type errors (#9926)
* CDK #9500 - fixing type errors

* CDK #9500 - exclude singer from check

* CDK #9500 - type ignore in utils.schema_helpers.py

* CDK #9500 - type ignore in utils.sentry.py

* CDK #9500 - type ignore in singer.singer_helpers.py

* CDK #9500 - type ignore in singer.source

* CDK #9500 - fix according to PR review

* CDK #9500 - fix according to PR review

* CDK #9500 - bump version
2022-02-14 09:24:04 +02:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
Marcos Eliziario Santos
fd716d7a10 Adding unit tests to SAT full refresh test (#8589)
* WIP - Adding unit tests to SAT full refresh test

* unit tests to SAT full refresh test

* Format

* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_test_full_refresh.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* use symmetric_difference to check for differences between expected/actual when testing stream read on SAT

* bump version

* bump version

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-21 12:09:09 -03:00
Dmytro
fe954c1a16 Integrate Sentry for performance and errors tracking. (#8248)
* Integrate Sentry for performance and errors tracking.

* Add sentry sensitive data scrubbing.

* updated cdk version and changelog

* Integrate Sentry for performance and errors tracking

Add `SENTRY_DSN` environment variable

* Integrate Sentry for performance and errors tracking

Add `sentry_sdk` to install requirements

* format cdk

* enable Sentry for google-search-console

* updated connector version

* update spec and source yamls

Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-21 20:02:02 +06:00
augan-rymkhan
7fbdfc61d0 Airbyte CDK: Log http status code and content in backoff handlers (#8829)
* Log http status code and content in default backoff handler

* Log http status code and content in usef defined backoff handler

* updated cdk version and changelog

* make it clear: exc.response

Co-authored-by: auganbay <auganenu@gmail.com>
2021-12-16 20:20:12 +06:00
Eugene Kulak
d50ae47b49 🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace (#8704)
* print stacktrace for unhandled exceptions

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09 CDK: Add base pydantic model for connector config and schemas (#8485)
* add base spec model

* fix usage of state_checkpoint_interval in case it is dynamic

* add schema base models, fix spelling, signatures and polishing

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-08 01:14:59 +02:00
Marcos Eliziario Santos
34ff075ba7 flake8 fix 3764 obfuscate secrets logging (#8583)
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

* Format and flake8

* Fix build error/bump version/format

* Add airbyte prefix on test logger

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-07 11:35:50 -03:00
Marcos Eliziario Santos
1d493ec620 CDK: #3764 obfuscate secrets in logging (#8211)
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-07 10:45:22 -03:00
augan-rymkhan
8b5303e94f added __init__.py file to utils (#8530)
* added __init__.py file to utils

* bump the version

Co-authored-by: Auganbay <auganenu@gmail.com>
2021-12-06 15:14:43 +02:00
Sergei Solonitcyn
6621732fa6 Improve URL creation in the CDK (#8522)
Updated version.

Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-06 10:27:03 +02:00
Yevhenii
afaa963a7f CDK: Fix 'emitted_at' from 'seconds * 1000' to correct milliseconds. (#8295) 2021-11-29 14:32:20 +02:00
Yevhenii
d64b1134d5 🐛 CDK: Add logger inheritance for streams from airbyte (#8040)
* Fix broken logger in streams: add logger inheritance for streams from airbyte
2021-11-17 11:37:03 +02:00
Dmytro
e56e86dd28 Fix schema transformation false warnings. (#7863) 2021-11-12 18:20:53 +02:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759)
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
2021-10-22 20:47:48 +03:00
vitaliizazmic
0c7f06018b 🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels

* Airbyte CDK native logger #1279 - add test for level types transforming

* CDK standard logger #1279 - test for transform critical level to fatal
2021-10-22 17:14:01 +03:00
George Claireaux
d660661b64 add google sheets oauth flow to server + fix auth rootObject type in protocol (#7131)
* adding google sheets oauth flow to server

* fix oauth type in protocol yaml

* bump sheets version in definitions

* added GDrive scope

* update sheets to master changes

* update protocol incl. cdk

* protocol typing for oauth rootobject

* format
2021-10-21 18:28:50 +01:00
vitaliizazmic
741001ae92 🐛 Airbyte CDK: fixing integration test failing
* Airbyte CDK native logger #1279 - fix import logger error

* Airbyte CDK native logger #1279 - source-paypal-transaction: change level from "WARN" to logging.WARN in self.logger.log

* Airbyte CDK native logger #1279 - source-s3: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - source-zuora: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - fix get logger
2021-10-18 18:46:55 +03:00
Vadym
3adf811726 CDK: add option to source to skip config validation on read/discover (#6978)
* Add connector check_config_against_spec parameter

* Bump CDK version.
Add changelog record.

* Always enable spec check on check cmd

* Update airbyte-cdk/python/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-14 18:31:00 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Yevhenii
a28b3f562b Bump Airbyte CDK version to 0.1.25 (#6511) 2021-09-28 19:05:34 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139)
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057)
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
2021-09-22 20:23:27 +03:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219)
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731)
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833)
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-09 19:55:29 +03:00
oleh.zorenko
58c9b84c3d 🐛 CDK: fix number of records mismatch (#5767)
* CDK: fix number of records mismatch

Increment `record_counter` after we check whether we reached the limit or not.

* Update docs. Bump version to 0.1.17
2021-09-02 09:16:43 +03:00
Sherif A. Nada
75a6a75b43 [protocol] add oauth params to connector specification (#5776) 2021-09-01 11:27:11 -07:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617)
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621)
* Fix facebook marketing source SAT fail on invalid config file
2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707)
* Python CDK: fix retry attempts in case of user defined backoff time
2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363)
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
2021-08-25 10:31:24 +03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457)
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223)
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c Populate the PK from the Singer discovery run (#2713) (#4789)
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
2021-08-09 20:11:22 +03:00