1
0
mirror of synced 2025-12-30 12:04:43 -05:00
Commit Graph

76 Commits

Author SHA1 Message Date
George Claireaux
4d279f8238 Remove legacy sentry code from cdk (#14016)
* rip sentry out of cdk

* remove sentry dsn from gsc
2022-06-23 12:14:09 +01:00
Alexandre Girard
57dd7a9d4b [low-code connectors] use jello instead of jq (#13570)
* use jello instead of jq

* reset

* rename

* remove print
2022-06-07 14:10:26 -07:00
Alexandre Girard
d9fa24ffff Low code connectors: implement components for sendgrid (#12853)
* checkout from alex/cac

* checkout from alex/cac

* checkout from alex/cac

* Add missing tests

* Add missing files

* Add missing tests

* add missing file

* missing file

* missing file

* rename

* doc

* doc

* remove broken test

* rename

* jinja dependency

* Add comment

* comment

* comment

* pyjq dependency

* rename file

* delete unused file

* Revert "delete unused file"

This reverts commit 758e939367.

* fix

* rename

* abstract property

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* isort

* update state

* fix imports

* update dependency

* format

* rename file

* decoder

* Use decoder

* Update comment

* dict_state is actually backed by a dict

* Add a comment

* update state takes kwargs

* move state out of offset paginator

* update jq parameter order

* update

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* remove print

* move test

* delete duplicates

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Allow passing a string or an enum

* Define StateType enum

* convert state_type if not of type type

* convert state_type if not of type type

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* Delete dead code

* rename methods

* rename to declarative

* rename the classes too

* Try to install packages to build jq

* isort

* only automake

* Revert "only automake"

This reverts commit c8fe154ffc.

* remove git

* format

* Add jq dependency

* Use request header provider

* rename

* rename field

* remove get_context method

* rename
2022-06-01 07:02:35 -07:00
Denis Davydov
acd1533cec 🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections (#13260) 2022-05-27 22:05:39 +03:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Alexandre Girard
150ab593f8 Low code connectors: core structure (#12850)
* checkout from alex/cac

* doc

* doc

* remove broken test

* rename

* rename file

* delete unused file

* rename

* abstract property

* isort

* update state

* Update comment

* remove incremental mixin

* delete comment

* update comments

* update comments

* remove no_state

* rename package

* pass parameters through kwargs

* update interface to pass source in interface

* update interface to pass source in interface

* rename to stream_slicer

* Low code connectors: string interpolation with jinja (#12852)

* checkout from alex/cac

* Add missing tests

* Add missing files

* missing file

* rename

* jinja dependency

* Add comment

* comment

* comment

* Revert "delete unused file"

This reverts commit 758e939367.

* delete unused field

* delete unused field

* rename

* pass kwargs directly

* isort

* Revert "isort"

This reverts commit 4a79223944.

* format

* decoder

* better error handling

* remove nostate

* isort

* delete dead code

* Update mapping type to [str, Any]

* add comment

* Add comment

* pass parameters through kwargs

* move test to right module

* Add missing test

* Use authbase instead of deprecated class

* leverage generator

* rename to declarative

* rename the classes too
2022-05-25 17:27:54 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Serhii Chvaliuk
9a960c4ca6 CDK: TypeTransformer.default_convert catch TypeError (#12948)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-05-19 07:32:48 +03:00
Pedro S. Lopez
7ea27d0ecd CDK: publish version 0.1.57 (#12938) 2022-05-17 16:33:06 -04:00
Pedro S. Lopez
73c7fad7fc CDK: emit AirbyteTraceMessage with exception trace information (#12593) 2022-05-06 17:16:57 -04:00
Sherif A. Nada
cc46255190 cull base-python and base-python-test (#12596) 2022-05-05 13:48:15 -07:00
Pedro S. Lopez
53799cb9a2 🎉 CDK: support loading spec from yaml file (#12104)
* support loading spec from yaml file

* formatting

* remove commented code

* update comment

* remove unused file

* raise correct exception types

* bump version, update changelog
2022-04-20 13:18:46 -07:00
Pedro S. Lopez
a6a6be2588 allow importing IncrementalMixin from airbyte_cdk.sources.streams (#11858)
* allow importing IncrementalMixin from airbyte_cdk.sources.streams

* format file

* update docs with hint on where to import IncrementalMixin from

* bump cdk version to 0.1.54

* update changelog

* only build python 3.9

* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Brian Leonard
c302af45ff Upgrade to Python 3.9 (#11763)
* Dockerfile to 3.9

* Python version

* More python updates

* 3.9 on GitHub actions and lint updates

* Test out 3.9.11 on GitHub actions

* install python with an action

* formatting: newline

* Also has python code

* only check first level for changed modules
Previous example (source-google-search-console/credentials)

* Test failure: there is no logger.trace
2022-04-11 20:51:37 -07:00
Augustin
2d277747be cdk: remove false error logging in _send (#11650) 2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4 Python CDK - Fix BaseBackoffException constructor (#11653)
* - Fix BaseBackoffException constructor

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab CDK improve error handling responses (#11629)
* add response text and message to backoff error handling

* add response text and message to backoff error handling

* add response text case http error

* change response text to before raise error

* apply suggestions

* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
vitaliizazmic
936050847f 🐛 CDK: fixing type errors (#9926)
* CDK #9500 - fixing type errors

* CDK #9500 - exclude singer from check

* CDK #9500 - type ignore in utils.schema_helpers.py

* CDK #9500 - type ignore in utils.sentry.py

* CDK #9500 - type ignore in singer.singer_helpers.py

* CDK #9500 - type ignore in singer.source

* CDK #9500 - fix according to PR review

* CDK #9500 - fix according to PR review

* CDK #9500 - bump version
2022-02-14 09:24:04 +02:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
Marcos Eliziario Santos
fd716d7a10 Adding unit tests to SAT full refresh test (#8589)
* WIP - Adding unit tests to SAT full refresh test

* unit tests to SAT full refresh test

* Format

* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_test_full_refresh.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* use symmetric_difference to check for differences between expected/actual when testing stream read on SAT

* bump version

* bump version

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-21 12:09:09 -03:00
Dmytro
fe954c1a16 Integrate Sentry for performance and errors tracking. (#8248)
* Integrate Sentry for performance and errors tracking.

* Add sentry sensitive data scrubbing.

* updated cdk version and changelog

* Integrate Sentry for performance and errors tracking

Add `SENTRY_DSN` environment variable

* Integrate Sentry for performance and errors tracking

Add `sentry_sdk` to install requirements

* format cdk

* enable Sentry for google-search-console

* updated connector version

* update spec and source yamls

Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-21 20:02:02 +06:00
augan-rymkhan
7fbdfc61d0 Airbyte CDK: Log http status code and content in backoff handlers (#8829)
* Log http status code and content in default backoff handler

* Log http status code and content in usef defined backoff handler

* updated cdk version and changelog

* make it clear: exc.response

Co-authored-by: auganbay <auganenu@gmail.com>
2021-12-16 20:20:12 +06:00
Eugene Kulak
d50ae47b49 🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace (#8704)
* print stacktrace for unhandled exceptions

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09 CDK: Add base pydantic model for connector config and schemas (#8485)
* add base spec model

* fix usage of state_checkpoint_interval in case it is dynamic

* add schema base models, fix spelling, signatures and polishing

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-08 01:14:59 +02:00
Marcos Eliziario Santos
34ff075ba7 flake8 fix 3764 obfuscate secrets logging (#8583)
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

* Format and flake8

* Fix build error/bump version/format

* Add airbyte prefix on test logger

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-07 11:35:50 -03:00
Marcos Eliziario Santos
1d493ec620 CDK: #3764 obfuscate secrets in logging (#8211)
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
2021-12-07 10:45:22 -03:00
augan-rymkhan
8b5303e94f added __init__.py file to utils (#8530)
* added __init__.py file to utils

* bump the version

Co-authored-by: Auganbay <auganenu@gmail.com>
2021-12-06 15:14:43 +02:00
Sergei Solonitcyn
6621732fa6 Improve URL creation in the CDK (#8522)
Updated version.

Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-06 10:27:03 +02:00
Yevhenii
afaa963a7f CDK: Fix 'emitted_at' from 'seconds * 1000' to correct milliseconds. (#8295) 2021-11-29 14:32:20 +02:00
Yevhenii
d64b1134d5 🐛 CDK: Add logger inheritance for streams from airbyte (#8040)
* Fix broken logger in streams: add logger inheritance for streams from airbyte
2021-11-17 11:37:03 +02:00
Dmytro
e56e86dd28 Fix schema transformation false warnings. (#7863) 2021-11-12 18:20:53 +02:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759)
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
2021-10-22 20:47:48 +03:00
vitaliizazmic
0c7f06018b 🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels

* Airbyte CDK native logger #1279 - add test for level types transforming

* CDK standard logger #1279 - test for transform critical level to fatal
2021-10-22 17:14:01 +03:00
George Claireaux
d660661b64 add google sheets oauth flow to server + fix auth rootObject type in protocol (#7131)
* adding google sheets oauth flow to server

* fix oauth type in protocol yaml

* bump sheets version in definitions

* added GDrive scope

* update sheets to master changes

* update protocol incl. cdk

* protocol typing for oauth rootobject

* format
2021-10-21 18:28:50 +01:00
vitaliizazmic
741001ae92 🐛 Airbyte CDK: fixing integration test failing
* Airbyte CDK native logger #1279 - fix import logger error

* Airbyte CDK native logger #1279 - source-paypal-transaction: change level from "WARN" to logging.WARN in self.logger.log

* Airbyte CDK native logger #1279 - source-s3: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - source-zuora: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - fix get logger
2021-10-18 18:46:55 +03:00
Vadym
3adf811726 CDK: add option to source to skip config validation on read/discover (#6978)
* Add connector check_config_against_spec parameter

* Bump CDK version.
Add changelog record.

* Always enable spec check on check cmd

* Update airbyte-cdk/python/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-14 18:31:00 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Yevhenii
a28b3f562b Bump Airbyte CDK version to 0.1.25 (#6511) 2021-09-28 19:05:34 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139)
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057)
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
2021-09-22 20:23:27 +03:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219)
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044)
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731)
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833)
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-09-09 19:55:29 +03:00
oleh.zorenko
58c9b84c3d 🐛 CDK: fix number of records mismatch (#5767)
* CDK: fix number of records mismatch

Increment `record_counter` after we check whether we reached the limit or not.

* Update docs. Bump version to 0.1.17
2021-09-02 09:16:43 +03:00
Sherif A. Nada
75a6a75b43 [protocol] add oauth params to connector specification (#5776) 2021-09-01 11:27:11 -07:00