1
0
mirror of synced 2026-01-02 03:02:26 -05:00
Commit Graph

37 Commits

Author SHA1 Message Date
Brian Lai
be01b476ce Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472)
* write out new request options provider and refactor components and parts of the YAML config

* fix formatting

* pr feedback to consolidate body_data_provider to simplify the code

* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Alexandre Girard
1225f80bd8 Source hubspot: filter out streams we don't have access to (#13455)
* filter out streams we don't have access to

* cleanup

* fix associations

* fix for api key auth

* this doesn't need to be a property method

* Bump

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-03 12:06:51 -07:00
Denis Davydov
acd1533cec 🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections (#13260) 2022-05-27 22:05:39 +03:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Pedro S. Lopez
73c7fad7fc CDK: emit AirbyteTraceMessage with exception trace information (#12593) 2022-05-06 17:16:57 -04:00
Pedro S. Lopez
a6a6be2588 allow importing IncrementalMixin from airbyte_cdk.sources.streams (#11858)
* allow importing IncrementalMixin from airbyte_cdk.sources.streams

* format file

* update docs with hint on where to import IncrementalMixin from

* bump cdk version to 0.1.54

* update changelog

* only build python 3.9

* update changelog with python dep bump
2022-04-14 11:42:57 -07:00
Augustin
2d277747be cdk: remove false error logging in _send (#11650) 2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4 Python CDK - Fix BaseBackoffException constructor (#11653)
* - Fix BaseBackoffException constructor

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab CDK improve error handling responses (#11629)
* add response text and message to backoff error handling

* add response text and message to backoff error handling

* add response text case http error

* change response text to before raise error

* apply suggestions

* bump cdk version
2022-03-31 16:41:10 -03:00
LiRen Tu
049a11b2bc 🎉 Snowflake destination: reduce memory footprint (#10394)
* Add detailed logging for flushing

* Log sentry transaction event id

* Adjust logging

* Log memory usage

* Add jvm monitoring

* Remove log

* Remove port 9010

* Remove host network mode

* Sample record size

* Remove profiling code

* Add unit tests

* Use average estimation

* Rename variable

* Format code

* Bump version

* Revert unnecessary change

* Update doc

* Fix format

* Bump version in seed
2022-02-17 12:55:35 -08:00
Eugene Kulak
d173ce741f CDK: add support for streams with state attribute (#9746)
* add support for streams with state attribute
* fix pre-commit and format
* update state attribute docs and logic
* added IncrementalMixin

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2022-02-17 00:20:33 +04:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16 Integrate Sentry for performance and errors tracking. (#8248)
* Integrate Sentry for performance and errors tracking.

* Add sentry sensitive data scrubbing.

* updated cdk version and changelog

* Integrate Sentry for performance and errors tracking

Add `SENTRY_DSN` environment variable

* Integrate Sentry for performance and errors tracking

Add `sentry_sdk` to install requirements

* format cdk

* enable Sentry for google-search-console

* updated connector version

* update spec and source yamls

Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-21 20:02:02 +06:00
augan-rymkhan
7fbdfc61d0 Airbyte CDK: Log http status code and content in backoff handlers (#8829)
* Log http status code and content in default backoff handler

* Log http status code and content in usef defined backoff handler

* updated cdk version and changelog

* make it clear: exc.response

Co-authored-by: auganbay <auganenu@gmail.com>
2021-12-16 20:20:12 +06:00
Eugene Kulak
d50ae47b49 🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace (#8704)
* print stacktrace for unhandled exceptions

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
2021-12-14 00:04:29 +02:00
Sergei Solonitcyn
6ee922cef2 Improve URL creation in the CDK (#8513)
Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-05 22:30:16 +02:00
Yevhenii
d64b1134d5 🐛 CDK: Add logger inheritance for streams from airbyte (#8040)
* Fix broken logger in streams: add logger inheritance for streams from airbyte
2021-11-17 11:37:03 +02:00
LiRen Tu
c511abd20c Fix master build (#7873)
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
2021-11-11 09:47:51 -08:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759)
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
2021-10-22 20:47:48 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
2021-10-12 20:56:56 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139)
* Python CDK: add schema transformer class
2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7 correct CI applying gradlew format (#6390)
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057)
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
2021-09-22 20:23:27 +03:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219)
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
2021-09-22 07:45:05 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731)
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617)
* CDK: private configuration option _limit and _page_size
2021-08-31 12:16:48 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707)
* Python CDK: fix retry attempts in case of user defined backoff time
2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363)
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
2021-08-25 10:31:24 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223)
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-19 08:41:14 +03:00
Maksym Pavlenok
106c3cd641 🎉 CDK: Allow setting request non-JSON data (#5161)
* add the function request_body_data

* gradlew format

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add test for application/x-www-form-urlencoded

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-06 15:32:25 +03:00
Sherif A. Nada
3aebe55cd5 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests (#4493) 2021-07-07 15:17:58 -07:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Davin Chia
1d8e6da511 Various CDK build fixes/clean up. (#3312)
* Better MyPy command to not type check singer. Export UserDefinedException as that is used in tests.
2021-05-10 10:02:04 +08:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00