1
0
mirror of synced 2026-01-01 09:02:59 -05:00
Commit Graph

37 Commits

Author SHA1 Message Date
JJ Nilbodee
33227f5812 Allows user to specify grant_type in OAuthAuthenticator (#19139)
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-11-10 22:22:17 -08:00
Alexandre Girard
1a608f846a Stream returns AirbyteMessages (#18572)
* method yielding airbytemessage

* move to Stream

* update abstract source

* reset

* missing file

* add test docker image

* script to run acceptance tests with local cdk

* Update conftest to use a different image

* extract to method

* dont use a different image tag

* Always install local cdk

* break the cdk

* get path from current working directory

* or

* ignore unit test

* debug log

* Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)"

This reverts commit bf06decf73.

* build from the top

* Update source-acceptance-test

* fix

* copy setup

* some work on the gradle plugin

* reset to master

* delete unused file

* delete unused file

* reset to master

* optional argument

* delete dead code

* use latest cdk with sendgrid

* fix sendgrid dockerfile

* break the cdk

* use local file

* Revert "break the cdk"

This reverts commit 600c195541.

* dont raise an exception

* reset to master

* unit tests

* missing test

* more unit tests

* newline

* reset to master

* remove files

* reset

* Update abstract source

* remove method from stream

* convert to airbytemessage

* unittests

* Update

* unit test

* remove debug logs

* Revert "remove debug logs"

This reverts commit a1a139ef37.

* Revert "Revert "remove debug logs""

This reverts commit b1d62cdb60.

* Revert "reset to master"

This reverts commit 3fa6a004c1.

* Revert "Revert "reset to master""

This reverts commit 5dac7c2804.

* reset to master

* reset to master

* test

* Revert "test"

This reverts commit 2f91b803b0.

* test

* Revert "test"

This reverts commit 62d95ebbb5.

* test

* Revert "test"

This reverts commit 27150ba341.

* format
2022-11-07 21:23:11 -08:00
Serhii Chvaliuk
258b23cf79 CDK: VCR -> requests_cache + SQLite (#17990)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-10-19 19:31:28 +03:00
Brian Lai
bd31100774 initial first pass converting every component to dataclasses (#15189)
* initial first pass converting every component to dataclasses

* replace the hackier options pass through

* get rid of the hackier way

* fix issues w/ type hints by making options required and lots of fixes to the language to fix compatability for dataclasses

* add dataclasses-jsonschema to setup

* fix oauth authenticator to avoid dataclass name collisions

* fix spacing for CI tests

* remove property from oauth and fix a interpolation bug

* pr feedback and cleaning up the code a bit, attempt at avoiding renaming

* fix templates and bugs surfaced during greenhouse spec testing

* fix tests

* fix missing options in some declarative components

* fix tests related to pulling latest master

* fix issue w/ passing state, slice, and token to subcomponents

* switch name back to get_access_token() since no name collision anymore
2022-08-05 17:39:27 -04:00
Alexandre Girard
783923db76 [low-code CDK] Enable runtime string interpolation in authenticators (#14914)
* interpolatedauth

* fix tests

* fix import

* no need for default

* Bump version

* Missing docstrings

* example

* missing example

* more docstrings

* interpolated types
2022-07-25 19:04:05 -07:00
Brian Lai
7bff12aea5 [#3078] [CDK] Add support for enabling debug from command line and some basic general debug logs (#14521)
* allow for command line debug option and basic debug statements + declarative

* feedback from pr comments

* fix some tests w/ req/res mixed up and fixing logging tests

* formatting

* pr feedback: cleaning up traces in logger.py and update docs with debug configuration

* remove unneeded trace logger test

* remove extra print statement
2022-07-13 18:01:07 -04:00
Brian Lai
374e265fcb [Low Code CDK] configurable oauth request payload (#13993)
* configurable oauth request payload

* support interpolation for dictionaries that are not new subcomponents

* rewrite a declarative oauth authenticator that performs interpolation at runtime

* formatting

* whatever i don't know why factory gets flagged w/ the newline change

* we java now

* remove duplicate oauth

* add some comments

* parse time properly from string interpolation

* move declarative oauth to its own package in declarative module

* add changelog info
2022-07-08 16:49:16 -04:00
jordan-glitch
4395fbb4d2 cdk: Attach namespace to stream in catalog (#13923) 2022-07-04 18:04:31 +02:00
Alexandre Girard
94abef3acc [low-code connectors] BasicHttpAuthenticator (#13733)
* implement basichttpauthenticator

* add optional refresh access token authenticator

* remove prints

* type hints

* Fix and unit test

* missing test

* Add class to __init__ file

* Add comment
2022-06-22 07:35:32 -07:00
Brian Lai
be01b476ce Add new InterpolatedRequestOptionsProvider that encapsulates all variations of request arguments (#13472)
* write out new request options provider and refactor components and parts of the YAML config

* fix formatting

* pr feedback to consolidate body_data_provider to simplify the code

* pr feedback get rid of extraneous optional
2022-06-21 16:01:05 -04:00
Pedro S. Lopez
4f8f979045 Fix Connectors Base Build: CDK tests fail after dependency update (#13652) 2022-06-09 16:05:36 -04:00
Denis Davydov
acd1533cec 🎉 Python CDK: handle requests.exceptions.ChunkedEncodingError for broken connections (#13260) 2022-05-27 22:05:39 +03:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Pedro S. Lopez
41dc82f056 CDK: provide better user-friendly error messages (#12944)
* initial implementation

* add parse_response_error_message tests

* move error handler to existing try/catch in AbstractSource

* formatting

* var rename

* use isinstance for type checking

* add docstrings

* add more abstract_source and httpstream tests

* fix wrong httperror usage

* more test cases

* bump version, update changelog
2022-05-19 17:28:24 -04:00
Augustin
2d277747be cdk: remove false error logging in _send (#11650) 2022-04-04 10:36:05 -07:00
Serhii Chvaliuk
df12c9e4d4 Python CDK - Fix BaseBackoffException constructor (#11653)
* - Fix BaseBackoffException constructor

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-04-01 19:05:11 +03:00
Marcos Marx
78cafc58ab CDK improve error handling responses (#11629)
* add response text and message to backoff error handling

* add response text and message to backoff error handling

* add response text case http error

* change response text to before raise error

* apply suggestions

* bump cdk version
2022-03-31 16:41:10 -03:00
Eugene Kulak
f83eca58ea CDK: Fix typing errors (#9037)
* fix typing, drop AirbyteLogger

* format

* bump the version

* use logger instead of fixture logger

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: auganbay <auganenu@gmail.com>
2022-01-14 10:29:34 +06:00
Dmytro
fe954c1a16 Integrate Sentry for performance and errors tracking. (#8248)
* Integrate Sentry for performance and errors tracking.

* Add sentry sensitive data scrubbing.

* updated cdk version and changelog

* Integrate Sentry for performance and errors tracking

Add `SENTRY_DSN` environment variable

* Integrate Sentry for performance and errors tracking

Add `sentry_sdk` to install requirements

* format cdk

* enable Sentry for google-search-console

* updated connector version

* update spec and source yamls

Co-authored-by: auganbay <auganenu@gmail.com>
Co-authored-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
2021-12-21 20:02:02 +06:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Marcos Marx
5861b6faf7 correct CI applying gradlew format (#6390)
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057)
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063 Format code (#6381) 2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219)
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
2021-09-22 07:45:05 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731)
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
2021-09-15 19:23:31 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707)
* Python CDK: fix retry attempts in case of user defined backoff time
2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363)
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
2021-08-25 10:31:24 +03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457)
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223)
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-19 08:41:14 +03:00
Maksym Pavlenok
106c3cd641 🎉 CDK: Allow setting request non-JSON data (#5161)
* add the function request_body_data

* gradlew format

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add test for application/x-www-form-urlencoded

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-08-06 15:32:25 +03:00
Sherif A. Nada
3aebe55cd5 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests (#4493) 2021-07-07 15:17:58 -07:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Charles
a2f29f6024 fix formatting (#3370) 2021-05-11 15:46:59 -07:00
Sherif A. Nada
eb812eaecd Bump CDK consumers (#3368) 2021-05-11 15:27:44 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Davin Chia
1d8e6da511 Various CDK build fixes/clean up. (#3312)
* Better MyPy command to not type check singer. Export UserDefinedException as that is used in tests.
2021-05-10 10:02:04 +08:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00