jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-02 12:02:47 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,149 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

291 Commits

Author SHA1 Message Date
LiRen Tu
c511abd20c Fix master build (#7873) 
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
 2021-11-11 09:47:51 -08:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
vitaliizazmic
0c7f06018b 🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels 
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels

* Airbyte CDK native logger #1279 - add test for level types transforming

* CDK standard logger #1279 - test for transform critical level to fatal
 2021-10-22 17:14:01 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler 
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
 2021-10-12 20:56:56 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139) 
* Python CDK: add schema transformer class
 2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7 correct CI applying gradlew format (#6390) 
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
 2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057) 
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
 2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063 Format code (#6381) 2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219) 
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
 2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044) 
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731) 
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
 2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c #5796 silence printing full config when config validation fails (#5879) 
* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
 2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833) 
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-09 19:55:29 +03:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617) 
* CDK: private configuration option _limit and _page_size
 2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621) 
* Fix facebook marketing source SAT fail on invalid config file
 2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707) 
* Python CDK: fix retry attempts in case of user defined backoff time
 2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363) 
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
 2021-08-25 10:31:24 +03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457) 
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223) 
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c Populate the PK from the Singer discovery run (#2713) (#4789) 
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
 2021-08-09 20:11:22 +03:00
Maksym Pavlenok
106c3cd641 🎉 CDK: Allow setting request non-JSON data (#5161) 
* add the function request_body_data

* gradlew format

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add test for application/x-www-form-urlencoded

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-06 15:32:25 +03:00
Sherif A. Nada
cb4fe7254c CDK: Add initial Destination abstraction and tests (#4719) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
 2021-07-13 16:18:08 -07:00
Sherif A. Nada
3aebe55cd5 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests (#4493) 2021-07-07 15:17:58 -07:00
Davin Chia
9460cb4422 Update all of Pydantic to 1.6.2 per Dependabot. (#3408) 
* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.
 2021-05-14 16:44:49 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Charles
a2f29f6024 fix formatting (#3370) 2021-05-11 15:46:59 -07:00
Sherif A. Nada
eb812eaecd Bump CDK consumers (#3368) 2021-05-11 15:27:44 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Davin Chia
1d8e6da511 Various CDK build fixes/clean up. (#3312) 
* Better MyPy command to not type check singer. Export UserDefinedException as that is used in tests.
 2021-05-10 10:02:04 +08:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00
Davin Chia
ca4c1b8821 CDK: Test Http.py (#3279) 2021-05-09 19:29:19 +08:00
Sherif A. Nada
f5a76e8448 Test AbstractSource read behavior (#3276) 2021-05-07 14:55:41 -07:00
Sherif A. Nada
efd429d58f CDK: Test AbstractSource: Check and Discover (#3272) 2021-05-07 13:11:57 -07:00
Sherif A. Nada
715dfc5e34 Test AirbyteEntrypoint.run (#3261) 2021-05-07 22:13:03 +08:00
Sherif A. Nada
1fb6b9f911 CDK: test catalog helpers (#3271) 
* test catalog helpers

* Format.

Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-05-07 22:12:52 +08:00
Sherif A. Nada
f88e5de903 Test Source Entrypoint Arg Parsing (#3259) 2021-05-06 15:34:55 -07:00
Sherif A. Nada
e856502455 CDK: Test integration.py (#3243) 
* integration.py tests

* refactor integration.py

Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-05-06 18:25:10 +08:00
Davin Chia
b5c2329748 CDK: Add OAuth and Streams/Core tests. (#3228) 
Completely test the OAuth module and streams/core.py file.

Add pytest-mock as well as a convenience wrapper to run the type checker and the test together when developing.

Also configure pytest to pick up loggers at the INFO level and above, and pick up standard out for easier debugging.

Some of the script bits are unpolished - will improve them as we go along.
 2021-05-06 15:52:23 +08:00
Sherif A. Nada
c3356684dd Test JSONSchema resolver (#3223) 
closes #2722

I don't love the current structure. There is a number of improvements that can be made:

JsonSchemaResolver can be refactored to have the following signature:

def __init__(schemas: Mapping[str, any], refs: Mapping[str, any])
  # schemas are all json files in `schemas/`, refs are all json files in `schemas/shared/`
Then we can thoroughly test the resolving behavior in JsonSchemaResolver independently from the logic for where we actually load schemas from.

I was tempted to make this refactor now but I'd rather cover more ground with tests then revisit. I created an issue to track this: #3222
 2021-05-05 09:51:24 +08:00
Davin Chia
fa33c9c126 Initial CDK Test set up and one test file. (#3207) 
Co-authored-by: Sherif Nada <snadalive@gmail.com>
 2021-05-04 14:27:58 -07:00