jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-02 12:02:47 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,265 Commits 5,149 Branches 518 Tags
e4d3d60ca8b683c301dfd5408917e1bb5e261cc0
Commit Graph

149 Commits

Author SHA1 Message Date
Eugene Kulak
d50ae47b49 🐛 CDK: Fix the logging of unhandled exceptions to show stacktrace (#8704) 
* print stacktrace for unhandled exceptions

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-12-14 00:04:29 +02:00
Eugene Kulak
aa67604f09 CDK: Add base pydantic model for connector config and schemas (#8485) 
* add base spec model

* fix usage of state_checkpoint_interval in case it is dynamic

* add schema base models, fix spelling, signatures and polishing

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-12-08 01:14:59 +02:00
Marcos Eliziario Santos
34ff075ba7 flake8 fix 3764 obfuscate secrets logging (#8583) 
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

* Format and flake8

* Fix build error/bump version/format

* Add airbyte prefix on test logger

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
 2021-12-07 11:35:50 -03:00
Marcos Eliziario Santos
1d493ec620 CDK: #3764 obfuscate secrets in logging (#8211) 
* Secure logger implementation minus still broken new tests

* Secure logger implementation and unit tests

* code review changes

* filter text on uncaught exceptions

* auto-formatting

* Mistaken change left in code

* filter text on uncaught exceptions

* Simplify code, remove LoggingFilter and move obfuscation functionality to Formatter

* Update airbyte-cdk/python/airbyte_cdk/entrypoint.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Obfuscate Secrets in Logging, code review changes

* Obfuscate Secrets in Logging, code review changes, unit test fixes

* CHANGELOG.md

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
 2021-12-07 10:45:22 -03:00
augan-rymkhan
8b5303e94f added __init__.py file to utils (#8530) 
* added __init__.py file to utils

* bump the version

Co-authored-by: Auganbay <auganenu@gmail.com>
 2021-12-06 15:14:43 +02:00
Sergei Solonitcyn
6621732fa6 Improve URL creation in the CDK (#8522) 
Updated version.

Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
 2021-12-06 10:27:03 +02:00
Sergei Solonitcyn
6ee922cef2 Improve URL creation in the CDK (#8513) 
Signed-off-by: Sergei Solonitcyn <sergei.solonitcyn@zazmic.com>
 2021-12-05 22:30:16 +02:00
Marcos Eliziario Santos
65d471abf4 Output run time per stream (#8173) 
* Output run time per stream

* Code review changes: Remove multithreaded support.
Port code to cdk

* Formatting

* remove extraneous try block
 2021-11-30 10:20:51 -03:00
Yevhenii
afaa963a7f CDK: Fix 'emitted_at' from 'seconds * 1000' to correct milliseconds. (#8295) 2021-11-29 14:32:20 +02:00
Yevhenii
d64b1134d5 🐛 CDK: Add logger inheritance for streams from airbyte (#8040) 
* Fix broken logger in streams: add logger inheritance for streams from airbyte
 2021-11-17 11:37:03 +02:00
Christophe Duong
fa040da084 Implement protocol change for OAuth outputs (#7917) 
* Change OAuth API

* Change protocol for new OAuthConfigSpecification

* Refactor OAuth classes and tests

* Remove webbackend source/destination creation

* Change from webback to normal API

* Implement new protocol change with OAuth specs

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* format

* format

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-11-15 14:36:50 +01:00
Christophe Duong
cdb476ecad Add auth config to oauth api (#7798) 
* Change OAuth API

* Change protocol for new OAuth Spec (#7827)

* Add examples

* Add protocol object to api too

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-11-15 12:56:48 +01:00
Dmytro
e56e86dd28 Fix schema transformation false warnings. (#7863) 2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c Fix master build (#7873) 
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
 2021-11-11 09:47:51 -08:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759) 
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
 2021-10-22 20:47:48 +03:00
vitaliizazmic
0c7f06018b 🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels 
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels

* Airbyte CDK native logger #1279 - add test for level types transforming

* CDK standard logger #1279 - test for transform critical level to fatal
 2021-10-22 17:14:01 +03:00
George Claireaux
d660661b64 add google sheets oauth flow to server + fix auth rootObject type in protocol (#7131) 
* adding google sheets oauth flow to server

* fix oauth type in protocol yaml

* bump sheets version in definitions

* added GDrive scope

* update sheets to master changes

* update protocol incl. cdk

* protocol typing for oauth rootobject

* format
 2021-10-21 18:28:50 +01:00
Jared Rhizor
1affb14827 even more link fixes (#7148) 
* even more link fixes

* add remaining exchangerates items
 2021-10-20 12:56:55 -07:00
LiRen Tu
e3be3d2ad0 Update M1 doc and fix doc links (#7205) 
* Add note about jdk 14 vs 17 for m1

* Fix doc links
 2021-10-19 22:31:16 -07:00
vitaliizazmic
741001ae92 🐛 Airbyte CDK: fixing integration test failing 
* Airbyte CDK native logger #1279 - fix import logger error

* Airbyte CDK native logger #1279 - source-paypal-transaction: change level from "WARN" to logging.WARN in self.logger.log

* Airbyte CDK native logger #1279 - source-s3: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - source-zuora: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - fix get logger
 2021-10-18 18:46:55 +03:00
Vadym
3adf811726 CDK: add option to source to skip config validation on read/discover (#6978) 
* Add connector check_config_against_spec parameter

* Bump CDK version.
Add changelog record.

* Always enable spec check on check cmd

* Update airbyte-cdk/python/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-10-14 18:31:00 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler 
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
 2021-10-12 20:56:56 +03:00
Christophe Duong
960214a48a Change Airbyte Protocol for OAuth and Make protocol more permissive (#6542) 2021-09-29 16:27:10 +02:00
Yevhenii
a28b3f562b Bump Airbyte CDK version to 0.1.25 (#6511) 2021-09-28 19:05:34 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Sherif A. Nada
89dccf3897 Modify oauthSpecification to allow working with oneOfs (#6456) 2021-09-27 18:00:36 +02:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139) 
* Python CDK: add schema transformer class
 2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7 correct CI applying gradlew format (#6390) 
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
 2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057) 
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
 2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063 Format code (#6381) 2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219) 
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
 2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044) 
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731) 
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
 2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c #5796 silence printing full config when config validation fails (#5879) 
* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
 2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833) 
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-09 19:55:29 +03:00
Abhi Vaidyanatha
e7413de0dc Add a variant of the CDK speedrun that uses the PokeAPI (#5780) 
* Add a variant of the CDK speedrun that uses the PokeAPI

* Make a lot of clarifications to improve the learning experience.

* Gradle format

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-08 14:56:13 -07:00
Baz
9c05308d1f 🐛 Source Shopify: Fixed line_items/discount allocations & duties parts of orders schema (#5801) 
* edited orders schema
* added changelog
* formated

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
 2021-09-02 13:58:47 +03:00
oleh.zorenko
58c9b84c3d 🐛 CDK: fix number of records mismatch (#5767) 
* CDK: fix number of records mismatch

Increment `record_counter` after we check whether we reached the limit or not.

* Update docs. Bump version to 0.1.17
 2021-09-02 09:16:43 +03:00
Sherif A. Nada
75a6a75b43 [protocol] add oauth params to connector specification (#5776) 2021-09-01 11:27:11 -07:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617) 
* CDK: private configuration option _limit and _page_size
 2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621) 
* Fix facebook marketing source SAT fail on invalid config file
 2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707) 
* Python CDK: fix retry attempts in case of user defined backoff time
 2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363) 
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
 2021-08-25 10:31:24 +03:00
Marcos Marx
7225187fa1 run gradlew format (#5552) 2021-08-20 15:38:28 -03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457) 
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223) 
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c Populate the PK from the Singer discovery run (#2713) (#4789) 
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
 2021-08-09 20:11:22 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00