jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-01 09:02:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,968 Commits 5,135 Branches 518 Tags
9f6963ccfcddd7a573c4ca0ff94de41ce6175cc0
Commit Graph

538 Commits

Author SHA1 Message Date
Christophe Duong
cdb476ecad Add auth config to oauth api (#7798) 
* Change OAuth API

* Change protocol for new OAuth Spec (#7827)

* Add examples

* Add protocol object to api too

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-11-15 12:56:48 +01:00
Dmytro
e56e86dd28 Fix schema transformation false warnings. (#7863) 2021-11-12 18:20:53 +02:00
LiRen Tu
c511abd20c Fix master build (#7873) 
* Check in format diff

* Remove cache http yaml files

* Ignore tmp files generated by unit tests
 2021-11-11 09:47:51 -08:00
Eugene Kulak
319492791e CDK: fix logging configuration inside source and streams (#7808) 2021-11-11 16:07:38 +02:00
Dmytro
16b626cc91 Resolve discover catalog ref fields (#7734) 2021-11-09 12:42:15 +02:00
Yevhenii
5eb2af6c08 CDK: Autogenerate reference docs (#4759) 
* CDK Autogenerated reference docs: base version

* Update docs config.
Add .readthedocs.yaml file.
Update build html files.

* Update .gitignore.
Remove sphinx build files.

* Add newline at the end of .gitignore

* Update setup.py requirements.
Update .readthedocs.yaml config.

* Update rst files.
Add Makefile rst config.

* Update CDK docstring format.
Change rst layout.
Update sphinx config.

* Add Sphinx docs.
Update index.rst.
Update abstract_source.py docstrings.

* Override master_doc and package templates.
Add docs schema enerator script.
Update sphinx docs.

* Add `Publishing to Read the Docs` section to sphinx-docs.md".
Replace sphinx-docs.md to `airbyte-cdk` module.

* Update sphinx-docs.md section name

* Bump airbyte-cdk version.
Update CHANGELOG.md.

Co-authored-by: ykurochkin <y.kurochkin@zazmic.com>
Co-authored-by: Vadym Hevlich <vege1wgw@gmail.com>
 2021-10-22 20:47:48 +03:00
vitaliizazmic
0c7f06018b 🐛 Airbyte CDK: transforming Python log levels to Airbyte protocol log levels 
* Airbyte CDK native logger #1279 - transforming Python log levels to Airbyte protocol log levels

* Airbyte CDK native logger #1279 - add test for level types transforming

* CDK standard logger #1279 - test for transform critical level to fatal
 2021-10-22 17:14:01 +03:00
George Claireaux
d660661b64 add google sheets oauth flow to server + fix auth rootObject type in protocol (#7131) 
* adding google sheets oauth flow to server

* fix oauth type in protocol yaml

* bump sheets version in definitions

* added GDrive scope

* update sheets to master changes

* update protocol incl. cdk

* protocol typing for oauth rootobject

* format
 2021-10-21 18:28:50 +01:00
Jared Rhizor
1affb14827 even more link fixes (#7148) 
* even more link fixes

* add remaining exchangerates items
 2021-10-20 12:56:55 -07:00
LiRen Tu
e3be3d2ad0 Update M1 doc and fix doc links (#7205) 
* Add note about jdk 14 vs 17 for m1

* Fix doc links
 2021-10-19 22:31:16 -07:00
vitaliizazmic
741001ae92 🐛 Airbyte CDK: fixing integration test failing 
* Airbyte CDK native logger #1279 - fix import logger error

* Airbyte CDK native logger #1279 - source-paypal-transaction: change level from "WARN" to logging.WARN in self.logger.log

* Airbyte CDK native logger #1279 - source-s3: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - source-zuora: use native logger instead AirbyteLogger

* Airbyte CDK native logger #1279 - fix get logger
 2021-10-18 18:46:55 +03:00
Vadym
3adf811726 CDK: add option to source to skip config validation on read/discover (#6978) 
* Add connector check_config_against_spec parameter

* Bump CDK version.
Add changelog record.

* Always enable spec check on check cmd

* Update airbyte-cdk/python/CHANGELOG.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-10-14 18:31:00 +03:00
vitaliizazmic
d5c04991bc 🎉 CDK: use standard logger with custom handler 
* CDK - using native logger with custom formatter

* CDK - using native logger(fix exception and add trace)

* CDK - using native logger in AirbyteEntrypoint

* CDK - CHANGELOG.md

* CDK - remove unnecessary imports

* CDK - fixing according to PR review

* CDK native logger #1279 - annotations

* CDK native logger #1279 - fixing according to PR review

* CDK standard logger #1279 - tests

* CDK standard logger #1279 - reformat

* Airbyte CDK #1279 - improve docstrings

* Airbyte CDK #1279 - improve log levels

* Airbyte CDK #1279 - fix init get source name

* Airbyte CDK #1279 - update test licence

* Airbyte CDK #1279 - bump version
 2021-10-12 20:56:56 +03:00
Christophe Duong
960214a48a Change Airbyte Protocol for OAuth and Make protocol more permissive (#6542) 2021-09-29 16:27:10 +02:00
Yevhenii
a28b3f562b Bump Airbyte CDK version to 0.1.25 (#6511) 2021-09-28 19:05:34 +03:00
Michel Tricot
1773e41e47 Shorten our headers + adds contributors file (#6478) 2021-09-27 10:45:50 -07:00
Sherif A. Nada
89dccf3897 Modify oauthSpecification to allow working with oneOfs (#6456) 2021-09-27 18:00:36 +02:00
Dmytro
1c5ac5b5ea 🏗️ Python CDK: add schema transformer class (#6139) 
* Python CDK: add schema transformer class
 2021-09-27 13:07:17 +03:00
Marcos Marx
5861b6faf7 correct CI applying gradlew format (#6390) 
Co-authored-by: Marcos Marx <marcosmarx@MacBook-Pro-de-Marcos.local>
 2021-09-23 01:55:06 -03:00
Arthur Galuza
9aa5a5a52d 🎉 CDK: Added support for efficient parent/child streams using cache (#6057) 
* Add caching

* Upd cache file handling

* Upd slices, sync mode, docs

* Bump version

* Use SyncMode.full_refresh for parent stream_slices

* Refactor
 2021-09-22 20:23:27 +03:00
LiRen Tu
548a3a3063 Format code (#6381) 2021-09-22 09:30:14 -07:00
oleh.zorenko
4dca32713b 🎉 CDK: Add support for custom headers passing to the request in OAuth2Authenticator. refresh_access_token (#6219) 
* Add support for headers to OAuth2Authenticator

Send custom headers in `refresh_access_token()`.

* Bump version + update CHANGELOG.md

* Add tests

* Update tests for refresh_access_token()

* Assert that mock_refresh_token_call was called

* Remove init file
 2021-09-22 07:45:05 +03:00
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044) 
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731) 
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
 2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c #5796 silence printing full config when config validation fails (#5879) 
* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
 2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833) 
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-09 19:55:29 +03:00
Abhi Vaidyanatha
e7413de0dc Add a variant of the CDK speedrun that uses the PokeAPI (#5780) 
* Add a variant of the CDK speedrun that uses the PokeAPI

* Make a lot of clarifications to improve the learning experience.

* Gradle format

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-08 14:56:13 -07:00
Baz
9c05308d1f 🐛 Source Shopify: Fixed line_items/discount allocations & duties parts of orders schema (#5801) 
* edited orders schema
* added changelog
* formated

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
 2021-09-02 13:58:47 +03:00
oleh.zorenko
58c9b84c3d 🐛 CDK: fix number of records mismatch (#5767) 
* CDK: fix number of records mismatch

Increment `record_counter` after we check whether we reached the limit or not.

* Update docs. Bump version to 0.1.17
 2021-09-02 09:16:43 +03:00
Sherif A. Nada
75a6a75b43 [protocol] add oauth params to connector specification (#5776) 2021-09-01 11:27:11 -07:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617) 
* CDK: private configuration option _limit and _page_size
 2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621) 
* Fix facebook marketing source SAT fail on invalid config file
 2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707) 
* Python CDK: fix retry attempts in case of user defined backoff time
 2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363) 
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
 2021-08-25 10:31:24 +03:00
Marcos Marx
7225187fa1 run gradlew format (#5552) 2021-08-20 15:38:28 -03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457) 
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223) 
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c Populate the PK from the Singer discovery run (#2713) (#4789) 
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
 2021-08-09 20:11:22 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
Maksym Pavlenok
106c3cd641 🎉 CDK: Allow setting request non-JSON data (#5161) 
* add the function request_body_data

* gradlew format

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add test for application/x-www-form-urlencoded

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-06 15:32:25 +03:00
Sherif A. Nada
f499026550 CDK: use approximate versions for dependencies (#5013) 2021-07-27 07:19:04 -07:00
Sherif A. Nada
9d8b1731b0 Remove outdated note about CDK not supporting destinations (#4995) 2021-07-26 13:52:42 -07:00
Sherif A. Nada
8e8b86b035 Update CHANGELOG.md 2021-07-19 20:54:57 -07:00
Eugene Kulak
d753dfce49 🐛 CDK: Fix logging of initial state value (#4795) 
* Update abstract_source.py

* bump

* CHANGELOG.md

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-18 07:34:03 -04:00
Sherif A. Nada
cb4fe7254c CDK: Add initial Destination abstraction and tests (#4719) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
 2021-07-13 16:18:08 -07:00
Sherif A. Nada
3aebe55cd5 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests (#4493) 2021-07-07 15:17:58 -07:00
Sherif A. Nada
21116cad97 📚 add doc to describe how CDK handles schemas (#4537) 2021-07-06 15:58:27 -07:00
Davin Chia
683a67c023 Upgrade Gradle Python plugin + explicitly install PyEnv to fix transient checkPython error. (#3362) 
This is what the plugin's maintainer suggested: xvik/gradle-use-python-plugin#19.

I ran the build 10 times in a row without any checkPython error so this does seem to have fixed it.
 2021-06-25 17:32:41 +08:00
Christophe Duong
7c26305865 Add supportsDBT and supportsNormalization to API objects (#4031) 
* Add supportsDBT and supportsNormalization to API objects
 2021-06-11 10:11:42 +02:00
Dmytro
d5d0c8b252 Replace npm install + generate -> ./generate.sh in docs (#4048) 2021-06-11 02:17:35 +03:00