jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

1266 Commits

Author SHA1 Message Date
Dmytro
474b8f1065 CDK: support nested refs resolving (#6044) 
Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-16 18:38:08 +03:00
Vadym
4a0d364ea1 🎉 CDK: Add requests native authenticator support (#5731) 
* Add requests native auth class

* Update init file.
Update type annotations.
Bump version.

* Update TokenAuthenticator implementation.
Update Oauth2Authenticator implemetation.
Add CHANGELOG.md record.

* Update Oauth2Authenticator default value setting.
Update CHANGELOG.md

* Add requests native authenticator tests

* Add CDK requests native __call__ method tests.
Update CHANGELOG.md

* Add outdated auth deprication messages

* Update requests native auth __call__ method tests

* Bump CDK version to 0.1.20
 2021-09-15 19:23:31 +03:00
Marcos Eliziario Santos
1314a6a04c #5796 silence printing full config when config validation fails (#5879) 
* - #5796 silence printing full config when config validation fails

* fix unit tests after config validation check changes

Co-authored-by: Marcos Eliziario Santos <marcos@coremarcos.com>
 2021-09-14 11:00:47 -03:00
Dmytro
6041f3df37 🐛 CDK: fix bug with limit parameter for incremental stream (#5833) 
* CDK: fix bug with limit parameter for incremental stream

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-09-09 19:55:29 +03:00
Abhi Vaidyanatha
e7413de0dc Add a variant of the CDK speedrun that uses the PokeAPI (#5780) 
* Add a variant of the CDK speedrun that uses the PokeAPI

* Make a lot of clarifications to improve the learning experience.

* Gradle format

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-08 14:56:13 -07:00
Baz
9c05308d1f 🐛 Source Shopify: Fixed line_items/discount allocations & duties parts of orders schema (#5801) 
* edited orders schema
* added changelog
* formated

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
 2021-09-02 13:58:47 +03:00
oleh.zorenko
58c9b84c3d 🐛 CDK: fix number of records mismatch (#5767) 
* CDK: fix number of records mismatch

Increment `record_counter` after we check whether we reached the limit or not.

* Update docs. Bump version to 0.1.17
 2021-09-02 09:16:43 +03:00
Sherif A. Nada
75a6a75b43 [protocol] add oauth params to connector specification (#5776) 2021-09-01 11:27:11 -07:00
Dmytro
7584440515 CDK: private configuration option _limit and _page_size (#5617) 
* CDK: private configuration option _limit and _page_size
 2021-08-31 12:16:48 +03:00
Dmytro
3aabd92137 Fix facebook marketing source SAT fail on invalid config file (#5621) 
* Fix facebook marketing source SAT fail on invalid config file
 2021-08-31 11:48:03 +03:00
Dmytro
eed2e107fd Python CDK: fix retry attempts in case of user defined backoff time (#5707) 
* Python CDK: fix retry attempts in case of user defined backoff time
 2021-08-31 09:55:45 +03:00
Yaroslav Dudar
8ddce6f355 🎉 Python CDK: Allow to ignore http status errors and override retry parameters (#5363) 
added auto_fail_on_errors, max_retries, retry_factor properties to python cdk
 2021-08-25 10:31:24 +03:00
Marcos Marx
7225187fa1 run gradlew format (#5552) 2021-08-20 15:38:28 -03:00
Dmytro
b1f2bf5665 4776: Python CDK: Validate input config.py against spec (#5457) 
Python CDK: Validate input config.py against spec

Co-authored-by: Dmytro Rezchykov <dmitry.rezchykov@zazmic.com>
 2021-08-19 13:14:37 +03:00
Arthur Galuza
51d406d6f2 🎉 Source Github: Add MultipleTokenAuthenticator (#5223) 
* Add multiple token authenticator

* Add MultipleTokenAuthenticator

* Upd docs

* Refactor

* Upd docs

* Fix merge typo

* Upd multiple token support: switch to list of tokens

* Upd multiple token support: refactoring

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-integrations/connectors/source-github/source_github/spec.json

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Cleanup, switch to MultipleTokenAuthenticator completely

* Upd changelog

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-19 08:41:14 +03:00
Artjoms Iskovs
f790fee57c Populate the PK from the Singer discovery run (#2713) (#4789) 
When running Singer discovery, use the `key_properties` field to populate the `source_defined_primary_key` stream meta.
 2021-08-09 20:11:22 +03:00
LiRen Tu
2906ec287a CI: Add action to check broken doc links (#5254) 
* Add action to check broken doc links

* Ignore localhost

* Update config

* Fix broken links

* Use quiet mode

* Ignore PR link

* Fix more broken links

* Fix more broken links

* Fix more broken links

* Verify pattern

* Fix more broken links

* Separate full and pr check

* Update pattern

* Test invalid link

* Remove invalid link
 2021-08-07 14:28:02 -07:00
Maksym Pavlenok
106c3cd641 🎉 CDK: Allow setting request non-JSON data (#5161) 
* add the function request_body_data

* gradlew format

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/http.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update airbyte-cdk/python/airbyte_cdk/sources/streams/http/exceptions.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* add test for application/x-www-form-urlencoded

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-08-06 15:32:25 +03:00
Sherif A. Nada
f499026550 CDK: use approximate versions for dependencies (#5013) 2021-07-27 07:19:04 -07:00
Sherif A. Nada
9d8b1731b0 Remove outdated note about CDK not supporting destinations (#4995) 2021-07-26 13:52:42 -07:00
Sherif A. Nada
8e8b86b035 Update CHANGELOG.md 2021-07-19 20:54:57 -07:00
Eugene Kulak
d753dfce49 🐛 CDK: Fix logging of initial state value (#4795) 
* Update abstract_source.py

* bump

* CHANGELOG.md

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2021-07-18 07:34:03 -04:00
Sherif A. Nada
cb4fe7254c CDK: Add initial Destination abstraction and tests (#4719) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
 2021-07-13 16:18:08 -07:00
Sherif A. Nada
3aebe55cd5 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests (#4493) 2021-07-07 15:17:58 -07:00
Sherif A. Nada
21116cad97 📚 add doc to describe how CDK handles schemas (#4537) 2021-07-06 15:58:27 -07:00
Davin Chia
683a67c023 Upgrade Gradle Python plugin + explicitly install PyEnv to fix transient checkPython error. (#3362) 
This is what the plugin's maintainer suggested: xvik/gradle-use-python-plugin#19.

I ran the build 10 times in a row without any checkPython error so this does seem to have fixed it.
 2021-06-25 17:32:41 +08:00
Christophe Duong
7c26305865 Add supportsDBT and supportsNormalization to API objects (#4031) 
* Add supportsDBT and supportsNormalization to API objects
 2021-06-11 10:11:42 +02:00
Dmytro
d5d0c8b252 Replace npm install + generate -> ./generate.sh in docs (#4048) 2021-06-11 02:17:35 +03:00
Dmytro
66d96967df 🎉 Containerize connector code generator (#3763) 2021-06-08 12:25:27 +03:00
Sherif A. Nada
30c1e95d9e Add migration guide from legacy CDK to current CDK (#3860) 2021-06-03 22:40:01 -07:00
Subodh Kant Chaturvedi
6adad7d98e destination-specification: add supportsNormalization and supportsDBT attributes (#3779) 
* destination-specification: add supportsNormalization and supportsDBT attributes

* address review comment

* missed this one

* output after gradle format
 2021-06-01 17:40:28 +05:30
Yaroslav Dudar
c49ff40754 🎉 Python CDK: change minimum python version to 3.7.0 (#3692) 
* changed python version to 3.7.0 in cdk

* updated Python version in Github Actions
 2021-05-28 21:08:53 +03:00
Sherif A. Nada
fc45e8f165 Fix typo in CDK publish command format 2021-05-27 14:30:49 -07:00
Sherif A. Nada
137845a28e rename toy connector tutorial to "Build a connector the hard way" (#3421) 2021-05-17 13:31:25 -07:00
Sherif A. Nada
6beb4eb7e7 add npm install before all npm run generates' (#3442) 2021-05-17 09:39:37 -07:00
Abhi Vaidyanatha
576d932ddb Add a CDK speedrun tutorial doc (#3403) 
* Add CDK Speedrun document.

* Finish speedrun doc.

* Address review comments

* Add to SUMMARY.md

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-05-14 16:43:34 -07:00
Davin Chia
9460cb4422 Update all of Pydantic to 1.6.2 per Dependabot. (#3408) 
* Update all to 1.6.2.

* Publish new airbyte-cdk version.

* Use repr instead of str for exceptions.

* Use rc.

* Edit test.

* Bump for SAT.

* Format.
 2021-05-14 16:44:49 +08:00
Charles
0df53170c9 Stop formatting python with spotless (#3388) 2021-05-13 17:46:34 -07:00
Charles
a2f29f6024 fix formatting (#3370) 2021-05-11 15:46:59 -07:00
Sherif A. Nada
eb812eaecd Bump CDK consumers (#3368) 2021-05-11 15:27:44 -07:00
Sherif A. Nada
535d83e0af bugfix infinite pagination in CDK (#3366) 2021-05-11 14:45:27 -07:00
Davin Chia
08cd57e679 Release CDK 0.1.1. (#3360) 
Release CDK 0.1.1. Unfortunately I can't release 0.1.0 because we previously released it while testing and I didn't realise once a package is released on PyPi, a package with the same name/version cannot be released regardless of whether it's deleted or not.

Updated:
- source-exchange-rates
- source-github-singer
- source-slack
- source-smartsheets (this is at 0.1.4 since someone published 0.1.3 but the change was not checked in)
- source-stripe

Only source-github-singer and source-smartsheets need to be updated. The rest are either not already published or have singer equivalents present.
 2021-05-11 20:16:01 +08:00
Davin Chia
9a92da06b9 Update CDK Tutorial to use new CDK. (#3358) 
As the title suggests.

Some intricacies around how various directories have diverged.

The HTTP templates now use integration_tests to house catalogs and config files instead of sample_files. After this PR, everything should be updated to use integration_tests for all files except config files. Config files use secrets since they often contain secrets. Anything, besides actual implementations that have yet to be updated, not following this is my mistake.

Even though it diverges from the generated templates, I've decided to leave the tutorial code and the tutorial as using sample_files for now. I think it's more straightforward than telling people to create a new secrets directory. We can change this at a later date.

Confirmed the checked in http python tutorial code works by running ./gradlew :airbyte-integrations:connectors:source-python-http-tutorial:standardSourceTestFile.
 2021-05-11 16:52:13 +08:00
Davin Chia
94f6d17796 CDK: Update conceptual docs. Update template docs. (#3352) 2021-05-11 11:02:51 +08:00
Davin Chia
1d8e6da511 Various CDK build fixes/clean up. (#3312) 
* Better MyPy command to not type check singer. Export UserDefinedException as that is used in tests.
 2021-05-10 10:02:04 +08:00
Sherif A. Nada
184dab77eb CDK: overhaul directory structure (#3295) 2021-05-09 15:27:38 -07:00
Davin Chia
ca4c1b8821 CDK: Test Http.py (#3279) 2021-05-09 19:29:19 +08:00
Abhi Vaidyanatha
4a397d2524 Rewrite and break out CDK Python overview to be more concise and digestible (#3291) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-05-07 20:30:21 -07:00
Sherif A. Nada
f5a76e8448 Test AbstractSource read behavior (#3276) 2021-05-07 14:55:41 -07:00
Sherif A. Nada
efd429d58f CDK: Test AbstractSource: Check and Discover (#3272) 2021-05-07 13:11:57 -07:00