jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-31 19:01:59 -05:00
Code Issues Packages Projects Releases Wiki Activity
15,541 Commits 5,438 Branches 518 Tags
b60bbc72fdfd195b700236b58490f5c71857edcd
Commit Graph

98 Commits

Author SHA1 Message Date
Maxime Carbonneau-Leclerc
48af92ad78 Concurrent CDK: if exception is AirbyteTracedException, raise this an… (#37443) 2024-04-19 20:32:46 +00:00
Ella Rohm-Ensing
0439cbcd10 Clarify cdk parity test behavior (#37407) 2024-04-19 08:47:02 -07:00
Maxime Carbonneau-Leclerc
b49c5006ea Break Python application with status 1 on exception (#37390) 2024-04-18 16:54:55 -04:00
Maxime Carbonneau-Leclerc
d2c8e63ea5 Fix partitioned state saving issue (#37389) 2024-04-17 22:57:45 -04:00
Maxime Carbonneau-Leclerc
a900c7841f Add StreamDescriptor as params of AirbyteTracedException.__init__ (#37108) 2024-04-12 17:11:12 -04:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Maxime Carbonneau-Leclerc
bbf69aea7f Concurrent CDK: support partitioned states (#36811) 2024-04-09 16:48:12 -04:00
Brian Lai
728c92cba1 [airbyte-cdk] Stream should not extract state using legacy get_updated_state if no cursor (#36342) 2024-03-20 23:47:23 -04:00
Ella Rohm-Ensing
2ac5248387 Emit record counts in state messages for concurrent streams (#35907) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2024-03-08 19:08:59 -05:00
Ella Rohm-Ensing
acbdc2d6e1 Introduce FinalStateCursor to emit state messages at the end of full refresh syncs (#35905) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2024-03-08 16:58:26 -05:00
Brian Lai
ef98194673 Emit final state message for full refresh syncs and consolidate read flows (#35622) 2024-03-05 01:05:06 -05:00
Artem Inzhyyants
0954ad3d3a Airbyte CDK: add interpolation for request options (#35485) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-02-22 19:40:44 +01:00
Catherine Noll
e8910e427a File-based CDK: make incremental syncs concurrent (#34540) 2024-02-07 20:41:04 -05:00
Maxime Carbonneau-Leclerc
ca8590e2b4 Have StateBuilder return our actual state object and not simply a dict (#34625) 2024-01-30 08:46:03 -05:00
Maxime Carbonneau-Leclerc
2c8b47b100 Emit state when no partitions are generated for ccdk (#34605) 2024-01-30 08:45:49 -05:00
Catherine Noll
eb31e4d2ba File-based CDK: make full refresh concurrent (#34411) 2024-01-29 19:33:50 -05:00
Maxime Carbonneau-Leclerc
b9c1897cfc Fix concurrent deadlock (#34454) 2024-01-24 12:35:21 -05:00
Catherine Noll
e3e58cc063 Concurrent CDK: fix state message ordering (#34131) 2024-01-18 11:35:40 -05:00
Alexandre Girard
0faa69d899 concurrent cdk: improve resource usage and stop waiting on the main thread (#33669) 
Co-authored-by: Augustin <augustin@airbyte.io>
 2024-01-17 23:54:02 -08:00
Catherine Noll
86321b7945 Concurrent CDK: add state converter for ISO timestamps with milliseco… (#33531) 2023-12-20 11:53:59 -05:00
Augustin
0b33caecda Revert "[skip ci] formatting: add missing license headers (#33250)" (#33289) 2023-12-11 11:38:37 +01:00
Augustin
60c1cc01ad [skip ci] formatting: add missing license headers (#33250) 2023-12-11 10:15:18 +01:00
Maxime Carbonneau-Leclerc
ba83309bb1 [ISSUE #32870] Adding entrypoint wrapper and migrating file based and… (#33103) 2023-12-06 08:46:38 -05:00
Alexandre Girard
a84902e8be ✨ concurrent cdk: Read multiple streams concurrently (#32411) 2023-11-28 15:00:00 -08:00
Maxime Carbonneau-Leclerc
b3f0e059d5 Fixing futures pruning (#32399) 2023-11-09 17:52:58 -05:00
Maxime Carbonneau-Leclerc
d0fd57ebf8 [ISSUE-32072] increase connection pool size (#32246) 2023-11-08 09:38:41 -05:00
Maxime Carbonneau-Leclerc
71d50635cc [ISSUE #32070] concurrent cdk improve futures handling (#32277) 2023-11-08 09:16:39 -05:00
Eugene Kulak
6c7ba28d75 API Call Rate limiter (#31276) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: keu <keu@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-11-07 23:32:53 +02:00
Catherine Noll
4f44e33f5c Concurrent CDK: handle legacy state messages (#31964) 2023-11-02 08:21:08 -04:00
Maxime Carbonneau-Leclerc
32fdd7fd72 [ISSUE #29573] Concurrent CDK: incremental syncs (#31466) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-11-01 12:00:25 -04:00
Alexandre Girard
b8ad0c6a91 🐛 CDK: use in memory caching if ENV_REQUEST_CACHE_PATH is not set (#31887) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-26 19:28:39 -07:00
Alexandre Girard
7da2822488 Concurrent CDK: catch exceptions from worker thread and add integration test scenarios (#31245) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-23 08:39:58 -07:00
Alexandre Girard
04c4fea5cc 🐛 Concurrent CDK bug fixes (#31402) 2023-10-16 12:06:35 -07:00
Anton Karpets
51fa2b3c31 🐛Airbyte CDK: wrap HTTP error with status code 400 in AirbyteTracedException (#31207) 2023-10-16 11:15:04 +03:00
Alexandre Girard
25fc396cdf CDK: ThreadBasedConcurrentStream skeleton and top-level AbstractStream (#30111) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-10-11 16:46:02 -07:00
Yevhenii
c17fae5855 CDK: create new method for parsing refresh token lifespan (#30698) 
Co-authored-by: yevhenii-ldv <yevhenii-ldv@users.noreply.github.com>
 2023-10-10 17:08:41 +03:00
Ben Church
4c97b2994a CDK: coerce read records to an iterator (#31122) 
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-10-06 10:01:29 -07:00
Eugene Kulak
5eba3c3b57 CDK: Fix request_cache clearing and move it to tmp folder (#30719) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2023-09-28 21:27:40 +03:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Alexandre Girard
df01616951 [Issue #23497] Deduplicate query parameters for declarative connectors (#28550) 
* remove duplicate param

* remove duplicate params

* fix some of the typing issues

* fix typing issues

* newline

* format

* Enable by default

* Add missing file

* refactor and remove flag

* none check

* move line of code

* fix typing in rate_limiting

* comment

* use typedef

* else branch

* format

* gate the feature

* rename test

* fix the test

* only dedupe if the values are the same

* Add some tests

* convert values to strings

* Document the change

* implement in requester too
 2023-07-25 14:22:25 -07:00
Maxime Carbonneau-Leclerc
df2a6e50bb Issue 21014/oauth requests (#27973) 
* [ISSUE #27494] fix type issue caused by connector builder logging

* [ISSUE #21014] log request/response for oauth as 'global_requests'

* formatcdk

* [ISSUE #21014] support DeclarativeOauth2Authenticator as well

* [ISSUE #21014] improving message grouper tests

* formatcdk

* Test solution with logic in MessageRepository (#27990)

* Test solution with logic in MessageRepository

* Solution without creating a new ModelToComponentFactory

* [ISSUE #21014] adding tests

* [ISSUE #21014] add title and description to global requests

* Revert "Solution without creating a new ModelToComponentFactory"

This reverts commit f17799ecff.

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* [ISSUE #21014] do not break on log appender conflict

* Automated Commit - Formatting Changes

* [ISSUE #21014] code review

* formatcdk

* [ISSUE #21014] moving is_global to is_auxiliary
 2023-07-11 13:37:38 -04:00
Alexandre Girard
dee2e9a905 🐛 Use url encoding in oauth refresh request (#27523) 
* Revert "🐛 CDK: replace `data` with `json` when making OAuth calls (#27350)"

This reverts commit 780f4415d9.

* Revert "Set content-type header on oauth request (#27225)"

This reverts commit 2864f72ff4.
 2023-06-20 14:41:06 -07:00
Denys Davydov
780f4415d9 🐛 CDK: replace data with json when making OAuth calls (#27350) 
* Connector health: source hubspot, gitlab, snapchat-marketing: fix builds

* Airbyte CDK: replace data with json when making OAuth calls
 2023-06-14 22:51:37 +03:00
Maxime Carbonneau-Leclerc
f48849fdb4 [ISSUE #26909] adding message repository (#27158) 
* [ISSUE #26909] adding message repository

* Automated Commit - Formatting Changes

* [ISSUE #26909] improve entrypoint error handling

* format CDK

* [ISSUE #26909] adding an integration test
 2023-06-13 08:40:55 -04:00
Alexandre Girard
2864f72ff4 Set content-type header on oauth request (#27225) 
* Set content-type header on oauth authenticator

* Revert "Set content-type header on oauth authenticator"

This reverts commit 1e6815e9bb.

* Set header on oauth request

* Fix test

* Verify header is set

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-06-12 13:29:53 -04:00
Joe Reuter
b34fb00660 Extend low code OAuthAuthenticator with token refresh capabilities (#26966) 
* wip

* Automated Commit - Formatting Changes

* add documentation

* tests and fixes

* fix tests

* more documentation

* revert

* changes as discussed

* fix case

* add docstring

* add details to schema

* format

* fix bug

---------

Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-06-07 10:51:59 +02:00
Alexandre Girard
5fecc7e626 add missing newline (#24035) 
* add missing newline

* Automated Commit - Formatting Changes

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-03-14 00:37:34 +00:00
Alexandre Girard
02f771b422 Do not remove trailing slash from path (#24003) 
* Do not remove trailing slash from path

* Add a breaking test

* Add some tests on HttpStream
 2023-03-13 23:59:24 +00:00
Leo Schick
09955423db support reading "expires_in" when the API passes the value as string (#23921) 
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2023-03-10 09:02:14 -08:00
Sherif A. Nada
ec9707c71f Low-code CDK: Override refresh_access_token logic DeclarativeOAuthAuthenticator (#23927) 
Co-authored-by: sherifnada <sherifnada@users.noreply.github.com>
 2023-03-09 17:28:33 -08:00