jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-30 16:01:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
17,931 Commits 5,433 Branches 518 Tags
0f086de2ca75dbfc0bafa2149bf317ac48a7e366
Commit Graph

108 Commits

Author SHA1 Message Date
Serhii Lazebnyi
f00ed4a925 [airbyte-cdk] add running stream status with rate limit reason to backoff aproach (#40681) 2024-07-10 14:00:01 +02:00
Artem Inzhyyants
02c5f59ccf ref(airbyte-cdk): use http_client inside HttpStream (#39811) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-07-09 12:01:03 +02:00
Boris Staal
6dd9b7ab25 chore(cdk): Avoid using time.sleep in unit tests for backoff of http stream (#40239) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-07-02 11:47:56 -07:00
Artem Inzhyyants
f49c8054ad feat(airbyte-cdk): add json_schema from ConfiguredCatalog to Stream (#39522) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-06-19 11:12:33 +02:00
Patrick Nilan
18e82d949a [airbyte-cdk] - Integrate HttpClient into HttpRequester (#38906) 2024-06-18 01:03:15 +00:00
Brian Lai
29d615080a [airbyte-cdk] Fix a bug so that successful Python RFR streams are not synced on subsequent attempts (#38608) 2024-05-24 15:58:18 -04:00
Brian Lai
040f1415e5 [low-code CDK] Rsumable full refresh support for low-code streams (#38300) 2024-05-22 16:23:31 -04:00
Alexandre Girard
0ceb76920a refactor!(airbyte-cdk): Delete deprecated AirbyteLogger, AirbyteSpec, and Authenticators + move public classes to the top level init file (#37805) 
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
 2024-05-20 07:18:37 -07:00
Patrick Nilan
8396fd2d7f airbyte-cdk: Improve Error Handling in Legacy CDK (#37576) 2024-05-16 19:07:21 -07:00
Brian Lai
d74125bf10 [RFR for API Sources] New Python interfaces to support resumable full refresh (#37429) 2024-05-06 18:41:29 -04:00
Maxime Carbonneau-Leclerc
48af92ad78 Concurrent CDK: if exception is AirbyteTracedException, raise this an… (#37443) 2024-04-19 20:32:46 +00:00
Ella Rohm-Ensing
0439cbcd10 Clarify cdk parity test behavior (#37407) 2024-04-19 08:47:02 -07:00
Maxime Carbonneau-Leclerc
b49c5006ea Break Python application with status 1 on exception (#37390) 2024-04-18 16:54:55 -04:00
Maxime Carbonneau-Leclerc
d2c8e63ea5 Fix partitioned state saving issue (#37389) 2024-04-17 22:57:45 -04:00
Maxime Carbonneau-Leclerc
a900c7841f Add StreamDescriptor as params of AirbyteTracedException.__init__ (#37108) 2024-04-12 17:11:12 -04:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Maxime Carbonneau-Leclerc
bbf69aea7f Concurrent CDK: support partitioned states (#36811) 2024-04-09 16:48:12 -04:00
Brian Lai
728c92cba1 [airbyte-cdk] Stream should not extract state using legacy get_updated_state if no cursor (#36342) 2024-03-20 23:47:23 -04:00
Ella Rohm-Ensing
2ac5248387 Emit record counts in state messages for concurrent streams (#35907) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2024-03-08 19:08:59 -05:00
Ella Rohm-Ensing
acbdc2d6e1 Introduce FinalStateCursor to emit state messages at the end of full refresh syncs (#35905) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2024-03-08 16:58:26 -05:00
Brian Lai
ef98194673 Emit final state message for full refresh syncs and consolidate read flows (#35622) 2024-03-05 01:05:06 -05:00
Artem Inzhyyants
0954ad3d3a Airbyte CDK: add interpolation for request options (#35485) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-02-22 19:40:44 +01:00
Catherine Noll
e8910e427a File-based CDK: make incremental syncs concurrent (#34540) 2024-02-07 20:41:04 -05:00
Maxime Carbonneau-Leclerc
ca8590e2b4 Have StateBuilder return our actual state object and not simply a dict (#34625) 2024-01-30 08:46:03 -05:00
Maxime Carbonneau-Leclerc
2c8b47b100 Emit state when no partitions are generated for ccdk (#34605) 2024-01-30 08:45:49 -05:00
Catherine Noll
eb31e4d2ba File-based CDK: make full refresh concurrent (#34411) 2024-01-29 19:33:50 -05:00
Maxime Carbonneau-Leclerc
b9c1897cfc Fix concurrent deadlock (#34454) 2024-01-24 12:35:21 -05:00
Catherine Noll
e3e58cc063 Concurrent CDK: fix state message ordering (#34131) 2024-01-18 11:35:40 -05:00
Alexandre Girard
0faa69d899 concurrent cdk: improve resource usage and stop waiting on the main thread (#33669) 
Co-authored-by: Augustin <augustin@airbyte.io>
 2024-01-17 23:54:02 -08:00
Catherine Noll
86321b7945 Concurrent CDK: add state converter for ISO timestamps with milliseco… (#33531) 2023-12-20 11:53:59 -05:00
Augustin
0b33caecda Revert "[skip ci] formatting: add missing license headers (#33250)" (#33289) 2023-12-11 11:38:37 +01:00
Augustin
60c1cc01ad [skip ci] formatting: add missing license headers (#33250) 2023-12-11 10:15:18 +01:00
Maxime Carbonneau-Leclerc
ba83309bb1 [ISSUE #32870] Adding entrypoint wrapper and migrating file based and… (#33103) 2023-12-06 08:46:38 -05:00
Alexandre Girard
a84902e8be ✨ concurrent cdk: Read multiple streams concurrently (#32411) 2023-11-28 15:00:00 -08:00
Maxime Carbonneau-Leclerc
b3f0e059d5 Fixing futures pruning (#32399) 2023-11-09 17:52:58 -05:00
Maxime Carbonneau-Leclerc
d0fd57ebf8 [ISSUE-32072] increase connection pool size (#32246) 2023-11-08 09:38:41 -05:00
Maxime Carbonneau-Leclerc
71d50635cc [ISSUE #32070] concurrent cdk improve futures handling (#32277) 2023-11-08 09:16:39 -05:00
Eugene Kulak
6c7ba28d75 API Call Rate limiter (#31276) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: keu <keu@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-11-07 23:32:53 +02:00
Catherine Noll
4f44e33f5c Concurrent CDK: handle legacy state messages (#31964) 2023-11-02 08:21:08 -04:00
Maxime Carbonneau-Leclerc
32fdd7fd72 [ISSUE #29573] Concurrent CDK: incremental syncs (#31466) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-11-01 12:00:25 -04:00
Alexandre Girard
b8ad0c6a91 🐛 CDK: use in memory caching if ENV_REQUEST_CACHE_PATH is not set (#31887) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-26 19:28:39 -07:00
Alexandre Girard
7da2822488 Concurrent CDK: catch exceptions from worker thread and add integration test scenarios (#31245) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-23 08:39:58 -07:00
Alexandre Girard
04c4fea5cc 🐛 Concurrent CDK bug fixes (#31402) 2023-10-16 12:06:35 -07:00
Anton Karpets
51fa2b3c31 🐛Airbyte CDK: wrap HTTP error with status code 400 in AirbyteTracedException (#31207) 2023-10-16 11:15:04 +03:00
Alexandre Girard
25fc396cdf CDK: ThreadBasedConcurrentStream skeleton and top-level AbstractStream (#30111) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-10-11 16:46:02 -07:00
Yevhenii
c17fae5855 CDK: create new method for parsing refresh token lifespan (#30698) 
Co-authored-by: yevhenii-ldv <yevhenii-ldv@users.noreply.github.com>
 2023-10-10 17:08:41 +03:00
Ben Church
4c97b2994a CDK: coerce read records to an iterator (#31122) 
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
 2023-10-06 10:01:29 -07:00
Eugene Kulak
5eba3c3b57 CDK: Fix request_cache clearing and move it to tmp folder (#30719) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
 2023-09-28 21:27:40 +03:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Alexandre Girard
df01616951 [Issue #23497] Deduplicate query parameters for declarative connectors (#28550) 
* remove duplicate param

* remove duplicate params

* fix some of the typing issues

* fix typing issues

* newline

* format

* Enable by default

* Add missing file

* refactor and remove flag

* none check

* move line of code

* fix typing in rate_limiting

* comment

* use typedef

* else branch

* format

* gate the feature

* rename test

* fix the test

* only dedupe if the values are the same

* Add some tests

* convert values to strings

* Document the change

* implement in requester too
 2023-07-25 14:22:25 -07:00