jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-26 05:05:18 -05:00
Code Issues Packages Projects Releases Wiki Activity
22,176 Commits 5,122 Branches 518 Tags
25802b6c4e6bc0eec94de09f40913366af42d2da
Commit Graph

130 Commits

Author SHA1 Message Date
Lake Mossman
b590a21629 return better structured error logs to connector builder (#46963) 2024-10-23 15:49:50 -07:00
Serhii Lazebnyi
dd5adef0d4 feat(concurrent-cdk): add to concurrent per slice tracking of the most recent cursor (#45180) 2024-10-07 23:26:48 +02:00
Serhii Lazebnyi
25e21bc395 feat(concurrent-cdk): add cursor partition generator (#45712) 2024-10-02 13:19:45 +02:00
Serhii Lazebnyi
1569c0288e feat(concurrent-cdk): replace availability strategy to always available (#46269) 2024-10-01 20:34:11 +02:00
Serhii Lazebnyi
d727af0fa0 fix(concurrent-cdk): follow-up #44946 update state to state_manager state in convert to concurrent stream (#45718) 2024-10-01 18:28:23 +02:00
Maxime Carbonneau-Leclerc
53af0c2e45 [ONCALL #6327] have headers match during HTTP cache hit (#45926) 2024-09-27 09:23:10 -04:00
Serhii Lazebnyi
91c2c750e2 refactor[concurrent-cdk]: add _convert_to_concurrent_stream to ConcurrentSourceAdapter (#44946) 2024-09-19 19:04:24 +02:00
Brian Lai
199a8078f2 [airbyte-cdk] Decouple request_options_provider from datetime_based_cursor + concurrent_cursor features for low-code (#45413) 2024-09-17 14:06:41 -04:00
Artem Inzhyyants
df34893b63 feat(airbyte-cdk): replace pydantic BaseModel with dataclasses + serpyco-rs in protocol (#44444) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-09-02 17:48:17 +02:00
Patrick Nilan
10601618c0 [airbyte-cdk] - Update JSON Error Message Parser to return additional error message fields or default to dict (#44010) 2024-08-28 12:38:26 -07:00
Brian Lai
fca0460030 [airbyte-cdk] tech-debt Remove support for parsing legacy state message format (#43459) 2024-08-16 21:06:37 -04:00
Maxime Carbonneau-Leclerc
61c07e8bf6 feat(airbyte-cdk): Have better fallback error message on HTTP error (#43399) 2024-08-12 20:43:23 -04:00
Brian Lai
197cb810b0 [RFR for API Sources] Add SubstreamResumableFullRefreshCursor to the Python CDK (#42429) 2024-08-01 16:39:14 -04:00
Maxime Carbonneau-Leclerc
36a6f35a61 feat(airbyte-cdk) Align BackoffStrategy interfaces to take attempt_count as a full-fled… (#42889) 2024-07-31 11:41:15 -04:00
Patrick Nilan
9537394887 [airbyte-cdk] Update HttpClient to return correct error message in case of FAIL/IGNORE (#42512) 2024-07-26 12:56:26 -07:00
Artem Inzhyyants
5056e67826 refactor!(airbyte-cdk): deprecate availability strategy (#42039) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-07-23 10:22:11 +02:00
Serhii Lazebnyi
ad67ccbda6 [airbyte-cdk]: follow up for #41333 (#42029) 2024-07-18 16:26:40 +02:00
Brian Lai
b2d53f552d [RFR for Python Sources] Make it easier for Python sources to automatically use RFR for eligible streams (#39450) 2024-07-18 01:17:14 -04:00
Serhii Lazebnyi
39b8e3da19 [airbyte-cdk] not exiting when rate limited (#41333) 2024-07-17 00:56:59 +02:00
Aldo Gonzalez
51cacb3e92 🐛 bug(Airbyte CDK): clean invalid fields from configured catalog (#41678) 2024-07-15 10:44:53 -06:00
Brian Lai
e456bca099 🐛 [RFR for API Sources] Fix bug where checkpoint reader stops syncing too early if first partition is complete (#41658) 2024-07-12 19:00:25 -04:00
Brian Lai
9e23b3f89b 🐛 [airbyte-cdk] Fix bug where substreams depending on an RFR parent stream don't paginate or use existing state (#40671) 2024-07-11 02:53:20 -04:00
Serhii Lazebnyi
f00ed4a925 [airbyte-cdk] add running stream status with rate limit reason to backoff aproach (#40681) 2024-07-10 14:00:01 +02:00
Artem Inzhyyants
02c5f59ccf ref(airbyte-cdk): use http_client inside HttpStream (#39811) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-07-09 12:01:03 +02:00
Boris Staal
6dd9b7ab25 chore(cdk): Avoid using time.sleep in unit tests for backoff of http stream (#40239) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-07-02 11:47:56 -07:00
Artem Inzhyyants
f49c8054ad feat(airbyte-cdk): add json_schema from ConfiguredCatalog to Stream (#39522) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-06-19 11:12:33 +02:00
Patrick Nilan
18e82d949a [airbyte-cdk] - Integrate HttpClient into HttpRequester (#38906) 2024-06-18 01:03:15 +00:00
Brian Lai
29d615080a [airbyte-cdk] Fix a bug so that successful Python RFR streams are not synced on subsequent attempts (#38608) 2024-05-24 15:58:18 -04:00
Brian Lai
040f1415e5 [low-code CDK] Rsumable full refresh support for low-code streams (#38300) 2024-05-22 16:23:31 -04:00
Alexandre Girard
0ceb76920a refactor!(airbyte-cdk): Delete deprecated AirbyteLogger, AirbyteSpec, and Authenticators + move public classes to the top level init file (#37805) 
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
 2024-05-20 07:18:37 -07:00
Patrick Nilan
8396fd2d7f airbyte-cdk: Improve Error Handling in Legacy CDK (#37576) 2024-05-16 19:07:21 -07:00
Brian Lai
d74125bf10 [RFR for API Sources] New Python interfaces to support resumable full refresh (#37429) 2024-05-06 18:41:29 -04:00
Maxime Carbonneau-Leclerc
48af92ad78 Concurrent CDK: if exception is AirbyteTracedException, raise this an… (#37443) 2024-04-19 20:32:46 +00:00
Ella Rohm-Ensing
0439cbcd10 Clarify cdk parity test behavior (#37407) 2024-04-19 08:47:02 -07:00
Maxime Carbonneau-Leclerc
b49c5006ea Break Python application with status 1 on exception (#37390) 2024-04-18 16:54:55 -04:00
Maxime Carbonneau-Leclerc
d2c8e63ea5 Fix partitioned state saving issue (#37389) 2024-04-17 22:57:45 -04:00
Maxime Carbonneau-Leclerc
a900c7841f Add StreamDescriptor as params of AirbyteTracedException.__init__ (#37108) 2024-04-12 17:11:12 -04:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Maxime Carbonneau-Leclerc
bbf69aea7f Concurrent CDK: support partitioned states (#36811) 2024-04-09 16:48:12 -04:00
Brian Lai
728c92cba1 [airbyte-cdk] Stream should not extract state using legacy get_updated_state if no cursor (#36342) 2024-03-20 23:47:23 -04:00
Ella Rohm-Ensing
2ac5248387 Emit record counts in state messages for concurrent streams (#35907) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2024-03-08 19:08:59 -05:00
Ella Rohm-Ensing
acbdc2d6e1 Introduce FinalStateCursor to emit state messages at the end of full refresh syncs (#35905) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2024-03-08 16:58:26 -05:00
Brian Lai
ef98194673 Emit final state message for full refresh syncs and consolidate read flows (#35622) 2024-03-05 01:05:06 -05:00
Artem Inzhyyants
0954ad3d3a Airbyte CDK: add interpolation for request options (#35485) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-02-22 19:40:44 +01:00
Catherine Noll
e8910e427a File-based CDK: make incremental syncs concurrent (#34540) 2024-02-07 20:41:04 -05:00
Maxime Carbonneau-Leclerc
ca8590e2b4 Have StateBuilder return our actual state object and not simply a dict (#34625) 2024-01-30 08:46:03 -05:00
Maxime Carbonneau-Leclerc
2c8b47b100 Emit state when no partitions are generated for ccdk (#34605) 2024-01-30 08:45:49 -05:00
Catherine Noll
eb31e4d2ba File-based CDK: make full refresh concurrent (#34411) 2024-01-29 19:33:50 -05:00
Maxime Carbonneau-Leclerc
b9c1897cfc Fix concurrent deadlock (#34454) 2024-01-24 12:35:21 -05:00
Catherine Noll
e3e58cc063 Concurrent CDK: fix state message ordering (#34131) 2024-01-18 11:35:40 -05:00