jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-26 14:02:10 -05:00
Code Issues Packages Projects Releases Wiki Activity
22,176 Commits 5,122 Branches 518 Tags
25802b6c4e6bc0eec94de09f40913366af42d2da
Commit Graph

35 Commits

Author SHA1 Message Date
Serhii Lazebnyi
dd5adef0d4 feat(concurrent-cdk): add to concurrent per slice tracking of the most recent cursor (#45180) 2024-10-07 23:26:48 +02:00
Serhii Lazebnyi
25e21bc395 feat(concurrent-cdk): add cursor partition generator (#45712) 2024-10-02 13:19:45 +02:00
Serhii Lazebnyi
1569c0288e feat(concurrent-cdk): replace availability strategy to always available (#46269) 2024-10-01 20:34:11 +02:00
Serhii Lazebnyi
d727af0fa0 fix(concurrent-cdk): follow-up #44946 update state to state_manager state in convert to concurrent stream (#45718) 2024-10-01 18:28:23 +02:00
Serhii Lazebnyi
91c2c750e2 refactor[concurrent-cdk]: add _convert_to_concurrent_stream to ConcurrentSourceAdapter (#44946) 2024-09-19 19:04:24 +02:00
Brian Lai
199a8078f2 [airbyte-cdk] Decouple request_options_provider from datetime_based_cursor + concurrent_cursor features for low-code (#45413) 2024-09-17 14:06:41 -04:00
Artem Inzhyyants
df34893b63 feat(airbyte-cdk): replace pydantic BaseModel with dataclasses + serpyco-rs in protocol (#44444) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
 2024-09-02 17:48:17 +02:00
Brian Lai
fca0460030 [airbyte-cdk] tech-debt Remove support for parsing legacy state message format (#43459) 2024-08-16 21:06:37 -04:00
Maxime Carbonneau-Leclerc
48af92ad78 Concurrent CDK: if exception is AirbyteTracedException, raise this an… (#37443) 2024-04-19 20:32:46 +00:00
Maxime Carbonneau-Leclerc
b49c5006ea Break Python application with status 1 on exception (#37390) 2024-04-18 16:54:55 -04:00
Maxime Carbonneau-Leclerc
d2c8e63ea5 Fix partitioned state saving issue (#37389) 2024-04-17 22:57:45 -04:00
Maxime Carbonneau-Leclerc
a900c7841f Add StreamDescriptor as params of AirbyteTracedException.__init__ (#37108) 2024-04-12 17:11:12 -04:00
Ella Rohm-Ensing
b7819d9f6c python: assert actual == expected ordering (#36980) 2024-04-11 15:16:33 +00:00
Maxime Carbonneau-Leclerc
bbf69aea7f Concurrent CDK: support partitioned states (#36811) 2024-04-09 16:48:12 -04:00
Ella Rohm-Ensing
2ac5248387 Emit record counts in state messages for concurrent streams (#35907) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
 2024-03-08 19:08:59 -05:00
Ella Rohm-Ensing
acbdc2d6e1 Introduce FinalStateCursor to emit state messages at the end of full refresh syncs (#35905) 
Co-authored-by: brianjlai <brian.lai@airbyte.io>
 2024-03-08 16:58:26 -05:00
Brian Lai
ef98194673 Emit final state message for full refresh syncs and consolidate read flows (#35622) 2024-03-05 01:05:06 -05:00
Artem Inzhyyants
0954ad3d3a Airbyte CDK: add interpolation for request options (#35485) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2024-02-22 19:40:44 +01:00
Catherine Noll
e8910e427a File-based CDK: make incremental syncs concurrent (#34540) 2024-02-07 20:41:04 -05:00
Maxime Carbonneau-Leclerc
ca8590e2b4 Have StateBuilder return our actual state object and not simply a dict (#34625) 2024-01-30 08:46:03 -05:00
Maxime Carbonneau-Leclerc
2c8b47b100 Emit state when no partitions are generated for ccdk (#34605) 2024-01-30 08:45:49 -05:00
Catherine Noll
eb31e4d2ba File-based CDK: make full refresh concurrent (#34411) 2024-01-29 19:33:50 -05:00
Maxime Carbonneau-Leclerc
b9c1897cfc Fix concurrent deadlock (#34454) 2024-01-24 12:35:21 -05:00
Catherine Noll
e3e58cc063 Concurrent CDK: fix state message ordering (#34131) 2024-01-18 11:35:40 -05:00
Alexandre Girard
0faa69d899 concurrent cdk: improve resource usage and stop waiting on the main thread (#33669) 
Co-authored-by: Augustin <augustin@airbyte.io>
 2024-01-17 23:54:02 -08:00
Catherine Noll
86321b7945 Concurrent CDK: add state converter for ISO timestamps with milliseco… (#33531) 2023-12-20 11:53:59 -05:00
Maxime Carbonneau-Leclerc
ba83309bb1 [ISSUE #32870] Adding entrypoint wrapper and migrating file based and… (#33103) 2023-12-06 08:46:38 -05:00
Alexandre Girard
a84902e8be ✨ concurrent cdk: Read multiple streams concurrently (#32411) 2023-11-28 15:00:00 -08:00
Maxime Carbonneau-Leclerc
b3f0e059d5 Fixing futures pruning (#32399) 2023-11-09 17:52:58 -05:00
Maxime Carbonneau-Leclerc
71d50635cc [ISSUE #32070] concurrent cdk improve futures handling (#32277) 2023-11-08 09:16:39 -05:00
Catherine Noll
4f44e33f5c Concurrent CDK: handle legacy state messages (#31964) 2023-11-02 08:21:08 -04:00
Maxime Carbonneau-Leclerc
32fdd7fd72 [ISSUE #29573] Concurrent CDK: incremental syncs (#31466) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-11-01 12:00:25 -04:00
Alexandre Girard
7da2822488 Concurrent CDK: catch exceptions from worker thread and add integration test scenarios (#31245) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-23 08:39:58 -07:00
Alexandre Girard
04c4fea5cc 🐛 Concurrent CDK bug fixes (#31402) 2023-10-16 12:06:35 -07:00
Alexandre Girard
25fc396cdf CDK: ThreadBasedConcurrentStream skeleton and top-level AbstractStream (#30111) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-10-11 16:46:02 -07:00