jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-20 21:06:36 -05:00
Code Issues Packages Projects Releases Wiki Activity
13,734 Commits 5,351 Branches 518 Tags
d3f2aa548a1ff87f2aedf3e25faf6c53feb0de78
Commit Graph

395 Commits

Author SHA1 Message Date
Maxime Carbonneau-Leclerc
d3f2aa548a [ISSUE #33202] allow for loose query params validation (#33226) 2023-12-11 08:57:41 -05:00
Augustin
0b33caecda Revert "[skip ci] formatting: add missing license headers (#33250)" (#33289) 2023-12-11 11:38:37 +01:00
Augustin
60c1cc01ad [skip ci] formatting: add missing license headers (#33250) 2023-12-11 10:15:18 +01:00
Joe Reuter
aa220fc515 Stop sync on traced exception (#33246) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-12-08 18:07:25 +01:00
Joe Reuter
f5ac5cfd80 File CDK: Add file processing via API to document file type parser (#32781) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-12-08 15:48:37 +01:00
Joe Reuter
7fd92e2a03 File CDK: Parser defined primary key (#33009) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-12-08 15:15:33 +01:00
Yevhenii
32ebd88402 CDK: low-code enable caching for parent streams (#32726) 2023-12-08 13:54:31 +00:00
Joe Reuter
5b682ef74f Unstructured parser: Handle parsing errors better (#32700) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-12-08 11:47:05 +01:00
Joe Reuter
21b3b2f638 Vector DB CDK: Fix special tokens (#33065) 2023-12-08 11:46:46 +01:00
Brian Lai
f8182bc18e airbyte-cdk: failed stream does not end the sync (#33136) 
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
 2023-12-07 14:11:39 -05:00
Catherine Noll
7ed47ee7d9 File-based CDK: hide the primary key field from config (#33172) 2023-12-06 11:12:50 -05:00
Maxime Carbonneau-Leclerc
ba83309bb1 [ISSUE #32870] Adding entrypoint wrapper and migrating file based and… (#33103) 2023-12-06 08:46:38 -05:00
Maxime Carbonneau-Leclerc
69cb3a571e [ISSUE #32868] create HttpMocker (#32937) and [ISSUE #32869] response builder (#32983) 
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
 2023-12-05 08:48:45 -05:00
Joe Reuter
28e8692624 Vector DB CDK: Add omit_raw_text flag (#32698) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-11-30 09:49:02 +01:00
Alexandre Girard
a84902e8be ✨ concurrent cdk: Read multiple streams concurrently (#32411) 2023-11-28 15:00:00 -08:00
Alexandre Girard
9d52007c31 🐛 low-code: Fix injection of page token in first request (#32666) 
Co-authored-by: Lake Mossman <lake@airbyte.io>
 2023-11-28 12:49:59 -08:00
Yevhenii
d475beba19 🐛CDK: Fix of generate the error message using _try_get_error based on list of errors (#32545) 2023-11-17 12:22:20 +00:00
Joe Reuter
aa111d2bea Vector DB CDK: Delete cdc records (#32496) 2023-11-16 15:32:15 +00:00
Alexandre Girard
c979f70490 pass airbytestream instead of configured stream to as_airbyte_message (#32412) 2023-11-15 11:36:58 -08:00
Joe Reuter
f8b0b3e99e File CDK: Improve stream config appearance (#32420) 2023-11-14 11:49:19 +01:00
Ella Rohm-Ensing
ac3eb28de2 airbyte-ci: add format commands (#31831) 
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
 2023-11-14 02:17:48 -06:00
Maxime Carbonneau-Leclerc
b3f0e059d5 Fixing futures pruning (#32399) 2023-11-09 17:52:58 -05:00
Joe Reuter
f1a11e1927 File CDK: Allow skipping unparseable file types (#32092) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-11-09 16:48:24 +01:00
Joe Reuter
e113ff66c5 CDK: Make consts required in Pydantic generated json schemas (#32251) 2023-11-09 16:12:11 +01:00
Maxime Carbonneau-Leclerc
d0fd57ebf8 [ISSUE-32072] increase connection pool size (#32246) 2023-11-08 09:38:41 -05:00
Maxime Carbonneau-Leclerc
71d50635cc [ISSUE #32070] concurrent cdk improve futures handling (#32277) 2023-11-08 09:16:39 -05:00
Alexandre Girard
139deeb081 Implement max_time on error handler (#32272) 2023-11-08 00:46:26 +00:00
Eugene Kulak
6c7ba28d75 API Call Rate limiter (#31276) 
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: keu <keu@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-11-07 23:32:53 +02:00
Catherine Noll
4f44e33f5c Concurrent CDK: handle legacy state messages (#31964) 2023-11-02 08:21:08 -04:00
Joe Reuter
66dd29f764 File CDK unstructured parser: Improve file type detection (#31997) 2023-11-02 12:19:27 +01:00
Maxime Carbonneau-Leclerc
32fdd7fd72 [ISSUE #29573] Concurrent CDK: incremental syncs (#31466) 
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-11-01 12:00:25 -04:00
Martin Hwasser
bc4b7198a9 ✨ Add pptx support in file based cdk (#31912) 
Co-authored-by: Joe Reuter <joe@airbyte.io>
 2023-10-30 14:42:39 +01:00
Artem Inzhyyants
ecd6d89b9a Airbyte CDK: make max_time optional for backoff handler external usage (#31889) 2023-10-27 13:56:36 +02:00
Alexandre Girard
b8ad0c6a91 🐛 CDK: use in memory caching if ENV_REQUEST_CACHE_PATH is not set (#31887) 
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-26 19:28:39 -07:00
Joe Reuter
e3793c1491 Move over unstructured parser (#31390) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-10-26 17:50:57 +02:00
Anatolii Yatsuk
c719137df3 🐛 Airbyte CDK: Fix flake errors in file-based CDK (#31771) 2023-10-24 16:15:11 +03:00
Anatolii Yatsuk
ce2342dde8 🎉 Airbyte CDK: Add CustomFileBasedException for custom errors in file-based CDK (#31704) 2023-10-24 11:09:50 +00:00
Alexandre Girard
7a764f8bbc ✨ low-code CDK: Allow connector developers to specify the type of an added field (#31638) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: erohmensing <erohmensing@gmail.com>
 2023-10-23 14:12:59 -07:00
Alexandre Girard
7da2822488 Concurrent CDK: catch exceptions from worker thread and add integration test scenarios (#31245) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
 2023-10-23 08:39:58 -07:00
Joe Reuter
d474827068 File CDK: Don't fetch full file list for availability check (#31651) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-10-23 16:14:41 +02:00
Joe Reuter
bb07939646 File CDK: Add analytics messages for parser usage (#31498) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-10-19 15:42:51 +02:00
Martin Hwasser
40b0e05526 ✨ vector_based_cdk: Add option to rename field names (#31524) 
Co-authored-by: Joe Reuter <joe@airbyte.io>
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-10-19 15:37:47 +02:00
Yevhenii
b951898c20 CDK: Support base64 encode and decode in Jinja Interpolation (#31387) 2023-10-19 13:55:45 +03:00
Alexandre Girard
ef9bd72a7e Parameterize ScenarioBuilder on Source type (#31244) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
 2023-10-16 17:12:18 -07:00
Alexandre Girard
04c4fea5cc 🐛 Concurrent CDK bug fixes (#31402) 2023-10-16 12:06:35 -07:00
Anton Karpets
51fa2b3c31 🐛Airbyte CDK: wrap HTTP error with status code 400 in AirbyteTracedException (#31207) 2023-10-16 11:15:04 +03:00
Joe Reuter
e35a1f2cd9 File CDK: Allow configuration of parsed records during check and discover from parser (#31281) 
Co-authored-by: flash1293 <flash1293@users.noreply.github.com>
 2023-10-13 09:50:22 +02:00
Catherine Noll
8536725944 CDK: URL-encode query parameters and request body (#30407) 2023-10-12 09:56:55 -04:00
Joe Reuter
67324a4b5b Vector DB CDK: Batch by documents separately for each stream and namespace (#31158) 2023-10-12 13:47:27 +00:00
Alexandre Girard
25fc396cdf CDK: ThreadBasedConcurrentStream skeleton and top-level AbstractStream (#30111) 
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <maxi297@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
 2023-10-11 16:46:02 -07:00