jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 11:01:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

304 Commits

Author SHA1 Message Date
Subodh Kant Chaturvedi
a15bd21714 destination-async-framework: use the value from stats counter for global state conversion (#35111) 2024-02-12 14:18:54 +05:30
Edward Gao
cb81cb4016 Destination snowflake: update test fixtures (#34574) 2024-02-09 15:06:02 -08:00
Marius Posta
ae2198df3a destination-snowflake: adopt cleaned-up cdk (#34747) 2024-02-09 11:08:44 -08:00
Subodh Kant Chaturvedi
e691b66ffa async-destination-framework: make GlobalAsyncStateManager entirely thread safe (#35027) 2024-02-09 02:45:59 +05:30
Edward Gao
c32c2f20a0 🐛 Destination snowflake: use 200MB batches (#34502) 2024-02-08 15:29:59 -04:00
Edward Gao
96b9f5a059 Destinations bigquery, snowflake, redshift: log better message when failing to parse an AirbyteMessage (#34451) 2024-01-25 18:33:33 -04:00
Edward Gao
180da5d698 🐛 Destination snowflake: fix check (#34528) 2024-01-25 13:25:27 -04:00
Edward Gao
a0345ae732 Destinations snowflake, bigquery, redshift: improve error reporting (#34458) 2024-01-24 19:20:45 -04:00
Cynthia Yin
f2c846188a Destination Snowflake: set isV2Destination (#34501) 2024-01-24 14:03:56 -08:00
Gireesh Sreepathi
950fea1bca Snowflake Destination: Upgrade CDK (#34462) 
## What
* Upgrade Snowflake to pick the reverted CDK changes to move forward.
 2024-01-24 10:09:09 -08:00
Cynthia Yin
13e32a6e87 ✨ Destination Redshift: GA Destinations V2 (#34077) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-23 16:02:00 -08:00
Gireesh Sreepathi
026f5a7ddd Destination Snowflake: Cleanup dependencies and upgrade CDK (#34227) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-12 17:15:27 -04:00
Subodh Kant Chaturvedi
7d9f636142 snowflake-destination: upgrade cdk version to start emitting destination stats as part of state (#34083) 2024-01-11 23:00:17 +05:30
Gireesh Sreepathi
7503dc78f1 destination-redshift: Heartbeats for bastion and other minor bugfixes (#33948) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2024-01-08 09:09:17 -08:00
Edward Gao
29a3da04ca Destinations Bigquery, snowflake, redshift: internal CDK refactors for DV2 (#33730) 2024-01-04 21:20:15 -04:00
Joe Bell
3fbc900c0d Destinations V2 - Default to Type and Dedupe once at the end of a sync (#33728) 2024-01-02 21:29:22 -04:00
Edward Gao
1d76fd2618 🎉 Destination redshift: reenable T+D optimizations; bigquery+snowflake: bump to latest CDK version (#33704) 2023-12-20 21:09:44 -06:00
Joe Bell
5b826ecbed Destination Snowflake, BigQuery, JavaCdk - Move Schema Creation out of each table creation (#33124) 
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
 2023-12-20 17:40:49 -04:00
Cynthia Yin
73b7e7bb49 📚 Destinations V2: update data types for BigQuery + Snowflake (#33471) 2023-12-14 00:59:15 +00:00
Edward Gao
d27ea33ffe DV2: Only run T+D if we have zero records or the previous sync left behind un-T+D-ed records (#33232) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-12-13 19:47:36 -04:00
Joe Reuter
e2789b4f03 Update docusaurus to 3 (#33041) 2023-12-11 17:03:18 +01:00
Edward Gao
a9b1e8d483 Destinations bigquery+snowflake+s3: Chore: upgrade CDK version (#33263) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-12-08 18:24:18 -04:00
Cynthia Yin
6a643629d3 ✨ Destinations V2 in Redshift - Opt in beta (#32326) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: gisripa <gisripa@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Gireesh Sreepathi <gireesh@airbyte.io>
 2023-12-07 19:19:09 -06:00
Evan Tahler
95a8e42a4b T&D SQL statements moved to debug log level (#33084) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
 2023-12-04 21:11:02 -04:00
Edward Gao
0536e89170 Destination snowflake: update docs to reflect that we removed s3/gcs staging (#32887) 2023-11-28 14:48:53 -08:00
Evan Tahler
0e166ebd24 Async Destinations - Make debug messages less scary (#32526) 2023-11-15 15:32:48 -04:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00
Joe Bell
04c067d01d Destinations BigQuery and Snowflake - Move SAFE/TRY casting to seperate transaction (#32026) 
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-06 20:33:54 -04:00
Edward Gao
c81abe51a8 Destination bigquery + Snowflake: Further error grouping improvements (#32190) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-06 13:25:06 -04:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Edward Gao
2f931d824e Destination snowflake: Revert merge statement (#32124) 2023-11-03 14:04:24 -07:00
Edward Gao
453232b366 Destinations bigquery+snowflake: better sentry error grouping (#31983) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-02 16:32:11 +00:00
Edward Gao
82679a679c Destinations snowflake+bigquery: delay upgrade deadline to nov 7 (#31985) 
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
 2023-10-31 11:47:48 -04:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Edward Gao
877200de94 Destination snowflake: filter raw table commit on extracted_at (#31897) 2023-10-27 13:14:50 -04:00
Edward Gao
c6005b17e9 Destination snowflake: T+D via merge (#31683) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-27 15:58:00 +00:00
Gireesh Sreepathi
3e764712f4 Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec (#31686) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: gisripa <gisripa@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2023-10-26 13:46:54 -07:00
Edward Gao
148dda1c1f Destinations Bigquery+Snowflake: Do not dedup raw table (#31520) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-25 12:19:58 -04:00
Edward Gao
f1baf2a2f2 Destinations snowflake+bigquery: Improve performance by filtering raw table on extracted_at (#31191) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-17 17:48:18 +00:00
Ryan Br
5f9bec01bd Remove entry and allocated memory for empty buffers. (#31194) 
Co-authored-by: tryangul <tryangul@users.noreply.github.com>
 2023-10-11 11:22:46 -07:00
Cynthia Yin
e3d126b5e7 🐛 Fixing rounding of numeric values for async destinations (#31083) 2023-10-10 22:13:33 -04:00
Evan Tahler
898846d40a Remove Dv2 Null checks (#31149) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-10-10 15:36:00 -04:00
Benoit Moriceau
8fddc21c35 Add missing java options (#31153) 2023-10-07 01:07:21 +00:00
Benoit Moriceau
9e2b3f4990 Bump CDk (#31139) 2023-10-06 16:50:21 -05:00
Benoit Moriceau
6e1a56d6b1 reduce block size (#31129) 2023-10-06 11:44:37 -05:00
Edward Gao
ad5ce75be9 Revert "T&D: Use non-null columns for PKs rather then scanning raw ta… (#31082) 2023-10-04 17:25:41 -05:00
Evan Tahler
4992fce7ba T&D: Use non-null columns for PKs rather then scanning raw table to ensure all records have PKs (#30779) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-03 16:23:01 +00:00
Edward Gao
83949efd1e Destination Snowflake: upgrade to latest snowflake jdbc driver (#30938) 2023-10-02 11:18:32 -07:00
Edward Gao
6d831f1a0d Destination snowflake: Fix regression from cdk publish race condition (#30835) 2023-09-28 14:32:42 -04:00
Benoit Moriceau
0537dde7d2 Bmoric/async memory bump (#30775) 
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
 2023-09-27 18:44:20 -05:00