jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-22 11:31:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

321 Commits

Author SHA1 Message Date
Gireesh Sreepathi
0e4ffb561a Destination BigQuery: add row ids for dummy inserts in check (#34021) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-08 19:13:30 -04:00
Gireesh Sreepathi
7503dc78f1 destination-redshift: Heartbeats for bastion and other minor bugfixes (#33948) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2024-01-08 09:09:17 -08:00
Edward Gao
29a3da04ca Destinations Bigquery, snowflake, redshift: internal CDK refactors for DV2 (#33730) 2024-01-04 21:20:15 -04:00
Edward Gao
1d76fd2618 🎉 Destination redshift: reenable T+D optimizations; bigquery+snowflake: bump to latest CDK version (#33704) 2023-12-20 21:09:44 -06:00
Joe Bell
f1099f5b53 🐛 Destination BigQuery - Remove unnecessary temp tables (#33697) 2023-12-20 14:55:45 -08:00
Joe Bell
5b826ecbed Destination Snowflake, BigQuery, JavaCdk - Move Schema Creation out of each table creation (#33124) 
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
 2023-12-20 17:40:49 -04:00
Edward Gao
dca6be0778 Destination bigquery: remove unused option from spec? (#33451) 2023-12-14 17:20:44 +00:00
Cynthia Yin
73b7e7bb49 📚 Destinations V2: update data types for BigQuery + Snowflake (#33471) 2023-12-14 00:59:15 +00:00
Edward Gao
d27ea33ffe DV2: Only run T+D if we have zero records or the previous sync left behind un-T+D-ed records (#33232) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-12-13 19:47:36 -04:00
Edward Gao
a9b1e8d483 Destinations bigquery+snowflake+s3: Chore: upgrade CDK version (#33263) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-12-08 18:24:18 -04:00
Cynthia Yin
8c592fec4d Destination BigQuery: bump version (follow-up) (#33259) 2023-12-08 11:49:03 -08:00
Evan Tahler
95a8e42a4b T&D SQL statements moved to debug log level (#33084) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
 2023-12-04 21:11:02 -04:00
Edward Gao
7863298966 Destination bigquery: further increase gcs COPY timeout (#33078) 2023-12-04 16:34:50 -08:00
Evan Tahler
0e166ebd24 Async Destinations - Make debug messages less scary (#32526) 2023-11-15 15:32:48 -04:00
Edward Gao
a79fb06f8a Destination bigquery error grouping: even more fixes (#32468) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-13 20:04:33 -04:00
Edward Gao
53bbac8656 Destination Bigquery docs: Clarify service account usage (#18642) 2023-11-13 19:48:04 +00:00
Evan Tahler
4d5d259fb6 Destination BigQuery - remove file_buffer_count (#32234) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
 2023-11-08 20:03:47 -04:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00
Edward Gao
c5d183bce4 Destination bigquery: revert typed config object (#32309) 2023-11-08 13:10:37 -06:00
Gireesh Sreepathi
6e10ee3051 Destination Bigquery: Replace usages of untyped JsonNode with Typed execution config (#32147) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: gisripa <gisripa@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
 2023-11-08 09:35:57 -08:00
Edward Gao
b7eab7564d Destination bigquery: don't error on malformed creds (#32261) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-07 19:06:12 -04:00
Edward Gao
37172c1261 Destination bigquery: reduce commit frequency in GCS staging mode (#32112) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-07 18:23:31 -04:00
Joe Bell
04c067d01d Destinations BigQuery and Snowflake - Move SAFE/TRY casting to seperate transaction (#32026) 
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-06 20:33:54 -04:00
Edward Gao
c81abe51a8 Destination bigquery + Snowflake: Further error grouping improvements (#32190) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-06 13:25:06 -04:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Edward Gao
453232b366 Destinations bigquery+snowflake: better sentry error grouping (#31983) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-11-02 16:32:11 +00:00
Al-Moatasem
1ce84d6396 ✨-Destination BigQuery: Updating Dataset Locations List (#32010) 2023-10-31 14:58:30 -05:00
Edward Gao
82679a679c Destinations snowflake+bigquery: delay upgrade deadline to nov 7 (#31985) 
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
 2023-10-31 11:47:48 -04:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Edward Gao
45505dd45c Destination bigquery: Use merge statement to do T+D (#31529) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-27 15:55:50 +00:00
Gireesh Sreepathi
3e764712f4 Destination BigQuery + Snowflake : Disable type and dedupe step controlled by flag in Connector spec (#31686) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: gisripa <gisripa@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2023-10-26 13:46:54 -07:00
Edward Gao
148dda1c1f Destinations Bigquery+Snowflake: Do not dedup raw table (#31520) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-25 12:19:58 -04:00
Joe Bell
500a7d9a75 Destination Bigquery - Remove v2 check (#31717) 2023-10-23 11:07:54 -07:00
Ignacio Valdelvira
06dc186531 Update bigquery.md with output schemas of V2 destination (#31548) 
Co-authored-by: Evan Tahler <evantahler@gmail.com>
 2023-10-19 08:29:29 -05:00
Benoit Moriceau
25dc4f6df2 Add standard async (#29812) 
Co-authored-by: tryangul <ryan.broughan@gmail.com>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: tryangul <tryangul@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-10-17 14:56:00 -07:00
Edward Gao
f1baf2a2f2 Destinations snowflake+bigquery: Improve performance by filtering raw table on extracted_at (#31191) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-17 17:48:18 +00:00
Gireesh Sreepathi
c4ff29e23a Destinations bigquery: Fix string-json auto conversion for BigQuery (#31358) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: gisripa <gisripa@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-10-12 17:13:39 -07:00
Ryan Br
5f9bec01bd Remove entry and allocated memory for empty buffers. (#31194) 
Co-authored-by: tryangul <tryangul@users.noreply.github.com>
 2023-10-11 11:22:46 -07:00
Cynthia Yin
e3d126b5e7 🐛 Fixing rounding of numeric values for async destinations (#31083) 2023-10-10 22:13:33 -04:00
Evan Tahler
898846d40a Remove Dv2 Null checks (#31149) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-10-10 15:36:00 -04:00
Evan Tahler
589ea37004 destination-bigquery - better config groups (#31198) 
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
 2023-10-10 10:48:01 -07:00
Evan Tahler
f4f97e3e65 destination-bigquery: Increase staging file upload timeout to 5 minutes (#31185) 2023-10-09 17:28:48 +00:00
Benoit Moriceau
9e2b3f4990 Bump CDk (#31139) 2023-10-06 16:50:21 -05:00
Benoit Moriceau
6e1a56d6b1 reduce block size (#31129) 2023-10-06 11:44:37 -05:00
Edward Gao
ad5ce75be9 Revert "T&D: Use non-null columns for PKs rather then scanning raw ta… (#31082) 2023-10-04 17:25:41 -05:00
Joe Bell
4ebac2e032 Destination Bigquery - Triple timeout (#31028) 2023-10-04 18:49:11 +00:00
Evan Tahler
4992fce7ba T&D: Use non-null columns for PKs rather then scanning raw table to ensure all records have PKs (#30779) 
Co-authored-by: evantahler <evantahler@users.noreply.github.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-10-03 16:23:01 +00:00
Evan Tahler
8d1bc7305c Remove mention of bigquery-denormalized from bigquery docs (#30848) 2023-09-28 19:33:19 +00:00
Benoit Moriceau
0537dde7d2 Bmoric/async memory bump (#30775) 
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
 2023-09-27 18:44:20 -05:00
Edward Gao
4c26d4cceb DV2: Fix column name collision detection (#30739) 
Co-authored-by: edgao <edgao@users.noreply.github.com>
 2023-09-27 15:57:43 -07:00