1
0
mirror of synced 2025-12-23 11:57:55 -05:00
Commit Graph

2421 Commits

Author SHA1 Message Date
Gireesh Sreepathi
e0225c18b2 Destination BigQuery: Consolidation of objects to StreamConfig, cleanup (#38131)
## What
Removing redundant references and duplicate information passed around using `WriteConfig` objects. No functional changes and resurrected all the information needed through `StreamConfig` and adapted changes accordingly. 

This PR should be in a mergeable state with no functional changes after the ones down the stack are published. 


## Review guide
* Removed references of `BigQueryWriteConfig` and reused already built `StreamConfig`
* Removing unnecessary `StagingOperations` interface and made concrete class, this will help for later adding a shim on this and refactoring without large changes 
* Removed other unnecessary references of getting dynamic schema, `WriteDispostion` etc. Probably remnant of bigquery-denormalized bespoke connector. 

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [x] YES 💚
- [ ] NO 
2024-05-13 11:09:19 -07:00
Gireesh Sreepathi
13e16de71f redshift_catalog_parser_update (#38126)
## What
Adapting to CDK signature changes in `StreamConfig`

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [x] YES 💚
- [ ] NO 
2024-05-13 11:07:52 -07:00
Edward Gao
9b0db78e7f Destination Snowflake: Remove incremental T+D option (#37715) 2024-05-10 16:16:18 -04:00
Stephane Geneix
eadbd41b95 destination-bigquery: bump CDK version (#38125)
the latest CDK should prevent the destination from hanging when encountering an error in the exception handler.
2024-05-10 12:38:54 -07:00
Gireesh Sreepathi
1188221473 Destination BigQuery: Nuking old remnants (#38111) 2024-05-10 10:54:42 -07:00
Edward Gao
3525225e6e Destination dev null: support dedup (#38118) 2024-05-10 10:28:24 -07:00
Edward Gao
e950c56cc5 Destination bigquery: switch to artifact cdk (#38098) 2024-05-09 17:26:52 -04:00
Edward Gao
4c7ebfa4cb Destination e2e test: enable dedup (#38097) 2024-05-09 16:44:41 -04:00
Edward Gao
985c839673 Destination Redshift: Remove incremental T+D option (#37713) 2024-05-08 14:29:14 -04:00
Edward Gao
47c86a9ea6 Destination Bigquery: Remove reference to TDValve (#37714) 2024-05-08 14:22:40 -04:00
Stephane Geneix
16ed6bf3c6 revert contention-reducing change in destination-snowflake (#38052)
as part of the move of destination-snowflake to the kotlin CDK, we tried improve concurrency by only `DELETE`ing from `_airbyte_destination_state` if it has some data to delete (by issuing an `IF EXISTS` in the same transaction.
Looks like it might be causing some stuck syncs, so we're reverting that "improvement"
2024-05-08 11:00:49 -07:00
Marcos Marx
327e010673 Marcos/destination firebolt up sdk (#38060)
Co-authored-by: ptiurin <petro.tiurin@firebolt.io>
Co-authored-by: Petro Tiurin <93913847+ptiurin@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2024-05-08 13:35:45 -03:00
Edward Gao
3e609f1403 Destination snowflake: Update to latest cdk (#34612) 2024-05-07 19:14:05 -04:00
Edward Gao
813510a71f Destination bigquery: Update to latest CDK (#34611) 2024-05-07 18:21:03 -04:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Bindi Pankhudi
0c145d9db0 Bindi/support for pinecone serverless (#37756)
Co-authored-by: Roie Schwaber-Cohen <@rschwabco>
2024-05-06 16:22:26 -07:00
Gireesh Sreepathi
3cca1c0ddd Destination Snowflake: Use safe executeMetadataQuery call (#37857) 2024-05-06 15:11:10 -07:00
Edward Gao
d31f0dd7a7 Destination redshift: Update to latest cdk (#34613)
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
2024-05-06 11:48:16 -07:00
Edward Gao
e8e85eb778 Destination Mysql: DV2 (#36936) 2024-05-03 16:44:46 -04:00
Stephane Geneix
7c0a6c569d convert destination-snowflake to Kotlin CDK (#36910)
not only bringing snowflake to the latest CDK but also:
1) Bringing the `SourceOperation` into production code from the test code. There's really no reason those improvements should stay out of production (and they're present in the source-snowflake)
2) adding `putTimestamp` into the `SourceOperation`, so that snowflake doesn't throw an exception at every call, which implies it also creates a new thread
3) make use of the newly added ability to filter orphan thread on shutdown. We filter all the threads created during calls to `SFStatement.close()`
4) don't always take a lock when deleting destinationStates. We now check if there's any states to delete by doing a `SELECT` (and not taking any table lock) before issuing the `DELETE` (the old behavior was causing test contention, and it's a bad idea in general)
5) only execute `airbyte_internal._airbyte_destination_state`
2024-05-03 11:20:07 -07:00
Alex
4a49615364 Update destinations/gcs.md (#37757) 2024-05-01 20:03:30 -05:00
Joe Bell
994e8c9015 fix admonition rendering for bq (#37730) 2024-04-30 22:41:31 +00:00
Benoit Moriceau
b8fad80f90 Cloud e2e support refreshes (#37694) 2024-04-30 08:50:00 -07:00
Mark Cusack
39f036a8a3 🎉 New Destination: Yellowbrick (#35775)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-04-30 12:09:30 -03:00
Benoit Moriceau
bfb7f8b73c e2e support refreshes (#37675)
Trivial change on non critical connector
2024-04-29 15:12:06 -07:00
menishak
3b5c44aaf3 🐛 Destination AWS Datalake: Enable STS Role Credential Refresh for Long Syncs (#33853)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2024-04-29 10:05:31 -03:00
Evan Tahler
e8859b2121 Exclude zookeeper dependencies (#37584) 2024-04-26 16:28:26 -04:00
Stefano Lottini
d84aa48f53 [destination-astra] Add user-agent with caller name (#37405)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2024-04-26 15:08:17 -03:00
Stephane Geneix
c4ad3d9660 convert destination-bigquery to kotlin CDK (#36899) 2024-04-25 14:46:47 -07:00
Joe Bell
fce34098fe bump s3 cdk version (#37546) 2024-04-24 10:34:38 -07:00
Aaron ("AJ") Steers
5f29e8fd25 Fix (destination-duckdb): Declare min mem constraints in manifest (#37515) 2024-04-23 15:04:50 -07:00
Louisa H
d4944a38e4 [destination-DuckDB] Improve performance, use pyarrow batch insert as replacement of executemany (#36715) 2024-04-22 15:22:05 -07:00
Joe Bell
54b0a7b86b MSSQL remove normalization (#36050) 2024-04-22 13:23:17 -07:00
Benoit Moriceau
a49f205ccb Fix NPE (#37382) 2024-04-17 11:51:19 -04:00
Benoit Moriceau
6536a390b5 Fix log trace message (#37366) 2024-04-16 18:06:49 -07:00
Bindi Pankhudi
a15e88cc20 Update CDK & pytest versions for Vector DB destinations (#37333) 2024-04-16 14:28:09 -07:00
Edward Gao
a02c342500 update destination postgres docs for column name truncation (#37352) 2024-04-16 17:59:01 +00:00
Edward Gao
faad484e08 Destination mysql: upgrade to latest cdk (#36926) 2024-04-12 14:41:58 -04:00
Joe Bell
b0d0dcd297 oracle_x_norm (#36048)
The testing framework for oracle is not functional and will not be fixed for this release
2024-04-11 16:01:50 -07:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Gireesh Sreepathi
e16b0d2119 Destination Redshift: Limit Standard insert statement size < 16MB (#36973) 2024-04-10 16:36:07 -07:00
Edward Gao
4fc7f1a82d Destination snowflake: Add config option to enable time travel (#35754) 2024-04-10 15:54:02 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Gireesh Sreepathi
5ea2b93c38 Destination S3: Adapt kotlin CDK (#36869) 2024-04-08 12:15:01 -07:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Aaron ("AJ") Steers
4ac078b382 Fix(destination-duckdb): Re-lock project dependencies, resolve failing check (#36884) 2024-04-07 15:27:47 -07:00
Edward Gao
9be3a8ec56 Destination redshift: delete unused spec option (#36365) 2024-04-05 19:04:00 -04:00
Evan Tahler
4342182cae Remove duplicate S3 regions from destination specs (#36846) 2024-04-05 16:24:26 -05:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00