jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 11:01:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

238 Commits

Author SHA1 Message Date
Benoit Moriceau
894fd3a136 Bump version (#70895) 2025-12-12 10:25:44 -08:00
Benoit Moriceau
8f929b0767 chore(destination-postgres): Update CDK to 0.1.84 and fix API changes (#70338) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-09 20:16:53 +00:00
Benoit Moriceau
b232d2caed fix(destination-postgres): Force Append mode in raw tables mode for Dedupe streams (#70364) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-05 15:25:29 -08:00
Benoit Moriceau
1eca9b940f fix(destination-postgres): Fix index recreation on non-existent columns in raw tables mode (#70347) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-04 14:43:23 -08:00
Benoit Moriceau
88209fa39f refactor(destination-postgres): Move raw tables mode check to index creation (#70337) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-04 11:19:19 -08:00
Benoit Moriceau
edf538e3f9 Fix(destination-postgres): Prevent index creation on non-existent columns in raw table mode (#70326) 2025-12-04 07:52:07 -08:00
Benoit Moriceau
d264b0f215 Feature/destination postgres (#69846) 
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodhchaturvedi <subodh1810@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ian Alton <ian.alton@airbyte.io>
 2025-12-03 13:26:54 -08:00
Edward Gao
3991e28549 Destination Clickhouse: increase test timeout (#69717) 2025-11-19 15:17:28 -08:00
Aaron ("AJ") Steers
a7be6e6a0f chore: Sort externalDocumentationUrls by type then title (do not merge) (#69742) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 21:51:00 -08:00
Aaron ("AJ") Steers
d9b08c06d9 feat: Add API release history URLs to 44 certified connectors (#69741) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 20:20:34 -08:00
Aaron ("AJ") Steers
16aaee0b9e feat: Add external documentation URLs to all destination connectors (Group A) (#69353) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-11-17 21:11:07 -08:00
Edward Gao
881fb09fc0 Destination Postgres: finalize RC (#65620) 2025-08-29 15:14:27 -07:00
Edward Gao
e38a1ca126 Destination Postgres: start RC to test publish pipeline (#65617) 2025-08-29 11:24:38 -07:00
qingemeng
7e672b9fe3 Trim double quotes for numeric value (#63769) 2025-08-26 09:53:58 -07:00
Davin Chia
2844fbfeb8 chore: Update these destinations. (#61359) 
After this PR all JVM destinations will be on the new non-dagger flow.

Note on the failed builds:
destination-csv - docker only, which is why the tests are failing. I am going to deprecate this if I cannot convince Junie to fix this for me.
destination-local-json - same as destination csv.
destination-postgres-strict-encrypt - passing for me locally, looks like a certificate error
destination-postgres - passing for me locall
destination-oracle - these are failing on container errors - the test containers they are trying to download aren't able to start up
destination-oracle-strict-encrypt - same as base oracle
destination-starburst-galaxy - actual test failures
destination-yellowbrick - actual test failures
destination-redis - actual test failures
The failed tests mirror master so we aren't adding any new failures here.
 2025-06-11 19:42:38 -07:00
Davin Chia
54a7dc5bc7 chore: move remaining certified Destinations new build flow. (#60282) 
Destination S3 errors are due to flaky tests.

Destination Databricks tests are erroring out for other reasons.
 2025-05-14 17:13:04 -04:00
Aaron ("AJ") Steers
05965c781e chore: add poe_tasks.toml for all java connectors (#59670) 2025-05-05 17:28:27 -07:00
Edward Gao
402d44db70 Destinations Snowflake, Postgres: Upgrade to latest CDK for security patches (#58652) 2025-04-24 16:02:01 -07:00
devin-ai-integration[bot]
ceaf322225 chore(destination): Upgrade all Java destination connectors to use java-connector-base:2.0.1 (#56355) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: davin@airbyte.io <davin@airbyte.io>
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2025-03-24 18:59:36 -04:00
Yue Li
15b08e2f27 [destination-postgres] Add option to use unconstrained precision when a Airbyte NUMBER gets converted to a decimal type (#53686) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-02-14 19:47:30 +02:00
Augustin
f065d2ef91 destination-postgres-strict-encrypt: Use airbyte/java-connector-base:2.0.0 (#51483) 
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-01-17 10:16:51 +01:00
Augustin
c0e458d3be destination-postgres: Use airbyte/java-connector-base:1.0.0 (#49887) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-01-09 16:18:49 +01:00
Stephane Geneix
21ec465d47 destination-postgres: update all dependencies (#45434) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-09-18 14:41:26 -07:00
Augustin
601c3c455c Rename registries to registryOverrides in connector metadata (#43757) 2024-08-13 16:10:56 +02:00
Stephane Geneix
d8c5a93c8f destination-postgres: bump CDK (#43331) 
Refactored `PostgresDestination`, `PostgresSqlOperations`, and `PostgresDestinationHandler` to support the new `PostgresGenerationHandler` class for handling generation IDs.
 2024-08-07 16:57:31 -07:00
Yue Li
961149568a [destination-postgres/postgres-strict-encrypt] bump up CDK for postgres destination to use error translation framework (#43363) 
Closes airbytehq/oncall#6191
 2024-08-07 09:39:14 -07:00
Stephane Geneix
c8b95c7a3a destination-postgres: add refreshes (#42540) 
This is the original PR, with all known bugs fixed. Some tests were added in #42514, and some were parameterized here.
This has also been tested with our test connection, and the following was run in our postgres cluster:

before the version upgrade:
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       700 |         700 | 2024-07-30 01:03:07.739+00
 purchases |       700 |         100 | 2024-07-30 01:03:07.769+00
 users     |       100 |         100 | 2024-07-30 01:03:07.695+00
(3 rows)
```
after the version upgrade (sync1):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       800 |         800 | 2024-07-30 01:07:44.094+00
 purchases |       800 |         100 | 2024-07-30 01:07:44.128+00
 users     |       100 |         100 | 2024-07-30 01:07:44.052+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 100 |      0
 products  | 700 |       
 products  | 100 |      0
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 700 |       
 purchases | 100 |      0
 products  | 100 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
after the connector upgrade (sync2):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       900 |         900 | 2024-07-30 01:12:57.448+00
 purchases |       900 |         100 | 2024-07-30 01:12:57.475+00
 users     |       100 |         100 | 2024-07-30 01:12:57.421+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 100 |      0
 products  | 200 |      0
 products  | 700 |       
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 700 |       
 purchases | 200 |      0
 products  | 200 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
 2024-07-31 14:38:54 -07:00
Stephane Geneix
df1cad1102 cdk-java: add tests for interrupted refreshes (#42514) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-07-31 14:21:31 -07:00
Stephane Geneix
faae5491e5 destination-postgres: bump version to 2.2.1 (#42470) 2024-07-23 15:11:27 -07:00
Stephane Geneix
034150c13d destination-postgres: republish 2.0.15 (#42469) 
we had a publishing issue where destination-postgres-2.0.15 was overriden with code from 2.1.1 (which was broken).
We did a hard-rollback in https://github.com/airbytehq/airbyte/pull/42460 but 2.0.15 is still a bad image.
This change is only intended to republish 2.0.15 so that it has the correct (old) code and behavior
 2024-07-23 14:12:08 -07:00
Edward Gao
8ca9d22113 Destination postgres: hard revert refreshes (#42460) 2024-07-23 16:21:53 -04:00
Edward Gao
1ac98ac06d Destination postgres: remove supports refreshes (#42457) 2024-07-23 14:50:03 -04:00
Edward Gao
14bfd1458f Destination postgres: pin back but better (#42424) 2024-07-22 19:12:51 -07:00
Edward Gao
9953cf3b39 Destination postgres: rollback (#42420) 2024-07-22 21:38:53 -04:00
Stephane Geneix
3d53fb38a0 destination-postgres: close streams in PostgresSqlOperations.isOtherGenerationIdInTable (#42415) 2024-07-22 16:10:21 -07:00
Stephane Geneix
34e4fe0134 destination-postgres: implement refreshes (#41954) 
implementing refreshes for destination-postgres

we're bumping the CDK version to the latest, and modifying a whole lot of jsonl files for tests (in both regular and strict-encrypt)
 2024-07-22 13:59:33 -07:00
Stephane Geneix
591d0630a0 destination-postgres: fix PostgresDestinationTest to use isDestinationV2=true (#40616) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-07-15 12:42:35 -07:00
Stephane Geneix
7df8a9430f destination-postgres: convert stric-encrypt prod code to kotlin (#40563) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-28 10:09:46 -07:00
Stephane Geneix
661876f26d destination-postgres: convert all tests to kotlin (#40554) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-27 15:17:29 -07:00
Johnny Schmidt
14233beb19 Issue 7990: postgres destination throws config error when drop fails due to dependency and cascadeDrop=false (#40159) 2024-06-21 17:58:43 +00:00
Gireesh Sreepathi
7fb208bc82 Destination Postgres: Kotlin auto conversion src/main/java -> src/main/kotlin (#39388) 2024-06-13 10:36:18 -07:00
Enrique Alcázar Garzás
356fb8129f 🐛 Postgres Destination fails when the connection is persisted across steps (#39372) 2024-06-12 12:58:29 -03:00
Augustin
31c95da499 [skip ci] Add connectorTestSuitesOptions to metadata (#38188) 2024-05-15 15:04:10 -05:00
Evan Tahler
ec78124ad7 Delete airbyte-integrations/connectors/destination-postgres/encrypt (#38193) 2024-05-14 13:00:49 -05:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00
Gireesh Sreepathi
98e99565a9 Destination Postgres: airbyte_meta, CDK upgrade and other fixes (#36363) 2024-03-28 10:01:24 -07:00