jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

121 Commits

Author SHA1 Message Date
Aaron ("AJ") Steers
a7be6e6a0f chore: Sort externalDocumentationUrls by type then title (do not merge) (#69742) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 21:51:00 -08:00
Aaron ("AJ") Steers
16aaee0b9e feat: Add external documentation URLs to all destination connectors (Group A) (#69353) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-11-17 21:11:07 -08:00
Edward Gao
881fb09fc0 Destination Postgres: finalize RC (#65620) 2025-08-29 15:14:27 -07:00
Edward Gao
e38a1ca126 Destination Postgres: start RC to test publish pipeline (#65617) 2025-08-29 11:24:38 -07:00
Edward Gao
90f5ea9df9 Destination Postgres: publish strict-encrypt to pull in NUMERIC workaround (#65577) 2025-08-28 07:55:58 -07:00
Davin Chia
2844fbfeb8 chore: Update these destinations. (#61359) 
After this PR all JVM destinations will be on the new non-dagger flow.

Note on the failed builds:
destination-csv - docker only, which is why the tests are failing. I am going to deprecate this if I cannot convince Junie to fix this for me.
destination-local-json - same as destination csv.
destination-postgres-strict-encrypt - passing for me locally, looks like a certificate error
destination-postgres - passing for me locall
destination-oracle - these are failing on container errors - the test containers they are trying to download aren't able to start up
destination-oracle-strict-encrypt - same as base oracle
destination-starburst-galaxy - actual test failures
destination-yellowbrick - actual test failures
destination-redis - actual test failures
The failed tests mirror master so we aren't adding any new failures here.
 2025-06-11 19:42:38 -07:00
Aaron ("AJ") Steers
05965c781e chore: add poe_tasks.toml for all java connectors (#59670) 2025-05-05 17:28:27 -07:00
Edward Gao
402d44db70 Destinations Snowflake, Postgres: Upgrade to latest CDK for security patches (#58652) 2025-04-24 16:02:01 -07:00
devin-ai-integration[bot]
ceaf322225 chore(destination): Upgrade all Java destination connectors to use java-connector-base:2.0.1 (#56355) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: davin@airbyte.io <davin@airbyte.io>
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2025-03-24 18:59:36 -04:00
Yue Li
15b08e2f27 [destination-postgres] Add option to use unconstrained precision when a Airbyte NUMBER gets converted to a decimal type (#53686) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-02-14 19:47:30 +02:00
Augustin
f065d2ef91 destination-postgres-strict-encrypt: Use airbyte/java-connector-base:2.0.0 (#51483) 
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-01-17 10:16:51 +01:00
Augustin
c0e458d3be destination-postgres: Use airbyte/java-connector-base:1.0.0 (#49887) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-01-09 16:18:49 +01:00
Stephane Geneix
21ec465d47 destination-postgres: update all dependencies (#45434) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-09-18 14:41:26 -07:00
Augustin
601c3c455c Rename registries to registryOverrides in connector metadata (#43757) 2024-08-13 16:10:56 +02:00
Yue Li
961149568a [destination-postgres/postgres-strict-encrypt] bump up CDK for postgres destination to use error translation framework (#43363) 
Closes airbytehq/oncall#6191
 2024-08-07 09:39:14 -07:00
Stephane Geneix
c8b95c7a3a destination-postgres: add refreshes (#42540) 
This is the original PR, with all known bugs fixed. Some tests were added in #42514, and some were parameterized here.
This has also been tested with our test connection, and the following was run in our postgres cluster:

before the version upgrade:
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       700 |         700 | 2024-07-30 01:03:07.739+00
 purchases |       700 |         100 | 2024-07-30 01:03:07.769+00
 users     |       100 |         100 | 2024-07-30 01:03:07.695+00
(3 rows)
```
after the version upgrade (sync1):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       800 |         800 | 2024-07-30 01:07:44.094+00
 purchases |       800 |         100 | 2024-07-30 01:07:44.128+00
 users     |       100 |         100 | 2024-07-30 01:07:44.052+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 100 |      0
 products  | 700 |       
 products  | 100 |      0
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 700 |       
 purchases | 100 |      0
 products  | 100 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
after the connector upgrade (sync2):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       900 |         900 | 2024-07-30 01:12:57.448+00
 purchases |       900 |         100 | 2024-07-30 01:12:57.475+00
 users     |       100 |         100 | 2024-07-30 01:12:57.421+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 100 |      0
 products  | 200 |      0
 products  | 700 |       
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 700 |       
 purchases | 200 |      0
 products  | 200 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
 2024-07-31 14:38:54 -07:00
Stephane Geneix
faae5491e5 destination-postgres: bump version to 2.2.1 (#42470) 2024-07-23 15:11:27 -07:00
Stephane Geneix
034150c13d destination-postgres: republish 2.0.15 (#42469) 
we had a publishing issue where destination-postgres-2.0.15 was overriden with code from 2.1.1 (which was broken).
We did a hard-rollback in https://github.com/airbytehq/airbyte/pull/42460 but 2.0.15 is still a bad image.
This change is only intended to republish 2.0.15 so that it has the correct (old) code and behavior
 2024-07-23 14:12:08 -07:00
Edward Gao
8ca9d22113 Destination postgres: hard revert refreshes (#42460) 2024-07-23 16:21:53 -04:00
Edward Gao
1ac98ac06d Destination postgres: remove supports refreshes (#42457) 2024-07-23 14:50:03 -04:00
Edward Gao
14bfd1458f Destination postgres: pin back but better (#42424) 2024-07-22 19:12:51 -07:00
Edward Gao
9953cf3b39 Destination postgres: rollback (#42420) 2024-07-22 21:38:53 -04:00
Stephane Geneix
3d53fb38a0 destination-postgres: close streams in PostgresSqlOperations.isOtherGenerationIdInTable (#42415) 2024-07-22 16:10:21 -07:00
Stephane Geneix
34e4fe0134 destination-postgres: implement refreshes (#41954) 
implementing refreshes for destination-postgres

we're bumping the CDK version to the latest, and modifying a whole lot of jsonl files for tests (in both regular and strict-encrypt)
 2024-07-22 13:59:33 -07:00
Stephane Geneix
5bf2eb6a2b destination-postgres: convert strict-encrypt tests to kotlin (#40564) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-07-15 12:28:21 -07:00
Stephane Geneix
7df8a9430f destination-postgres: convert stric-encrypt prod code to kotlin (#40563) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-28 10:09:46 -07:00
Stephane Geneix
661876f26d destination-postgres: convert all tests to kotlin (#40554) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-27 15:17:29 -07:00
Johnny Schmidt
6839d0d4af Increment the postgres-strict metadata version to match unstrict (#40200) 2024-06-21 16:27:54 -07:00
Gireesh Sreepathi
7fb208bc82 Destination Postgres: Kotlin auto conversion src/main/java -> src/main/kotlin (#39388) 2024-06-13 10:36:18 -07:00
Enrique Alcázar Garzás
356fb8129f 🐛 Postgres Destination fails when the connection is persisted across steps (#39372) 2024-06-12 12:58:29 -03:00
Augustin
31c95da499 [skip ci] Add connectorTestSuitesOptions to metadata (#38188) 2024-05-15 15:04:10 -05:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00
Gireesh Sreepathi
98e99565a9 Destination Postgres: airbyte_meta, CDK upgrade and other fixes (#36363) 2024-03-28 10:01:24 -07:00
Marius Posta
ef473c5338 gradle: add kotlin support in all projects, warnings as errors (#36018) 2024-03-13 10:56:17 -05:00
Gireesh Sreepathi
55e6fbd191 Destination Snowflake: Handle NPE for state value. (#35899) 2024-03-07 18:25:38 -04:00
Edward Gao
8b83f14283 Destination postgres: upgrade cdk (#35528) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-03-05 11:54:06 -08:00
Joe Bell
7063ea3fd9 Postgres Strict Encrypt (#35762) 2024-03-01 13:39:38 -08:00
Gireesh Sreepathi
a13bd80f48 Destination Postgres: CDK T+D initial state gathering (#35385) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-02-23 12:30:10 -08:00
Gireesh Sreepathi
23cf31e80f ✨ Destination Postgres: DV2 GA (#35042) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-02-12 13:03:21 -08:00
Gireesh Sreepathi
01b6f4bede Destination Postgres: Remove varchar limit of 64k, defaults to 10MiB limit (#34891) 2024-02-06 12:56:03 -08:00
Gireesh Sreepathi
f5cec3a299 Destination Postgres: Upgrade CDK with fixed dependency and unpin cloud (#34683) 2024-01-31 09:58:18 -08:00
Joe Bell
db3055e774 Revert Default Cloud Version (#34646) 2024-01-29 22:18:29 -08:00
Gireesh Sreepathi
1ad7155092 Destination Postgres: Fix casing for raw table in T+D query (#34630) 
## What
* Fixes: #34632 

## How
* Change the raw table convention to always use lowercase, since that is the default when created as unquoted identifier.
This still preserves the mixed case identifers in Final tables.
 2024-01-29 17:25:35 -08:00
Cynthia Yin
7d89fc9112 📚 Destinations V2: update typing & deduping doc link (#34505) 2024-01-24 17:50:42 -06:00
Gireesh Sreepathi
6b4f215c6d Destination Postgres: DV2 Beta (#34372) 
## What
* Adding postgres DV2 beta launch option
 2024-01-22 09:07:13 -08:00
Edward Gao
d610ad1f48 Destination postgres (dv2): add indexes to raw table (#34236) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-18 16:21:24 -08:00