jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,485 Commits 5,109 Branches 518 Tags
bmoric/schema-mapping-postgres
Commit Graph

104 Commits

Author SHA1 Message Date
Benoit Moriceau
894fd3a136 Bump version (#70895) 2025-12-12 10:25:44 -08:00
Benoit Moriceau
8f929b0767 chore(destination-postgres): Update CDK to 0.1.84 and fix API changes (#70338) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-09 20:16:53 +00:00
Benoit Moriceau
b232d2caed fix(destination-postgres): Force Append mode in raw tables mode for Dedupe streams (#70364) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-05 15:25:29 -08:00
Benoit Moriceau
1eca9b940f fix(destination-postgres): Fix index recreation on non-existent columns in raw tables mode (#70347) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-04 14:43:23 -08:00
Benoit Moriceau
88209fa39f refactor(destination-postgres): Move raw tables mode check to index creation (#70337) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-12-04 11:19:19 -08:00
Benoit Moriceau
edf538e3f9 Fix(destination-postgres): Prevent index creation on non-existent columns in raw table mode (#70326) 2025-12-04 07:52:07 -08:00
Benoit Moriceau
d264b0f215 Feature/destination postgres (#69846) 
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodhchaturvedi <subodh1810@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ian Alton <ian.alton@airbyte.io>
 2025-12-03 13:26:54 -08:00
Edward Gao
881fb09fc0 Destination Postgres: finalize RC (#65620) 2025-08-29 15:14:27 -07:00
Edward Gao
e38a1ca126 Destination Postgres: start RC to test publish pipeline (#65617) 2025-08-29 11:24:38 -07:00
qingemeng
7e672b9fe3 Trim double quotes for numeric value (#63769) 2025-08-26 09:53:58 -07:00
Edward Gao
402d44db70 Destinations Snowflake, Postgres: Upgrade to latest CDK for security patches (#58652) 2025-04-24 16:02:01 -07:00
Ian Alton
01cd16654e 11059 multi-instance, versioned docs (#58095) 
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-04-24 02:58:09 +03:00
devin-ai-integration[bot]
ceaf322225 chore(destination): Upgrade all Java destination connectors to use java-connector-base:2.0.1 (#56355) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: davin@airbyte.io <davin@airbyte.io>
Co-authored-by: Davin Chia <davinchia@gmail.com>
 2025-03-24 18:59:36 -04:00
Yue Li
15b08e2f27 [destination-postgres] Add option to use unconstrained precision when a Airbyte NUMBER gets converted to a decimal type (#53686) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-02-14 19:47:30 +02:00
Augustin
f065d2ef91 destination-postgres-strict-encrypt: Use airbyte/java-connector-base:2.0.0 (#51483) 
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-01-17 10:16:51 +01:00
Augustin
c0e458d3be destination-postgres: Use airbyte/java-connector-base:1.0.0 (#49887) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-01-09 16:18:49 +01:00
Stephane Geneix
21ec465d47 destination-postgres: update all dependencies (#45434) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-09-18 14:41:26 -07:00
Stephane Geneix
d8c5a93c8f destination-postgres: bump CDK (#43331) 
Refactored `PostgresDestination`, `PostgresSqlOperations`, and `PostgresDestinationHandler` to support the new `PostgresGenerationHandler` class for handling generation IDs.
 2024-08-07 16:57:31 -07:00
Yue Li
961149568a [destination-postgres/postgres-strict-encrypt] bump up CDK for postgres destination to use error translation framework (#43363) 
Closes airbytehq/oncall#6191
 2024-08-07 09:39:14 -07:00
Stephane Geneix
c8b95c7a3a destination-postgres: add refreshes (#42540) 
This is the original PR, with all known bugs fixed. Some tests were added in #42514, and some were parameterized here.
This has also been tested with our test connection, and the following was run in our postgres cluster:

before the version upgrade:
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       700 |         700 | 2024-07-30 01:03:07.739+00
 purchases |       700 |         100 | 2024-07-30 01:03:07.769+00
 users     |       100 |         100 | 2024-07-30 01:03:07.695+00
(3 rows)
```
after the version upgrade (sync1):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       800 |         800 | 2024-07-30 01:07:44.094+00
 purchases |       800 |         100 | 2024-07-30 01:07:44.128+00
 users     |       100 |         100 | 2024-07-30 01:07:44.052+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 100 |      0
 products  | 700 |       
 products  | 100 |      0
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 700 |       
 purchases | 100 |      0
 products  | 100 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
after the connector upgrade (sync2):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       900 |         900 | 2024-07-30 01:12:57.448+00
 purchases |       900 |         100 | 2024-07-30 01:12:57.475+00
 users     |       100 |         100 | 2024-07-30 01:12:57.421+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 100 |      0
 products  | 200 |      0
 products  | 700 |       
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 700 |       
 purchases | 200 |      0
 products  | 200 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
 2024-07-31 14:38:54 -07:00
Stephane Geneix
faae5491e5 destination-postgres: bump version to 2.2.1 (#42470) 2024-07-23 15:11:27 -07:00
Edward Gao
8ca9d22113 Destination postgres: hard revert refreshes (#42460) 2024-07-23 16:21:53 -04:00
Stephane Geneix
3d53fb38a0 destination-postgres: close streams in PostgresSqlOperations.isOtherGenerationIdInTable (#42415) 2024-07-22 16:10:21 -07:00
Stephane Geneix
34e4fe0134 destination-postgres: implement refreshes (#41954) 
implementing refreshes for destination-postgres

we're bumping the CDK version to the latest, and modifying a whole lot of jsonl files for tests (in both regular and strict-encrypt)
 2024-07-22 13:59:33 -07:00
Stephane Geneix
7df8a9430f destination-postgres: convert stric-encrypt prod code to kotlin (#40563) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-28 10:09:46 -07:00
Stephane Geneix
661876f26d destination-postgres: convert all tests to kotlin (#40554) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-27 15:17:29 -07:00
Johnny Schmidt
14233beb19 Issue 7990: postgres destination throws config error when drop fails due to dependency and cascadeDrop=false (#40159) 2024-06-21 17:58:43 +00:00
Gireesh Sreepathi
7fb208bc82 Destination Postgres: Kotlin auto conversion src/main/java -> src/main/kotlin (#39388) 2024-06-13 10:36:18 -07:00
Enrique Alcázar Garzás
356fb8129f 🐛 Postgres Destination fails when the connection is persisted across steps (#39372) 2024-06-12 12:58:29 -03:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Evan Tahler
a914027d74 [docs] Vendor-Specific limitations added to troubleshooting sections (#38351) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
 2024-05-21 14:42:22 +00:00
Evan Tahler
5ecaef0b8a more destination postgres warnings (#38219) 2024-05-15 09:06:14 -07:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Edward Gao
a02c342500 update destination postgres docs for column name truncation (#37352) 2024-04-16 17:59:01 +00:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00
Gireesh Sreepathi
98e99565a9 Destination Postgres: airbyte_meta, CDK upgrade and other fixes (#36363) 2024-03-28 10:01:24 -07:00
Evan Tahler
4b1ebb7b44 [docs] update pg destination warnings (#36454) 2024-03-25 17:30:49 +00:00
Evan Tahler
101bd4378d postgres is a bad datawarehouse - column size limitations (#36453) 2024-03-25 17:08:16 +00:00
Gireesh Sreepathi
55e6fbd191 Destination Snowflake: Handle NPE for state value. (#35899) 2024-03-07 18:25:38 -04:00
Edward Gao
8b83f14283 Destination postgres: upgrade cdk (#35528) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-03-05 11:54:06 -08:00
Joe Bell
2a32688970 Certify Postgres (#35760) 2024-03-01 12:16:12 -08:00
Gireesh Sreepathi
a13bd80f48 Destination Postgres: CDK T+D initial state gathering (#35385) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-02-23 12:30:10 -08:00
Gireesh Sreepathi
23cf31e80f ✨ Destination Postgres: DV2 GA (#35042) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-02-12 13:03:21 -08:00
Gireesh Sreepathi
01b6f4bede Destination Postgres: Remove varchar limit of 64k, defaults to 10MiB limit (#34891) 2024-02-06 12:56:03 -08:00
Gireesh Sreepathi
f5cec3a299 Destination Postgres: Upgrade CDK with fixed dependency and unpin cloud (#34683) 2024-01-31 09:58:18 -08:00
Gireesh Sreepathi
1ad7155092 Destination Postgres: Fix casing for raw table in T+D query (#34630) 
## What
* Fixes: #34632 

## How
* Change the raw table convention to always use lowercase, since that is the default when created as unquoted identifier.
This still preserves the mixed case identifers in Final tables.
 2024-01-29 17:25:35 -08:00