jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 11:57:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
22,318 Commits 5,118 Branches 518 Tags
baf4633ab8b51a55276115ea083837d8632fe217
Commit Graph

88 Commits

Author SHA1 Message Date
Stephane Geneix
21ec465d47 destination-postgres: update all dependencies (#45434) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-09-18 14:41:26 -07:00
Stephane Geneix
d8c5a93c8f destination-postgres: bump CDK (#43331) 
Refactored `PostgresDestination`, `PostgresSqlOperations`, and `PostgresDestinationHandler` to support the new `PostgresGenerationHandler` class for handling generation IDs.
 2024-08-07 16:57:31 -07:00
Yue Li
961149568a [destination-postgres/postgres-strict-encrypt] bump up CDK for postgres destination to use error translation framework (#43363) 
Closes airbytehq/oncall#6191
 2024-08-07 09:39:14 -07:00
Stephane Geneix
c8b95c7a3a destination-postgres: add refreshes (#42540) 
This is the original PR, with all known bugs fixed. Some tests were added in #42514, and some were parameterized here.
This has also been tested with our test connection, and the following was run in our postgres cluster:

before the version upgrade:
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       700 |         700 | 2024-07-30 01:03:07.739+00
 purchases |       700 |         100 | 2024-07-30 01:03:07.769+00
 users     |       100 |         100 | 2024-07-30 01:03:07.695+00
(3 rows)
```
after the version upgrade (sync1):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       800 |         800 | 2024-07-30 01:07:44.094+00
 purchases |       800 |         100 | 2024-07-30 01:07:44.128+00
 users     |       100 |         100 | 2024-07-30 01:07:44.052+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 100 |      0
 products  | 700 |       
 products  | 100 |      0
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 700 |       
 purchases | 100 |      0
 products  | 100 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
after the connector upgrade (sync2):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       900 |         900 | 2024-07-30 01:12:57.448+00
 purchases |       900 |         100 | 2024-07-30 01:12:57.475+00
 users     |       100 |         100 | 2024-07-30 01:12:57.421+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 100 |      0
 products  | 200 |      0
 products  | 700 |       
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 700 |       
 purchases | 200 |      0
 products  | 200 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
 2024-07-31 14:38:54 -07:00
Stephane Geneix
faae5491e5 destination-postgres: bump version to 2.2.1 (#42470) 2024-07-23 15:11:27 -07:00
Edward Gao
8ca9d22113 Destination postgres: hard revert refreshes (#42460) 2024-07-23 16:21:53 -04:00
Stephane Geneix
3d53fb38a0 destination-postgres: close streams in PostgresSqlOperations.isOtherGenerationIdInTable (#42415) 2024-07-22 16:10:21 -07:00
Stephane Geneix
34e4fe0134 destination-postgres: implement refreshes (#41954) 
implementing refreshes for destination-postgres

we're bumping the CDK version to the latest, and modifying a whole lot of jsonl files for tests (in both regular and strict-encrypt)
 2024-07-22 13:59:33 -07:00
Stephane Geneix
7df8a9430f destination-postgres: convert stric-encrypt prod code to kotlin (#40563) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-28 10:09:46 -07:00
Stephane Geneix
661876f26d destination-postgres: convert all tests to kotlin (#40554) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-06-27 15:17:29 -07:00
Johnny Schmidt
14233beb19 Issue 7990: postgres destination throws config error when drop fails due to dependency and cascadeDrop=false (#40159) 2024-06-21 17:58:43 +00:00
Gireesh Sreepathi
7fb208bc82 Destination Postgres: Kotlin auto conversion src/main/java -> src/main/kotlin (#39388) 2024-06-13 10:36:18 -07:00
Enrique Alcázar Garzás
356fb8129f 🐛 Postgres Destination fails when the connection is persisted across steps (#39372) 2024-06-12 12:58:29 -03:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Evan Tahler
a914027d74 [docs] Vendor-Specific limitations added to troubleshooting sections (#38351) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
 2024-05-21 14:42:22 +00:00
Evan Tahler
5ecaef0b8a more destination postgres warnings (#38219) 2024-05-15 09:06:14 -07:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Edward Gao
a02c342500 update destination postgres docs for column name truncation (#37352) 2024-04-16 17:59:01 +00:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00
Gireesh Sreepathi
98e99565a9 Destination Postgres: airbyte_meta, CDK upgrade and other fixes (#36363) 2024-03-28 10:01:24 -07:00
Evan Tahler
4b1ebb7b44 [docs] update pg destination warnings (#36454) 2024-03-25 17:30:49 +00:00
Evan Tahler
101bd4378d postgres is a bad datawarehouse - column size limitations (#36453) 2024-03-25 17:08:16 +00:00
Gireesh Sreepathi
55e6fbd191 Destination Snowflake: Handle NPE for state value. (#35899) 2024-03-07 18:25:38 -04:00
Edward Gao
8b83f14283 Destination postgres: upgrade cdk (#35528) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-03-05 11:54:06 -08:00
Joe Bell
2a32688970 Certify Postgres (#35760) 2024-03-01 12:16:12 -08:00
Gireesh Sreepathi
a13bd80f48 Destination Postgres: CDK T+D initial state gathering (#35385) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-02-23 12:30:10 -08:00
Gireesh Sreepathi
23cf31e80f ✨ Destination Postgres: DV2 GA (#35042) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-02-12 13:03:21 -08:00
Gireesh Sreepathi
01b6f4bede Destination Postgres: Remove varchar limit of 64k, defaults to 10MiB limit (#34891) 2024-02-06 12:56:03 -08:00
Gireesh Sreepathi
f5cec3a299 Destination Postgres: Upgrade CDK with fixed dependency and unpin cloud (#34683) 2024-01-31 09:58:18 -08:00
Gireesh Sreepathi
1ad7155092 Destination Postgres: Fix casing for raw table in T+D query (#34630) 
## What
* Fixes: #34632 

## How
* Change the raw table convention to always use lowercase, since that is the default when created as unquoted identifier.
This still preserves the mixed case identifers in Final tables.
 2024-01-29 17:25:35 -08:00
Gireesh Sreepathi
6b4f215c6d Destination Postgres: DV2 Beta (#34372) 
## What
* Adding postgres DV2 beta launch option
 2024-01-22 09:07:13 -08:00
Edward Gao
d610ad1f48 Destination postgres (dv2): add indexes to raw table (#34236) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-18 16:21:24 -08:00
Edward Gao
006338257c Destination postgres: DV2 beta implementation (#34177) 2024-01-17 10:32:21 -08:00
Gireesh Sreepathi
d29cb2d41e Publish CDK and fix postgres to use cdk (#34135) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-10 20:48:14 -04:00
Gireesh Sreepathi
a0ece1231e destination-postgres: Add tunnel heartbeats and keepalive (#33875) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-10 15:42:32 -08:00
Edward Gao
216d2b2e70 Destination postgres: enable in-connector normalization (#33873) 2024-01-04 14:15:33 -04:00
Gireesh Sreepathi
af4f5ae570 Destination Postgres - fix tests (#33507) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-12-18 17:51:16 -08:00
Evan Tahler
9210547af5 [Docs] No Deduped + Hostory, Append + Deduped is the future! (#29114) 
* [Docs] No `Deduped + Hostory`, `Append + Deduped` is the future!

* fix links
 2023-08-08 15:07:49 -07:00
Evan Tahler
f455c1288d Java DB Destination connector licenses to Elv2 (#27781) 
* Java DB Destination connector licenses to Elv2

* PR id for docs

* fix redshift tagging
 2023-06-29 12:26:24 -05:00
Evan Tahler
4c26efb6b8 destination-postgres docs warning for large data (#25817) 
* destination-postgres warning for large data

* :::caution
 2023-05-05 09:15:29 -07:00
Ryan Fu
73900fad3e JDBC connections to support checkpointing (#24604) 
* Shifted logic within JDBC connections to support checkpointing

* Bumped version numbers and updated changelog
 2023-04-05 18:32:28 +00:00
Sophia Wiley
5512befeb1 Docs: updated links from .io to .com (#23652) 
* updated links

* edited contributors link

* deleted line about CDK in docs
 2023-03-06 17:27:55 +01:00
Werner Fouché
05390991ec Update postgres.md (#18432) 2022-10-25 15:09:53 -03:00
Tyler B
40896d00c0 Re-direct Postgres guide to active page, current link is 404'd (#18103) 
The current  "Refer to this guide..." link goes to a 404 page on postgres. The new link i put in is the equivalent of the old site. It can be compared using the wayback machine: https://web.archive.org/web/20220121223117/https://jdbc.postgresql.org/documentation/head/connect.html
 2022-10-18 10:47:39 -04:00
Rodi Reich Zilberman
248da78ba7 Make error more descriptive in case of no encryption state (#17299) 
* Make error more descriptive in case of no encryption state

* Fix failing test

* Change error message on strict-encrypt postgres destination as well. Fix failing test

* update versions and changelogs

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-28 15:26:09 -07:00
Amruta Ranade
a6af9611a6 added SSL Tunnel method rec (#16782) 2022-09-15 17:36:32 -04:00