jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-22 11:31:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

61 Commits

Author SHA1 Message Date
Benoit Moriceau
d264b0f215 Feature/destination postgres (#69846) 
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodhchaturvedi <subodh1810@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ian Alton <ian.alton@airbyte.io>
 2025-12-03 13:26:54 -08:00
Davin Chia
54a7dc5bc7 chore: move remaining certified Destinations new build flow. (#60282) 
Destination S3 errors are due to flaky tests.

Destination Databricks tests are erroring out for other reasons.
 2025-05-14 17:13:04 -04:00
Edward Gao
402d44db70 Destinations Snowflake, Postgres: Upgrade to latest CDK for security patches (#58652) 2025-04-24 16:02:01 -07:00
Augustin
f065d2ef91 destination-postgres-strict-encrypt: Use airbyte/java-connector-base:2.0.0 (#51483) 
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2025-01-17 10:16:51 +01:00
Stephane Geneix
21ec465d47 destination-postgres: update all dependencies (#45434) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-09-18 14:41:26 -07:00
Stephane Geneix
d8c5a93c8f destination-postgres: bump CDK (#43331) 
Refactored `PostgresDestination`, `PostgresSqlOperations`, and `PostgresDestinationHandler` to support the new `PostgresGenerationHandler` class for handling generation IDs.
 2024-08-07 16:57:31 -07:00
Yue Li
961149568a [destination-postgres/postgres-strict-encrypt] bump up CDK for postgres destination to use error translation framework (#43363) 
Closes airbytehq/oncall#6191
 2024-08-07 09:39:14 -07:00
Stephane Geneix
c8b95c7a3a destination-postgres: add refreshes (#42540) 
This is the original PR, with all known bugs fixed. Some tests were added in #42514, and some were parameterized here.
This has also been tested with our test connection, and the following was run in our postgres cluster:

before the version upgrade:
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       700 |         700 | 2024-07-30 01:03:07.739+00
 purchases |       700 |         100 | 2024-07-30 01:03:07.769+00
 users     |       100 |         100 | 2024-07-30 01:03:07.695+00
(3 rows)
```
after the version upgrade (sync1):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       800 |         800 | 2024-07-30 01:07:44.094+00
 purchases |       800 |         100 | 2024-07-30 01:07:44.128+00
 users     |       100 |         100 | 2024-07-30 01:07:44.052+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 100 |      0
 products  | 700 |       
 products  | 100 |      0
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all
select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      8
 purchases | 700 |       
 purchases | 100 |      0
 products  | 100 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
after the connector upgrade (sync2):
```
postgres=> select 'products' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_products ) raw_count, (select count(*) from dest_v2_perf.products) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.products ) max_extracted_at
union all
select 'purchases' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_purchases ) raw_count, (select count(*) from dest_v2_perf.purchases) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.purchases ) max_extracted_at
union all
select 'users' stream, (select count(*) from airbyte_internal.dest_v2_perf_raw__stream_users ) raw_count, (select count(*) from dest_v2_perf.users) final_count, (select max(_airbyte_extracted_at) from dest_v2_perf.users ) max_extracted_at;
  stream   | raw_count | final_count |      max_extracted_at      
-----------+-----------+-------------+----------------------------
 products  |       900 |         900 | 2024-07-30 01:12:57.448+00
 purchases |       900 |         100 | 2024-07-30 01:12:57.475+00
 users     |       100 |         100 | 2024-07-30 01:12:57.421+00
(3 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from dest_v2_perf.purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from dest_v2_perf.products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 100 |      0
 products  | 200 |      0
 products  | 700 |       
(4 rows)

postgres=> select 'users' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_users group by 1, 3
union all
select 'purchases', count(*), _airbyte_generation_id from airbyte_internal.dest_v2_perf_raw__stream_purchases group by 1, 3
union all                                                                                                                                                                                                                                          select 'products' name, count(*) ct, _airbyte_generation_id gen_id from airbyte_internal.dest_v2_perf_raw__stream_products group by 1, 3
;
   name    | ct  | gen_id 
-----------+-----+--------
 users     | 100 |      9
 purchases | 700 |       
 purchases | 200 |      0
 products  | 200 |      0
 products  | 700 |       
(5 rows)

postgres=> 
```
 2024-07-31 14:38:54 -07:00
Edward Gao
8ca9d22113 Destination postgres: hard revert refreshes (#42460) 2024-07-23 16:21:53 -04:00
Stephane Geneix
34e4fe0134 destination-postgres: implement refreshes (#41954) 
implementing refreshes for destination-postgres

we're bumping the CDK version to the latest, and modifying a whole lot of jsonl files for tests (in both regular and strict-encrypt)
 2024-07-22 13:59:33 -07:00
Gireesh Sreepathi
7fb208bc82 Destination Postgres: Kotlin auto conversion src/main/java -> src/main/kotlin (#39388) 2024-06-13 10:36:18 -07:00
Edward Gao
b8838a4210 Destination Postgres: Upgrade to latest CDK (#37660) 2024-05-07 17:32:45 -04:00
Edward Gao
9ac5233738 Destination postgres: add option for cascade drop (#36974) 2024-04-11 14:48:39 -04:00
Edward Gao
7ecb1d313c Destination Postgres: improve handling for column name truncation (#36805) 2024-04-10 12:12:19 -04:00
Edward Gao
a9238d9513 Destination Postgres: improve destination state handling (#36768) 2024-04-08 13:57:28 -04:00
Gireesh Sreepathi
beda34bb1d Destination Postgres: Adapt to kotlin CDK (#36620) 2024-04-05 13:02:32 -07:00
Gireesh Sreepathi
98e99565a9 Destination Postgres: airbyte_meta, CDK upgrade and other fixes (#36363) 2024-03-28 10:01:24 -07:00
Marius Posta
ef473c5338 gradle: add kotlin support in all projects, warnings as errors (#36018) 2024-03-13 10:56:17 -05:00
Gireesh Sreepathi
55e6fbd191 Destination Snowflake: Handle NPE for state value. (#35899) 2024-03-07 18:25:38 -04:00
Edward Gao
8b83f14283 Destination postgres: upgrade cdk (#35528) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-03-05 11:54:06 -08:00
Gireesh Sreepathi
a13bd80f48 Destination Postgres: CDK T+D initial state gathering (#35385) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-02-23 12:30:10 -08:00
Gireesh Sreepathi
23cf31e80f ✨ Destination Postgres: DV2 GA (#35042) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-02-12 13:03:21 -08:00
Gireesh Sreepathi
f5cec3a299 Destination Postgres: Upgrade CDK with fixed dependency and unpin cloud (#34683) 2024-01-31 09:58:18 -08:00
Gireesh Sreepathi
1ad7155092 Destination Postgres: Fix casing for raw table in T+D query (#34630) 
## What
* Fixes: #34632 

## How
* Change the raw table convention to always use lowercase, since that is the default when created as unquoted identifier.
This still preserves the mixed case identifers in Final tables.
 2024-01-29 17:25:35 -08:00
Marius Posta
d01bb65dcb java CDK: build no longer downloads files from connector registry (#34441) 2024-01-25 11:44:48 -08:00
Edward Gao
d610ad1f48 Destination postgres (dv2): add indexes to raw table (#34236) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-18 16:21:24 -08:00
Edward Gao
006338257c Destination postgres: DV2 beta implementation (#34177) 2024-01-17 10:32:21 -08:00
Gireesh Sreepathi
d29cb2d41e Publish CDK and fix postgres to use cdk (#34135) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-10 20:48:14 -04:00
Gireesh Sreepathi
a0ece1231e destination-postgres: Add tunnel heartbeats and keepalive (#33875) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-01-10 15:42:32 -08:00
Gireesh Sreepathi
af4f5ae570 Destination Postgres - fix tests (#33507) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
 2023-12-18 17:51:16 -08:00
Stephane Geneix
c642d81b84 Stephane/enable javac errors 1 (#32467) 2023-11-13 18:07:58 -06:00
Stephane Geneix
9c6ed93a9d Revert "update all connectors for new CDK version and more stringent javac checks" 
This reverts commit 18ad87bbd1.
 2023-11-02 15:42:28 -07:00
Stephane Geneix
18ad87bbd1 update all connectors for new CDK version and more stringent javac checks 2023-11-02 15:24:58 -07:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Marius Posta
7ae97175a6 gradle: fix repo wide behaviour (#30607) 2023-09-28 05:01:13 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383) 
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
 2023-09-27 15:46:02 -07:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Ryan Fu
0dcc653bd1 Shuffled destination-jdbc into bases folder (#25120) 
* Testing changes for reshuffle

* Resolves dependency/race-condition with base classes and docker creation

* Removes dockerfile

* Updates java connectors gradle dependencies
 2023-04-17 12:29:03 -07:00
Peter Hu
e5eac0a4cc use published protocol models jar (#22498) 
* add airbyte-protocol to deps.toml

* use published protocol jar for platform

* use published protocol jar for connectors

* point at published jar

* fix dep

* bump gcs storage

* fix build failures in standard-source-test

* fix deps

* downgrade alloy db because it is missing strictness tests

* Revert "downgrade alloy db because it is missing strictness tests"

This reverts commit cc6089d053.

---------

Co-authored-by: cgardens <charles@airbyte.io>
 2023-02-13 12:50:43 -06:00
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642) 
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
 2022-06-10 09:34:31 -04:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476) 
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
 2022-06-06 00:35:43 +08:00
Jonathan Pearlin
ebb9f3e1ac Prepare Database Access Layer for Dependency Injection (#12546) 
* Prepare database access objects for dependency injection

* Replace duplicate code

* Remove unused imports

* Remove redundant validation call

* Remove unused imports

* Use constants

* Disable fast fail during connection pool initialization

* Remove typo

* Add missing test dependency

* Add missing test dependency

* Add missing test dependency

* Fix issue caused by rebase

* Add method for cloud

* Autoclose DSL context during migration

* Better connection close handling

* Fix typo in dependency

* Fix SpotBugs issue

* React to rebase

* Fix typo

* Update JavaDoc

* Fix database close calls

* Pass configs to getServer

* Fix typo

* Fix call to removed method

* Fix typo

* Use catalog to manage versions

* PR feedback

* Centralize shutdown hook

* Fix rebase issues

* Document test cases

* Document test cases

* Formatting

* Properly close database resources

* Rebase cleanup
 2022-05-09 15:26:54 -04:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104) 
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
 2021-10-19 11:20:21 -07:00
Subodh Kant Chaturvedi
7591324351 introduce jvm flag MaxRAMPercentage for java connectors (#6001) 
* introduce jvm flag MaxRAMPercentage for java connectors

* temporary commit to test this out on GKE via kube acceptance test

* undo temp commit
 2021-09-22 19:59:39 +05:30
Andrii Leonets
b18bd439d0 🐛 Destination Postgres: fix \u0000(NULL) value processing (#5336) 
* fix \u0000(NULL) value processing for Postgres + move postgres impl of SqlOperations to PostgresSqlOperations.

* changelog + format

* incr release version

* Add generic solution to adopt messages for a destination + remove unnecessary serialization

* revert version for build

* minor review fixes

* format

* add comments

* format

* incr version
 2021-08-30 21:41:02 +03:00
LiRen Tu
b9e1997d2f Split airbyte-db and move db dev commands to gradle (#5616) 
# Summary

- A follow-up PR for #5543.
- This PR separates the `airbyte-db` project to two modules:
  - `lib` is the original `airbyte-db`.
  - `jooq` is for jOOQ code generation.
- This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project.

# Commits
* Separate db to lib and jooq modules
* Update dependencies
* Add jobs db migrator test
* Fix compose build
* Add migration dev center
* Add schema dump task
* Update airbyte-db/lib/README.md
  * Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update readme
* Remove bom dependency
* Update readme
* Use jooq code in db config persistence
* Remove AirbyteConfigsTable

Co-authored-by: Davin Chia <davinchia@gmail.com>
 2021-08-26 10:44:09 -07:00
Charles
e4d0707781 Destination Checkpointing: Add StateMessage handing to BufferedStreamConsumer (#3230) 2021-05-07 13:05:52 -07:00
Charles
f9b63d0a9e Update Destination Abstractions (#1456) 2021-01-07 10:37:10 -08:00