1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

121 Commits

Author SHA1 Message Date
Edward Gao
006338257c Destination postgres: DV2 beta implementation (#34177) 2024-01-17 10:32:21 -08:00
Gireesh Sreepathi
d29cb2d41e Publish CDK and fix postgres to use cdk (#34135)
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
2024-01-10 20:48:14 -04:00
Gireesh Sreepathi
a0ece1231e destination-postgres: Add tunnel heartbeats and keepalive (#33875)
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
2024-01-10 15:42:32 -08:00
Edward Gao
216d2b2e70 Destination postgres: enable in-connector normalization (#33873) 2024-01-04 14:15:33 -04:00
Gireesh Sreepathi
a764be74df Destination-mysql, destination-postgres: Pin old versions for oss and cloud (#33649)
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
2023-12-19 15:07:37 -06:00
Gireesh Sreepathi
af4f5ae570 Destination Postgres - fix tests (#33507)
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2023-12-18 17:51:16 -08:00
Stephane Geneix
c642d81b84 Stephane/enable javac errors 1 (#32467) 2023-11-13 18:07:58 -06:00
Marius Posta
ef6dbd0cb9 gradle: test task configuration changes (#32108)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-03 12:54:09 -05:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Augustin
ac6f44cf2c remove Dockerfile and .dockerignore for Java connectors (#31769) 2023-10-24 15:08:13 -05:00
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383)
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
2023-09-27 15:46:02 -07:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Edward Gao
ab04c3b2da DATs: Fix spurious failures (#29734) 2023-08-22 17:44:04 -05:00
Evan Tahler
9f515a867f DAT tearDown should cleanup *all* schemas used (#29312)
* DAT `tearDown` should cleanup *all* schemas used

* redshift too

* push all method signature changes

* Automated Commit - Format and Process Resources Changes

* use hashset

* TEST_SCHEMAS on setup as well

* fix call

* Automated Commit - Format and Process Resources Changes

---------

Co-authored-by: evantahler <evantahler@users.noreply.github.com>
2023-08-11 11:40:49 -07:00
Evan Tahler
f455c1288d Java DB Destination connector licenses to Elv2 (#27781)
* Java DB Destination connector licenses to Elv2

* PR id for docs

* fix redshift tagging
2023-06-29 12:26:24 -05:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Augustin
fd3655707e connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489) 2023-05-25 09:32:19 +02:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Akash Kulkarni
889a5bc50b Run formatter (#25901) 2023-05-09 10:03:22 -07:00
Ben Church
efbdb73e24 Bnchrch/metadata/add missing metadata files (#25899)
* Add source fullstory metadata

* Add source merge metadata

* Add source yotpo metadata

* Update normalization tag

* Update oss diffs

* Add aircall to cloud

* Pin Source sales force

* Ensure strict encrypt matches

* Remove zurora pin

* Make metadata validation mandatory

* Remove top level third party folders
2023-05-08 16:55:11 -05:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Ryan Fu
0dcc653bd1 Shuffled destination-jdbc into bases folder (#25120)
* Testing changes for reshuffle

* Resolves dependency/race-condition with base classes and docker creation

* Removes dockerfile

* Updates java connectors gradle dependencies
2023-04-17 12:29:03 -07:00
Augustin
aa43bfc196 fix strict-encrypt metadata dockerRepository (#25159) 2023-04-13 20:48:23 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
Rodi Reich Zilberman
0bab1756b8 Rename airbyte-config module (#24885)
* rename airbyte-config module

* Automated Commit - Formatting Changes

* sanity

* update import

* update import

* update script

* update script

* update script

* update script

* Automated Change

* Automated Change

* Automated Change

* Automated Change

* update awsdatalake icon

* point slash commands to new path

* sanity

* Automated Commit - Formatting Changes

* sanity

* Automated Change

* Automated Change

* sanity

---------

Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-04-06 10:47:30 -07:00
Ryan Fu
73900fad3e JDBC connections to support checkpointing (#24604)
* Shifted logic within JDBC connections to support checkpointing

* Bumped version numbers and updated changelog
2023-04-05 18:32:28 +00:00
Joe Bell
ba7f6d2fca Add Equality check to Clickhouse Destination Acceptance Test (#24646)
* add equality check

* fixing another failing test

* update acceptance tests

* Automated Change

* formatting

---------

Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
2023-03-31 11:19:04 -07:00
Greg Solovyev
5f1eb87444 Fix destination integration tests (#24431)
* MS SQL does not support schema change in incremental model
* make schema change test optional
* fix compilation errors in postgres-strict-encrypt destination
* deparallelize integration tests for destination postgres
* deparalellize MS SQL integration tests
* remove broken SSH tunnel test from destination-postgres-strict-encrypt
2023-03-24 10:41:47 -07:00
Sherif A. Nada
e85eda088e Remove ExtendedNameTrasformer (#23655) 2023-03-07 17:22:08 -08:00
Charles
f83ef9eea7 Remove workers (#23422) 2023-02-24 17:45:44 -08:00
Peter Hu
e5eac0a4cc use published protocol models jar (#22498)
* add airbyte-protocol to deps.toml

* use published protocol jar for platform

* use published protocol jar for connectors

* point at published jar

* fix dep

* bump gcs storage

* fix build failures in standard-source-test

* fix deps

* downgrade alloy db because it is missing strictness tests

* Revert "downgrade alloy db because it is missing strictness tests"

This reverts commit cc6089d053.

---------

Co-authored-by: cgardens <charles@airbyte.io>
2023-02-13 12:50:43 -06:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Augustin
6c2b9a164b destination acceptance tests: get normalization and dbt support from definition (#21299) 2023-01-18 20:00:47 +01:00
Jimmy Ma
094aff1520 Java connectors use protocol v0 objects (#20404)
* Dupl v0 objects

* v0 namespace swap

* Introduce some object conversions in the tests

* Update source-java-jdbc template

* Use v0 objects in db-lib

* Fix more protocol object references

* Fix arg capture in tests
2022-12-15 13:39:50 -08:00
Benoit Moriceau
adf2c5ff8e Bmoric/remove dep connector worker (#17977)
* test [ci skip]

* Autogenerated files

* Add missing annotation

* Remove unused json2Schema block from worker

* Move tess

* Missing deps and format

* Fix test build

* TMP

* Add missing dependencies

* PR comments

* Tmp

* [ci skip] Tmp

* Fix acceptance test and add the seed dependency

* Fix build

* For diff

* tmp

* Build pass

* make the worker to be  on the platform only

* fix setting.yaml

* remove dependency

* Fix pmd

* Fix Cron

* Add chart

* Fix cron

* Fix server build.gradle

* Fix jar conflict

* PR comments

* Add cron micronaut environemnt

* Update required build path

* remove all dependencies on worker

* Typo

* Add gradle exclusions

* Add exclusion to avoid conflict

* more exclusion

* Move more class

* Exclude worker from the general build

* Fix connector build
2022-10-18 11:51:25 -07:00
Eugene
33dfd5d6be Fixed "Ensure no file change" build issue. Only checkstyle changes, no logic chnages (#17885) 2022-10-12 15:06:28 +03:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
Rodi Reich Zilberman
248da78ba7 Make error more descriptive in case of no encryption state (#17299)
* Make error more descriptive in case of no encryption state

* Fix failing test

* Change error message on strict-encrypt postgres destination as well. Fix failing test

* update versions and changelogs

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-28 15:26:09 -07:00
Yurii Bidiuk
10e4aae891 Use LOG4J2 to wrap connectors logs to JSON format (#15668)
* Use LOG4J2 to wrap connectors logs to JSON format

* log connector's stack trace directly as a message

* add stack_trace field to json template

* bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* revert versions for destinations: postgres, mssql, mysql

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-14 23:27:13 +03:00
Adam Fletcher
80397adbe1 🐛 Destination Postgres: fix missing database name URL Encoding (#16046)
* Fix missing database name URL Encoding in Postgres destination

* add unit test

* bump connector version and update doc

* auto-bump connector version [ci skip]

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-08 21:17:29 -03:00
Rodi Reich Zilberman
fc13413c9b Enforce secure connection in destination-postgres-strict-encrypt (#16260)
* Enforce secure connection in destination-postgres-strict-encrypt

* bump docker image values and update changelog.

* Typo

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-02 00:47:19 -07:00
Ryan Fu
4572b75845 Moved getConnection into a try-with block and ran gradlew format (#15805) 2022-08-22 19:54:00 -07:00
andriikorotkov
d63a24ed41 🎉 Destination postgres: Add SSL certificates and update normalization (#14743)
* added ssl certificates for postgres source

* added command for remove client private key after transformation to encrypted key with .pk8 extension

* added connection with CA and client certificates for postgres destination

* updated code style

* moved common methods to the common class

* moved common methods to the common class

* fixed remarks

* updated postgres source tests

* added minor changes to spec and added fixes to password mechanism

* updated postgres source tests

* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors

* fixed check style

* updated documentation and versions of connectors

* updated ordrs in test spec

* fixed minor remarks in specs and expected_specs

* fixed minor remarks in specs and expected_specs

* fixed Dockerfile

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed code style

* fixed connectors version in definition file

* updated postgres destination normalization

* updated postgres destination tests

* fixed code style for postgres source and destination

* pulled master changes

* removed allow mode for destination-postgres-strect-encrypt

* updated connectors version

* fixed custom DBT transformation and enabled test for it

* updated normalization version

* updated keystore password generation method

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-15 16:32:37 +03:00
Greg Solovyev
a7a1847d47 Align Postgres Destination regular and strict encrypt versions (#15261) 2022-08-03 14:12:16 -07:00
Benoit Moriceau
318e79f777 bump postgres destination (#14950) 2022-07-22 09:21:55 -07:00
Ryan Fu
063b01f51c De-duplicate config parameters (#14781)
* Migrated 'ssl' string to become a constant in JdbcUtils

* Migrated 'database' string to become a constant in JdbcUtils

* Migrated 'username' string to become a constant in JdbcUtils

* Migrated 'password' string to become a constant in JdbcUtils

* Migrated 'schema' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'jdbc_url' string to become a constant in JdbcUtils

* Migrated 'connection_properties' string to become a constant in JdbcUtils

* Migrated list of 'host_key' string to become a constant in JdbcUtils

* Migrated list of 'port_key' string to become a constant in JdbcUtils

* Migrated 'host_key' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'port_key' string to become a constant in JdbcUtils

* Migrated 'tls' string to become a constant in JdbcUtils

* Migrated 'schemas' string to become a constant in JdbcUtils

* Migrated 'encryption' string to become a constant in JdbcUtils

* Migrated 'jdbc_url_params' string to become a constant in JdbcUtils and fixed uncaught parenthesis

* Fixed linter issues with comments on AcceptanceTest and failed import statement
2022-07-21 16:05:21 -07:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00