1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

121 Commits

Author SHA1 Message Date
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642)
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
2022-06-10 09:34:31 -04:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476)
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
2022-06-06 00:35:43 +08:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Oleksandr Sheheda
44a43f0338 Improve check connection performance for all JDBC-related DBs (#12820)
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
2022-05-24 16:06:01 +03:00
Jonathan Pearlin
fdaf335279 Better database connection handling for connectors (#12743)
* Better database connection handling for connectors

* Log connection error

* Properly close connection

* Remove unused method

* Close data source

* Use utility to close data source

* Use utility to close data source

* PR feedback

* Add Databricks driver

* Use driver class enum

* Use correct config

* Ensure config created before use

* Fix failing integration test

* Create DSLContext before use

* Address integration test failures

* Ensure DSLContext is closed

* Fix compile error

* Use correct datasource

* Use correct connection properties

* Close DSLContext

* Close DSLContext

* Fix integration test failures

* Properly close datasource

* Fix compilation issues

* Use existing database object

* Wrap close in try/finally

* Update test

* Wrap close in try/finally

* Ensure DSLContext is created

* Revert change to test

* Use correct data source

* Remove unused import

* More cleanup

* Add missing annotation

* Only initialize data source once

* Remove unused import

* Force testcontainers version

* Fix testcontainer issue

* Fix failing test

* Properly close all data sources

* Clear data sources after closing

* Fix compile error

* Fix compilation error

* Add missing method
2022-05-13 16:28:38 -04:00
Jonathan Pearlin
ebb9f3e1ac Prepare Database Access Layer for Dependency Injection (#12546)
* Prepare database access objects for dependency injection

* Replace duplicate code

* Remove unused imports

* Remove redundant validation call

* Remove unused imports

* Use constants

* Disable fast fail during connection pool initialization

* Remove typo

* Add missing test dependency

* Add missing test dependency

* Add missing test dependency

* Fix issue caused by rebase

* Add method for cloud

* Autoclose DSL context during migration

* Better connection close handling

* Fix typo in dependency

* Fix SpotBugs issue

* React to rebase

* Fix typo

* Update JavaDoc

* Fix database close calls

* Pass configs to getServer

* Fix typo

* Fix call to removed method

* Fix typo

* Use catalog to manage versions

* PR feedback

* Centralize shutdown hook

* Fix rebase issues

* Document test cases

* Document test cases

* Formatting

* Properly close database resources

* Rebase cleanup
2022-05-09 15:26:54 -04:00
noahkawasaki-airbyte
9f577bb027 Update specs and definitions files for destination-postgres 0.3.19 (#12317)
* Generate specs and definitions files after destination-postgres 0.3.19

* Bump destination-postgres-strict-encrypt to 0.1.5
2022-04-27 07:07:54 -07:00
Yurii Bidiuk
af1d087d22 Revert "DAT: verify that a destination is able to write any ISO8601-compliant date string (#9816)" (#11802)
This reverts commit 2f850b98ac.
2022-04-08 19:39:40 +03:00
Yurii Bidiuk
2f850b98ac DAT: verify that a destination is able to write any ISO8601-compliant date string (#9816)
* added format_reference.txt, edited edge_case_messages.txt, edge_case_catalog.json

* updated due to comments

* fix message order for destiantion acceptance tests

* remove unused import

* handle date-time messages for destiantion acceptance tests

* refactoring

* format code

* remove non iso8601 date

* add comments

* fixed pubsub tests

* add test date with microseconds

* fix comments

* fixed PR remarks

* fix assertion for nested objects

* format code

* fix remarks

* format code

* add additional check to convert only datetime values

* changed key structure for saved date-time fieds

* format code

* add comments

* refactored iteration

* format code

* pubsub bump version

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
2022-04-05 14:04:02 +03:00
girarda
5d8cb41150 Refactor to enable support for optional JDBC parameters for all JDBC destinations (#10421)
* refactoring to allow testing

* MySQLDestination uses connection property map instead of url arguments

* Update jdbc destinations

* A little more generic

* reset to master

* reset to master

* move to jdbcutils

* Align when multiline

* Align when multiline

* Update postgres to use property map

* Move tests to AbstractJdbcDestinationTest

* clean

* Align when multiline

* return property map

* Add postgres tests

* update clickhouse

* reformat

* reset

* reformat

* fix test

* reformat

* fix bug

* Add mssql tests

* refactor test

* fix oracle destination test

* oracle tests

* fix redshift acceptance test

* Pass string

* Revert "Pass string"

This reverts commit 697821738c.

* Double deserialization

* Revert "Double deserialization"

This reverts commit ee8d75245b.

* try updating json_operations

* Revert "try updating json_operations"

This reverts commit c8022c2994.

* json parse

* Revert "json parse"

This reverts commit 11a6725eaa.

* Revert "Revert "Double deserialization""

This reverts commit 213f47acc4.

* Revert "Revert "Revert "Double deserialization"""

This reverts commit 66822454af.

* move to constant

* Add comment

* map can be constant

* Add comment

* move map

* hide in method

* no need to create new map

* no need to create new map

* no need to create new map

* enably mysql test

* Update changelogs

* Update changelog

* update changelog

* Bump versions

* bump version

* disable dbt support

* update spec

* update other oracle tests

* update doc

* bump seed

* fix source test

* update seed spec file

* fix expected spec
2022-02-28 09:25:04 -08:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
Serhii Chvaliuk
844dd93122 Use multi-stage builds in dockerfiles to reduce java images (#9077)
* use multi-stage to reduce image size

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2021-12-23 22:27:05 +02:00
Oleksandr Sheheda
f0a9945f80 Revert "Change copy to add in dockerfiles to reduce container size (#8516)" (#8997)
This reverts commit 8ac2c6f4
2021-12-21 16:01:17 +02:00
Haoran Yu
8ac2c6f4a7 Change copy to add in dockerfiles to reduce container size (#8516)
Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
2021-12-16 23:52:59 -03:00
oneshcheret
fcf7491fee 🐛 Validate incorrect handling '\n' symbols in ssh key (#8371)
* 🐛 Validate \n in ssh key

* bump versions for ssh key connectors

* update versions for ssh key connectors

* temporal fix for checking failed tests

* revert temp changes and destination oracle version

* bump versions in config for ssh key related connectors
2021-12-03 20:11:51 +02:00
Sherif A. Nada
efb5151011 🐛 Make all JDBC destinations (SF, RS, PG, MySQL, MSSQL, Oracle) handle wide rows by using byte-based record buffering (#7719) 2021-11-08 19:26:32 -08:00
Haoran Yu
92fcdea5bb Refactor JDBC JSON_Format into JDBC utils (#7504) 2021-11-02 20:20:33 -07:00
Harsha Teja Kanna
3e7f95c25a 🎉 Support build on MacOS M1 (Apple Silicon) (#7104)
- See this doc for details: https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md
- Unit test does not work yet.
2021-10-19 11:20:21 -07:00
Eugene
05e09aff7d 🎉Destination MySQl - added ssl strict encrypt connector (#6763)
* [6423] Destination MySQl - added ssl strict encrypt connector
2021-10-14 08:28:43 +03:00
andriikorotkov
d746f4922b Exposing SSL-only version of MySQL Source (#6605)
* Source MySQL - added option to connect using ssl

* Source MySQL - updated changelog

* dummy commit

* Updated PR as per comment

* added source mysql strict encrypt

* updated code style for MySQL source

* updated code style for MySQL source

* fixed remarks

* fixed code style

* fixed remarks

Co-authored-by: ievgeniit <ievgeniit>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
2021-10-05 09:47:10 +03:00
Charles
5e750164ac Publish SSL-only version of Postgres Destination (#6496)
* try to publish new normalization version

* default to using ssl in postgres destinatoin

* tidy up

* Run normalization tests using postgres DB with SSL support

* bump version

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
2021-09-30 12:55:26 +02:00