1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

482 Commits

Author SHA1 Message Date
Ryan Fu
063b01f51c De-duplicate config parameters (#14781)
* Migrated 'ssl' string to become a constant in JdbcUtils

* Migrated 'database' string to become a constant in JdbcUtils

* Migrated 'username' string to become a constant in JdbcUtils

* Migrated 'password' string to become a constant in JdbcUtils

* Migrated 'schema' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'jdbc_url' string to become a constant in JdbcUtils

* Migrated 'connection_properties' string to become a constant in JdbcUtils

* Migrated list of 'host_key' string to become a constant in JdbcUtils

* Migrated list of 'port_key' string to become a constant in JdbcUtils

* Migrated 'host_key' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'port_key' string to become a constant in JdbcUtils

* Migrated 'tls' string to become a constant in JdbcUtils

* Migrated 'schemas' string to become a constant in JdbcUtils

* Migrated 'encryption' string to become a constant in JdbcUtils

* Migrated 'jdbc_url_params' string to become a constant in JdbcUtils and fixed uncaught parenthesis

* Fixed linter issues with comments on AcceptanceTest and failed import statement
2022-07-21 16:05:21 -07:00
VitaliiMaltsev
df445713aa Remove additionalProperties from JDBC source connectors (#14574)
* Remove additional properties from JDBC specs

* fixed SnowflakeSourceAuthAcceptanceTest

* fixed OracleStrictEncryptJdbcSourceAcceptanceTest

* fixed Oracle Strict encrypt expected spec

* updated Changelog

* fixed CockroachDbSourceTest

* fixed CockroachDb Tests

* fixed CockroachDb Tests for Linux OS

* bump versions

* bump clickhouse-source-strict-encrypt

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed dependencies for Clickhouse Source

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed OracleSourceDatatypeTest

* auto-bump connector version [ci skip]

* rollback ClickHouse Source changelog

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-21 11:01:34 +03:00
Liren Tu
07b132986b Mysql source: test utf8 char in json column (#14865)
* Add test case in mysql data type

* Add one more test case for utf8 char
2022-07-20 10:55:06 -07:00
Subodh Kant Chaturvedi
83b0f81982 upgrade debezium version to 1.9.2 for mysql source (#14542)
* upgrade debezium version to 1.9.2 for mysql source

* new debezium version doen't require special string handling

* fix log message

* add test tor date time data type

* gracefully handle the case when logs are purged

* fix test

* address review comment
2022-07-20 16:15:02 +05:30
Marcos Marx
8bbe9939f6 run gradlew format (#14721) 2022-07-14 15:17:12 -03:00
Jonathan Pearlin
969521215e Release source connectors (#14077)
* Release source connectors

* Fix issue with database connection in test

* Fix failing tests due to authentication

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-23 14:33:22 -04:00
Subodh Kant Chaturvedi
62cf441827 improve cdc check for connectors (#14005)
* improve should use cdc check

* Revert "improve should use cdc check"

This reverts commit 7d01727279.

* improve should use cdc check

* add unit test
2022-06-23 14:22:51 +05:30
Lake Mossman
cb90d7be0c Revert "Prepare release of JDBC connectors (#13987)" (#14029)
This reverts commit df759b3077.
2022-06-22 13:45:46 -07:00
Jonathan Pearlin
df759b3077 Prepare release of JDBC connectors (#13987)
* Prepare release of JDBC connectors

* Update source definitions manually
2022-06-22 10:59:56 -04:00
Jonathan Pearlin
6ca4017d28 Per-stream state support for Postgres source (#13609)
* WIP Per-stream state support for Postgres source

* Fix failing test

* Improve code coverage

* Make global the default state manager

* Add legacy adapter state manager

* Formatting

* Include legacy state for backwards compatibility

* Add global state manager

* Implement Global/CDC state handling

* Fix test issues

* Fix issue with updated method signature

* Handle empty state case in global state manager

* Adjust to protocol changes

* Fix failing acceptance tests

* Fix failing test

* Fix unmodifiable list issue

* Fix unmodifiable exception

* PR feedback

* Abstract global state manager selection

* Handle conversion between different state types

* Handle invalid conversion

* Rename parameter

* Refactor state manager creation

* Fix failing tests

* Fix failing integration tests

* Add CDC test

* Fix failing integration test

* Revert change

* Fix failing integration test

* Use per-stream for postgres tests

* Formatting

* Correct stream descriptor validation

* Correct permalink

* PR feedback
2022-06-21 15:35:40 -04:00
Eugene
393397290f 🎉Source-mysql: aligned datatype test (#13945)
* [13607] source-mysql: aligned datatype tests for regular and CDC ways + added CHAR fix to CDC processing
2022-06-21 14:05:11 +03:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
Subodh Kant Chaturvedi
cdb2d8f15e Fix mysql cdc acceptance test (#13795)
* fix mysql cdc acceptance test

* remove comments
2022-06-15 20:51:43 +05:30
Subodh Kant Chaturvedi
1ab08e5be2 upgrade debezium version for postgres to 1.9.2 (#13368)
* upgrade debezium version for postgres to 1.9.2

* fix test+build

* fix build

* address review comments

* update link to docs

* fix test
2022-06-15 16:17:36 +05:30
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642)
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
2022-06-10 09:34:31 -04:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476)
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
2022-06-06 00:35:43 +08:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Topher Lubaway
013a886f4f Fixes Spotless and runs spotless (#13040)
zipped files with JSON extension made this task sad
2022-05-20 07:26:55 -05:00
Jonathan Pearlin
fdaf335279 Better database connection handling for connectors (#12743)
* Better database connection handling for connectors

* Log connection error

* Properly close connection

* Remove unused method

* Close data source

* Use utility to close data source

* Use utility to close data source

* PR feedback

* Add Databricks driver

* Use driver class enum

* Use correct config

* Ensure config created before use

* Fix failing integration test

* Create DSLContext before use

* Address integration test failures

* Ensure DSLContext is closed

* Fix compile error

* Use correct datasource

* Use correct connection properties

* Close DSLContext

* Close DSLContext

* Fix integration test failures

* Properly close datasource

* Fix compilation issues

* Use existing database object

* Wrap close in try/finally

* Update test

* Wrap close in try/finally

* Ensure DSLContext is created

* Revert change to test

* Use correct data source

* Remove unused import

* More cleanup

* Add missing annotation

* Only initialize data source once

* Remove unused import

* Force testcontainers version

* Fix testcontainer issue

* Fix failing test

* Properly close all data sources

* Clear data sources after closing

* Fix compile error

* Fix compilation error

* Add missing method
2022-05-13 16:28:38 -04:00
Jonathan Pearlin
ebb9f3e1ac Prepare Database Access Layer for Dependency Injection (#12546)
* Prepare database access objects for dependency injection

* Replace duplicate code

* Remove unused imports

* Remove redundant validation call

* Remove unused imports

* Use constants

* Disable fast fail during connection pool initialization

* Remove typo

* Add missing test dependency

* Add missing test dependency

* Add missing test dependency

* Fix issue caused by rebase

* Add method for cloud

* Autoclose DSL context during migration

* Better connection close handling

* Fix typo in dependency

* Fix SpotBugs issue

* React to rebase

* Fix typo

* Update JavaDoc

* Fix database close calls

* Pass configs to getServer

* Fix typo

* Fix call to removed method

* Fix typo

* Use catalog to manage versions

* PR feedback

* Centralize shutdown hook

* Fix rebase issues

* Document test cases

* Document test cases

* Formatting

* Properly close database resources

* Rebase cleanup
2022-05-09 15:26:54 -04:00
Edward Gao
3d416129c7 🐛 Prevent sources from hanging if they have orphaned threads (#12544) 2022-05-03 18:48:43 -07:00
LiRen Tu
e15ae56389 Close all unsafe queries (#12495)
* Add helper methods to return lists of json nodes

* Close all unsafe queries

* Add one more helper method

* Simplify helper names

* Format code
2022-05-03 13:45:02 -07:00
LiRen Tu
35f2aa9aed 🎉 Jdbc sources: publish new version with adaptive fetch size (#12480)
* Default scaffold to use adaptive streaming config

* Switch more connectors to use adaptive streaming config

* Bump version for cockroach db

* Bump version for db2

* Bump mssql version

* Bump mysql version

* Bump oracle version

* Bump postgres version

* Bump redshift version

* Bump snowflake version

* Bump tidb version

* auto-bump connector version

* Fix db2 findbug issue

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix more findbug issues

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix findbug issue for mysql-strict-encrypt

* Fix findbugs issue for oracle source

* auto-bump connector version

* Remove suppress warnings annotation

* Fix oracle encrypt tests

* Fix oracle encrypt acceptance test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 23:14:58 -07:00
LiRen Tu
55a0db7c67 🎉 JDBC source: adjust streaming query fetch size dynamically (#12400)
* Merge all streaming configs to one

* Implement new streaming query config

* Format code

* Fix comparison

* Use double for mean byte size

* Update fetch size only when changed

* Calculate mean size by sampling n rows

* Add javadoc

* Change min fetch size to 1

* Add comment by buffer size

* Update java connector template

* Perform division first

* Add unit test for fetching large rows

* Format code

* Fix connector compilation error
2022-04-28 22:36:17 -07:00
Greg Solovyev
597d6ccbb8 Reconcile versions in git with versions in docker hub (#12368) 2022-04-26 11:41:59 -07:00
LiRen Tu
e5ae3b3990 Rename jdbc db methods that require manual closure (#11300)
* Rename jdbc db methods to raise awareness of potential connection leak

* Format code
2022-03-21 15:00:10 -07:00
Christophe Duong
84660a27a8 fix some spotbugsTest errors in source connectors (#11197)
* fix ./gradlew :airbyte-integrations:connectors:source-mysql:spotbugsTest

* Fix more connectors

* Fix tests

* format code

* fix spotbugTest

* Fix cdc msssql tests
2022-03-18 16:08:05 +01:00
Davin Chia
5a575e42e0 Format connectors. (#11231) 2022-03-17 19:08:01 +08:00
Charles
5fde59fdbd add spotbugs (#10522) 2022-03-11 12:05:17 -08:00
Edward Gao
b7e9c6bd5f jdbc build fixes (#10799) 2022-03-02 09:59:16 -08:00
andriikorotkov
1d66672fad 🐛 Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats (#10508)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style

* updated cursor for jdbc sources

* updated cursor for jdbc sources

* updated cursor for jdbc sources
2022-02-22 13:26:10 +02:00
andriikorotkov
9d401fc9f7 🎉 Updated timestamp transformation with microseconds (#10242)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style
2022-02-18 14:51:58 +02:00
LiRen Tu
d6747abd8f Clean up jdbc databases (#10371)
* Make streaming jdbc db extend from default impl

* Remove connection supplier

* Fix unit test

* Remove mysql database creation method

* Format code

* Revert "Remove mysql database creation method"

This reverts commit cb0ca3575a.

* Fix compilation
2022-02-16 12:19:14 -08:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
VitaliiMaltsev
e30d8348b2 Change JsonSchemaPrimitive to a class (#9913)
* fix for jdk 17

* add JsonSchemaType class

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix tests

* fix Oracle tests

* fix Redshift tests

* fix Redshift tests

* fix checkstyle

* fix MSSQL tests

* fix cockroachdb tests

* fix checkstyle

* fix checkstyle

* replace star imports

* replace star imports

* replace star imports

* update JsonSchemaType | fixed checkstyle

* Remove unused variables in test

* Fix imports

* Expand imports

* Fix more imports

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2022-02-14 02:12:37 -08:00
Andrii Leonets
5795d1386d 🐛 Source MySQL CDC: sync failed when has Zero-date value in mandatory column (#10251)
* replace null to epoch if column is mandatory

* format

* add tests

* incr mysql encrypt version

* bump version

* bump version

* fix changelog
2022-02-11 17:53:41 +02:00
mkhokh-33
b230543ba2 MySQL CDC sync fails because starting binlog position not found in DB #6425 (#9514)
* Check binlog position on mysql server before run sync job, add error description into log

* fix MySqlStrictEncryptSourceAcceptanceTest

* fix formatting

* fix review comments

* added java docs and fixed few minor comments

* fix formatting

* update versions

* update source_specs.yaml
2022-02-10 16:41:52 +02:00
LiRen Tu
a094142825 Format code (#9892) 2022-01-29 19:56:00 -08:00
Noah Kawasaki
5391880f7a 🎉 Source and Destination Snowflake: Add jdbc_url_params support for optional JDBC parameters (#9623) 2022-01-28 22:07:12 +01:00
LiRen Tu
e4661fb92a Remove regex check from Java source acceptance test (#9829)
* Move getRegexTests to python source acceptance test

* Remove unused imports

* Update test template
2022-01-26 17:51:37 -08:00
andriikorotkov
208c14682d updated benchmarks databases and scripts (#9597) 2022-01-19 17:22:21 +02:00
Serhii Chvaliuk
844dd93122 Use multi-stage builds in dockerfiles to reduce java images (#9077)
* use multi-stage to reduce image size

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2021-12-23 22:27:05 +02:00
Oleksandr Sheheda
f0a9945f80 Revert "Change copy to add in dockerfiles to reduce container size (#8516)" (#8997)
This reverts commit 8ac2c6f4
2021-12-21 16:01:17 +02:00
LiRen Tu
9b75a76720 📑 Update source database performance test docs (#8980)
* Add missing pr number

* Update source database performance test docs

* Update doc

* Update script paths
2021-12-20 20:49:22 -08:00
andriikorotkov
0c99c5d7a5 🎉 Setting CPU and Memory limits for performance tests via github (#8807)
* updated ci performance test sh script

* updated ci performance test sh script

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* set all params to one arg

* run tests

* update ci creds

* update performance test yaml file

* update performance test yaml file

* update performance test yaml file

* update performance test yaml file

* update performance test sh file

* update performance test sh file

* update performance test sh file

* update performance test sh file

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* update java to 17 version in yaml file for performance tests

* clean test logs

* remove test logs

* remove test logs

* remove test logs

* return performance test logs

* add readme by performance tests
2021-12-17 12:47:45 +02:00
Haoran Yu
8ac2c6f4a7 Change copy to add in dockerfiles to reduce container size (#8516)
Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
2021-12-16 23:52:59 -03:00
Marcos Marx
e45dc12bdc format files (#8756) 2021-12-13 16:13:48 -03:00
Iryna Grankova
62c3c4e93e Update fields in source-connectors specifications: mysql, onesignal, oracle, outreach, paystack, pipedrive, pokeapi (#8582)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-12-13 11:14:51 +02:00
LiRen Tu
4276b2f3d6 Bump the minor version for mysql source (#8731)
* Bump the minor version for mysql source

* Setup java 17 in the publish command

* Bump version in seed
2021-12-12 16:30:54 -08:00
andriikorotkov
7591883e21 Added benchmarks scripts with small instruction (#8529)
* updated source-mongodb-v2 performance

* updated code style

* fixed remarks

* fixed remarks

* fixed remarks

* updated strict encrypt source mongodb version

* updated source mongodb work with empty collections

* updated source mongodb timestamp cursor

* updated mongodb source perfomance

* fix code style

* fix code style

* updated tests and documentation

* updated tests and documentation

* updated tests and documentation

* added vudangngoc changes

* updated code style

* updated code style

* added benchmarks scripts with small instruction

* fixed remarks

* updated ci performance test sh script
2021-12-12 10:30:26 -08:00