1
0
mirror of synced 2025-12-20 18:39:31 -05:00
Commit Graph

59 Commits

Author SHA1 Message Date
Aaron ("AJ") Steers
ec5f3bb3dd feat: Add external documentation URLs to Group C source connectors (source-box-data-extract through source-commcare) (#69730)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2025-11-18 17:28:04 -08:00
Matt Bayley
267f5ad295 Mbayley/revert 66482 upgrade source clickhouse jdbc version (#66715) 2025-09-26 00:07:39 -07:00
Matt Bayley
6c17477515 source-clickhouse: strict-encrypt metadata 0.2.5 (#66705) 2025-09-25 13:47:55 -07:00
Charles
50a97db804 chore: migrate connectors from MIT to ELv2 (#63723) 2025-07-22 14:26:58 -07:00
devin-ai-integration[bot]
bec0ac8c91 chore(source-clickhouse-strict-encrypt): Convert to new gradle build flow (do not merge) (#62913)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yue.li@airbyte.io <yue.li@airbyte.io>
2025-07-10 16:21:44 -07:00
Aaron ("AJ") Steers
05965c781e chore: add poe_tasks.toml for all java connectors (#59670) 2025-05-05 17:28:27 -07:00
Augustin
7f93412250 source-clickhouse-strict-encrypt: Use airbyte/java-connector-base:1.0.0 (#49906)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2024-12-19 09:34:00 +01:00
Augustin
601c3c455c Rename registries to registryOverrides in connector metadata (#43757) 2024-08-13 16:10:56 +02:00
Augustin
31c95da499 [skip ci] Add connectorTestSuitesOptions to metadata (#38188) 2024-05-15 15:04:10 -05:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Marius Posta
9e7936e0b0 source-clickhouse: adopt CDK 0.20.4 (#35235) 2024-02-14 11:53:16 -08:00
Chandler Prall
9a67b7fa10 🐛 bump cdk versions for java sources (#34453) 2024-02-01 14:26:38 -07:00
Subodh Kant Chaturvedi
0c3d12ba2b jdbc-sources: migrate to PER-STREAM state (#33485) 2023-12-19 15:38:39 +05:30
Stephane Geneix
c642d81b84 Stephane/enable javac errors 1 (#32467) 2023-11-13 18:07:58 -06:00
Augustin
368ba78b64 🧹doc: update connectors README and remove acceptance-test-docker.sh (#32209) 2023-11-06 12:36:07 -06:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Augustin
ac6f44cf2c remove Dockerfile and .dockerignore for Java connectors (#31769) 2023-10-24 15:08:13 -05:00
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383)
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
2023-09-27 15:46:02 -07:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Marius Posta
ef2849e35e gradle: fix airbyteDocker task inputs (#30187) 2023-09-07 03:46:31 -07:00
Ben Church
1dabc6208e Metadata: add tags field (#26320)
* Add optional tags field

* Remove duplicate icons

* Add programming tags to all

* Update docs

* supportUrl -> documentationUrl

* Ensure one language tag is applied

* Add keyvalue check

* rebase and fix tests

* Format

* Add cache buster

* Improve test

* Automated Commit - Formatting Changes

* Update error

* Fix missing tags

* Fix scaffold

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com>
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-05-26 16:13:09 -07:00
Augustin
fd3655707e connectors-ci: add finalize_build logic to handle custom Dockerfiles (#26489) 2023-05-25 09:32:19 +02:00
Ben Church
248bbf94c1 HACKDAY: Icon CDN (#26158)
* Move icons to connector folder

* Delete old icons

* Update upload logic

* Add icon url to definitions

* Update registry model

* Populate cdn url

* DNC butcher the pipeline

* Low hanging fruit fixes

* Fix bucket name

* Merge old and new approaches

* Fix metadata upload step

* Format

* Fix test
2023-05-24 17:25:41 -07:00
Ben Church
5563179782 Dagster: rename catalog to registry (#25254)
* rename catalog to registry in metadata service

* rename catalog to registry in metadata files

* Run generate models

* Fix missed renames

* Add github personal access token

* Run black

* Automated Change

---------

Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
2023-04-18 22:15:11 +02:00
Augustin
aa43bfc196 fix strict-encrypt metadata dockerRepository (#25159) 2023-04-13 20:48:23 +02:00
Augustin
ebc907cdf7 create metadata files for all connectors (#24964) 2023-04-13 07:45:04 +02:00
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Prateek Mukhedkar
33ca7f3451 Adding allowHosts to all non-GA source database connectors (#23455)
Adding allowedHosts to clickhouse, cockroachdb, db2, oracle, snowflake, tidb source Connectors. 
---------

Co-authored-by: prateekmukhedkar <prateekmukhedkar@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-07 19:52:15 -08:00
Peter Hu
e5eac0a4cc use published protocol models jar (#22498)
* add airbyte-protocol to deps.toml

* use published protocol jar for platform

* use published protocol jar for connectors

* point at published jar

* fix dep

* bump gcs storage

* fix build failures in standard-source-test

* fix deps

* downgrade alloy db because it is missing strictness tests

* Revert "downgrade alloy db because it is missing strictness tests"

This reverts commit cc6089d053.

---------

Co-authored-by: cgardens <charles@airbyte.io>
2023-02-13 12:50:43 -06:00
Cole Snodgrass
2e099acc52 update headers from 2022 -> 2023 (#22594)
* It's 2023!

* 2022 -> 2023

---------

Co-authored-by: evantahler <evan@airbyte.io>
2023-02-08 13:01:16 -08:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846)
Rename source acceptance test to connector acceptance test
2023-02-02 11:38:19 +01:00
VitaliiMaltsev
f5f4de2e5f JDBC sources: consolidate date/time values mapping (#20346)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* resolved merge conflicts

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* bump postgres version

* auto-bump connector version

* updated seed definitions

* revert generated definitions for Postgres

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-18 19:21:11 +02:00
Jimmy Ma
094aff1520 Java connectors use protocol v0 objects (#20404)
* Dupl v0 objects

* v0 namespace swap

* Introduce some object conversions in the tests

* Update source-java-jdbc template

* Use v0 objects in db-lib

* Fix more protocol object references

* Fix arg capture in tests
2022-12-15 13:39:50 -08:00
Pedro S. Lopez
938436bcc9 update connector specs and definitions with new .com documentation urls (#17585)
* update definitions with new .com docs urls

* update docs urls in specs

* update generators

* regenerate scaffold connectors

* remove unrelated changes

* update more urls

* update specs

* fix tests

* run `:airbyte-config:specs:generateSeedConnectorSpecs` to fix formatting

* revert docs changes to make pr more reviewable

* revert generator readme changes to make more reviewable

* fix mysql strict encrypt expected spec

* fix postgres expected spec
2022-10-11 11:04:23 -04:00
oneshcheret
765bd1bfeb Source ClickHouse: add custom jdbc params (#17031)
* Source clickhouse: add jdbc params

* Source clickhouse: add jdbc params

* Source clickhouse: add jdbc params

* Source clickhouse: updated tests

* Source clickhouse: format

* Source clickhouse: format

* Source clickhouse: format

* Source clickhouse: format

* Source clickhouse: bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-27 14:03:52 +03:00
Liren Tu
c3f3702522 🎉 Jdbc sources: emit state messages more frequently (#16256)
* Emit intermediate states in clickhouse

* Add pr id

* Emit intermediate states in cockroachdb

* Emit intermediate states in db2

* Emit intermediate states in jdbc source

* Emit intermediate states in tidb

* Revert "Emit intermediate states in jdbc source"

This reverts commit 7a913bbae7.

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-01 21:44:48 -07:00
mkhokh-33
5776dcc949 Mkhokh/8858 cursor fields for incremental update (#14356)
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor

* added tests for cursor fields incremental updates

* remove not used methods

* fix tests

* fix formatting

* revert RedshiftInsertDestinationAcceptanceTest

* fix ssl clickhouse test

* fix codereview comments

* fix clickhouse strict encrypt test

* fix postgres source after merge from master

* optimize ssl clickhouse test logic

* fix db2 test

* added check for cursor field should be not nullable, fix formatting

* fixed tests, formatting, issue after merge from master

* fix merge issues

* fix test for source-jdbc and remove some code used for local testing

* added description for cockroach db tests

* test changes

* refactoring for cursor field tests after code review comments

* refactoring for cursor field tests after code review comments

* fix code review comments

* remove unused imports

* fix formatting

* fix compilation error

* fix postgres and postgress ssl test

* Fixed bucket naming for S3

* remove CHAR, NCHAR from possible cursor types

* remove SYMBOL from possible cursor types

* removed redundant configs

* fixed mysql-strict-encrypt tests

* fixed mongodb-v2 tests

* minor format changes

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-18 19:48:22 +03:00
Yevhen Sukhomud
436de264cb 13539 Fix integration tests source-clickhouse Mac OS (#14701)
* 13539 Fix integration tests source-clickhouse Mac OS

* 13539 Updated clickhouse jdbc driver

* 13539 Updated destination-clickhouse-strict-encrypt

* 13539 Updated SSL configuration and tests for clickhouse-destination

* 13539 Updated SSL for source-clickhouse-strict-encrypt

* 13539 Resolved host by ip

* 13539 Fixed code formatting

* 13539 Bump up source-clickhouse-strict-encrypt version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-26 13:08:44 +07:00
Ryan Fu
063b01f51c De-duplicate config parameters (#14781)
* Migrated 'ssl' string to become a constant in JdbcUtils

* Migrated 'database' string to become a constant in JdbcUtils

* Migrated 'username' string to become a constant in JdbcUtils

* Migrated 'password' string to become a constant in JdbcUtils

* Migrated 'schema' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'jdbc_url' string to become a constant in JdbcUtils

* Migrated 'connection_properties' string to become a constant in JdbcUtils

* Migrated list of 'host_key' string to become a constant in JdbcUtils

* Migrated list of 'port_key' string to become a constant in JdbcUtils

* Migrated 'host_key' string to become a constant in JdbcUtils and updated corresponding config params

* Migrated 'port_key' string to become a constant in JdbcUtils

* Migrated 'tls' string to become a constant in JdbcUtils

* Migrated 'schemas' string to become a constant in JdbcUtils

* Migrated 'encryption' string to become a constant in JdbcUtils

* Migrated 'jdbc_url_params' string to become a constant in JdbcUtils and fixed uncaught parenthesis

* Fixed linter issues with comments on AcceptanceTest and failed import statement
2022-07-21 16:05:21 -07:00
VitaliiMaltsev
df445713aa Remove additionalProperties from JDBC source connectors (#14574)
* Remove additional properties from JDBC specs

* fixed SnowflakeSourceAuthAcceptanceTest

* fixed OracleStrictEncryptJdbcSourceAcceptanceTest

* fixed Oracle Strict encrypt expected spec

* updated Changelog

* fixed CockroachDbSourceTest

* fixed CockroachDb Tests

* fixed CockroachDb Tests for Linux OS

* bump versions

* bump clickhouse-source-strict-encrypt

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed dependencies for Clickhouse Source

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed OracleSourceDatatypeTest

* auto-bump connector version [ci skip]

* rollback ClickHouse Source changelog

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-21 11:01:34 +03:00
Oleksandr Sheheda
0a6a630bb4 [13539] Fix integration tests source-clickhouse Mac OS (#14201)
* [13539] Fix integration tests source-clickhouse Mac OS
fixed unit tests

* [13524] Fix integration tests destination-clickhouse Mac OS
fixed unit tests
2022-06-29 14:18:47 +03:00
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642)
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
2022-06-10 09:34:31 -04:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476)
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
2022-06-06 00:35:43 +08:00
Alexandre Girard
3894134d11 Bump year in license short to 2022 (#13191)
* Bump to 2022

* format
2022-05-25 17:56:49 -07:00
Topher Lubaway
013a886f4f Fixes Spotless and runs spotless (#13040)
zipped files with JSON extension made this task sad
2022-05-20 07:26:55 -05:00
Jonathan Pearlin
ebb9f3e1ac Prepare Database Access Layer for Dependency Injection (#12546)
* Prepare database access objects for dependency injection

* Replace duplicate code

* Remove unused imports

* Remove redundant validation call

* Remove unused imports

* Use constants

* Disable fast fail during connection pool initialization

* Remove typo

* Add missing test dependency

* Add missing test dependency

* Add missing test dependency

* Fix issue caused by rebase

* Add method for cloud

* Autoclose DSL context during migration

* Better connection close handling

* Fix typo in dependency

* Fix SpotBugs issue

* React to rebase

* Fix typo

* Update JavaDoc

* Fix database close calls

* Pass configs to getServer

* Fix typo

* Fix call to removed method

* Fix typo

* Use catalog to manage versions

* PR feedback

* Centralize shutdown hook

* Fix rebase issues

* Document test cases

* Document test cases

* Formatting

* Properly close database resources

* Rebase cleanup
2022-05-09 15:26:54 -04:00
Nikolai Korolev
22810c3937 Source Clickhouse: Fix exception in case password field is not provided in configuration (#10214)
* Fix exception in case password is not provided for Clickhouse source

Exception while checking connection:
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.JsonNode.asText()" because the return value of "com.fasterxml.jackson.databind.JsonNode.get(String)" is null

* Update changelog

* Bump version for Source Clickhouse

* Bump version of source-clickhouse-strict-encrypt

* Update version in docs

* correct conflict in docs

* bump connector version

* correct doc

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2022-02-16 13:06:20 -03:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00