1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

25 Commits

Author SHA1 Message Date
devin-ai-integration[bot]
dca8f42a10 chore(source-tidb): Convert to new gradle build flow (do not merge) (#62927)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yue.li@airbyte.io <yue.li@airbyte.io>
2025-07-10 16:48:53 -07:00
Augustin
c1d9286ef9 source-tidb: Use airbyte/java-connector-base:2.0.0 (#51499) 2025-01-13 17:37:04 +02:00
Augustin
dce085ad86 source-tidb: Use airbyte/java-connector-base:1.0.0 (#49896)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2024-12-19 10:25:33 +02:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Marius Posta
16821f0d5f source-tidb: adopt CDK 0.20.4 (#35218) 2024-02-14 11:55:06 -08:00
Chandler Prall
9a67b7fa10 🐛 bump cdk versions for java sources (#34453) 2024-02-01 14:26:38 -07:00
Subodh Kant Chaturvedi
0c3d12ba2b jdbc-sources: migrate to PER-STREAM state (#33485) 2023-12-19 15:38:39 +05:30
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Prateek Mukhedkar
33ca7f3451 Adding allowHosts to all non-GA source database connectors (#23455)
Adding allowedHosts to clickhouse, cockroachdb, db2, oracle, snowflake, tidb source Connectors. 
---------

Co-authored-by: prateekmukhedkar <prateekmukhedkar@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-07 19:52:15 -08:00
VitaliiMaltsev
f5f4de2e5f JDBC sources: consolidate date/time values mapping (#20346)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* resolved merge conflicts

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* bump postgres version

* auto-bump connector version

* updated seed definitions

* revert generated definitions for Postgres

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-01-18 19:21:11 +02:00
Liren Tu
888347a0d0 🎉 JDBC sources: store cursor record count in db state (#15535)
* Add cursor_record_count to db stream state

* Add cursor record count to cursor info

* Emit max cursor record count

* Add original cursor record count

* Unify logging format

* Add backward compatible methods

* Update unit tests for state decorating iterator

* Update test (not done yet)

* Fix one more unit test

* Change where clause operator according to record count

* Add branch for null cursor

* Skip saving record count when it is 0

* Fix log wording

* Set mock record count in test

* Check cursor value instead of cursor info

* Fix source jdbc test

* Read record count from state

* Fix tests

* Add an acceptance test case

* Fix npe

* Change record count from int to long to avoid type conversion

* Fix references

* Fix oracle container

* Use uppercase for snowflake

* Use uppercase for db2

* Fix and use uppercase

* Update test case to include the edge case

* Format code

* Remove extra assertion in clickhouse

* Merge ms sql incremental query method

* Log query for debugging

* Clean up name_and_timestamp table

* Fix db2 tests

* Fix mssql tests

* Fix oracle tests

* Fix oracle tests

* Fix cockroachdb tests

* Fix snowflake tests

* Add changelog

* Fix mssql tests

* Fix db2-strict-encrypt tests

* Fix oracle-strict-encrypt tests

* Bump postgres version

* Fix oracle-strict-encrypt tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-14 01:09:12 -07:00
Liren Tu
c3f3702522 🎉 Jdbc sources: emit state messages more frequently (#16256)
* Emit intermediate states in clickhouse

* Add pr id

* Emit intermediate states in cockroachdb

* Emit intermediate states in db2

* Emit intermediate states in jdbc source

* Emit intermediate states in tidb

* Revert "Emit intermediate states in jdbc source"

This reverts commit 7a913bbae7.

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-01 21:44:48 -07:00
Daemonxiao
d4524032ae 🎉 New Destination: TiDB (#15592)
* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* fmt

* fix integration test

* Update docs/integrations/destinations/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update doc

* Update doc

* Update doc

* bump normalization version

* update normalization changelog

* run format

* add dest def

* generat spec

Co-authored-by: Xiang Zhang <angwerzx@126.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-08-31 16:50:27 -03:00
Xiang Zhang
c05b3f0ceb Docs: Add "External resources" section to TiDB source connector doc (#15238) 2022-08-03 13:45:35 -03:00
Edward Gao
b2dd470d3d Handle ints and longs in normalization (#14362)
* generate airbyte_type:integer

* normalization accepts `airbyte_type: integer`

* handles ints+longs

* update avro for consistency

* delete long type for now, treat all ints as longs

* update avro type mappings

{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")

* fix test

* remove long handling

* Revert "remove long handling"

This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.

* Revert "update avro type mappings"

This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.

* Revert "delete long type for now, treat all ints as longs"

This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.

* Revert "update avro for consistency"

This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.

* newline@eof

* update test

* slightly better local tests

* fix test

* missed a few cases

* postgres tests use correct hostnames

* fix normalization

* fix int macro

* add test case

* normalization test output

* handle int/long correctly

* fix types for other DBs

* uint32 -> bigint; tests

* add type value assertions

* more test updates

* regenerate output

* reconcile big_integer to match docs

* update comment

* fix type

* fix mysql constructor call

* bigint only has 38 digits

* fix s3 ints, fix DAT test case

* big_integer should be string

* reduce to 28 digit big_ints

* fix test setup, mysql

* kill big_integer tests

* regenerate output

* version bumps

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-26 16:40:14 -07:00
VitaliiMaltsev
8ebed61163 Source TiDb Removed additionalProperties: false from spec (#14996)
* Source TiDb Removed additionalProperties: false from spec

* updated changelog

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-25 12:04:22 +03:00
Edward Gao
24efcedade 📝 improve error message when using unsupported JDBC type as cursor (#14714)
* improve error message

* also in JdbcSourceOperations

* bump versions + changelog

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-22 18:29:03 -07:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
Daemonxiao
76a1e86363 Source TiDB: correct jdbc string builder (#14243)
* add icon for tidb-connector

* Fix TiDB source connector

* bump connector version

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-04 14:56:44 -03:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
LiRen Tu
35f2aa9aed 🎉 Jdbc sources: publish new version with adaptive fetch size (#12480)
* Default scaffold to use adaptive streaming config

* Switch more connectors to use adaptive streaming config

* Bump version for cockroach db

* Bump version for db2

* Bump mssql version

* Bump mysql version

* Bump oracle version

* Bump postgres version

* Bump redshift version

* Bump snowflake version

* Bump tidb version

* auto-bump connector version

* Fix db2 findbug issue

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix more findbug issues

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix findbug issue for mysql-strict-encrypt

* Fix findbugs issue for oracle source

* auto-bump connector version

* Remove suppress warnings annotation

* Fix oracle encrypt tests

* Fix oracle encrypt acceptance test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 23:14:58 -07:00
Daemonxiao
5164e12fc4 🎉 New source: TiDB (#11283)
* add new source tidb

* formate java code style and add item in SUMMARY.md

* update doc

* Update airbyte-integrations/connectors/source-tidb/src/main/resources/spec.json

uptdate doc

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update airbyte-integrations/connectors/source-tidb/README.md

* Update docs/integrations/sources/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update docs/integrations/sources/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* add seed and doc changelog

* run format

* regenerate seed file

Co-authored-by: Xiang Zhang <angwerzx@126.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-04-20 13:14:25 -03:00