1
0
mirror of synced 2025-12-21 19:11:14 -05:00
Commit Graph

195 Commits

Author SHA1 Message Date
Alexandre Cuoci
ecb56bbbc2 mysql doc rewrite (#32189)
Co-authored-by: Lake Mossman <lake@airbyte.io>
2023-11-07 01:48:51 +00:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Rodi Reich Zilberman
1e8e6402bc Revert "Automated Commit - Formatting Changes" (#32077) 2023-11-01 15:38:25 -05:00
Rodi Reich Zilberman
f5c87968ca URL encode username and password fields (#32028)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-11-01 15:22:27 -05:00
Akash Kulkarni
85d266ac20 Bump DBZ version from 2.2.0 -> 2.4.0 for mssql, mysql (#32024) 2023-11-01 13:55:13 -04:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Rodi Reich Zilberman
7e92375438 bump cdk version for postgres, mysql sources (#31322) 2023-10-12 10:38:54 -07:00
Rodi Reich Zilberman
3d9578fdac Cap each log field (message, stacktrace) to max 32K characters (#30806)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-10-11 13:34:53 -07:00
Akash Kulkarni
2097c0e4e4 Update MySQL streaming config (#30744) 2023-09-26 15:45:13 -07:00
Duy Nguyen
11ad938152 [Source-MySQL] Enable Enhanced Standard sync with PK -> Cursor switch (#30556)
Enable enhanced standard sync
2023-09-21 13:38:05 -07:00
Prateek Mukhedkar
6beba9d4d9 Source MySQL: Release Airbyte Certified MySQL Source connector (#30620) 2023-09-20 14:03:56 -07:00
Duy Nguyen
fca0b6f62e [Source-MySQL] Correctly set Debezium MySQL connector database timezone property (#30333)
Co-authored-by: subodh <subodh1810@gmail.com>
2023-09-14 16:10:14 -05:00
Rodi Reich Zilberman
0936143f1e 🐛 Only take table size info for streams syncing via primary key (#30375)
Co-authored-by: subodh <subodh1810@gmail.com>
2023-09-13 14:35:10 -05:00
Subodh Kant Chaturvedi
6f0c4e2594 mysql-source-cdc: enable compression logic for schema history (#30308)
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
2023-09-13 13:14:01 -05:00
Subodh Kant Chaturvedi
5a4cc0d8b0 source-mysql-cdc: introduce logic to compress schema history if greater than 3MB (#30289)
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
2023-09-13 12:41:28 -05:00
Akash Kulkarni
09df4f4875 Enable MySQL CDC + PK 100% (#30213) 2023-09-07 15:32:29 -05:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-08-31 12:19:02 -05:00
Bumsik Kim
13928271aa doc(mysql.md): binlog_expire_log_seconds -> binlog_expire_logs_seconds (#30019) 2023-08-31 09:57:05 -05:00
Akash Kulkarni
da80408a53 Log storage engines used in MySQL (#30015)
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-08-30 18:35:46 -05:00
Duy Nguyen
b7dc462c4c [Source-MySQL]CDC: Enable frequent state emission during incremental runs (#29308) 2023-08-21 20:13:31 -05:00
Duy Nguyen
6ee426a261 [Source-Mysql] Set default cursor value for cdc mode (#28756) 2023-08-17 13:13:39 -05:00
Lake Mossman
cb863bb552 🎨 Source MySQL: Add 'Recommended' before CDC description (#29220)
* add 'Recommended' before CDC description

* bump version
2023-08-08 16:58:50 -05:00
Lake Mossman
96df471940 🎨 Improve replication method selection UX (#28882)
* update replication method in MySQL source

* bump version

* update expected specs

* update registries

* bump strict encrypt version

* make password always_show

* change url

* update registries
2023-08-03 15:18:34 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
Subodh Kant Chaturvedi
9b9809b006 fix(cdc): limit queue size to lower memory consumption (#26473)
* fix(cdc): limit queue size to lower memory consumption

* add queue size attribute in spec

* disable retries

* fix log

* review comments

* add validation test for queue size

* update expected spec

* bump version + changelog

* update metadata files

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-05-25 22:34:40 +05:30
Rodi Reich Zilberman
804087a052 Don't convert a string to binary base64 unnecessarily (#25586)
* Don't convert a string to binary base64 unnecessarily

* update doc

* update doc
2023-05-24 14:33:20 +00:00
Rodi Reich Zilberman
2aa8a0dbbb Allow sessionvariables jdbc url param in source-mysql (#25859)
* initial commit

* cleanup

* cleanup

* Throw a configuration exception in case of bad jdbc url param

* End-to-End test session variable jdbc param

* reafctoring sanity

* sanity

* bump version and update note

* cherry pick fix to unrelated build error

* fix failing test

---------

Co-authored-by: Ryan Fu <ryan.fu@airbyte.io>
2023-05-22 18:53:33 -07:00
Rodi Reich Zilberman
e598cc9a6c Convert source mysql decimal number with 0 decimal digits to an int (#25460)
* Convert source mysql decimal number with 0 decimal point to an int

* Put int representation in json for decimal with 0 decimal digits

* bump docker image ver and release note

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-10 16:19:57 -07:00
Akash Kulkarni
931911dceb 🐛 Source MySQL/MsSQL Disable index logging for MySQL (#25740)
* Disable index logging for MySQL/MsSQL

* Update mysql.md

* Update mysql.md

* auto-bump connector version

* Disable for MsSQL as well

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-03 08:44:21 -07:00
Subodh Kant Chaturvedi
94ed6554ea upgrade debezium version to 2.2.0.Final (#25401)
* upgrade debezium version to 2.2.0.Final

* bump version + changelog

* auto-bump connector version

* manual bump version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-27 22:12:20 +05:30
Akash Kulkarni
395de77a50 [JDBC sources] : Log database indexes per stream (#25345)
* [JDBC sources] : Log database indexes per stream

* Docker & doc bump

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-20 01:05:15 +00:00
Sergio Ropero
819bba174f postgres-cdc checkpointing: fix LSN parsing bug + refactor for efficiency (#24582)
* Fix LSN parsing from Integer to Long

* rebasing

* Rebase

* Rebase

* Other casting

* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.

* Update load function documentation

* bump mysql and mssql

* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* resolve master merge conflicts

* Automated Change

* minor changes

* resolve merge conflicts

* avoid deserializing multiple times

* simplify

* enable checkpointing for Postgres

* more improvements

* enable assertions

* changelog + bump version

* auto-bump connector version

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-19 21:05:50 +05:30
Akash Kulkarni
f23b3ad88e Add logging functionality to understand Xmin wraparound (#25220)
* Add logging functionality

- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values

* Automated Change

* Renaming + comments

* More comments

* Bump docker version + docs

* auto-bump connector version

* Update failed source_specs.yaml

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-17 22:39:04 +00:00
Lake Mossman
076356c4d4 Always show client cert and key (#25131)
* make CA cert and key always show

* make same change for mysql

* update readmes

* update strict-encrypt version and expected spec

* fix expected spec for mysql strict encrypt

* auto-bump connector version

* manually bump seed for postgres and mysql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-13 21:10:08 +00:00
Subodh Kant Chaturvedi
b5a2ceb7fe cdc: refactor to remove debezium dependency from connector packages (#24656)
* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* Automated Change

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-11 21:32:17 +05:30
Akash Kulkarni
c0341a3271 Fix emitted intermediate state for initial incremental non-CDC syncs (#24820)
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808

* Revert pg

* Adding a more robust unit test for PostgresSource

* Log the query

* Bump dockerfiles + docs

* Bump docs

* auto-bump connector version

* Revert alloydb changes

* Manually update source specs for ones that failed

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-06 22:49:32 +00:00
Subodh Kant Chaturvedi
8748a8d6c7 fix-cdc: errors retry debezium property should be less than max retry (#24833)
* fix-cdc: errors retry debezium property should be less than max retry

* add comment

* version bump + changelog

* auto-bump connector version

* Update source-alloydb versions to match source-postgres

* rebump to 2.0.15

* auto-bump connector version

* definitions + regenerate manually

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
2023-04-04 19:23:00 -07:00
oneshcheret
3c343f5255 Source mysql: update docs to include increase slave_net_timeout and thread_pool_idle_timeout in case of errors (#20438)
* Source mysql: mark unknown column exception as config error

* Source mysql: format code

* Source mysql: increase slave_net_timeout and thread_pool_idle_timeout to 120

* update doc rather than attempting to set global server properties

---------

Co-authored-by: nataly <nataly@airbyte.io>
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
2023-03-29 09:45:45 -07:00
Subodh Kant Chaturvedi
29a7e2ebf5 cdc: fix debezium shutdown interruption bug (#24166)
* cdc: fix debezium shutdown interruption bug

* update retry properties

* add test for queue transfer

* refactor to simplify

* rename parameter

* fix test

* add comments

* add check for engine already closed

* disable checkpointing

* bump version

* auto-bump connector version

* manual bump

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-28 19:47:40 +05:30
Sergio Ropero
80da900d70 feat: Enable CDC checkpointing on Postgres (#24373)
* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.

* Add final statement on exception ;)

* Add comments to CDC Checkpoint tests.
Clean a bit.

* Bump connector versioning

* Add log message

* Fix changelog

* auto-bump connector version

* Manually generate definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-27 14:00:49 +02:00
Lake Mossman
38534b70e0 Set ssl_mode to require on remaining strict encrypt db sources (#24529)
* set ssl_mode to require on remaining db sources

* update readmes

* update expected specs

* auto-bump connector version

* bump mysql version manually

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-25 02:01:11 +00:00
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Duy Nguyen
c1323f615d Verify catalog schema fieldNames is a subset of DB schema (#24207)
* Update current schema vs catalog validation to omit airbyte metadata fields
* Bumped versions for postgres, mssql, and mySQL + changelog
* Bumped dockerfile version of strict encrypt
* Manually generate definitions for mysql
* Manually generate definitions for mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-22 16:41:15 +00:00
VitaliiMaltsev
f41d8219df MySQL Source : add CDC heartbeat support (#23984)
* MySQL Source : add CDC heartbeat support

* removed logs

* fixed DebeziumRecordIteratorTest

* Automated Change

* use generics to simplify implementation

* use Duration

* more refactoring

* removed redundant null check

* Automated Change

* bump version

* auto-bump connector version

---------

Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-21 16:54:45 +00:00
Sergio Ropero
ca2d9c733f fix: Disable CDC checkpointing using a flag (#24275)
* Disable CDC checkpointing until finished the investigation with reading Offset file during syncs

* Bump connector versions

* auto-bump connector version

* Publish postgres manually

* Definitions manually for mysql

* Generated definitions manually mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-21 15:32:20 +01:00
VitaliiMaltsev
e5f96a3678 MySQL Source : support different table structure during "DESCRIBE" query (#24147)
* MySQL Source : support different table structure during "DESCRIBE" query

* bump version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-20 14:28:29 +00:00
VitaliiMaltsev
9e94126fd6 MySQL/MsSQL sources fixed NPE during cursor values validation (#24082)
* MySQL/MsSQL sources  fixed NPE during cursor values validation

* added logging

* bump versions

* auto-bump connector version

* manually update mssql definitions

* Automated Change

* Automated Commit - Formatting Changes

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-15 19:27:31 +00:00
VitaliiMaltsev
ace229f971 MySQL/MsSQL Source: stop sync on null cursor value (#23908)
* MySQL Source: stop sync on null cursor value

* updated MsSQL test

* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql

* fixed MsSQL and MySQL tests

* modified MsSQL query

* fixed typo

* fixed typo

* simplified Mysql query logic

* simplified Mssql query logic

* add quotes to Mssql query

* add mysql view test

* MySQL and MsSQL source throw a warning if a cursor column contains null value

* disabled tests; log warn if null cursor values found

* removed logging in AbstractDbSource

* bump version

* bump version

* auto-bump connector version

* updated changelog

* Automated Change

* updated definitions

* Automated Change

---------

Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-14 22:34:58 +02:00
Prateek Mukhedkar
f6e6ae6c95 Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation (#23939)
* Add allowedHosts to MySQL and MSSQL DB Sources, for network isolation.
2023-03-14 11:04:55 -07:00