* Proof of concept parallel source stream reading implementation for MySQL
* Automated Change
* Add read method that supports concurrent execution to Source interface
* Remove parallel iterator
* Ensure that executor service is stopped
* Automated Commit - Format and Process Resources Changes
* Expose method to fix compilation issue
* Use concurrent map to avoid access issues
* Automated Commit - Format and Process Resources Changes
* Ensure concurrent streams finish before closing source
* Fix compile issue
* Formatting
* Exclude concurrent stream threads from orphan thread watcher
* Automated Commit - Format and Process Resources Changes
* Refactor orphaned thread logic to account for concurrent execution
* PR feedback
* Implement readStreams in wrapper source
* Automated Commit - Format and Process Resources Changes
* Add readStream override
* Automated Commit - Format and Process Resources Changes
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* Debug logging
* Reduce logging level
* Replace synchronized calls to System.out.println when concurrent
* Close consumer
* Flush before close
* Automated Commit - Format and Process Resources Changes
* Remove charset
* Use ASCII and flush periodically for parallel streams
* Test performance harness patch
* Automated Commit - Format and Process Resources Changes
* Cleanup
* Logging to identify concurrent read enabled
* Mark parameter as final
---------
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
* use LSN as default cursor for postgres CDC
* Fixed static constant
* Set lsn default cursor value for postgres sync
* Bumped metadata and dockerfile versions
* Disable acceptance backwards compatibility discovery test as this is a breaking change
---------
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
* wip
* wip 2
* undo unwatned change
* some refactoring
* fix few tests
* more fixing
* more fixes
* Automated Commit - Format and Process Resources Changes
* more improvements
* 1 more test
* another test
* add flag for ctid enabling
* fix conflicts
* else block is not required
* use emittedAt
* skip WAL processing if streams under vacuuming
---------
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
* Adds support for incremental syncs with GLOBAL state message emission
* Adds args to Source to not break, future work is to add same features to Source perf-harness and ./gradlew :spotlessJavaApply
* initial commit
* cleanup
* cleanup
* Throw a configuration exception in case of bad jdbc url param
* End-to-End test session variable jdbc param
* reafctoring sanity
* sanity
* bump version and update note
* cherry pick fix to unrelated build error
* fix failing test
---------
Co-authored-by: Ryan Fu <ryan.fu@airbyte.io>
Split out the smallest set of reasonable changes from #26086 .
My goal was to split out the interface, as well as show how the interface it's meant to be used.
Follow up PRs:
- Split out classes from BufferManager and add more tests there.
- Add in the AsyncConsumer with tests.
- Add in the StagingConsumer factory.
* Convert source mysql decimal number with 0 decimal point to an int
* Put int representation in json for decimal with 0 decimal digits
* bump docker image ver and release note
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* make CA cert and key always show
* make same change for mysql
* update readmes
* update strict-encrypt version and expected spec
* fix expected spec for mysql strict encrypt
* auto-bump connector version
* manually bump seed for postgres and mysql
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* test docker behavior on CI env
* Automated Change
* test docker behavior on CI env
* Make all unit and integration tests in source-postgres pass locally
* Fix mysql ssh integration test
* Fix failing test
* Fix source-mssql build
* source-mssql runss tests locally.
Fix compilation errors
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
* MySQL Source : support different table structure during "DESCRIBE" query
* bump version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* MySQL Source: stop sync on null cursor value
* updated MsSQL test
* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql
* fixed MsSQL and MySQL tests
* modified MsSQL query
* fixed typo
* fixed typo
* simplified Mysql query logic
* simplified Mssql query logic
* add quotes to Mssql query
* add mysql view test
* MySQL and MsSQL source throw a warning if a cursor column contains null value
* disabled tests; log warn if null cursor values found
* removed logging in AbstractDbSource
* bump version
* bump version
* auto-bump connector version
* updated changelog
* Automated Change
* updated definitions
* Automated Change
---------
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.
* Reformat code
* Reformat code
* Reformat code
* Reformat code
* Second attempt with ugly if statement
* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>
* Code formatting
* Add additional check if the record is part of the snapshot load to skip state message.
* Remove comments
* Fix imports
* Fix format
* Add check if the iterator has extra elements so we don't send state message twice (edge case)
* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out
* Modify MSSQL and MySQL implementations
* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states
* Formatting
* Improve code documentation and use default for CdcStateHandler new functions
* Sort out missing `final` and types from comments
* Minor improve in checkpoint validation
* format files
* It's 2023!
* Import issues
* Changes after merging master
* Upgrade Debezium version in MySQL
* Bump Postgres and Alloydb
* auto-bump connector version
* Manually bumping version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add column.include.list for debezium column selection support
* sanity
* Fix escaping of regex to support all unicode characters in table and column names, including regex control characters (command, dot, asterisk etc.)
* sanity
* Fix failing test
* Test that directly verifies that read doesn't return columns that are not selected
* Test mysql cdc column selection
* Test
* Check for REPLICATION CLIENT PRIVILEGE
* format
* undo format
* Fix test + comments
* Simplify checked function
* Bump up version + log
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial working commit
* Code sanity. Provide no-on implementation to mysql, MSSql to allow compilation.
* Update test
* sanity
* sanity
* sanity
* sanity
* sanity
* changes per review comments
* Make heartbeat change waittime configurable.
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* fix acceptance test config format
* add missing SAT test in config
* revert back changes in acceptance-test-config.yml
* Version and notes
* auto-bump connector version
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* mysql-source:tinyint-unsigned should not be identified as boolean
* upgrade version
* fix encrypted spec
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* mssql-source:upgrade debezium version to 1.9.6
* more improvements
* upgrade version
* auto-bump connector version
* fix test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
