* Separated Standard vs Xmin categorization
Added StandardCtidUtilsTest
Added versions to StandardStatus
* WIP incremental read test
* Added PostgresStandardStateManager to write state_type key to stream state
Added sync to verify state messages and cursor records on running multiple syncs of the same streams
* Isolate tables used for ctid test and enable checkpointing records to 1 for just CTID test
* Moved ctid cursor test to its own file CtidEnabledPostgresSourceTest
* Handle cases we cannot proceed with ctid sync
* improvements
* Test
* No longer throw error in StandardCtidUtils when there is no state type
Removed unused variable
* Updated "standard" sync type to "cursorBased"
* Removed stale code comment
* Test
* Fix merge error
* Added more logging during ctid startup.
changes cursorBased to cursor_based
* remove merge error
* Improve filenode query in case of view.
beautify code
---------
Co-authored-by: nguyenaiden <duy@airbyte.io>
* use LSN as default cursor for postgres CDC
* Fixed static constant
* Set lsn default cursor value for postgres sync
* Bumped metadata and dockerfile versions
* Disable acceptance backwards compatibility discovery test as this is a breaking change
---------
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
* Separated Standard vs Xmin categorization
Added StandardCtidUtilsTest
Added versions to StandardStatus
* WIP incremental read test
* Added PostgresStandardStateManager to write state_type key to stream state
Added sync to verify state messages and cursor records on running multiple syncs of the same streams
* Isolate tables used for ctid test and enable checkpointing records to 1 for just CTID test
* Moved ctid cursor test to its own file CtidEnabledPostgresSourceTest
* Gracefully handle case where there is no statetype
Update all "standard" to "cursorBased"
* Add logging functionality
- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values
* Automated Change
* Renaming + comments
* More comments
* Initial commit for xmin prototype
- Add spec.json
- Catalog
- Try to change incremental query
* Skeleton for sync
Decision is to use a whole new sync method
* With cursor
* Working catalog + attempt at xmin record iterator
* Unit tested code
* Fix build break
* Fixing state
* more xmin
* Revert unchanged files
* Addressing code review comments
* use generated pojo
* Add iniital XminStateManagerTest
* Additional unit tests
* Added unit test
* Update xmin state manager test
* Remove spec.json xmin option
* Fixed bug where full refresh streams were not being filtered out for Xmin
* Fix latest safe change boundary issue
* Xmin spec update
* Fix
* Automated Commit - Format and Process Resources Changes
* Remove full_refresh as an option for xmin syncs and always include incremental syncs
* Fix integ test
* revert unused files
* revert
* Bump docker + update change log
* Modify order of CDC
* Update expected_spec.json
* Update expected_json
Fix acceptance tests
* Reverse order for xmin replication
* Update expected strict encrypt json
* Fix broken XminCtidUtilsTest
- Logic was added to XminCtidUtils to filter for incremental syncs, but the test data wasn't updated to represent this. This doesn't require any publishing
---------
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* Add logging functionality
- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values
* Automated Change
* Renaming + comments
* More comments
* Initial commit for xmin prototype
- Add spec.json
- Catalog
- Try to change incremental query
* Skeleton for sync
Decision is to use a whole new sync method
* With cursor
* Working catalog + attempt at xmin record iterator
* Unit tested code
* Fix build break
* Fixing state
* more xmin
* Revert unchanged files
* Addressing code review comments
* use generated pojo
* Add iniital XminStateManagerTest
* Additional unit tests
* Added unit test
* Update xmin state manager test
* Remove spec.json xmin option
* Fixed bug where full refresh streams were not being filtered out for Xmin
* Fix latest safe change boundary issue
* Xmin spec update
* Fix
* Automated Commit - Format and Process Resources Changes
* Remove full_refresh as an option for xmin syncs and always include incremental syncs
* Fix integ test
* revert unused files
* revert
* Bump docker + update change log
* Modify order of CDC
* Update expected_spec.json
* Update expected_json
Fix acceptance tests
* Reverse order for xmin replication
* Update expected strict encrypt json
---------
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* wip
* wip 2
* undo unwatned change
* some refactoring
* fix few tests
* more fixing
* more fixes
* Automated Commit - Format and Process Resources Changes
* more improvements
* 1 more test
* another test
* add flag for ctid enabling
* fix conflicts
* else block is not required
* use emittedAt
* skip WAL processing if streams under vacuuming
---------
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
* initial ctid for testing
* initial ctid for testing
* initial ctid for testing
* Automated Commit - Format and Process Resources Changes
* add version and state type to xmin status
* add logic to swtich between xmin and ctid sync
* npe fixes
* use enum
* refactor
* add relation node logic + validation for vacuuming + more refactor
* refine test + make PR ready for review
* remove un-wanted changes
* missed this one
* remove irrelevant comments
* add more assertions
* remove jdbc log
* address review comments
* 🤖 Auto format source-postgres code [skip ci]
* skip streams under vacuum
* 🤖 Auto format source-postgres code [skip ci]
* update log message
* 🤖 Auto format source-postgres code [skip ci]
* comment
* latest round of review comments
* missed this file
* 🤖 Auto format source-postgres code [skip ci]
* initial drop for testing
* test
* refactor query plan
* Add testing for query plan
* sanity
* sanity
* sanity
* remove change in spec json
* sanity
* test
* comment
* remove hardcoded testing value
* add test case
* revert json spec
* small imrovements
* typo
* fix test
* bump docker image version
* revert bump docker image version
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
* Fix issue in streaming JDBC database
* Bump + publish
* 🤖 Auto format source-db2 code [skip ci]
* 🤖 Auto format source-mssql code [skip ci]
* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]
* 🤖 Auto format source-postgres code [skip ci]
* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]
* 🤖 Auto format source-oracle code [skip ci]
* 🤖 Auto format source-tidb code [skip ci]
* 🤖 Auto format source-snowflake code [skip ci]
* 🤖 Auto format source-redshift code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* Move icons to connector folder
* Delete old icons
* Update upload logic
* Add icon url to definitions
* Update registry model
* Populate cdn url
* DNC butcher the pipeline
* Low hanging fruit fixes
* Fix bucket name
* Merge old and new approaches
* Fix metadata upload step
* Format
* Fix test
* additional unit testing added for postgres source connector spec.
* Fixed character issues in postgres unit tests.
* Fix the removal of testing comment.
---------
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Split out the smallest set of reasonable changes from #26086 .
My goal was to split out the interface, as well as show how the interface it's meant to be used.
Follow up PRs:
- Split out classes from BufferManager and add more tests there.
- Add in the AsyncConsumer with tests.
- Add in the StagingConsumer factory.
* Disable index logging for MySQL/MsSQL
* Update mysql.md
* Update mysql.md
* auto-bump connector version
* Disable for MsSQL as well
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* upgrade debezium version to 2.2.0.Final
* bump version + changelog
* auto-bump connector version
* manual bump version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Fix LSN parsing from Integer to Long
* rebasing
* Rebase
* Rebase
* Other casting
* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.
* Update load function documentation
* bump mysql and mssql
* cdc: refactor to remove debezium dependency from connector packages
* use gradle's shared dependency
* more refactoring
* upgrade docker version
* resolve master merge conflicts
* Automated Change
* minor changes
* resolve merge conflicts
* avoid deserializing multiple times
* simplify
* enable checkpointing for Postgres
* more improvements
* enable assertions
* changelog + bump version
* auto-bump connector version
* auto-bump connector version
* manual bump
---------
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* rename catalog to registry in metadata service
* rename catalog to registry in metadata files
* Run generate models
* Fix missed renames
* Add github personal access token
* Run black
* Automated Change
---------
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
* make CA cert and key always show
* make same change for mysql
* update readmes
* update strict-encrypt version and expected spec
* fix expected spec for mysql strict encrypt
* auto-bump connector version
* manually bump seed for postgres and mysql
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Basic removal
* Removed SSL key static variables in PostgresSource
Updated expected_spec to match spec
* Removed autocommit for SelectDB
* Removed the line in PostgresSourceStrictEncrypt that removed "SSL_KEY" from properties
* Revert "Removed autocommit for SelectDB"
This reverts commit 14ff20d005.
* bumped Dockerfile versions and updated changelog
* Reverted removal of SSL key from the JDBC Connection query
* Removed extraneous keys
* auto-bump connector version
---------
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808
* Revert pg
* Adding a more robust unit test for PostgresSource
* Log the query
* Bump dockerfiles + docs
* Bump docs
* auto-bump connector version
* Revert alloydb changes
* Manually update source specs for ones that failed
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-cdc: errors retry debezium property should be less than max retry
* add comment
* version bump + changelog
* auto-bump connector version
* Update source-alloydb versions to match source-postgres
* rebump to 2.0.15
* auto-bump connector version
* definitions + regenerate manually
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
