* enable ctid feature flag for testing.
bump versions
* Removed PostgresSourceTest pre-ctid and turn CtidEnabledPostgresSourceTest back into PostgresJdbcSourceAcceptanceTest (#28341)
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
* Update PostgresSourceTest to no longer use Legacy AirbyteStateType
* Fix test.
Uncovered a bug in handling of money[] and float[]
* revert version change
* Orderly removal of ctid flag for cursor
* Handle correctly full refresh streams. (#28369)
* Handle correctly full refresh streams.
Handle xmin full refresh (currently blocked in ui).
remove unnecessary debug logs
* Test catalog with a mix of incremental and full refresh streams
* Filter out full refresh streams since they are not expected to be retuned by getIncrementalIterators()
* Filter out full refresh existing streams as well
* revert name to reflect incremental only streams
* 28505 xmin + ctid reclassify connection when ctid is not possible (#28548)
* reclassify to xmin stream we cannot read with ctid
* fix reclassify
* improve test
* Add initial load using xmin flow
* typo
* Added test for xmin initial load flow
* Improve testing for initial load with xmin flow
* sanity
* sanity
* bump versions and release notes
---------
Co-authored-by: Duy Nguyen <duy@airbyte.io>
* use LSN as default cursor for postgres CDC
* Fixed static constant
* Set lsn default cursor value for postgres sync
* Bumped metadata and dockerfile versions
* Disable acceptance backwards compatibility discovery test as this is a breaking change
---------
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
* Add logging functionality
- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values
* Automated Change
* Renaming + comments
* More comments
* Initial commit for xmin prototype
- Add spec.json
- Catalog
- Try to change incremental query
* Skeleton for sync
Decision is to use a whole new sync method
* With cursor
* Working catalog + attempt at xmin record iterator
* Unit tested code
* Fix build break
* Fixing state
* more xmin
* Revert unchanged files
* Addressing code review comments
* use generated pojo
* Add iniital XminStateManagerTest
* Additional unit tests
* Added unit test
* Update xmin state manager test
* Remove spec.json xmin option
* Fixed bug where full refresh streams were not being filtered out for Xmin
* Fix latest safe change boundary issue
* Xmin spec update
* Fix
* Automated Commit - Format and Process Resources Changes
* Remove full_refresh as an option for xmin syncs and always include incremental syncs
* Fix integ test
* revert unused files
* revert
* Bump docker + update change log
* Modify order of CDC
* Update expected_spec.json
* Update expected_json
Fix acceptance tests
* Reverse order for xmin replication
* Update expected strict encrypt json
---------
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* Fix issue in streaming JDBC database
* Bump + publish
* 🤖 Auto format source-db2 code [skip ci]
* 🤖 Auto format source-mssql code [skip ci]
* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]
* 🤖 Auto format source-postgres code [skip ci]
* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]
* 🤖 Auto format source-oracle code [skip ci]
* 🤖 Auto format source-tidb code [skip ci]
* 🤖 Auto format source-snowflake code [skip ci]
* 🤖 Auto format source-redshift code [skip ci]
* 🤖 Auto format source-mysql code [skip ci]
* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]
---------
Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
* upgrade debezium version to 2.2.0.Final
* bump version + changelog
* auto-bump connector version
* manual bump version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Fix LSN parsing from Integer to Long
* rebasing
* Rebase
* Rebase
* Other casting
* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.
* Update load function documentation
* bump mysql and mssql
* cdc: refactor to remove debezium dependency from connector packages
* use gradle's shared dependency
* more refactoring
* upgrade docker version
* resolve master merge conflicts
* Automated Change
* minor changes
* resolve merge conflicts
* avoid deserializing multiple times
* simplify
* enable checkpointing for Postgres
* more improvements
* enable assertions
* changelog + bump version
* auto-bump connector version
* auto-bump connector version
* manual bump
---------
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* make CA cert and key always show
* make same change for mysql
* update readmes
* update strict-encrypt version and expected spec
* fix expected spec for mysql strict encrypt
* auto-bump connector version
* manually bump seed for postgres and mysql
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Basic removal
* Removed SSL key static variables in PostgresSource
Updated expected_spec to match spec
* Removed autocommit for SelectDB
* Removed the line in PostgresSourceStrictEncrypt that removed "SSL_KEY" from properties
* Revert "Removed autocommit for SelectDB"
This reverts commit 14ff20d005.
* bumped Dockerfile versions and updated changelog
* Reverted removal of SSL key from the JDBC Connection query
* Removed extraneous keys
* auto-bump connector version
---------
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808
* Revert pg
* Adding a more robust unit test for PostgresSource
* Log the query
* Bump dockerfiles + docs
* Bump docs
* auto-bump connector version
* Revert alloydb changes
* Manually update source specs for ones that failed
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-cdc: errors retry debezium property should be less than max retry
* add comment
* version bump + changelog
* auto-bump connector version
* Update source-alloydb versions to match source-postgres
* rebump to 2.0.15
* auto-bump connector version
* definitions + regenerate manually
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.
* Add final statement on exception ;)
* Add comments to CDC Checkpoint tests.
Clean a bit.
* Bump connector versioning
* Add log message
* Fix changelog
* auto-bump connector version
* Manually generate definitions
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* make ssl mode default to require for postgres on cloud
* update dockerfiles and readme
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add pattern descriptors to more fields in github and postgres sources
* update dockerfiles and readmes
* auto-bump connector version
* manually bump github seed source files
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
* fix all postgres acceptance tests
* add newline
* undo other changes to postgres readme file
* add security group to tunnel_method in expected_spec.json
* bump version of strict encrypt
* manually bump versions in seed files
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Update current schema vs catalog validation to omit airbyte metadata fields
* Bumped versions for postgres, mssql, and mySQL + changelog
* Bumped dockerfile version of strict encrypt
* Manually generate definitions for mysql
* Manually generate definitions for mssql
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-postgres-cdc-npe:do not put null in properties
* version bump + change log
* auto-bump connector version
* manual bump
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version
---------
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.
* Reformat code
* Reformat code
* Reformat code
* Reformat code
* Second attempt with ugly if statement
* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>
* Code formatting
* Add additional check if the record is part of the snapshot load to skip state message.
* Remove comments
* Fix imports
* Fix format
* Add check if the iterator has extra elements so we don't send state message twice (edge case)
* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out
* Modify MSSQL and MySQL implementations
* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states
* Formatting
* Improve code documentation and use default for CdcStateHandler new functions
* Sort out missing `final` and types from comments
* Minor improve in checkpoint validation
* format files
* It's 2023!
* Import issues
* Changes after merging master
* Upgrade Debezium version in MySQL
* Bump Postgres and Alloydb
* auto-bump connector version
* Manually bumping version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
