* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.
* Add final statement on exception ;)
* Add comments to CDC Checkpoint tests.
Clean a bit.
* Bump connector versioning
* Add log message
* Fix changelog
* auto-bump connector version
* Manually generate definitions
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* set ssl_mode to require on remaining db sources
* update readmes
* update expected specs
* auto-bump connector version
* bump mysql version manually
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add pattern descriptors to more fields in github and postgres sources
* update dockerfiles and readmes
* auto-bump connector version
* manually bump github seed source files
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Cleanup createDatabase() code
Move methods from AbstractJdbcSource
Moved methods until createDataSource outside of the abstract class
* Quick cleanups/renames
* Add tests
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
* fix all postgres acceptance tests
* add newline
* undo other changes to postgres readme file
* add security group to tunnel_method in expected_spec.json
* bump version of strict encrypt
* manually bump versions in seed files
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version
---------
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.
* Reformat code
* Reformat code
* Reformat code
* Reformat code
* Second attempt with ugly if statement
* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>
* Code formatting
* Add additional check if the record is part of the snapshot load to skip state message.
* Remove comments
* Fix imports
* Fix format
* Add check if the iterator has extra elements so we don't send state message twice (edge case)
* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out
* Modify MSSQL and MySQL implementations
* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states
* Formatting
* Improve code documentation and use default for CdcStateHandler new functions
* Sort out missing `final` and types from comments
* Minor improve in checkpoint validation
* format files
* It's 2023!
* Import issues
* Changes after merging master
* Upgrade Debezium version in MySQL
* Bump Postgres and Alloydb
* auto-bump connector version
* Manually bumping version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* This commit fixes the issue when permission is granted at ROLE level instead of USER level.
Missing revoke privileges in the tests.
* Change the query to recursively look for all roles asigned.
Also improve testing.
* Add test for subrole with replication access
* formatting
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* Bump versions and update alloydb docs
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* improve comment
* Typo
* Change from checking the permisions in `pg_users` to execute `createConnection` and verifying the connection is fine for CDC.
* Remove unneeded import
* format
* Rename ReplicationConnection class
* Revert "source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)"
This reverts commit e9efd9878a.
* Revert "Add metadata to connector logs (log level, class name, method name and line number) (#23105)"
This reverts commit a2c80a1fdb.
* Change ConfigError throw point
* Include in try to autoclose the connection
* Bump versions
* auto-bump connector version
* fix SSL failure on check
* format + undo spec changes
* auto-bump connector version
* Manual interaction for source definitions
---------
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* Add initial commit.
Missing testing
* Add unit test revoking replication privilege.
* Small fix in the documentation
* Change from `RuntimeException` to `ConfigErrorException`
* Improve test validation
* Add commentodoc
* bump version!
* Bump strict script
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This changes allows to filter out system views created out of system namespaces
* Add extra view
* Fix issue
* Bump Postgres source version
* bump version
* Bump alloydb
* Bump versioning
* auto-bump connector version
* Set default implementation and remove from unneeded places
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-postgres-source: get rid of short lived objects
* cache column info as well
* get rid of constructor
* upgrade version
* auto-bump connector version
* update definition
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* WIP
* fix issues
* fix test
* format
* more tests
* one more test
* make lsn_commit_behaviour not required
* update expected spec
* update expected spec
* update expected spec
* PR comments
* PR comments
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Stop sync on a null value in a cursor column
* Fix quoting in query to solve a failing test
* Fix another escaping issue in query
* Fix failing test
* Fix failing tests
* Test view with null value cursor
* Improve error message
* bump dockerfile version and update note
* bump version to 1.0.31
* Fix failing test
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres Source Strict Encrypt: Allow connections with sslmodes 'allow' and 'prefer' if SSH tunnel established
* updated changelog
* fixed tests
* add test_strictness_level to acceptance-test-config.yml
* remove test_strictness_level to due to failed SAT test
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial working commit
* Code sanity. Provide no-on implementation to mysql, MSSql to allow compilation.
* Update test
* sanity
* sanity
* sanity
* sanity
* sanity
* changes per review comments
* Make heartbeat change waittime configurable.
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* fix acceptance test config format
* add missing SAT test in config
* revert back changes in acceptance-test-config.yml
* Version and notes
* auto-bump connector version
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
#19191 made me realise the DefaultReplicationWorker's metric tracking today has a bug where we aren't accounting for namespace when tracking metrics today. i.e. Streams with the same name and duplicate namespace will merge metrics.
While reading the code to figure out a fix, I realised we don't have a good conceptual representation of stream namespace <> name pairs within the platform today. We use a concatenated string. Though this works, it will become harder and harder to read/track as we do more operations that involve namespace i.e. progress bars and column selection.
This PR introduces the AirbyteStreamNameNamespacePair object into the platform code to make it more convenient to work with Streams in the future. (Especially if we proceed with the project to make streams a first-class citizen!)
The AirbyteStreamNameNamespacePair object was written to deal with the same issue of namespace <> name pair manipulation within the Java destination code. It implements the Comparable interface, which makes it convenient to use for Collections operations.
For an example of how this is consumed, see #19361.
* Source postgres: encode database name
* Source postgres, mysql: move encoding in util class, apply for mysql
* Source postgres, mysql: make var final
* Source postgres, mysql: bump version
* Source postgres, mysql: format code
* auto-bump connector version
* Source mysql: bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
