* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version
---------
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.
* Reformat code
* Reformat code
* Reformat code
* Reformat code
* Second attempt with ugly if statement
* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>
* Code formatting
* Add additional check if the record is part of the snapshot load to skip state message.
* Remove comments
* Fix imports
* Fix format
* Add check if the iterator has extra elements so we don't send state message twice (edge case)
* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out
* Modify MSSQL and MySQL implementations
* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states
* Formatting
* Improve code documentation and use default for CdcStateHandler new functions
* Sort out missing `final` and types from comments
* Minor improve in checkpoint validation
* format files
* It's 2023!
* Import issues
* Changes after merging master
* Upgrade Debezium version in MySQL
* Bump Postgres and Alloydb
* auto-bump connector version
* Manually bumping version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres Source: fixed bug with non readable double-quoted values within a database name or column name
* bump version
* auto-bump connector version
* bump AlloyDB version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Changes to max fetch row size
- Add frequent logging for fetch size
- Increase MAX_ROW_FETCH_SIZE to 1,000,000,000
* Bump version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* This commit fixes the issue when permission is granted at ROLE level instead of USER level.
Missing revoke privileges in the tests.
* Change the query to recursively look for all roles asigned.
Also improve testing.
* Add test for subrole with replication access
* formatting
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* Bump versions and update alloydb docs
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* improve comment
* Typo
* Change from checking the permisions in `pg_users` to execute `createConnection` and verifying the connection is fine for CDC.
* Remove unneeded import
* format
* Rename ReplicationConnection class
* Revert "source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)"
This reverts commit e9efd9878a.
* Revert "Add metadata to connector logs (log level, class name, method name and line number) (#23105)"
This reverts commit a2c80a1fdb.
* Change ConfigError throw point
* Include in try to autoclose the connection
* Bump versions
* auto-bump connector version
* fix SSL failure on check
* format + undo spec changes
* auto-bump connector version
* Manual interaction for source definitions
---------
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* Update docker image and release notes
* auto-bump connector version
* manually bump version on spec
---------
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Sergio Ropero <sergio@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add initial commit.
Missing testing
* Add unit test revoking replication privilege.
* Small fix in the documentation
* Change from `RuntimeException` to `ConfigErrorException`
* Improve test validation
* Add commentodoc
* bump version!
* Bump strict script
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This changes allows to filter out system views created out of system namespaces
* Add extra view
* Fix issue
* Bump Postgres source version
* bump version
* Bump alloydb
* Bump versioning
* auto-bump connector version
* Set default implementation and remove from unneeded places
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-postgres-source: get rid of short lived objects
* cache column info as well
* get rid of constructor
* upgrade version
* auto-bump connector version
* update definition
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Bump total allocation of the source's available memory for the JDBC buffer from 60% to 70%
* Profiling setup
* Update Dockerfile
* Fixes
* Update FetchSizeConstants.java
* Update dockerfiles
* Update JdbcStreamingQueryConfig.java
* auto-bump connector version
* Manually bump source-mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* revert back to erroring out on a null cursor value
* Update dockerfiles and release notes
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* postgres-source-cdc:handle null values for array data types
* fix test
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* WIP
* fix issues
* fix test
* format
* more tests
* one more test
* make lsn_commit_behaviour not required
* update expected spec
* update expected spec
* update expected spec
* PR comments
* PR comments
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-postgres:just throw a warning if a cursor column contains null value
* ignore test + update review comment
* use Disabled annotation
* upgrade version
* upgrade version for alloy db strict encrypt as well
* auto-bump connector version
* update definitions
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Stop sync on a null value in a cursor column
* Fix quoting in query to solve a failing test
* Fix another escaping issue in query
* Fix failing test
* Fix failing tests
* Test view with null value cursor
* Improve error message
* bump dockerfile version and update note
* bump version to 1.0.31
* Fix failing test
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres Source Strict Encrypt: Allow connections with sslmodes 'allow' and 'prefer' if SSH tunnel established
* updated changelog
* fixed tests
* add test_strictness_level to acceptance-test-config.yml
* remove test_strictness_level to due to failed SAT test
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial working commit
* Code sanity. Provide no-on implementation to mysql, MSSql to allow compilation.
* Update test
* sanity
* sanity
* sanity
* sanity
* sanity
* changes per review comments
* Make heartbeat change waittime configurable.
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* fix acceptance test config format
* add missing SAT test in config
* revert back changes in acceptance-test-config.yml
* Version and notes
* auto-bump connector version
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres Source: use native Postgres timeout if it's not set by the user
* refactoring
* updated connection timeout logic and added tests for different datasources creation
* fixed pmd
* refactoring
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres/MySQL Source Strict Encrypt: stop enforce SSL if ssl mode disabled
* fixed checkstyle
* updated changelog
* add tests
* replaced MySQL test to mysql-strict-encrypt module
* fixed Connection Refused for mysql test
* replaced Postgres Source strict-encrypt tests into new class
* bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Source postgres: encode database name
* Source postgres, mysql: move encoding in util class, apply for mysql
* Source postgres, mysql: make var final
* Source postgres, mysql: bump version
* Source postgres, mysql: format code
* auto-bump connector version
* Source mysql: bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Changes method from returning null to throwing Exception to match method structure
* Added javadoc explaining that KeyPair should not be returning null
* Moved ConnectionErrorException to commons package
* Adds error handling for SshWrappedDestinations and markdown to include changes
* Bumps version numbers for Postgres, MySQL, MSSQL
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Publishes Postgres, MySQL, MSSQL source with changes from #18041
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* postgres-source: handle 24:00:00 value for time column
* address review comment
* fix test + use LocalTime.MAX
* bump version
* update alloy db as well
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Match entry in client cert keystore to hardcoded postgres driver value
* Match entry in client cert keystore to hardcoded postgres driver value
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* upgrade debezium version to 1.9.6
* test override not required
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Make error more descriptive in case of no encryption state
* Fix failing test
* Change error message on strict-encrypt postgres destination as well. Fix failing test
* update versions and changelogs
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
