* postgres-source-cdc:handle null values for array data types
* fix test
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* WIP
* fix issues
* fix test
* format
* more tests
* one more test
* make lsn_commit_behaviour not required
* update expected spec
* update expected spec
* update expected spec
* PR comments
* PR comments
* upgrade version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-postgres:just throw a warning if a cursor column contains null value
* ignore test + update review comment
* use Disabled annotation
* upgrade version
* upgrade version for alloy db strict encrypt as well
* auto-bump connector version
* update definitions
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add column.include.list for debezium column selection support
* sanity
* Fix escaping of regex to support all unicode characters in table and column names, including regex control characters (command, dot, asterisk etc.)
* sanity
* Fix failing test
* Test that directly verifies that read doesn't return columns that are not selected
* Test mysql cdc column selection
* Stop sync on a null value in a cursor column
* Fix quoting in query to solve a failing test
* Fix another escaping issue in query
* Fix failing test
* Fix failing tests
* Test view with null value cursor
* Improve error message
* bump dockerfile version and update note
* bump version to 1.0.31
* Fix failing test
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Migrating remaining tests
* More tests
* set env vars before querying them; use stream state everywhere?
* fix strirct encrypt expected spec
* runRead after workerConfigs exist
* format
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* Postgres Source Strict Encrypt: Allow connections with sslmodes 'allow' and 'prefer' if SSH tunnel established
* updated changelog
* fixed tests
* add test_strictness_level to acceptance-test-config.yml
* remove test_strictness_level to due to failed SAT test
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial working commit
* Code sanity. Provide no-on implementation to mysql, MSSql to allow compilation.
* Update test
* sanity
* sanity
* sanity
* sanity
* sanity
* changes per review comments
* Make heartbeat change waittime configurable.
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* Trying to bypass test strictness test
* fix acceptance test config format
* add missing SAT test in config
* revert back changes in acceptance-test-config.yml
* Version and notes
* auto-bump connector version
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
#19191 made me realise the DefaultReplicationWorker's metric tracking today has a bug where we aren't accounting for namespace when tracking metrics today. i.e. Streams with the same name and duplicate namespace will merge metrics.
While reading the code to figure out a fix, I realised we don't have a good conceptual representation of stream namespace <> name pairs within the platform today. We use a concatenated string. Though this works, it will become harder and harder to read/track as we do more operations that involve namespace i.e. progress bars and column selection.
This PR introduces the AirbyteStreamNameNamespacePair object into the platform code to make it more convenient to work with Streams in the future. (Especially if we proceed with the project to make streams a first-class citizen!)
The AirbyteStreamNameNamespacePair object was written to deal with the same issue of namespace <> name pair manipulation within the Java destination code. It implements the Comparable interface, which makes it convenient to use for Collections operations.
For an example of how this is consumed, see #19361.
* Postgres Source: use native Postgres timeout if it's not set by the user
* refactoring
* updated connection timeout logic and added tests for different datasources creation
* fixed pmd
* refactoring
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Postgres/MySQL Source Strict Encrypt: stop enforce SSL if ssl mode disabled
* fixed checkstyle
* updated changelog
* add tests
* replaced MySQL test to mysql-strict-encrypt module
* fixed Connection Refused for mysql test
* replaced Postgres Source strict-encrypt tests into new class
* bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Source postgres: encode database name
* Source postgres, mysql: move encoding in util class, apply for mysql
* Source postgres, mysql: make var final
* Source postgres, mysql: bump version
* Source postgres, mysql: format code
* auto-bump connector version
* Source mysql: bump version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Changes method from returning null to throwing Exception to match method structure
* Added javadoc explaining that KeyPair should not be returning null
* Moved ConnectionErrorException to commons package
* Adds error handling for SshWrappedDestinations and markdown to include changes
* Bumps version numbers for Postgres, MySQL, MSSQL
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Publishes Postgres, MySQL, MSSQL source with changes from #18041
* auto-bump connector version
* auto-bump connector version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* postgres-source: handle 24:00:00 value for time column
* address review comment
* fix test + use LocalTime.MAX
* bump version
* update alloy db as well
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Match entry in client cert keystore to hardcoded postgres driver value
* Match entry in client cert keystore to hardcoded postgres driver value
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
