* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* postgres-source: handle 24:00:00 value for time column
* address review comment
* fix test + use LocalTime.MAX
* bump version
* update alloy db as well
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Match entry in client cert keystore to hardcoded postgres driver value
* Match entry in client cert keystore to hardcoded postgres driver value
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* upgrade debezium version to 1.9.6
* test override not required
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Make error more descriptive in case of no encryption state
* Fix failing test
* Change error message on strict-encrypt postgres destination as well. Fix failing test
* update versions and changelogs
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* allow nullable columns to be used as cursor
* add test
* fix test
* bump version
* Revert "bump version"
This reverts commit a3cf9aff84.
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Use LOG4J2 to wrap connectors logs to JSON format
* log connector's stack trace directly as a message
* add stack_trace field to json template
* bump versions
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* revert versions for destinations: postgres, mssql, mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Cleaned error messages factory PR
* Bumped MySQL and Postgres version
* Fixed messages and typos in test
* Fixes the changelog conflict with per-stream state
* Added note for flaky test
* Bumps mysql version to match changelog
* Added exception objects to all LOGGER.error for more visibility
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* enforce ssl in strict mysql and postgres source
* Add test to verify tunnel aware strict SSL behavior
* bump version numbers and changelogs
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* new ssl flow for connectors
* Bump versions and update changelogs
* Address review comments
* sanity
* Revert Dockerfile versions
* Fix error in test
* The test certificate we use do not contain a valid hostname so we can only verify ca trust but not hostname match
* verify-ca and verify-full SSL modes are not related to whether a client certificate is used. Make client certificate optional on both modes
* Bump Dockerfile versions and correct change notes.
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix temporal datatype bug for columns with default in postgres cdc
* fix test
* add test for date and time as well
* add more logs for unknown classes
* review comments
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor
* added tests for cursor fields incremental updates
* remove not used methods
* fix tests
* fix formatting
* revert RedshiftInsertDestinationAcceptanceTest
* fix ssl clickhouse test
* fix codereview comments
* fix clickhouse strict encrypt test
* fix postgres source after merge from master
* optimize ssl clickhouse test logic
* fix db2 test
* added check for cursor field should be not nullable, fix formatting
* fixed tests, formatting, issue after merge from master
* fix merge issues
* fix test for source-jdbc and remove some code used for local testing
* added description for cockroach db tests
* test changes
* refactoring for cursor field tests after code review comments
* refactoring for cursor field tests after code review comments
* fix code review comments
* remove unused imports
* fix formatting
* fix compilation error
* fix postgres and postgress ssl test
* Fixed bucket naming for S3
* remove CHAR, NCHAR from possible cursor types
* remove SYMBOL from possible cursor types
* removed redundant configs
* fixed mysql-strict-encrypt tests
* fixed mongodb-v2 tests
* minor format changes
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Allow additional properties in db stream state
* Bump postgres version
* Bump mysql version
* Bump mssql version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Rename record counter
* Rename method
* Emit intermediate state after all cursor records
* Emit intermediate state only when it is ready
* Merge two checks
* Add a testing message
* Fix unit tests
* Add one more testing record and add comments
* Add test case for multiple records with the same cursor value
* Revert irrelevant change
* Add explanation in javadoc
* Format code
* Rename testing methods
* Fix comment
* Bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Label source-postgres to GA
* Correct the PR number
* 📄 Postgres source: fix links in the spec (#15382)
* Add internal anchors back
* Remove internal anchor
* Use links for subtitles
* Remove one more anchor
* auto-bump connector version [ci skip]
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Extract connectionTimeout from jdbc_url_params along with corresponding tests
* Fixed linter issues
* Reverted createDataSourceWithConnectionTimeout and migrated logic to get operation
* Fixed dangling createDataSourceWithConnectionTimeout and linter issues
* Fixed import to use java standard library
* Bump Postgres Source and Postgres Source Strict Encrypt versions
* Fixed import ordering issues
* Bumped the connector version [CI fix] for definitions not generated
* Fix first record wait time parsing bug
* Bump version
* Add connection check for first record waiting time
* Update spec
* Override initial waiting time when it is too short
* Set is_test to true for cdc integration tests
* Fix integration test
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* work in progress
* cleanup
* add test
* introduce tests for state parsing util class
* enable test via feature flag
* review comments
* Bump versions
* auto-bump connector version [ci skip]
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* bump version
* 🐛 Source Postgres: Improve BCE date handling (#15187)
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* borked merge - re-delete deleted methods
* enable catalog tests for postgres
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* update BCE handling in JDBC
* reuse existing method
* handle bce dates
* inline methods
* fix JDBC BCE year inconsistency
* use correct data type in test
* format
* Update airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractPostgresSourceDatatypeTest.java
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* pmd fix
* use class.getname()
* fix pmd
* format
* bump version
* handle incremental mode
* clean up diff
* more comments
* unused imports
* format
* versions+changelog
Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com>
Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
* auto-bump connector version [ci skip]
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* generate airbyte_type:integer
* normalization accepts `airbyte_type: integer`
* handles ints+longs
* update avro for consistency
* delete long type for now, treat all ints as longs
* update avro type mappings
{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")
* fix test
* remove long handling
* Revert "remove long handling"
This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.
* Revert "update avro type mappings"
This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.
* Revert "delete long type for now, treat all ints as longs"
This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.
* Revert "update avro for consistency"
This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.
* newline@eof
* update test
* slightly better local tests
* fix test
* missed a few cases
* postgres tests use correct hostnames
* fix normalization
* fix int macro
* add test case
* normalization test output
* handle int/long correctly
* fix types for other DBs
* uint32 -> bigint; tests
* add type value assertions
* more test updates
* regenerate output
* reconcile big_integer to match docs
* update comment
* fix type
* fix mysql constructor call
* bigint only has 38 digits
* fix s3 ints, fix DAT test case
* big_integer should be string
* reduce to 28 digit big_ints
* fix test setup, mysql
* kill big_integer tests
* regenerate output
* version bumps
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add initial waiting time param
* Pass in first record wait time from config
* Set initial waiting time to 10s in tests
* Log first record waiting time
* Fix expected_spec.json
* Reduce initial waiting seconds in test more
* Add tests
* Complete tests
* Refactor code
* Bump version
* Fix pr id in changelog
* Set shorter initial waiting time in one more test
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* added ssl certificates for postgres source
* added command for remove client private key after transformation to encrypted key with .pk8 extension
* added connection with CA and client certificates for postgres destination
* updated code style
* moved common methods to the common class
* moved common methods to the common class
* fixed remarks
* updated postgres source tests
* added minor changes to spec and added fixes to password mechanism
* updated postgres source tests
* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors
* fixed check style
* updated documentation and versions of connectors
* updated ordrs in test spec
* fixed minor remarks in specs and expected_specs
* fixed minor remarks in specs and expected_specs
* fixed Dockerfile
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* rollback changes for postgres destination and move them to a new pull request
* rollback changes for postgres destination and move them to a new pull request
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* auto-bump connector version
* updated version of postgres strict-encrypt source
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Moved toDatabaseConfigStatic subfunction into the main toDatabaseConfig function
* Moved Postgres JDBC_URL_PARAMS_KEY into JdbcUtils to reduce redundancy
* Migrated useSsl method into JdbcUtils to remove duplication
* Created constants for config parameters to match PostgresDestination
* Add validation check for overwritten connection properties
* Added test coverage for useSsl and clarifying javadoc comments
* Updated useSsl tests to match linter and javadoc comments for useSsl
* Fixed isCDC method with proper PostgresUtil function and corresponding imports
* Bumping Dockerfile and adding changelog description
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Force legacy if feature flag not set
* Fix acceptance test
* bump version
* PR comments
* Revert Integration test
* Update some integration test
* asset legacy type and data is correct in source tests
* better diff
* disable test cause its not valid for LEGACY state (#14703)
* auto-bump connector version
Co-authored-by: lmossman <lake@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* use more simple and comprehensive query to get selectable tables
* cover case when schema is not specified
* add test to check discover with different ways of grants
* format
* incr ver
* incr ver
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Improve documentation for Postgres Source
* add information about additional JDBC params
* add anchors for doc sections
* fix link to CDC on Bare Metal
* add more details about parsing date/time values
* add doc link to SSH fields
