* fix temporal datatype bug for columns with default in postgres cdc
* fix test
* add test for date and time as well
* add more logs for unknown classes
* review comments
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor
* added tests for cursor fields incremental updates
* remove not used methods
* fix tests
* fix formatting
* revert RedshiftInsertDestinationAcceptanceTest
* fix ssl clickhouse test
* fix codereview comments
* fix clickhouse strict encrypt test
* fix postgres source after merge from master
* optimize ssl clickhouse test logic
* fix db2 test
* added check for cursor field should be not nullable, fix formatting
* fixed tests, formatting, issue after merge from master
* fix merge issues
* fix test for source-jdbc and remove some code used for local testing
* added description for cockroach db tests
* test changes
* refactoring for cursor field tests after code review comments
* refactoring for cursor field tests after code review comments
* fix code review comments
* remove unused imports
* fix formatting
* fix compilation error
* fix postgres and postgress ssl test
* Fixed bucket naming for S3
* remove CHAR, NCHAR from possible cursor types
* remove SYMBOL from possible cursor types
* removed redundant configs
* fixed mysql-strict-encrypt tests
* fixed mongodb-v2 tests
* minor format changes
* bump version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Allow additional properties in db stream state
* Bump postgres version
* Bump mysql version
* Bump mssql version
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Rename record counter
* Rename method
* Emit intermediate state after all cursor records
* Emit intermediate state only when it is ready
* Merge two checks
* Add a testing message
* Fix unit tests
* Add one more testing record and add comments
* Add test case for multiple records with the same cursor value
* Revert irrelevant change
* Add explanation in javadoc
* Format code
* Rename testing methods
* Fix comment
* Bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Label source-postgres to GA
* Correct the PR number
* 📄 Postgres source: fix links in the spec (#15382)
* Add internal anchors back
* Remove internal anchor
* Use links for subtitles
* Remove one more anchor
* auto-bump connector version [ci skip]
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Extract connectionTimeout from jdbc_url_params along with corresponding tests
* Fixed linter issues
* Reverted createDataSourceWithConnectionTimeout and migrated logic to get operation
* Fixed dangling createDataSourceWithConnectionTimeout and linter issues
* Fixed import to use java standard library
* Bump Postgres Source and Postgres Source Strict Encrypt versions
* Fixed import ordering issues
* Bumped the connector version [CI fix] for definitions not generated
* Fix first record wait time parsing bug
* Bump version
* Add connection check for first record waiting time
* Update spec
* Override initial waiting time when it is too short
* Set is_test to true for cdc integration tests
* Fix integration test
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* work in progress
* cleanup
* add test
* introduce tests for state parsing util class
* enable test via feature flag
* review comments
* Bump versions
* auto-bump connector version [ci skip]
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* bump version
* 🐛 Source Postgres: Improve BCE date handling (#15187)
* 13608 & 12026 - align regular and CDC integration tests and data mappers
* format code
* update int handling
* borked merge - re-delete deleted methods
* enable catalog tests for postgres
* fix build
* fix PR remarks
* revert changes for money type that are broken by #7338
* update BCE handling in JDBC
* reuse existing method
* handle bce dates
* inline methods
* fix JDBC BCE year inconsistency
* use correct data type in test
* format
* Update airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractPostgresSourceDatatypeTest.java
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* pmd fix
* use class.getname()
* fix pmd
* format
* bump version
* handle incremental mode
* clean up diff
* more comments
* unused imports
* format
* versions+changelog
Co-authored-by: Yurii Bidiuk <yura.bidyuk@gmail.com>
Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
* auto-bump connector version [ci skip]
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* generate airbyte_type:integer
* normalization accepts `airbyte_type: integer`
* handles ints+longs
* update avro for consistency
* delete long type for now, treat all ints as longs
* update avro type mappings
{type:number, airbyte_type:integer} -> long
{type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer")
* fix test
* remove long handling
* Revert "remove long handling"
This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9.
* Revert "update avro type mappings"
This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84.
* Revert "delete long type for now, treat all ints as longs"
This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947.
* Revert "update avro for consistency"
This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74.
* newline@eof
* update test
* slightly better local tests
* fix test
* missed a few cases
* postgres tests use correct hostnames
* fix normalization
* fix int macro
* add test case
* normalization test output
* handle int/long correctly
* fix types for other DBs
* uint32 -> bigint; tests
* add type value assertions
* more test updates
* regenerate output
* reconcile big_integer to match docs
* update comment
* fix type
* fix mysql constructor call
* bigint only has 38 digits
* fix s3 ints, fix DAT test case
* big_integer should be string
* reduce to 28 digit big_ints
* fix test setup, mysql
* kill big_integer tests
* regenerate output
* version bumps
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add initial waiting time param
* Pass in first record wait time from config
* Set initial waiting time to 10s in tests
* Log first record waiting time
* Fix expected_spec.json
* Reduce initial waiting seconds in test more
* Add tests
* Complete tests
* Refactor code
* Bump version
* Fix pr id in changelog
* Set shorter initial waiting time in one more test
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* added ssl certificates for postgres source
* added command for remove client private key after transformation to encrypted key with .pk8 extension
* added connection with CA and client certificates for postgres destination
* updated code style
* moved common methods to the common class
* moved common methods to the common class
* fixed remarks
* updated postgres source tests
* added minor changes to spec and added fixes to password mechanism
* updated postgres source tests
* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors
* fixed check style
* updated documentation and versions of connectors
* updated ordrs in test spec
* fixed minor remarks in specs and expected_specs
* fixed minor remarks in specs and expected_specs
* fixed Dockerfile
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* fixed remarks
* rollback changes for postgres destination and move them to a new pull request
* rollback changes for postgres destination and move them to a new pull request
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* fixed code style
* auto-bump connector version
* updated version of postgres strict-encrypt source
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Moved toDatabaseConfigStatic subfunction into the main toDatabaseConfig function
* Moved Postgres JDBC_URL_PARAMS_KEY into JdbcUtils to reduce redundancy
* Migrated useSsl method into JdbcUtils to remove duplication
* Created constants for config parameters to match PostgresDestination
* Add validation check for overwritten connection properties
* Added test coverage for useSsl and clarifying javadoc comments
* Updated useSsl tests to match linter and javadoc comments for useSsl
* Fixed isCDC method with proper PostgresUtil function and corresponding imports
* Bumping Dockerfile and adding changelog description
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Force legacy if feature flag not set
* Fix acceptance test
* bump version
* PR comments
* Revert Integration test
* Update some integration test
* asset legacy type and data is correct in source tests
* better diff
* disable test cause its not valid for LEGACY state (#14703)
* auto-bump connector version
Co-authored-by: lmossman <lake@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* use more simple and comprehensive query to get selectable tables
* cover case when schema is not specified
* add test to check discover with different ways of grants
* format
* incr ver
* incr ver
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Improve documentation for Postgres Source
* add information about additional JDBC params
* add anchors for doc sections
* fix link to CDC on Bare Metal
* add more details about parsing date/time values
* add doc link to SSH fields
* Postgres Source: fixed truncated precision if the value of the millisecond or second is 0
* check CI with 1.15.3 testcontainer
* check CI with 1.15.3 testcontainer
* returned latest version of testcontainer
* fixed checkstyle
* fixed checkstyle
* returned latest testcontainer version
* updated CHANGELOG
* bump version
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Switch to measure max row byte size
* Reduce fetch size change logs
* Update unit tests
* Determine jdbc buffer size based on max memory
* Bump postgres version
* Bump postgres version
* Bump mysql version
* Bump mssql version
* Format java code
* Increase hikari connection timeout
* Update data source default parameters
* auto-bump connector version
* Mark postgres 0.4.21 as not published
* Revert "Bump mysql version"
This reverts commit ad9135258c.
* Fix unit test
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* cdc - add support of tsquery type
* rebase on master
* Add changes made by the build
* Update version in Dockerfile
* auto-bump connector version
Co-authored-by: grishick <greg@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Remove host in azure username
* Bump version
* Fix test
* Replace host with jdbc_url
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
