* fix-cdc: errors retry debezium property should be less than max retry
* add comment
* version bump + changelog
* auto-bump connector version
* Update source-alloydb versions to match source-postgres
* rebump to 2.0.15
* auto-bump connector version
* definitions + regenerate manually
---------
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
* test docker behavior on CI env
* Automated Change
* test docker behavior on CI env
* Make all unit and integration tests in source-postgres pass locally
* Fix mysql ssh integration test
* Fix failing test
* Fix source-mssql build
* source-mssql runss tests locally.
Fix compilation errors
---------
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.
* Add final statement on exception ;)
* Add comments to CDC Checkpoint tests.
Clean a bit.
* Bump connector versioning
* Add log message
* Fix changelog
* auto-bump connector version
* Manually generate definitions
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* set ssl_mode to require on remaining db sources
* update readmes
* update expected specs
* auto-bump connector version
* bump mysql version manually
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* make ssl mode default to require for postgres on cloud
* update dockerfiles and readme
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add pattern descriptors to more fields in github and postgres sources
* update dockerfiles and readmes
* auto-bump connector version
* manually bump github seed source files
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Cleanup createDatabase() code
Move methods from AbstractJdbcSource
Moved methods until createDataSource outside of the abstract class
* Quick cleanups/renames
* Add tests
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
* fix all postgres acceptance tests
* add newline
* undo other changes to postgres readme file
* add security group to tunnel_method in expected_spec.json
* bump version of strict encrypt
* manually bump versions in seed files
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Update current schema vs catalog validation to omit airbyte metadata fields
* Bumped versions for postgres, mssql, and mySQL + changelog
* Bumped dockerfile version of strict encrypt
* Manually generate definitions for mysql
* Manually generate definitions for mssql
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-postgres-cdc-npe:do not put null in properties
* version bump + change log
* auto-bump connector version
* manual bump
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add grouping and collapsing fields to postgres source
* add auth group to github source connector
* revert postgres field order changes and adjust group of schemas field
* inject group into ssh tunnel spec for postgres only, through overloaded methods
* Automated Change
* bump Dockerfile versions and update changelogs
* bump strict encrypt version as well
* fix postgres acceptance test
* fix acceptance test again
---------
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
* Removed extraneous read on the check() method for CockroachDB and Postgres
* Bumped dockerfile version for relevant conenctors
* Bumped versions for strick-encrypt tests
* Removed withoutPublication and withoutReplicationSlot tests for postgres cdc
* Automated Change
* Updated sources docs
* Update incorrect year in change logs doc for Cockroach and Postgres
* Removed read override method from cockroachDB
* Removed read method override for source-postgres
* auto-bump connector version
---------
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* This commit adds new functionality that generates checkpoints when doing CDC synchronization.
For that purpose we encapsulate an AirbyteMessage Iterator on a new iterator that handles the
checkpoint messaging.
* Reformat code
* Reformat code
* Reformat code
* Reformat code
* Second attempt with ugly if statement
* Add `isRecordBehindOffset` function to make sure is safe to send the state.
Tests are failing as now it has more state messages:
expected: <1> but was: <3>
* Code formatting
* Add additional check if the record is part of the snapshot load to skip state message.
* Remove comments
* Fix imports
* Fix format
* Add check if the iterator has extra elements so we don't send state message twice (edge case)
* Add a new check to avoid sending multiple state messages with same offset.
Fix PR comments.
Not sending checkpoints... figuring out
* Modify MSSQL and MySQL implementations
* Adds better control on Maps and include a test for time checkpoint.
Also adds extra assert to verify there are no duplicate states
* Formatting
* Improve code documentation and use default for CdcStateHandler new functions
* Sort out missing `final` and types from comments
* Minor improve in checkpoint validation
* format files
* It's 2023!
* Import issues
* Changes after merging master
* Upgrade Debezium version in MySQL
* Bump Postgres and Alloydb
* auto-bump connector version
* Manually bumping version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Split test for readability and increase waiting time as possible culprit of random failure
* Improve testDataContent() output and test all the types without instead of stopping the test in the first one.
* Format and add documentation
* Postgres Source: fixed bug with non readable double-quoted values within a database name or column name
* bump version
* auto-bump connector version
* bump AlloyDB version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Scripts to
* Run CATs against the local CDK for one connector
* Run CATs against the local CDK for multiple connectors
* Create a connecter image with the local CDK
---------
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* Changes to max fetch row size
- Add frequent logging for fetch size
- Increase MAX_ROW_FETCH_SIZE to 1,000,000,000
* Bump version
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* This commit fixes the issue when permission is granted at ROLE level instead of USER level.
Missing revoke privileges in the tests.
* Change the query to recursively look for all roles asigned.
Also improve testing.
* Add test for subrole with replication access
* formatting
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* Bump versions and update alloydb docs
* Roles don't share attributes, only accesses.
That means that the REPLICATION or SUPERUSER can not be inhered to the user. Because of that, we need to make the user have REPLICATION access directly.
* improve comment
* Typo
* Change from checking the permisions in `pg_users` to execute `createConnection` and verifying the connection is fine for CDC.
* Remove unneeded import
* format
* Rename ReplicationConnection class
* Revert "source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)"
This reverts commit e9efd9878a.
* Revert "Add metadata to connector logs (log level, class name, method name and line number) (#23105)"
This reverts commit a2c80a1fdb.
* Change ConfigError throw point
* Include in try to autoclose the connection
* Bump versions
* auto-bump connector version
* fix SSL failure on check
* format + undo spec changes
* auto-bump connector version
* Manual interaction for source definitions
---------
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
* source-snowflake: use a safer method for parsing a BigInteger cursor value (#22358)
* use a safer method for parsing a BigInteger cursor value
* Add testing
* fix format change
* Fix failing integration tests
* Try removing the failing incremental test
* Try removing the failing incremental test
* Fix failing test
* Add metadata to connector logs (log level, class name, method name and line number) (#23105)
* Issue #17861 Add labels, class, method name and line numbers to connector logs
* Refactored unit test
* fix for warning about UTF8 charset in test class
---------
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
* Update docker image and release notes
* auto-bump connector version
* manually bump version on spec
---------
Co-authored-by: Prateek Mukhedkar <123108018+prateekmukhedkar@users.noreply.github.com>
Co-authored-by: prateekmukhedkar <prateek@airbyte.io>
Co-authored-by: Sergio Ropero <sergio@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Add initial commit.
Missing testing
* Add unit test revoking replication privilege.
* Small fix in the documentation
* Change from `RuntimeException` to `ConfigErrorException`
* Improve test validation
* Add commentodoc
* bump version!
* Bump strict script
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* add airbyte-protocol to deps.toml
* use published protocol jar for platform
* use published protocol jar for connectors
* point at published jar
* fix dep
* bump gcs storage
* fix build failures in standard-source-test
* fix deps
* downgrade alloy db because it is missing strictness tests
* Revert "downgrade alloy db because it is missing strictness tests"
This reverts commit cc6089d053.
---------
Co-authored-by: cgardens <charles@airbyte.io>
* This changes allows to filter out system views created out of system namespaces
* Add extra view
* Fix issue
* Bump Postgres source version
* bump version
* Bump alloydb
* Bump versioning
* auto-bump connector version
* Set default implementation and remove from unneeded places
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* fix-postgres-source: get rid of short lived objects
* cache column info as well
* get rid of constructor
* upgrade version
* auto-bump connector version
* update definition
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Bump total allocation of the source's available memory for the JDBC buffer from 60% to 70%
* Profiling setup
* Update Dockerfile
* Fixes
* Update FetchSizeConstants.java
* Update dockerfiles
* Update JdbcStreamingQueryConfig.java
* auto-bump connector version
* Manually bump source-mysql
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* revert back to erroring out on a null cursor value
* Update dockerfiles and release notes
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
