* Separated Standard vs Xmin categorization
Added StandardCtidUtilsTest
Added versions to StandardStatus
* WIP incremental read test
* Added PostgresStandardStateManager to write state_type key to stream state
Added sync to verify state messages and cursor records on running multiple syncs of the same streams
* Isolate tables used for ctid test and enable checkpointing records to 1 for just CTID test
* Moved ctid cursor test to its own file CtidEnabledPostgresSourceTest
* Gracefully handle case where there is no statetype
Update all "standard" to "cursorBased"
* wip
* wip 2
* undo unwatned change
* some refactoring
* fix few tests
* more fixing
* more fixes
* Automated Commit - Format and Process Resources Changes
* more improvements
* 1 more test
* another test
* add flag for ctid enabling
* fix conflicts
* else block is not required
* use emittedAt
* skip WAL processing if streams under vacuuming
---------
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
* initial commit
* cleanup
* cleanup
* Throw a configuration exception in case of bad jdbc url param
* End-to-End test session variable jdbc param
* reafctoring sanity
* sanity
* bump version and update note
* cherry pick fix to unrelated build error
* fix failing test
---------
Co-authored-by: Ryan Fu <ryan.fu@airbyte.io>
* Convert source mysql decimal number with 0 decimal point to an int
* Put int representation in json for decimal with 0 decimal digits
* bump docker image ver and release note
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Disable index logging for MySQL/MsSQL
* Update mysql.md
* Update mysql.md
* auto-bump connector version
* Disable for MsSQL as well
* auto-bump connector version
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808
* Revert pg
* Adding a more robust unit test for PostgresSource
* Log the query
* Bump dockerfiles + docs
* Bump docs
* auto-bump connector version
* Revert alloydb changes
* Manually update source specs for ones that failed
---------
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* Cleanup createDatabase() code
Move methods from AbstractJdbcSource
Moved methods until createDataSource outside of the abstract class
* Quick cleanups/renames
* Add tests
Scripts to
* Run CATs against the local CDK for one connector
* Run CATs against the local CDK for multiple connectors
* Create a connecter image with the local CDK
---------
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
* add airbyte-protocol to deps.toml
* use published protocol jar for platform
* use published protocol jar for connectors
* point at published jar
* fix dep
* bump gcs storage
* fix build failures in standard-source-test
* fix deps
* downgrade alloy db because it is missing strictness tests
* Revert "downgrade alloy db because it is missing strictness tests"
This reverts commit cc6089d053.
---------
Co-authored-by: cgardens <charles@airbyte.io>
* Stop sync on a null value in a cursor column
* Fix quoting in query to solve a failing test
* Fix another escaping issue in query
* Fix failing test
* Fix failing tests
* Test view with null value cursor
* Improve error message
* bump dockerfile version and update note
* bump version to 1.0.31
* Fix failing test
* auto-bump connector version
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
#19191 made me realise the DefaultReplicationWorker's metric tracking today has a bug where we aren't accounting for namespace when tracking metrics today. i.e. Streams with the same name and duplicate namespace will merge metrics.
While reading the code to figure out a fix, I realised we don't have a good conceptual representation of stream namespace <> name pairs within the platform today. We use a concatenated string. Though this works, it will become harder and harder to read/track as we do more operations that involve namespace i.e. progress bars and column selection.
This PR introduces the AirbyteStreamNameNamespacePair object into the platform code to make it more convenient to work with Streams in the future. (Especially if we proceed with the project to make streams a first-class citizen!)
The AirbyteStreamNameNamespacePair object was written to deal with the same issue of namespace <> name pair manipulation within the Java destination code. It implements the Comparable interface, which makes it convenient to use for Collections operations.
For an example of how this is consumed, see #19361.
* Refactor attempt 1
Increasing readability:
- Performed audit of methods that are marked as public/protected but should be private
- Performed audit of methods that are marked as public but should be protected
- Removing @Override from methods that don't need them (are the very first)
* Remove .sqlite files
* Remove AbstractRelationDbSource class
- Removing AbstractRelationalDbSource abstract class and moving contents of it to a Util class
* Rename utility class
* Add overrides back
* Add cursor_record_count to db stream state
* Add cursor record count to cursor info
* Emit max cursor record count
* Add original cursor record count
* Unify logging format
* Add backward compatible methods
* Update unit tests for state decorating iterator
* Update test (not done yet)
* Fix one more unit test
* Change where clause operator according to record count
* Add branch for null cursor
* Skip saving record count when it is 0
* Fix log wording
* Set mock record count in test
* Check cursor value instead of cursor info
* Fix source jdbc test
* Read record count from state
* Fix tests
* Add an acceptance test case
* Fix npe
* Change record count from int to long to avoid type conversion
* Fix references
* Fix oracle container
* Use uppercase for snowflake
* Use uppercase for db2
* Fix and use uppercase
* Update test case to include the edge case
* Format code
* Remove extra assertion in clickhouse
* Merge ms sql incremental query method
* Log query for debugging
* Clean up name_and_timestamp table
* Fix db2 tests
* Fix mssql tests
* Fix oracle tests
* Fix oracle tests
* Fix cockroachdb tests
* Fix snowflake tests
* Add changelog
* Fix mssql tests
* Fix db2-strict-encrypt tests
* Fix oracle-strict-encrypt tests
* Bump postgres version
* Fix oracle-strict-encrypt tests
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* mysql source : implement support for snapshot of new tables in cdc mode
* undo unwanted changes
* add more assertions
* format
* fix build
* fix build
* revert acceptance test changes
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
* allow nullable columns to be used as cursor
* add test
* fix test
* bump version
* Revert "bump version"
This reverts commit a3cf9aff84.
* bump version
* auto-bump connector version [ci skip]
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
