1
0
mirror of synced 2025-12-23 21:03:15 -05:00

🎉 JDBC sources: store cursor record count in db state (#15535)

* Add cursor_record_count to db stream state

* Add cursor record count to cursor info

* Emit max cursor record count

* Add original cursor record count

* Unify logging format

* Add backward compatible methods

* Update unit tests for state decorating iterator

* Update test (not done yet)

* Fix one more unit test

* Change where clause operator according to record count

* Add branch for null cursor

* Skip saving record count when it is 0

* Fix log wording

* Set mock record count in test

* Check cursor value instead of cursor info

* Fix source jdbc test

* Read record count from state

* Fix tests

* Add an acceptance test case

* Fix npe

* Change record count from int to long to avoid type conversion

* Fix references

* Fix oracle container

* Use uppercase for snowflake

* Use uppercase for db2

* Fix and use uppercase

* Update test case to include the edge case

* Format code

* Remove extra assertion in clickhouse

* Merge ms sql incremental query method

* Log query for debugging

* Clean up name_and_timestamp table

* Fix db2 tests

* Fix mssql tests

* Fix oracle tests

* Fix oracle tests

* Fix cockroachdb tests

* Fix snowflake tests

* Add changelog

* Fix mssql tests

* Fix db2-strict-encrypt tests

* Fix oracle-strict-encrypt tests

* Bump postgres version

* Fix oracle-strict-encrypt tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
This commit is contained in:
Liren Tu
2022-10-14 01:09:12 -07:00
committed by GitHub
parent 6a48da9226
commit 888347a0d0
54 changed files with 537 additions and 189 deletions

View File

@@ -341,6 +341,7 @@ WHERE actor_definition_id ='b5ea17b1-f170-46dc-bc31-cc744ca984c1' AND (configura
| Version | Date | Pull Request | Subject |
|:--------|:-----------| :----------------------------------------------------- |:-------------------------------------------------------------------------------------------------------|
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
| 0.4.20 | 2022-09-14 | [15668](https://github.com/airbytehq/airbyte/pull/15668) | Wrap logs in AirbyteLogMessage |
| 0.4.19 | 2022-09-05 | [16002](https://github.com/airbytehq/airbyte/pull/16002) | Added ability to specify schemas for discovery during setting connector up |
| 0.4.18 | 2022-09-03 | [14910](https://github.com/airbytehq/airbyte/pull/14910) | Standardize spec for CDC replication. Replace the `replication_method` enum with a config object with a `method` enum field. |