1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

224 Commits

Author SHA1 Message Date
Stephane Geneix
e01c0a3780 re-release source mssql with logger fixes (#35596) 2024-02-23 13:19:43 -06:00
Stephane Geneix
184d10967c push new source-mssql version (#35564) 2024-02-22 16:37:27 -06:00
Rodi Reich Zilberman
c9ba9df789 🐛 Change null cursor value query to not use IIF sql function (#35405) 2024-02-21 11:44:55 -06:00
Rodi Reich Zilberman
c8d5927401 Checkpointing source-mssql (#34182)
Co-authored-by: nguyenaiden <duy@airbyte.io>
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
2024-02-08 17:20:38 -06:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Ash-v-G
f70a6a5e41 Docs: Update mssql.md (#34432)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-01-23 10:57:26 -03:00
Scott Sinclair
a0623c54b4 fix link in MSSQL source documentation to pr (#33961)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2024-01-15 17:40:19 -03:00
Akash Kulkarni
804a34b212 [Source-mssql] : Remove options for data_to_sync & snapshot_isolation (#33700) 2024-01-10 14:06:38 -08:00
Marius Posta
c284588d99 source-mssql: shorten capture job polling interval in tests (#33510) 2024-01-08 13:05:28 -08:00
Stephane Geneix
15490f6cd9 add SSL options to mssql (#33071)
Co-authored-by: Marius Posta <marius@airbyte.io>
2024-01-04 11:50:03 -08:00
Xiaohan Song
609bcb2799 Use base64 to encode mssql binary field (#33755) 2024-01-03 16:32:14 -08:00
Subodh Kant Chaturvedi
7c9394f8da mssql-source: remove LEGACY state flag (#33481) 2023-12-19 23:46:19 +05:30
Stephane Geneix
7054db7ad8 bump airbyte-cdk version (#33505) 2023-12-14 16:28:07 -04:00
Stephane Geneix
6e375f4e60 pull MSSql debezium classes out of the CDK (#33270)
This is just relocating classes from inside the CDK to the individual container. Bumping the CDK version, using the new CDK in the source-mssql and bumping the source-mssql connector version
2023-12-13 15:10:28 -08:00
Duy Nguyen
ce7554021f [Source-MSSQL] Enable per-stream and Global states and migrate away from Legacy states (#33018) 2023-12-12 17:53:29 -08:00
Duy Nguyen
44ce0022d2 [Source-mssql] Datetimeoffset field used as cursor error patch (#33330) 2023-12-12 12:52:26 -08:00
Marius Posta
3026cc59a8 source-mssql: enable state compression (#33225)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-12-12 11:08:31 -08:00
Marius Posta
b17f73be14 source-mssql: enforce SSL on Airbyte Cloud (#32882)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-12-01 10:18:34 -08:00
Duy Nguyen
5acca55f3c [Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision (#32573)
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
2023-11-30 12:17:39 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-27 12:49:06 -06:00
Tim Roes
c5db1e947b 📚 Documentation November 2023 overhaul (#32811)
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: timroes <timroes@users.noreply.github.com>
Co-authored-by: nataliekwong <nataliekwong@users.noreply.github.com>
2023-11-26 11:57:38 -08:00
Duy Nguyen
1e59751bc4 [Source-Mssql] Assess and map all SQLServer supported datatypes to the correct JDBC and Airbyte types for Standard and CDC mode (#31531)
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
2023-11-06 16:30:08 -08:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Rodi Reich Zilberman
1e8e6402bc Revert "Automated Commit - Formatting Changes" (#32077) 2023-11-01 15:38:25 -05:00
Rodi Reich Zilberman
f5c87968ca URL encode username and password fields (#32028)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-11-01 15:22:27 -05:00
Akash Kulkarni
85d266ac20 Bump DBZ version from 2.2.0 -> 2.4.0 for mssql, mysql (#32024) 2023-11-01 13:55:13 -04:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-08-31 12:19:02 -05:00
Duy Nguyen
f68a828d1e [Source-MsSQL] Set default cursor for cdc mode (#29493) 2023-08-23 14:01:49 -05:00
sh4sh
6df99b24af 🐛 source-mssql: support Read Committed snapshot isolation level (#28545)
* make getSnapshotIsolationConfig grab the correct config

* add object case for getDataToSyncConfig and getSnapshotIsolationConfig

* update tests to use current config parameters

* bump connector version

* add changelog
2023-08-01 16:37:45 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
Akash Kulkarni
d30906d89d Remove noisy logging for LegacyStateManager (#27335)
* Remove noisy logging for legacy state manager

Context : https://airbytehq-team.slack.com/archives/C043JHEEYKG/p1686654588724759

* Bump version + dockerfile
2023-06-16 17:13:33 -05:00
Nipuna Prashan
d95b497dd5 Update mssql.md (#27000)
Fix pull request ID.

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2023-06-05 15:57:04 -03:00
Subodh Kant Chaturvedi
9b9809b006 fix(cdc): limit queue size to lower memory consumption (#26473)
* fix(cdc): limit queue size to lower memory consumption

* add queue size attribute in spec

* disable retries

* fix log

* review comments

* add validation test for queue size

* update expected spec

* bump version + changelog

* update metadata files

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-05-25 22:34:40 +05:30
Akash Kulkarni
931911dceb 🐛 Source MySQL/MsSQL Disable index logging for MySQL (#25740)
* Disable index logging for MySQL/MsSQL

* Update mysql.md

* Update mysql.md

* auto-bump connector version

* Disable for MsSQL as well

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-05-03 08:44:21 -07:00
Subodh Kant Chaturvedi
94ed6554ea upgrade debezium version to 2.2.0.Final (#25401)
* upgrade debezium version to 2.2.0.Final

* bump version + changelog

* auto-bump connector version

* manual bump version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-27 22:12:20 +05:30
Akash Kulkarni
395de77a50 [JDBC sources] : Log database indexes per stream (#25345)
* [JDBC sources] : Log database indexes per stream

* Docker & doc bump

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-20 01:05:15 +00:00
Sergio Ropero
819bba174f postgres-cdc checkpointing: fix LSN parsing bug + refactor for efficiency (#24582)
* Fix LSN parsing from Integer to Long

* rebasing

* Rebase

* Rebase

* Other casting

* Lock the file only when reading, so the file is free when parsing the object.
Increased from 1 to 166 checkpoints, and from skipping hundreds of checkpoints to never skip a state.

* Update load function documentation

* bump mysql and mssql

* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* resolve master merge conflicts

* Automated Change

* minor changes

* resolve merge conflicts

* avoid deserializing multiple times

* simplify

* enable checkpointing for Postgres

* more improvements

* enable assertions

* changelog + bump version

* auto-bump connector version

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-19 21:05:50 +05:30
Akash Kulkarni
f23b3ad88e Add logging functionality to understand Xmin wraparound (#25220)
* Add logging functionality

- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values

* Automated Change

* Renaming + comments

* More comments

* Bump docker version + docs

* auto-bump connector version

* Update failed source_specs.yaml

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-17 22:39:04 +00:00
Subodh Kant Chaturvedi
b5a2ceb7fe cdc: refactor to remove debezium dependency from connector packages (#24656)
* cdc: refactor to remove debezium dependency from connector packages

* use gradle's shared dependency

* more refactoring

* upgrade docker version

* Automated Change

* auto-bump connector version

* manual bump

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-11 21:32:17 +05:30
Akash Kulkarni
c0341a3271 Fix emitted intermediate state for initial incremental non-CDC syncs (#24820)
* Initial fix for issue https://github.com/airbytehq/airbyte/issues/24808

* Revert pg

* Adding a more robust unit test for PostgresSource

* Log the query

* Bump dockerfiles + docs

* Bump docs

* auto-bump connector version

* Revert alloydb changes

* Manually update source specs for ones that failed

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-06 22:49:32 +00:00
Subodh Kant Chaturvedi
8748a8d6c7 fix-cdc: errors retry debezium property should be less than max retry (#24833)
* fix-cdc: errors retry debezium property should be less than max retry

* add comment

* version bump + changelog

* auto-bump connector version

* Update source-alloydb versions to match source-postgres

* rebump to 2.0.15

* auto-bump connector version

* definitions + regenerate manually

---------

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: nguyenaiden <duy@airbyte.io>
2023-04-04 19:23:00 -07:00
Subodh Kant Chaturvedi
29a7e2ebf5 cdc: fix debezium shutdown interruption bug (#24166)
* cdc: fix debezium shutdown interruption bug

* update retry properties

* add test for queue transfer

* refactor to simplify

* rename parameter

* fix test

* add comments

* add check for engine already closed

* disable checkpointing

* bump version

* auto-bump connector version

* manual bump

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-28 19:47:40 +05:30
Sergio Ropero
80da900d70 feat: Enable CDC checkpointing on Postgres (#24373)
* Fix the error reading offset file meanwhile Debezium is writing.
Enable CDC checkpointing to Postgres.
Minor change in the variable name to fit the type.

* Add final statement on exception ;)

* Add comments to CDC Checkpoint tests.
Clean a bit.

* Bump connector versioning

* Add log message

* Fix changelog

* auto-bump connector version

* Manually generate definitions

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-27 14:00:49 +02:00
VitaliiMaltsev
12eac421de JDBC Sources: remove redundant date-time datatypes formatting (#20760)
* JDBC sources: consolidate date/time values mapping

* fixed Snowflake acceptance tests

* fixed TestJdbcUtils

* refactoring

* update setDate for incremental sync

* fixed CockroachDbJdbcSourceAcceptanceTest

* fixed CockroachDbSourceDatatypeTest

* fixed CockroachDbSourceDatatypeTest

* fixed MsSQLSourceDatatypeTest

* fixed MsSQL cdc datatype test

* refactoring

* fixed mssql cdc test

* fixed Redshift tests

* fixed Redshift tests

* updated db2 datatype tests

* fixed DB2 tests

* fixed DB2 strict-encrypt tests

* fixed Oracle strict-encrypt tests

* refactoring

* updated changelog

* refactoring

* removed redundant overrides

* bump versions

* auto-bump connector version

* JDBC Sources: remove redundant date parsing

* updated Snowflake test

* updated cockroachdb test

* updated date-time datatypes parsing in Postgres Source

* updated formatting in JDBC sources

* updated test cases

* updated Snowflake test

* updated Snowflake, Redshift, Cockroach, DB2 tests

* updated Snowflake, Postgres tests

* updated Redshift tests

* updated MySQL, MsSQL, DB2 tests

* updated MsSQL tests

* updated Oracle tests

* updated Oracle tests

* updated Oracle tests

* fixed some tests

* refactoring

* fixed checkstyle

* fixed TestJdbcUtils

* randomize Reshift test username

* removed redundant test

* updated sanity test

* Automated Change

* format time and bump versions

* auto-bump connector version

* bump versions

* auto-bump connector version

* auto-bump connector version

* manually update definitions

* Automated Change

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-23 14:31:50 +00:00
Duy Nguyen
c1323f615d Verify catalog schema fieldNames is a subset of DB schema (#24207)
* Update current schema vs catalog validation to omit airbyte metadata fields
* Bumped versions for postgres, mssql, and mySQL + changelog
* Bumped dockerfile version of strict encrypt
* Manually generate definitions for mysql
* Manually generate definitions for mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-22 16:41:15 +00:00
Sergio Ropero
ca2d9c733f fix: Disable CDC checkpointing using a flag (#24275)
* Disable CDC checkpointing until finished the investigation with reading Offset file during syncs

* Bump connector versions

* auto-bump connector version

* Publish postgres manually

* Definitions manually for mysql

* Generated definitions manually mssql

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-03-21 15:32:20 +01:00
VitaliiMaltsev
9e94126fd6 MySQL/MsSQL sources fixed NPE during cursor values validation (#24082)
* MySQL/MsSQL sources  fixed NPE during cursor values validation

* added logging

* bump versions

* auto-bump connector version

* manually update mssql definitions

* Automated Change

* Automated Commit - Formatting Changes

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-15 19:27:31 +00:00
VitaliiMaltsev
ace229f971 MySQL/MsSQL Source: stop sync on null cursor value (#23908)
* MySQL Source: stop sync on null cursor value

* updated MsSQL test

* Fixed testTableWithNullCursorValueShouldThrowException test for Mssql

* fixed MsSQL and MySQL tests

* modified MsSQL query

* fixed typo

* fixed typo

* simplified Mysql query logic

* simplified Mssql query logic

* add quotes to Mssql query

* add mysql view test

* MySQL and MsSQL source throw a warning if a cursor column contains null value

* disabled tests; log warn if null cursor values found

* removed logging in AbstractDbSource

* bump version

* bump version

* auto-bump connector version

* updated changelog

* Automated Change

* updated definitions

* Automated Change

---------

Co-authored-by: ievgeniit <etsybaev@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: VitaliiMaltsev <VitaliiMaltsev@users.noreply.github.com>
2023-03-14 22:34:58 +02:00