jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

362 Commits

Author SHA1 Message Date
Stephane Geneix
8ee36d690c add background threads to MssqlTestDatabase (#35727) 2024-03-06 12:20:25 -08:00
Rodi Reich Zilberman
9ce9217660 make query safer for servers that are case sensitive (#35816) 2024-03-06 13:47:50 -04:00
Stephane Geneix
cdf80f3bfa get source-mssql CI to green (#35721) 
TheAbstractMssqlSourceDatatypeTest was wrong for reals. 
CdcMssqlSourceTest and CdcMssqlSslSourceTest no longer use exclusive containers.
tests that change the SQLServer agent state are now in their own test class (which needs to use an exclusive container)

Besides that, the core of the problem can be grouped in 2 issues:
a) some tests are failing to enable CDC for tables. This is due to a timing issue. We moved that logic into its own function that will try for a total of 240seconds before giving up.
b) some tests are failing when trying to read the minLsn. There is a 1sec wait implemented in the production code. Instead we introduce a busy loop that will wait for a total of 240seconds for records to appear in the CDC table before giving up. That function is implemented in test code.

Unfortunately, for both cases, we sometimes needed to wait while in the middle of a function implemented in the CDK. We introduced a few hooks in the parent PR that are implemented in this PR for the source-mssql tests, and use the functions described above
 2024-03-06 09:01:37 -08:00
Rodi Reich Zilberman
f9e73cf62a add configuration for cdc queue size (#35739) 2024-03-05 18:45:09 -06:00
Akash Kulkarni
30190eae54 [Source-mssql] : Add config to throw an error on invalid CDC position (#35566) 2024-02-28 12:48:20 -06:00
Stephane Geneix
e01c0a3780 re-release source mssql with logger fixes (#35596) 2024-02-23 13:19:43 -06:00
Stephane Geneix
184d10967c push new source-mssql version (#35564) 2024-02-22 16:37:27 -06:00
David Wallace
3285fd2791 ✨ [source-mssql] skip sql server agent check if EngineEdition == 8 (#35368) 2024-02-22 15:57:41 -06:00
Rodi Reich Zilberman
c9ba9df789 🐛 Change null cursor value query to not use IIF sql function (#35405) 2024-02-21 11:44:55 -06:00
Marius Posta
b9ab3d1a90 migrate everything from java 17 to java 21 (#35103) 2024-02-09 19:26:19 -06:00
Marius Posta
d80a9fca8f source-mssql: adopt cleaned-up cdk (#34749) 2024-02-09 14:02:49 -06:00
Rodi Reich Zilberman
c8d5927401 Checkpointing source-mssql (#34182) 
Co-authored-by: nguyenaiden <duy@airbyte.io>
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
 2024-02-08 17:20:38 -06:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Akash Kulkarni
804a34b212 [Source-mssql] : Remove options for data_to_sync & snapshot_isolation (#33700) 2024-01-10 14:06:38 -08:00
Marius Posta
c284588d99 source-mssql: shorten capture job polling interval in tests (#33510) 2024-01-08 13:05:28 -08:00
Stephane Geneix
15490f6cd9 add SSL options to mssql (#33071) 
Co-authored-by: Marius Posta <marius@airbyte.io>
 2024-01-04 11:50:03 -08:00
Stephane Geneix
cb22376037 don't use MsSQLConfigBuilder.withSsl with strings, but instead properly named and typed methods (#33261) 2024-01-04 09:52:16 -08:00
Xiaohan Song
609bcb2799 ✨Use base64 to encode mssql binary field (#33755) 2024-01-03 16:32:14 -08:00
Subodh Kant Chaturvedi
7c9394f8da mssql-source: remove LEGACY state flag (#33481) 2023-12-19 23:46:19 +05:30
Stephane Geneix
7054db7ad8 bump airbyte-cdk version (#33505) 2023-12-14 16:28:07 -04:00
Stephane Geneix
6e375f4e60 pull MSSql debezium classes out of the CDK (#33270) 
This is just relocating classes from inside the CDK to the individual container. Bumping the CDK version, using the new CDK in the source-mssql and bumping the source-mssql connector version
 2023-12-13 15:10:28 -08:00
Duy Nguyen
ce7554021f [Source-MSSQL] Enable per-stream and Global states and migrate away from Legacy states (#33018) 2023-12-12 17:53:29 -08:00
Stephane Geneix
26f1dc3f7a limit memory for MSSQL containers (#33358) 
on my M2 laptop, sometimes tests fail due to timeouts or some random process getting killed (either a container, or the gradle cache inside dagger). After investigation, I realized that we were running up to 6 different containers with MSSQL db, with each taking upwards of 2GB of RAM (real, not virtual). So I'm now limiting the individual MSSQL instances to 384MB. That helps with swapping too
 2023-12-12 15:45:07 -08:00
Duy Nguyen
44ce0022d2 [Source-mssql] Datetimeoffset field used as cursor error patch (#33330) 2023-12-12 12:52:26 -08:00
Stephane Geneix
4c726c9c4e use latest image for mssql (#33357) 
In a [previous change](dccd48a896), we simplified our MSSQL images to only use latest, but I missed some places where we're still using strings. Here, we're bringing everything under latest. That should have the nice side effect to make CI a bit master, and less resource intensive
 2023-12-12 11:54:43 -08:00
Marius Posta
3026cc59a8 source-mssql: enable state compression (#33225) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-12-12 11:08:31 -08:00
Duy Nguyen
8563f5c135 [Source-mssql] Remove reference to strict encrypt in metadata (#33316) 2023-12-11 12:56:44 -08:00
Stephane Geneix
dccd48a896 simplify MSSQL images (#32951) 
remove MSSQL_2022_UBUNTU_2024. Rename into MSSQL_VERSION, get everything on latest
 2023-12-07 19:16:28 -08:00
Stephane Geneix
f6d03ec860 introduce ImageModifier to MS SQL source (#32898) 
introducing ImageModifier to MsSql so we don't use strings everywhere
 2023-12-07 15:26:52 -08:00
Stephane Geneix
d32bbc116e introduce BaseImage for MsSql (#32897) 
introducing the BaseImage paradigm to MsSql so we don't use strings everywhere
 2023-12-07 15:07:51 -08:00
Marius Posta
b17f73be14 source-mssql: enforce SSL on Airbyte Cloud (#32882) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-12-01 10:18:34 -08:00
Duy Nguyen
5acca55f3c [Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision (#32573) 
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
 2023-11-30 12:17:39 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-11-27 12:49:06 -06:00
Duy Nguyen
1e59751bc4 [Source-Mssql] Assess and map all SQLServer supported datatypes to the correct JDBC and Airbyte types for Standard and CDC mode (#31531) 
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
 2023-11-06 16:30:08 -08:00
Augustin
368ba78b64 🧹doc: update connectors README and remove acceptance-test-docker.sh (#32209) 2023-11-06 12:36:07 -06:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Marius Posta
d2e313985f source-mssql*: use CDK 0.4.0 (#32175) 2023-11-03 14:29:43 -07:00
Stephane Geneix
9c6ed93a9d Revert "update all connectors for new CDK version and more stringent javac checks" 
This reverts commit 18ad87bbd1.
 2023-11-02 15:42:28 -07:00
Stephane Geneix
18ad87bbd1 update all connectors for new CDK version and more stringent javac checks 2023-11-02 15:24:58 -07:00
Rodi Reich Zilberman
1e8e6402bc Revert "Automated Commit - Formatting Changes" (#32077) 2023-11-01 15:38:25 -05:00
Rodi Reich Zilberman
f5c87968ca ✨URL encode username and password fields (#32028) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
 2023-11-01 15:22:27 -05:00
Akash Kulkarni
85d266ac20 Bump DBZ version from 2.2.0 -> 2.4.0 for mssql, mysql (#32024) 2023-11-01 13:55:13 -04:00
Duy Nguyen
34b09e0490 [Source-mssql] Update source-mssql-strict-encrypt tests to work with airbyte-ci (#32025) 
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
 2023-10-31 18:07:23 -05:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Augustin
ac6f44cf2c remove Dockerfile and .dockerignore for Java connectors (#31769) 2023-10-24 15:08:13 -05:00
Marius Posta
085c497d01 source-mssql*: fix integration tests (#31504) 2023-10-17 13:05:03 -05:00
Akash Kulkarni
4787b7124f Pin cdk back to 0.1.0 for postgres, mysql, mssql (#31085) 2023-10-04 18:11:05 -05:00
Jonathan Pearlin
60b0c9d023 Set the user timezone for tests (#31057) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-04 16:29:41 -05:00
Marius Posta
f8edc18039 airbyte-ci,gradle: replace airbyte-docker with airbyte-ci (#30743) 2023-10-04 08:38:17 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383) 
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
 2023-09-27 15:46:02 -07:00