jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 03:47:05 -05:00
Code Issues Packages Projects Releases Wiki Activity
29,097 Commits 5,116 Branches 518 Tags
c7e17a49cb15dbd1ae7a19de10bb4173127bb30c
Commit Graph

196 Commits

Author SHA1 Message Date
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00
Akash Kulkarni
097709a187 [Source-mysql/mssql] : Remove legacy bad values handling code (#37476) 2024-04-22 20:10:54 -04:00
Akash Kulkarni
bae63044cc [Source-mysql/mssql] : Populate null values (#37111) 2024-04-16 14:25:27 -07:00
Evan Tahler
466abc2968 source-mssql: Allow 60min for MAX_FIRST_RECORD_WAIT_TIME (#37326) 2024-04-15 15:45:51 -07:00
Akash Kulkarni
ca394d20e3 [Source-mssql] : Add meta error handling in initial load path (#37325) 2024-04-15 16:17:26 -04:00
Akash Kulkarni
e75ea0fa70 [Source-mssql] : Remove override for queryTableFullRefresh (#37110) 2024-04-15 09:17:17 -07:00
Rodi Reich Zilberman
1e9ee1dbb8 Prevent conversion of null fields (#36919) 2024-04-11 20:30:38 -04:00
Rodi Reich Zilberman
27e851c5ca Add maxSecondsBetweenMessages to metadata yaml (#36872) 2024-04-05 19:57:44 -07:00
Rodi Reich Zilberman
814d95cfbb Adapt source-mssql to latest Kotlin converted CDK (#36772) 2024-04-04 15:53:36 -07:00
Xiaohan Song
1b99f1ab89 DBZ iterator migration to use SourceStateIterator (#36333) 2024-03-25 21:50:33 -07:00
Rodi Reich Zilberman
f173b37db8 Allow up to 40 minutes of initial wait for record (#36364) 2024-03-21 15:04:34 -07:00
Akash Kulkarni
34d36a2cad [Source-mssql] : Remove mssql initial source operations (#36325) 2024-03-20 14:41:27 -07:00
Rodi Reich Zilberman
e5aea95272 Custom convert incoming long for date and datetime types (#36263) 2024-03-19 15:32:45 -07:00
Rodi Reich Zilberman
838a50d844 Revert "sanity" 
This reverts commit 817dbe4d15.
 2024-03-19 09:00:57 -07:00
Rodi Reich Zilberman
817dbe4d15 sanity 2024-03-19 08:59:57 -07:00
Xiaohan Song
2d270f20da [source-mssql] mssql use SourceStateIterator to send out counts (#35792) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Anton Karpets <anton.karpets@globallogic.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: erohmensing <erohmensing@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Dan Lecocq <dlecocq@sofi.org>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com>
 2024-03-13 16:56:26 -07:00
Rodi Reich Zilberman
fd98267c4b configure debezium to correctly convert column with default value whe… (#36011) 2024-03-13 11:34:43 -07:00
Rodi Reich Zilberman
eabf39c2e7 GA release 4.0.0 (#35873) 2024-03-07 14:52:38 -08:00
Rodi Reich Zilberman
9ce9217660 make query safer for servers that are case sensitive (#35816) 2024-03-06 13:47:50 -04:00
Stephane Geneix
cdf80f3bfa get source-mssql CI to green (#35721) 
TheAbstractMssqlSourceDatatypeTest was wrong for reals. 
CdcMssqlSourceTest and CdcMssqlSslSourceTest no longer use exclusive containers.
tests that change the SQLServer agent state are now in their own test class (which needs to use an exclusive container)

Besides that, the core of the problem can be grouped in 2 issues:
a) some tests are failing to enable CDC for tables. This is due to a timing issue. We moved that logic into its own function that will try for a total of 240seconds before giving up.
b) some tests are failing when trying to read the minLsn. There is a 1sec wait implemented in the production code. Instead we introduce a busy loop that will wait for a total of 240seconds for records to appear in the CDC table before giving up. That function is implemented in test code.

Unfortunately, for both cases, we sometimes needed to wait while in the middle of a function implemented in the CDK. We introduced a few hooks in the parent PR that are implemented in this PR for the source-mssql tests, and use the functions described above
 2024-03-06 09:01:37 -08:00
Rodi Reich Zilberman
f9e73cf62a add configuration for cdc queue size (#35739) 2024-03-05 18:45:09 -06:00
Akash Kulkarni
30190eae54 [Source-mssql] : Add config to throw an error on invalid CDC position (#35566) 2024-02-28 12:48:20 -06:00
Stephane Geneix
e01c0a3780 re-release source mssql with logger fixes (#35596) 2024-02-23 13:19:43 -06:00
Stephane Geneix
184d10967c push new source-mssql version (#35564) 2024-02-22 16:37:27 -06:00
Rodi Reich Zilberman
c9ba9df789 🐛 Change null cursor value query to not use IIF sql function (#35405) 2024-02-21 11:44:55 -06:00
Rodi Reich Zilberman
c8d5927401 Checkpointing source-mssql (#34182) 
Co-authored-by: nguyenaiden <duy@airbyte.io>
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
 2024-02-08 17:20:38 -06:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Ash-v-G
f70a6a5e41 Docs: Update mssql.md (#34432) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2024-01-23 10:57:26 -03:00
Scott Sinclair
a0623c54b4 fix link in MSSQL source documentation to pr (#33961) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2024-01-15 17:40:19 -03:00
Akash Kulkarni
804a34b212 [Source-mssql] : Remove options for data_to_sync & snapshot_isolation (#33700) 2024-01-10 14:06:38 -08:00
Marius Posta
c284588d99 source-mssql: shorten capture job polling interval in tests (#33510) 2024-01-08 13:05:28 -08:00
Stephane Geneix
15490f6cd9 add SSL options to mssql (#33071) 
Co-authored-by: Marius Posta <marius@airbyte.io>
 2024-01-04 11:50:03 -08:00
Xiaohan Song
609bcb2799 ✨Use base64 to encode mssql binary field (#33755) 2024-01-03 16:32:14 -08:00
Subodh Kant Chaturvedi
7c9394f8da mssql-source: remove LEGACY state flag (#33481) 2023-12-19 23:46:19 +05:30
Stephane Geneix
7054db7ad8 bump airbyte-cdk version (#33505) 2023-12-14 16:28:07 -04:00
Stephane Geneix
6e375f4e60 pull MSSql debezium classes out of the CDK (#33270) 
This is just relocating classes from inside the CDK to the individual container. Bumping the CDK version, using the new CDK in the source-mssql and bumping the source-mssql connector version
 2023-12-13 15:10:28 -08:00
Duy Nguyen
ce7554021f [Source-MSSQL] Enable per-stream and Global states and migrate away from Legacy states (#33018) 2023-12-12 17:53:29 -08:00
Duy Nguyen
44ce0022d2 [Source-mssql] Datetimeoffset field used as cursor error patch (#33330) 2023-12-12 12:52:26 -08:00
Marius Posta
3026cc59a8 source-mssql: enable state compression (#33225) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-12-12 11:08:31 -08:00
Marius Posta
b17f73be14 source-mssql: enforce SSL on Airbyte Cloud (#32882) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-12-01 10:18:34 -08:00
Duy Nguyen
5acca55f3c [Source-Mssql] Format Datetime and Datetime2 datatypes to 6-digit microsecond precision (#32573) 
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
 2023-11-30 12:17:39 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-11-27 12:49:06 -06:00
Tim Roes
c5db1e947b 📚 Documentation November 2023 overhaul (#32811) 
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: timroes <timroes@users.noreply.github.com>
Co-authored-by: nataliekwong <nataliekwong@users.noreply.github.com>
 2023-11-26 11:57:38 -08:00
Duy Nguyen
1e59751bc4 [Source-Mssql] Assess and map all SQLServer supported datatypes to the correct JDBC and Airbyte types for Standard and CDC mode (#31531) 
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
 2023-11-06 16:30:08 -08:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Rodi Reich Zilberman
1e8e6402bc Revert "Automated Commit - Formatting Changes" (#32077) 2023-11-01 15:38:25 -05:00
Rodi Reich Zilberman
f5c87968ca ✨URL encode username and password fields (#32028) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
 2023-11-01 15:22:27 -05:00
Akash Kulkarni
85d266ac20 Bump DBZ version from 2.2.0 -> 2.4.0 for mssql, mysql (#32024) 2023-11-01 13:55:13 -04:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821) 
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-08-31 12:19:02 -05:00