jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 18:39:31 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

224 Commits

Author SHA1 Message Date
Akash Kulkarni
af0394150f [WASS] : Remove final transient error emitter iterators (#42421) 2024-07-23 14:27:15 -04:00
Yue Li
4cadb340b9 [source-postgres/mssql] Hide the "initial load timeout in hours" field by default in UI for postgres and mssql (#42411) 
Hide the field as the other advanced files were also hidden by default.
 2024-07-23 02:10:37 -07:00
Xiaohan Song
af28105efa [source-mssql/mysql/postgres] Fix and cleanup oc map (#42024) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
 2024-07-22 11:32:51 -07:00
Akash Kulkarni
000dc477cf [WASS] : Enhancements in logging & analytics (#42122) 2024-07-21 23:07:18 -04:00
Akash Kulkarni
8e9daf03bd [Source-mssql] : Adopt latest CDK (#42078) 2024-07-17 13:32:43 -07:00
Akash Kulkarni
c36d8e251e Source-mssql : Implement WASS algorithm (#41648) 2024-07-17 08:40:25 -07:00
Marius Posta
652cfd34d6 source-mssql: bump debezium-connector-sqlserver to latest patch version (#40750) 2024-07-05 15:38:25 -04:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Rodi Reich Zilberman
b66d528a4d Prefer a clustered over primary for initial load of table (#40638) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-01 13:16:58 -04:00
Akash Kulkarni
98b1080998 [Source-mssql] : TImeout for building schema should be tied to the configured wait time (#40558) 2024-06-26 14:30:27 -04:00
Xiaohan Song
131ee051ec [source-mssql] correctly parse type datetimeoffset (#39419) 2024-06-21 13:39:45 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Akash Kulkarni
5b62bdfbb3 [Source-mssql] : Bump cdk to latest (#39506) 2024-06-14 15:09:06 -07:00
Rodi Reich Zilberman
8b76e89ef3 Remove custom debezium mssql converted for datetimeoffset type (#39342) 2024-06-07 13:09:57 -07:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Marius Posta
25db2e0f2c source-mssql: fix cursor value query (#38292) 2024-05-30 07:26:38 -07:00
Akash Kulkarni
2bf0bf8d07 Bump versions for SQL server, Postgres (#38775) 2024-05-29 18:33:50 -04:00
Rodi Reich Zilberman
ccea0239ec Mssql stream status messages (#38640) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-28 14:11:19 -04:00
Subodh Kant Chaturvedi
073b94062c db-sources: disable counts for state messages for FULL_REFRESH streams (#38208) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-15 11:15:26 -07:00
Marius Posta
100f4e0f34 source-mssql: bump jdbc driver to 12.6.1.jre11 (#38196) 2024-05-14 13:51:39 -07:00
Rodi Reich Zilberman
714c836a93 ignore user defined Pk for resumable full refresh (#38054) 2024-05-08 08:17:55 -07:00
Rodi Reich Zilberman
8314f611d4 Bump debezium in mssql to latest (#38042) 2024-05-07 15:37:52 -07:00
Rodi Reich Zilberman
b3da013cf7 mssql rfr publish cdk (#38029) 2024-05-07 12:25:55 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Rodi Reich Zilberman
5432fab8af Resumable Full Refresh sync for mssql (#37451) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-07 07:36:24 -07:00
Rodi Reich Zilberman
bdb3556b8a bump connectors to latest CDK version (#37781) 2024-05-02 16:52:11 -04:00
Akash Kulkarni
914f044354 [Certified sources] : Bump cdk versions to disable DBZ retries (#37742) 2024-05-01 15:07:37 -04:00
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00
Akash Kulkarni
097709a187 [Source-mysql/mssql] : Remove legacy bad values handling code (#37476) 2024-04-22 20:10:54 -04:00
Akash Kulkarni
bae63044cc [Source-mysql/mssql] : Populate null values (#37111) 2024-04-16 14:25:27 -07:00
Evan Tahler
466abc2968 source-mssql: Allow 60min for MAX_FIRST_RECORD_WAIT_TIME (#37326) 2024-04-15 15:45:51 -07:00
Akash Kulkarni
ca394d20e3 [Source-mssql] : Add meta error handling in initial load path (#37325) 2024-04-15 16:17:26 -04:00
Akash Kulkarni
e75ea0fa70 [Source-mssql] : Remove override for queryTableFullRefresh (#37110) 2024-04-15 09:17:17 -07:00
Rodi Reich Zilberman
1e9ee1dbb8 Prevent conversion of null fields (#36919) 2024-04-11 20:30:38 -04:00
Rodi Reich Zilberman
27e851c5ca Add maxSecondsBetweenMessages to metadata yaml (#36872) 2024-04-05 19:57:44 -07:00
Rodi Reich Zilberman
814d95cfbb Adapt source-mssql to latest Kotlin converted CDK (#36772) 2024-04-04 15:53:36 -07:00
Xiaohan Song
1b99f1ab89 DBZ iterator migration to use SourceStateIterator (#36333) 2024-03-25 21:50:33 -07:00
Rodi Reich Zilberman
f173b37db8 Allow up to 40 minutes of initial wait for record (#36364) 2024-03-21 15:04:34 -07:00
Akash Kulkarni
34d36a2cad [Source-mssql] : Remove mssql initial source operations (#36325) 2024-03-20 14:41:27 -07:00
Rodi Reich Zilberman
e5aea95272 Custom convert incoming long for date and datetime types (#36263) 2024-03-19 15:32:45 -07:00
Rodi Reich Zilberman
838a50d844 Revert "sanity" 
This reverts commit 817dbe4d15.
 2024-03-19 09:00:57 -07:00
Rodi Reich Zilberman
817dbe4d15 sanity 2024-03-19 08:59:57 -07:00
Xiaohan Song
2d270f20da [source-mssql] mssql use SourceStateIterator to send out counts (#35792) 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Anton Karpets <anton.karpets@globallogic.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: erohmensing <erohmensing@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Dan Lecocq <dlecocq@sofi.org>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com>
 2024-03-13 16:56:26 -07:00
Rodi Reich Zilberman
fd98267c4b configure debezium to correctly convert column with default value whe… (#36011) 2024-03-13 11:34:43 -07:00
Rodi Reich Zilberman
eabf39c2e7 GA release 4.0.0 (#35873) 2024-03-07 14:52:38 -08:00
Rodi Reich Zilberman
9ce9217660 make query safer for servers that are case sensitive (#35816) 2024-03-06 13:47:50 -04:00
Stephane Geneix
cdf80f3bfa get source-mssql CI to green (#35721) 
TheAbstractMssqlSourceDatatypeTest was wrong for reals. 
CdcMssqlSourceTest and CdcMssqlSslSourceTest no longer use exclusive containers.
tests that change the SQLServer agent state are now in their own test class (which needs to use an exclusive container)

Besides that, the core of the problem can be grouped in 2 issues:
a) some tests are failing to enable CDC for tables. This is due to a timing issue. We moved that logic into its own function that will try for a total of 240seconds before giving up.
b) some tests are failing when trying to read the minLsn. There is a 1sec wait implemented in the production code. Instead we introduce a busy loop that will wait for a total of 240seconds for records to appear in the CDC table before giving up. That function is implemented in test code.

Unfortunately, for both cases, we sometimes needed to wait while in the middle of a function implemented in the CDK. We introduced a few hooks in the parent PR that are implemented in this PR for the source-mssql tests, and use the functions described above
 2024-03-06 09:01:37 -08:00
Rodi Reich Zilberman
f9e73cf62a add configuration for cdc queue size (#35739) 2024-03-05 18:45:09 -06:00
Akash Kulkarni
30190eae54 [Source-mssql] : Add config to throw an error on invalid CDC position (#35566) 2024-02-28 12:48:20 -06:00