jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

317 Commits

Author SHA1 Message Date
Xiaohan Song
86a0d52829 [source-mysql] fix for datetime in mysql (#48628) 2024-11-25 18:30:40 +00:00
Marius Posta
8aec55d738 bulk-cdk: CDC-related fixes and tweaks (#48602) 2024-11-22 20:58:51 +00:00
Rodi Reich Zilberman
e193fb30ce Re-sync in case of stale offset (#48578) 2024-11-21 09:27:27 -08:00
Rodi Reich Zilberman
6f25f8230e A saved state integer should map to the largest possible big int - re… (#48575) 2024-11-20 09:34:01 -08:00
Xiaohan Song
4b391439dc [source-mysql] No empty sample (#48545) 2024-11-18 23:02:55 +00:00
Rodi Reich Zilberman
754ca19b2d A saved state integer should map to the largest possible big int (#48529) 2024-11-18 21:20:48 +02:00
Rodi Reich Zilberman
59bcfc1331 remove the gtid_mode extra check (#48524) 2024-11-18 09:11:02 -08:00
Xiaohan Song
712ea82d05 [source-mysql] Relax GTID validation (#48521) 2024-11-15 21:53:42 +00:00
Xiaohan Song
33cecdd332 [source-mysql] fix for legacy datetime (#48517) 2024-11-15 19:37:10 +00:00
Rodi Reich Zilberman
37a1bcba00 Improve error message (#48514) 2024-11-15 11:02:46 -08:00
Rodi Reich Zilberman
f89371e03e make check query compatible with older versions of mysql (#48507) 2024-11-15 09:28:14 -08:00
Rodi Reich Zilberman
c6b7ac1450 10684 source mysql beta stream with id of type binary shows an error when queried (#48501) 2024-11-14 18:18:22 -08:00
Xiaohan Song
860b695baf [source-mysql] convert datetime in cursor state to v2 format (#48491) 2024-11-15 00:47:50 +00:00
Marius Posta
4ae0ce69e3 bulk-cdk-core-extract: fix mistaken assumption about non-global streams (#48466) 2024-11-12 14:09:30 -08:00
Rodi Reich Zilberman
d184317acc quote compressed db schema history (#48456) 2024-11-11 15:35:41 -08:00
Rodi Reich Zilberman
63cd395fbe handle pk value from saved state (#48423) 2024-11-08 19:47:59 +00:00
Rodi Reich Zilberman
c589cb5e9e migrate CAT tests to new source-mysql location (#48414) 2024-11-07 19:18:07 -08:00
Rodi Reich Zilberman
e42d056050 merge mysql v2 into v1 (#48369) 2024-11-07 10:39:55 -08:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Xiaohan Song
03584d5afb [source-mysql/mssql] Fix state manager on determining non-resumable streams (#45181) 2024-09-10 16:13:04 -07:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Yue Li
ed44c093b3 [source-mysql] Upgrade Debezium and mysql connector to 2.7.1.Final (#44013) 2024-08-14 10:15:05 -07:00
Xiaohan Song
ab005c6465 [source-mssql/postgres/mysql] fix rfr losing stream state (#42550) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-08-02 20:02:19 +03:00
Akash Kulkarni
af0394150f [WASS] : Remove final transient error emitter iterators (#42421) 2024-07-23 14:27:15 -04:00
Xiaohan Song
af28105efa [source-mssql/mysql/postgres] Fix and cleanup oc map (#42024) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
 2024-07-22 11:32:51 -07:00
Yue Li
03c57ea472 [source-postgres, source-mysql] Translating top opaque and unfriendly error messages (#42087) 
This patch includes multiple translation rules for additional top opaque and system error messages that can be converted to transient or config errors. Translation/error profiles are added to the exception handler class for Postgres and MySQL sources.
 2024-07-21 23:24:17 -07:00
Akash Kulkarni
000dc477cf [WASS] : Enhancements in logging & analytics (#42122) 2024-07-21 23:07:18 -04:00
Yue Li
4417769038 [cdk, source-postgres, source-mysql] a new error handling and translation framework (#40208) 
Fixes airbytehq/airbyte-internal-issues#8516

This set of changes mainly moves error translation to be part of each connector.

In general, each connector needs to implement its own error translation class that inherits from the abstract class ConnectorExceptionTranslator, which is part of the CDK. By implementing, it means the connector developer or our support will populate the error dictionary with error samples with matching rules (e.g., regex). See the example we created for the Postgres source.
 2024-07-17 17:19:13 -07:00
Akash Kulkarni
508638fcb7 [Source-mysql] : Adopt latest CDK + WASS fixes (#42043) 2024-07-17 13:33:05 -07:00
Akash Kulkarni
310e6bd315 [Source-mysql] : Implement WASS algo (#38240) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-11 11:54:54 -04:00
Yue Li
42124f6772 [source-mysql]: support Planetscale MySQL's per-query row limit (#40561) 
Fixes airbytehq/oncall#5051

Planet MySQL has defined multiple system limits. One of our customers has been hitting the 100K per-query row limit, which causes our connector to emit system errors. This patch defines a chunk size limit for our connector. By default, it is the largest long, and if we find it is a Planetscale MySQL, we will adjust to 100K.
 2024-06-26 17:29:02 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Akash Kulkarni
fcc2940d20 [DB sources] : Debezium properties accept list of streams (#39405) 2024-06-13 15:11:10 -04:00
Yue Li
30cd6f1a3d source-mysql: upgrade debezium to 2.5.4 (#39144) 
This patch attempts to fix the following issues:

airbytehq/oncall#5332
#32348
 2024-06-12 16:25:21 -07:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Rodi Reich Zilberman
7bd9423216 mysql stream statuses (#38198) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-23 19:25:51 -04:00
Subodh Kant Chaturvedi
073b94062c db-sources: disable counts for state messages for FULL_REFRESH streams (#38208) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-15 11:15:26 -07:00
Akash Kulkarni
fcbfaa376f [DB source errors] : Handle common transient errors (#38104) 2024-05-13 20:27:07 -04:00
Xiaohan Song
533cea8bb8 Only check sourcedefined pk for eligibility of rfr (#38046) 2024-05-07 16:54:08 -04:00
Xiaohan Song
d3864c29d2 [source-mysql] Fix NPE on cursor based full refresh (#37824) 2024-05-03 19:24:18 -04:00
Xiaohan Song
b9dc2053cf Mysql to emit stream initial loader (#36932) 
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
 2024-05-03 12:30:53 -07:00
Akash Kulkarni
2b260de9d0 [DB source error messages] : Better error messages when switching between global/per-stream modes (#37507) 2024-04-30 19:18:43 -04:00
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00
Akash Kulkarni
097709a187 [Source-mysql/mssql] : Remove legacy bad values handling code (#37476) 2024-04-22 20:10:54 -04:00
Akash Kulkarni
bae63044cc [Source-mysql/mssql] : Populate null values (#37111) 2024-04-16 14:25:27 -07:00
Akash Kulkarni
cb9eae3681 [Source-mysql] : Add meta error handling in initial load path (#37328) 2024-04-15 14:49:14 -07:00
Akash Kulkarni
5587167ddc [Source-mysql] : Remove mysql initial source operations (#37324) 2024-04-15 12:58:54 -04:00
Rodi Reich Zilberman
1e9ee1dbb8 Prevent conversion of null fields (#36919) 2024-04-11 20:30:38 -04:00
Xiaohan Song
3c35466413 MySQL to adapt to new cdk (#36742) 2024-04-05 13:10:17 -04:00
Xiaohan Song
2637e97e23 [source-mysql] merge state manager (#35529) 2024-02-29 14:30:25 -04:00