jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-22 11:31:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
17,230 Commits 5,109 Branches 518 Tags
42124f67722cbacbd90b09fa682ee972edb51f6a
Commit Graph

195 Commits

Author SHA1 Message Date
Yue Li
42124f6772 [source-mysql]: support Planetscale MySQL's per-query row limit (#40561) 
Fixes airbytehq/oncall#5051

Planet MySQL has defined multiple system limits. One of our customers has been hitting the 100K per-query row limit, which causes our connector to emit system errors. This patch defines a chunk size limit for our connector. By default, it is the largest long, and if we find it is a Planetscale MySQL, we will adjust to 100K.
 2024-06-26 17:29:02 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Akash Kulkarni
fcc2940d20 [DB sources] : Debezium properties accept list of streams (#39405) 2024-06-13 15:11:10 -04:00
Yue Li
30cd6f1a3d source-mysql: upgrade debezium to 2.5.4 (#39144) 
This patch attempts to fix the following issues:

airbytehq/oncall#5332
#32348
 2024-06-12 16:25:21 -07:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Akash Kulkarni
a2bcd4904c [DB sources] : Exit when encountering a config error (#38538) 2024-05-29 17:44:21 -04:00
Rodi Reich Zilberman
7bd9423216 mysql stream statuses (#38198) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-23 19:25:51 -04:00
Evan Tahler
a914027d74 [docs] Vendor-Specific limitations added to troubleshooting sections (#38351) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
 2024-05-21 14:42:22 +00:00
Subodh Kant Chaturvedi
073b94062c db-sources: disable counts for state messages for FULL_REFRESH streams (#38208) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-15 11:15:26 -07:00
Akash Kulkarni
fcbfaa376f [DB source errors] : Handle common transient errors (#38104) 2024-05-13 20:27:07 -04:00
Xiaohan Song
533cea8bb8 Only check sourcedefined pk for eligibility of rfr (#38046) 2024-05-07 16:54:08 -04:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Xiaohan Song
d3864c29d2 [source-mysql] Fix NPE on cursor based full refresh (#37824) 2024-05-03 19:24:18 -04:00
Xiaohan Song
b9dc2053cf Mysql to emit stream initial loader (#36932) 
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
 2024-05-03 12:30:53 -07:00
Rodi Reich Zilberman
bdb3556b8a bump connectors to latest CDK version (#37781) 2024-05-02 16:52:11 -04:00
Akash Kulkarni
914f044354 [Certified sources] : Bump cdk versions to disable DBZ retries (#37742) 2024-05-01 15:07:37 -04:00
Akash Kulkarni
2b260de9d0 [DB source error messages] : Better error messages when switching between global/per-stream modes (#37507) 2024-04-30 19:18:43 -04:00
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00
Akash Kulkarni
097709a187 [Source-mysql/mssql] : Remove legacy bad values handling code (#37476) 2024-04-22 20:10:54 -04:00
Akash Kulkarni
bae63044cc [Source-mysql/mssql] : Populate null values (#37111) 2024-04-16 14:25:27 -07:00
Akash Kulkarni
cb9eae3681 [Source-mysql] : Add meta error handling in initial load path (#37328) 2024-04-15 14:49:14 -07:00
Akash Kulkarni
5587167ddc [Source-mysql] : Remove mysql initial source operations (#37324) 2024-04-15 12:58:54 -04:00
Rodi Reich Zilberman
1e9ee1dbb8 Prevent conversion of null fields (#36919) 2024-04-11 20:30:38 -04:00
Rodi Reich Zilberman
27e851c5ca Add maxSecondsBetweenMessages to metadata yaml (#36872) 2024-04-05 19:57:44 -07:00
Xiaohan Song
9d53013112 not throwing config error (#36577) 2024-04-05 16:58:54 -04:00
Xiaohan Song
3c35466413 MySQL to adapt to new cdk (#36742) 2024-04-05 13:10:17 -04:00
Xiaohan Song
2637e97e23 [source-mysql] merge state manager (#35529) 2024-02-29 14:30:25 -04:00
Akash Kulkarni
ae39cbe82f Fix error message for saved offset not valid. (#35675) 2024-02-27 17:54:30 -04:00
Stephane Geneix
b2b9b7fa21 reduce interrupt and shutdown delays to 1 minutes and 2 minutes when stopping a connector (initially set at 60minutes and 70minutes) (#35527) 
Fixes #32348 
discussed here : https://airbytehq-team.slack.com/archives/C02U2SSHP9S/p1708552465201999
 2024-02-23 14:12:28 -08:00
Marius Posta
2b370f84c2 source-mysql: bump patch version following #35569 (#35573) 
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
 2024-02-23 00:19:56 -04:00
Akash Kulkarni
1162302069 [Source-mysql] : Adopt 0.21.4 and reduce cdc state compression threshold to 1MB (#35525) 2024-02-21 19:24:56 -08:00
Akash Kulkarni
40a5edc5fe [Source-mysql] : Add config to throw an error on invalid CDC position (#35338) 2024-02-20 14:09:05 -08:00
Akash Kulkarni
5bab3df367 Add airbyte trace utility to emit analytics messages & emit messages for MongoDB, Postgres & MySQL (#35036) 2024-02-13 19:10:38 -04:00
Akash Kulkarni
62e5528e57 Don't emit final state if there is an underlying stream failure (#34869) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-02-13 12:31:54 -04:00
Xiaohan Song
82d9197587 🐛 source-mysql Support special chars in dbname (#34580) 2024-02-12 11:06:16 -08:00
Marius Posta
796c0ee494 source-mysql: adopt cleaned-up cdk (#34750) 2024-02-09 14:04:41 -06:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Xiaohan Song
b290208391 Add count in state message for incremental syncs (#33005) 
Co-authored-by: xiaohansong <xiaohansong@users.noreply.github.com>
 2024-01-09 21:05:57 -04:00
Xiaohan Song
18e0e7741a ✨ Introduce StateIteratorProcessor in CDK (#33312) 2024-01-03 15:29:12 -08:00
Subodh Kant Chaturvedi
5b915c6837 mysql-source: remove LEGACY state flag (#33436) 2023-12-19 23:45:51 +05:30
Akash Kulkarni
4c9908496b Add debugger drivers for MongoDB & MySQL (#33356) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-12 20:07:46 -04:00
Rodi Reich Zilberman
f5ae28fc0c 🐛 Rodi/port zero date time behavior fix (#33210) 
Co-authored-by: Robert Tomczak <36033856+robertomczak@users.noreply.github.com>
Co-authored-by: robertomczak <robertomczak@users.noreply.github.com>
 2023-12-11 11:09:14 -08:00
Akash Kulkarni
ea8cf787cf [Source MySQL] : Improvements to schema snapshotting and error handling (#33082) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-07 11:24:38 -08:00
Xiaohan Song
2d26337bda Use parse base64Binary to parse binary related data (#32610) 
Co-authored-by: xiaohansong <xiaohansong@users.noreply.github.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
 2023-11-30 15:05:45 -04:00
Duy Nguyen
665e4f73ad source-mysql: merge strict encrypt variant into standard main mysql source (#31062) 
Co-authored-by: erohmensing <erohmensing@gmail.com>
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: erohmensing <erohmensing@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
 2023-11-29 10:45:28 -04:00
Xiaohan Song
d7c8f35c28 [Mysql] Apply initial setup time to Debezium setup time (#32662) 2023-11-28 13:54:35 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-11-27 12:49:06 -06:00
Tim Roes
c5db1e947b 📚 Documentation November 2023 overhaul (#32811) 
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: timroes <timroes@users.noreply.github.com>
Co-authored-by: nataliekwong <nataliekwong@users.noreply.github.com>
 2023-11-26 11:57:38 -08:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00