jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 11:01:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
22,318 Commits 5,109 Branches 518 Tags
baf4633ab8b51a55276115ea083837d8632fe217
Commit Graph

98 Commits

Author SHA1 Message Date
Xiaohan Song
657110b3c0 [source-mongodb] lazy init current iterator to prevent cursor timeout issue (#45883) 2024-09-24 13:00:52 -07:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Xiaohan Song
1b2c27e231 [source-mongodb] add subtype when converting binary (#42927) 2024-08-30 02:13:22 +03:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Xiaohan Song
6cbba8d19a [source-mongodb] use escaped dot in wass streamlist (#44846) 2024-08-28 07:46:03 -07:00
Xiaohan Song
ff907f0ae5 [source-mongodb] include list does not have escaped dot (#44839) 2024-08-28 02:27:19 +03:00
Yue Li
c68ec5ab2f [source-mongo-v2] revert permission check on oplog.rs for mongo source (#44779) 2024-08-26 11:44:11 -07:00
Yue Li
f565c0426c [source-mongodb-v2] Adding permission check on oplog.rs needed by CDC (#44490) 
The user will not be able to set up the connector if his/her airbyte user does not have read access to the oplog.rs collection.
 2024-08-20 18:19:44 -07:00
Akash Kulkarni
b43b1111a8 [cdk] : More logging for count mismatch (#43410) 2024-08-09 05:45:27 +03:00
Xiaohan Song
97d822f820 [cdk, source-mysql/postgres/mssql/mongodb-v2] log record count to state message (#42869) 
Co-authored-by: Yue Li <yue.li@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-07 04:35:22 +03:00
nora-airbyte
889cb65e21 Center MongoDb connector icon (#42549) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-05 11:24:49 -07:00
Yue Li
8f42de9c37 [source-mongodb-v2] : Implement WASS algorithm (#42561) 
Fixes airbytehq/airbyte-internal-issues#8730

Port of #38240 for MongoDB
 2024-08-05 06:36:23 -07:00
Evan Tahler
d2ae38209f source-mongo: Warn (vs fail) on different _id types in collection (#39145) 2024-07-22 16:34:11 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Akash Kulkarni
94234e57c6 [Source-mongodb-v2] : Adopt new CDK (#39530) 2024-06-17 13:57:57 -07:00
Xiaohan Song
3c9a6121c4 Upgrade mongodb to use dbz 2.6.2 (#38238) 2024-06-11 13:25:24 -04:00
Audrey Maldonado
b9690ba76e Wrapping changelog in details for a collapsible view (#38801) 2024-06-04 13:40:53 -04:00
Rodi Reich Zilberman
c1748b4efe MongoDB stream status traces (#38781) 2024-05-30 19:44:07 -04:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Evan Tahler
a914027d74 [docs] Vendor-Specific limitations added to troubleshooting sections (#38351) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
 2024-05-21 14:42:22 +00:00
Rodi Reich Zilberman
82961ad9f9 Add support for binary _id field in source-mongodb (#38103) 2024-05-10 10:46:42 -07:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Akash Kulkarni
ddc3d2b527 [Source-mongo] : Add options for CAPTURE_MODE (#36851) 2024-05-06 19:06:27 -04:00
Akash Kulkarni
48930e499a [Source-mongo] : Chunk size(limit) should correspond to ~1GB of data (#37753) 2024-05-02 15:04:53 -07:00
Rodi Reich Zilberman
bdb3556b8a bump connectors to latest CDK version (#37781) 2024-05-02 16:52:11 -04:00
Akash Kulkarni
914f044354 [Certified sources] : Bump cdk versions to disable DBZ retries (#37742) 2024-05-01 15:07:37 -04:00
Akash Kulkarni
993aece63d [Source-mongo] : Implement fixed chunk size (1million) (#37559) 2024-04-25 14:37:24 -07:00
Akash Kulkarni
8eb634c17f [Source-mongodb-v2] : Fix logic in determining whether a resume token is valid or not (#37557) 2024-04-24 15:54:20 -07:00
Akash Kulkarni
d9ee103f10 [Source-mongo] : Adding a new iterator in preparation of adding chunking subqueries (#37525) 2024-04-24 10:26:31 -07:00
Xiaohan Song
28369403c9 [source-MongoDB] Do not send estimate trace message if we don't have data (#37473) 2024-04-22 11:03:19 -07:00
Akash Kulkarni
70afb9d2c7 [Source-mongodb] : Populate null values in airbyte record message (#37348) 2024-04-16 13:41:21 -07:00
Rodi Reich Zilberman
27e851c5ca Add maxSecondsBetweenMessages to metadata yaml (#36872) 2024-04-05 19:57:44 -07:00
Akash Kulkarni
4a020a4078 [Source-mongo] : Migrate to Kotlin (#36845) 2024-04-04 20:29:23 -04:00
Akash Kulkarni
deb47ab707 [source-mongo] : Adopt CDK 0.28.0 (#36837) 2024-04-04 15:50:36 -07:00
Rodi Reich Zilberman
bcd32a0a8b Resumable full refresh source-mongodb (#35845) 2024-03-20 13:01:53 -07:00
Xiaohan Song
d5e91ae83a [source-mongodb] record count in state & initial iterator refactor (#35669) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Tim Roes <tim@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Anton Karpets <anton.karpets@globallogic.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Tyler B <104733644+tybernstein@users.noreply.github.com>
Co-authored-by: bgroff <bgroff@users.noreply.github.com>
Co-authored-by: mjgatz <86885812+mjgatz@users.noreply.github.com>
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Joe Reuter <joe@airbyte.io>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Roman Yermilov [GL] <86300758+roman-yermilov-gl@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: midavadim <midavadim@yahoo.com>
Co-authored-by: Julien COUTAND <julien.coutand@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.com>
Co-authored-by: Ben Drucker <bvdrucker@gmail.com>
Co-authored-by: TornadoContre <37258495+TornadoContre@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Thomas Dippel <dipth@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Alex Birdsall <ambirdsall@gmail.com>
Co-authored-by: ambirdsall <ambirdsall@users.noreply.github.com>
Co-authored-by: Jose Gerardo Pineda <jose.pineda@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Siarhei Ivanou <sinusu@gmail.com>
Co-authored-by: Anatolii Yatsuk <tolikyatsuk@gmail.com>
Co-authored-by: Ryan Waskewich <156025126+rwask@users.noreply.github.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: perangel <perangel@gmail.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: Obioma Anomnachi <onanomnachi@gmail.com>
Co-authored-by: maxi297 <maxi297@users.noreply.github.com>
Co-authored-by: SatishChGit <satishchinthanippu@gmail.com>
Co-authored-by: Brian Leonard <brian@bleonard.com>
Co-authored-by: David Wallace <dwallace0723@gmail.com>
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
Co-authored-by: Danny Tiesling <tiesling@gmail.com>
 2024-03-06 14:29:42 -08:00
Rodi Reich Zilberman
4390db5c77 [source-mongodb-v2] remove default connection string options (#35673) (#35677) 
Co-authored-by: David Wallace <dwallace0723@gmail.com>
 2024-02-27 14:22:28 -08:00
Akash Kulkarni
ae39cbe82f Fix error message for saved offset not valid. (#35675) 2024-02-27 17:54:30 -04:00
Marius Posta
aa084a9446 source-mongodb-v2: bump patch version following #35569 (#35571) 
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
 2024-02-23 00:30:12 -04:00
Rodi Reich Zilberman
f106d11d1c Mangle unhandled MongoCommandException to prevent creating grouping o… (#35526) 2024-02-21 18:57:48 -04:00
Akash Kulkarni
9a7da6dca1 [Source-Mongodb] : Add config to throw an error on invalid CDC position (#35375) 2024-02-20 14:46:44 -08:00
Akash Kulkarni
5bab3df367 Add airbyte trace utility to emit analytics messages & emit messages for MongoDB, Postgres & MySQL (#35036) 2024-02-13 19:10:38 -04:00
Rodi Reich Zilberman
0385a64200 🐛 Set cdc record subsequent record wait time to initial wait time as a workaround (#35114) 2024-02-13 14:04:23 -08:00
Marius Posta
5435df6d95 source-mongodb-v2: adopt cleaned-up cdk (#34748) 2024-02-09 11:38:55 -08:00
Akash Kulkarni
c7c51ea4e1 [source-mongodb-v2] : Fail sync if initial snapshot for any stream fails (#34759) 2024-02-05 09:15:03 -08:00
Rodi Reich Zilberman
eb5b23d669 Build a resume token with a pipeline consisting of selected streams (#34594) 
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
 2024-01-31 09:48:12 -08:00
Rodi Reich Zilberman
f9df3dff04 Support resuming initial snapshot when id type is String, Int, Long (#34641) 2024-01-30 15:58:57 -08:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Akash Kulkarni
25a0c80547 [Source-mongodb-v2] : Adding logs for resume token timestamps + reducing min document discovery size further (#34364) 2024-01-18 16:38:57 -08:00
Akash Kulkarni
be06855dc0 [Source-mongo] : Relax minimum document discovery size to 100 (#34314) 2024-01-17 09:04:31 -08:00