jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

115 Commits

Author SHA1 Message Date
Akash Kulkarni
deb47ab707 [source-mongo] : Adopt CDK 0.28.0 (#36837) 2024-04-04 15:50:36 -07:00
Rodi Reich Zilberman
bcd32a0a8b Resumable full refresh source-mongodb (#35845) 2024-03-20 13:01:53 -07:00
Xiaohan Song
d5e91ae83a [source-mongodb] record count in state & initial iterator refactor (#35669) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Tim Roes <tim@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Anton Karpets <anton.karpets@globallogic.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Tyler B <104733644+tybernstein@users.noreply.github.com>
Co-authored-by: bgroff <bgroff@users.noreply.github.com>
Co-authored-by: mjgatz <86885812+mjgatz@users.noreply.github.com>
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Joe Reuter <joe@airbyte.io>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Roman Yermilov [GL] <86300758+roman-yermilov-gl@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: midavadim <midavadim@yahoo.com>
Co-authored-by: Julien COUTAND <julien.coutand@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.com>
Co-authored-by: Ben Drucker <bvdrucker@gmail.com>
Co-authored-by: TornadoContre <37258495+TornadoContre@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Thomas Dippel <dipth@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Alex Birdsall <ambirdsall@gmail.com>
Co-authored-by: ambirdsall <ambirdsall@users.noreply.github.com>
Co-authored-by: Jose Gerardo Pineda <jose.pineda@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Siarhei Ivanou <sinusu@gmail.com>
Co-authored-by: Anatolii Yatsuk <tolikyatsuk@gmail.com>
Co-authored-by: Ryan Waskewich <156025126+rwask@users.noreply.github.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: perangel <perangel@gmail.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: Obioma Anomnachi <onanomnachi@gmail.com>
Co-authored-by: maxi297 <maxi297@users.noreply.github.com>
Co-authored-by: SatishChGit <satishchinthanippu@gmail.com>
Co-authored-by: Brian Leonard <brian@bleonard.com>
Co-authored-by: David Wallace <dwallace0723@gmail.com>
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
Co-authored-by: Danny Tiesling <tiesling@gmail.com>
 2024-03-06 14:29:42 -08:00
Rodi Reich Zilberman
4390db5c77 [source-mongodb-v2] remove default connection string options (#35673) (#35677) 
Co-authored-by: David Wallace <dwallace0723@gmail.com>
 2024-02-27 14:22:28 -08:00
Akash Kulkarni
ae39cbe82f Fix error message for saved offset not valid. (#35675) 2024-02-27 17:54:30 -04:00
Marius Posta
aa084a9446 source-mongodb-v2: bump patch version following #35569 (#35571) 
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
 2024-02-23 00:30:12 -04:00
Rodi Reich Zilberman
f106d11d1c Mangle unhandled MongoCommandException to prevent creating grouping o… (#35526) 2024-02-21 18:57:48 -04:00
Akash Kulkarni
9a7da6dca1 [Source-Mongodb] : Add config to throw an error on invalid CDC position (#35375) 2024-02-20 14:46:44 -08:00
Akash Kulkarni
5bab3df367 Add airbyte trace utility to emit analytics messages & emit messages for MongoDB, Postgres & MySQL (#35036) 2024-02-13 19:10:38 -04:00
Rodi Reich Zilberman
0385a64200 🐛 Set cdc record subsequent record wait time to initial wait time as a workaround (#35114) 2024-02-13 14:04:23 -08:00
Marius Posta
5435df6d95 source-mongodb-v2: adopt cleaned-up cdk (#34748) 2024-02-09 11:38:55 -08:00
Akash Kulkarni
c7c51ea4e1 [source-mongodb-v2] : Fail sync if initial snapshot for any stream fails (#34759) 2024-02-05 09:15:03 -08:00
Rodi Reich Zilberman
eb5b23d669 Build a resume token with a pipeline consisting of selected streams (#34594) 
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
 2024-01-31 09:48:12 -08:00
Rodi Reich Zilberman
f9df3dff04 Support resuming initial snapshot when id type is String, Int, Long (#34641) 2024-01-30 15:58:57 -08:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Akash Kulkarni
25a0c80547 [Source-mongodb-v2] : Adding logs for resume token timestamps + reducing min document discovery size further (#34364) 2024-01-18 16:38:57 -08:00
Akash Kulkarni
be06855dc0 [Source-mongo] : Relax minimum document discovery size to 100 (#34314) 2024-01-17 09:04:31 -08:00
Alexandre Cuoci
f9bfd6df50 MongoDB Oplog Troubleshooting (#34122) 2024-01-10 13:56:58 -05:00
Rodi Reich Zilberman
ca33d3099c Update doc with schema less mode (#33550) 2023-12-19 14:41:41 -08:00
Akash Kulkarni
279dd98e9a [Source-mongo] : Op log logging (#33549) 2023-12-18 17:40:04 -04:00
Rodi Reich Zilberman
743ab29b36 doc update to reflect minimum supported atlas tier. (#32186) 2023-12-18 11:54:47 -08:00
Subodh Kant Chaturvedi
c1c2a31920 mongo-db-source: upgrade to latest CDK (#33438) 2023-12-19 00:24:24 +05:30
Rodi Reich Zilberman
aee153a24a mongodb schema-less mode (#32328) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-14 20:05:16 -04:00
Akash Kulkarni
f2e0585508 [Source-MongoDb] : Better logging for MongoDB (#33430) 2023-12-13 10:13:48 -08:00
Akash Kulkarni
4c9908496b Add debugger drivers for MongoDB & MySQL (#33356) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-12 20:07:46 -04:00
Akash Kulkarni
1a09927d3e 🐛 Source MongoDB : Improve our discovery phase to not create large BSON objects (#32886) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-11-30 10:49:15 -04:00
Akash Kulkarni
e2e0aba489 [source-mongodb-v2] : Fix support to read UUIDs (#32285) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-11-13 23:51:05 -04:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00
Akash Kulkarni
ce415452cb [source-mongodb-v2] : Add support to read UUIDs (#32250) 2023-11-07 09:08:45 -08:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Rodi Reich Zilberman
f5c87968ca ✨URL encode username and password fields (#32028) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
 2023-11-01 15:22:27 -05:00
Rodi Reich Zilberman
62eeaee648 🐛 Allow mongodb-v2 discover operation to use temp disk file (#31629) 2023-10-19 18:49:32 -05:00
Rodi Reich Zilberman
1d4ee5b7b7 🐛 Allow disk use in case a sort goes over mem limit (#31596) 2023-10-19 14:20:28 -07:00
Jonathan Pearlin
5d6597e5b6 ✨ Source MongoDB V2: Fix nested object filtering (#31034) 2023-10-03 18:31:00 -05:00
Jonathan Pearlin
4c251c11b3 ✨ Source MongoDB V2: Fix formatting (#31031) 2023-10-03 18:13:12 -05:00
Jonathan Pearlin
26787a82e5 ✨ Source MongoDB V2: Support incremental syncs via Debezium (#29969) 
Co-authored-by: Cole Snodgrass <cole@airbyte.io>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
 2023-10-03 12:37:10 -05:00
Jonathan Pearlin
9ca8cdece6 🐛 Source Mongo: Revert recent changes (#28815) 
* revert changes

* revert changes

* revert changes

* revert changes

* revert changes

* Prepare release

* Add pull request to release notes

* Automated Commit - Formatting Changes

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-07-27 14:47:47 -05:00
Cole Snodgrass
0622860baf 🐛 Source Mongo: fix potential npe when collecting stats (#28760) 
* avoid NPE in getCollectionStats

* remove accidental whitespace

* update mongo changelog
 2023-07-27 08:18:58 -05:00
Jonathan Pearlin
3eb222fc89 ✨ Source MongoDB V2: Clean fieldPath before use in projection (#28733) 
* Clean fieldPath before use in projection

* Update release notes

* Automated Commit - Format and Process Resources Changes

* Remove prefix $ from field name

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-07-26 12:11:46 -05:00
Jonathan Pearlin
28bedf9675 ✨ Source MongoDB V2: Handle statistics collection failure gracefully (#28692) 
* Fix bug introduced by statistics collection

* Fix syntax

* Formatting

* Fix failing test

* Fix failing test

* Update release notes
 2023-07-25 15:22:02 -05:00
Jonathan Pearlin
496678df5f ✨ Source MongoDB V2: Record statistics about MongoDB source connection (#28527) 
* Record statistics about source database/collection

* Formatting

* Remove unnecessary log message

* Only record statistics during sync

* Emit structured Google log data

* Revert use of System.out.println

* Try using Log4j ObjectMessage

* Revert "Try using Log4j ObjectMessage"

This reverts commit d518c87aa8.

* Prepare release

* Update changelog

* Correct release version
 2023-07-24 12:41:40 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737) 
* Java DB Source connector licenses to Elv2

* Bump PR Ids
 2023-06-28 18:52:02 -07:00
Mickaël Andrieu
e38baf4997 Docs: highlight the requirement on MongoDB user access levels (#21462) 2023-01-17 07:34:02 +00:00
VitaliiMaltsev
84f02995ba MongoDb Source: Increase performance of discover (#17614) 
* MongoDb Source: Increase performance of discover

* bump version

* fixed tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-07 14:18:29 +03:00
Akash Kulkarni
4b990d0797 🚨 Add ability to enforce SSL in MongoDB source connector (#17590) 
* Check and test for if we are disabling SSL/TLS in source-mongodb-strict-encrypt

* Update docs

* Address comments

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-10-06 11:57:36 -07:00
VitaliiMaltsev
022b8db345 Destination: BigQuery (denormalized typed struct) - Could not evaluate union, field is expected to be one of these: NULL, STRING. (#16401) 
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Denormalized: Could not evaluate union on Mongo formatted type

* add test

* fixed BigQuery tests

* updated changelog

* replaced star import

* bump version

* auto-bump connector version [ci skip]

* fixed compilation error

* bump mongodb-strict-encrypt

* auto-bump connector version [ci skip]

* removed unused method

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-09-13 19:10:09 +03:00
mkhokh-33
5776dcc949 Mkhokh/8858 cursor fields for incremental update (#14356) 
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor

* added tests for cursor fields incremental updates

* remove not used methods

* fix tests

* fix formatting

* revert RedshiftInsertDestinationAcceptanceTest

* fix ssl clickhouse test

* fix codereview comments

* fix clickhouse strict encrypt test

* fix postgres source after merge from master

* optimize ssl clickhouse test logic

* fix db2 test

* added check for cursor field should be not nullable, fix formatting

* fixed tests, formatting, issue after merge from master

* fix merge issues

* fix test for source-jdbc and remove some code used for local testing

* added description for cockroach db tests

* test changes

* refactoring for cursor field tests after code review comments

* refactoring for cursor field tests after code review comments

* fix code review comments

* remove unused imports

* fix formatting

* fix compilation error

* fix postgres and postgress ssl test

* Fixed bucket naming for S3

* remove CHAR, NCHAR from possible cursor types

* remove SYMBOL from possible cursor types

* removed redundant configs

* fixed mysql-strict-encrypt tests

* fixed mongodb-v2 tests

* minor format changes

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-08-18 19:48:22 +03:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596) 
This reverts commit a2c194a11f.
 2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369) 
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
 2022-07-08 14:18:27 -05:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00