jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

57 Commits

Author SHA1 Message Date
Davin Chia
af8ce3aa6d chore: migrate mongo, mssql and oracle to new gradle flow. (#60311) 2025-05-27 17:12:40 -04:00
Yue Li
ad7aa7af4b [cdk, source-postgres, source-mysql, source-mongodb-v2] Fixing vulnerabilities in dependencies (#58132) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-04-24 08:28:30 -07:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Akash Kulkarni
b43b1111a8 [cdk] : More logging for count mismatch (#43410) 2024-08-09 05:45:27 +03:00
Xiaohan Song
97d822f820 [cdk, source-mysql/postgres/mssql/mongodb-v2] log record count to state message (#42869) 
Co-authored-by: Yue Li <yue.li@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-07 04:35:22 +03:00
Yue Li
8f42de9c37 [source-mongodb-v2] : Implement WASS algorithm (#42561) 
Fixes airbytehq/airbyte-internal-issues#8730

Port of #38240 for MongoDB
 2024-08-05 06:36:23 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Akash Kulkarni
94234e57c6 [Source-mongodb-v2] : Adopt new CDK (#39530) 2024-06-17 13:57:57 -07:00
Xiaohan Song
3c9a6121c4 Upgrade mongodb to use dbz 2.6.2 (#38238) 2024-06-11 13:25:24 -04:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Akash Kulkarni
ddc3d2b527 [Source-mongo] : Add options for CAPTURE_MODE (#36851) 2024-05-06 19:06:27 -04:00
Rodi Reich Zilberman
bdb3556b8a bump connectors to latest CDK version (#37781) 2024-05-02 16:52:11 -04:00
Akash Kulkarni
914f044354 [Certified sources] : Bump cdk versions to disable DBZ retries (#37742) 2024-05-01 15:07:37 -04:00
Akash Kulkarni
70afb9d2c7 [Source-mongodb] : Populate null values in airbyte record message (#37348) 2024-04-16 13:41:21 -07:00
Akash Kulkarni
4a020a4078 [Source-mongo] : Migrate to Kotlin (#36845) 2024-04-04 20:29:23 -04:00
Akash Kulkarni
deb47ab707 [source-mongo] : Adopt CDK 0.28.0 (#36837) 2024-04-04 15:50:36 -07:00
Marius Posta
ef473c5338 gradle: add kotlin support in all projects, warnings as errors (#36018) 2024-03-13 10:56:17 -05:00
Xiaohan Song
d5e91ae83a [source-mongodb] record count in state & initial iterator refactor (#35669) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Co-authored-by: Tim Roes <tim@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Anton Karpets <anton.karpets@globallogic.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Tyler B <104733644+tybernstein@users.noreply.github.com>
Co-authored-by: bgroff <bgroff@users.noreply.github.com>
Co-authored-by: mjgatz <86885812+mjgatz@users.noreply.github.com>
Co-authored-by: mgreene <michael.greene@gravie.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Joe Reuter <joe@airbyte.io>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Roman Yermilov [GL] <86300758+roman-yermilov-gl@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: girarda <girarda@users.noreply.github.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: brianjlai <brianjlai@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: midavadim <midavadim@yahoo.com>
Co-authored-by: Julien COUTAND <julien.coutand@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.io>
Co-authored-by: Bindi Pankhudi <bindi@airbyte.com>
Co-authored-by: Ben Drucker <bvdrucker@gmail.com>
Co-authored-by: TornadoContre <37258495+TornadoContre@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Thomas Dippel <dipth@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Alex Birdsall <ambirdsall@gmail.com>
Co-authored-by: ambirdsall <ambirdsall@users.noreply.github.com>
Co-authored-by: Jose Gerardo Pineda <jose.pineda@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Pedro S. Lopez <pedroslopez@me.com>
Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
Co-authored-by: Siarhei Ivanou <sinusu@gmail.com>
Co-authored-by: Anatolii Yatsuk <tolikyatsuk@gmail.com>
Co-authored-by: Ryan Waskewich <156025126+rwask@users.noreply.github.com>
Co-authored-by: Sajarin <sajarindider@gmail.com>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: perangel <perangel@gmail.com>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
Co-authored-by: Obioma Anomnachi <onanomnachi@gmail.com>
Co-authored-by: maxi297 <maxi297@users.noreply.github.com>
Co-authored-by: SatishChGit <satishchinthanippu@gmail.com>
Co-authored-by: Brian Leonard <brian@bleonard.com>
Co-authored-by: David Wallace <dwallace0723@gmail.com>
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
Co-authored-by: Stephane Geneix <stephane@airbyte.io>
Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com>
Co-authored-by: Danny Tiesling <tiesling@gmail.com>
 2024-03-06 14:29:42 -08:00
Akash Kulkarni
5bab3df367 Add airbyte trace utility to emit analytics messages & emit messages for MongoDB, Postgres & MySQL (#35036) 2024-02-13 19:10:38 -04:00
Marius Posta
b9ab3d1a90 migrate everything from java 17 to java 21 (#35103) 2024-02-09 19:26:19 -06:00
Marius Posta
5435df6d95 source-mongodb-v2: adopt cleaned-up cdk (#34748) 2024-02-09 11:38:55 -08:00
Rodi Reich Zilberman
eb5b23d669 Build a resume token with a pipeline consisting of selected streams (#34594) 
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
 2024-01-31 09:48:12 -08:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Marius Posta
482bf97086 source-mongodb-v2: remove dependency on airbyte-commons source (#34455) 2024-01-24 10:30:49 -08:00
Akash Kulkarni
25a0c80547 [Source-mongodb-v2] : Adding logs for resume token timestamps + reducing min document discovery size further (#34364) 2024-01-18 16:38:57 -08:00
Akash Kulkarni
279dd98e9a [Source-mongo] : Op log logging (#33549) 2023-12-18 17:40:04 -04:00
Subodh Kant Chaturvedi
c1c2a31920 mongo-db-source: upgrade to latest CDK (#33438) 2023-12-19 00:24:24 +05:30
Rodi Reich Zilberman
aee153a24a mongodb schema-less mode (#32328) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-14 20:05:16 -04:00
Akash Kulkarni
4c9908496b Add debugger drivers for MongoDB & MySQL (#33356) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-12 20:07:46 -04:00
Stephane Geneix
c454e9723f remove debezium-bundles from deps.toml (#33267) 
in deps.toml, we define a debezium-bundle which include all the debezium dependencies for all databases. That's being used in several subprojects, including CDK, which means CDK also depends on specific databases (mysql, postgres, mongo, mssql). As I want to remove that dependency in the future, this is preparing this by doing a no-op except for refactoring and clarifying dependencies. Note that mongo used to import debezium-bundle, and rather than risking a regression, I'm still importing all corresponding debezium libraries there.
The CDK still has a bunch of uses of DB-specific debezium libraries, so I left those there too.

I haven't bumped the CDK version nor the mongo connector version because this is a no-op. I will do that if the CI yells at me or if anyone feels strongly about it
 2023-12-11 12:08:46 -08:00
Akash Kulkarni
e2e0aba489 [source-mongodb-v2] : Fix support to read UUIDs (#32285) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-11-13 23:51:05 -04:00
Akash Kulkarni
ce415452cb [source-mongodb-v2] : Add support to read UUIDs (#32250) 2023-11-07 09:08:45 -08:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Marius Posta
53ba826712 source-mongodb-v2: use CDK 0.4.0 (#32176) 2023-11-03 14:31:53 -07:00
Rodi Reich Zilberman
f5c87968ca ✨URL encode username and password fields (#32028) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
 2023-11-01 15:22:27 -05:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Jonathan Pearlin
60b0c9d023 Set the user timezone for tests (#31057) 
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-04 16:29:41 -05:00
Jonathan Pearlin
26787a82e5 ✨ Source MongoDB V2: Support incremental syncs via Debezium (#29969) 
Co-authored-by: Cole Snodgrass <cole@airbyte.io>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
 2023-10-03 12:37:10 -05:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383) 
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
 2023-09-27 15:46:02 -07:00
Ben Church
5c56ac1d84 Airbyte-ci: Remove gradle task connectorAcceptanceTest (#30326) 2023-09-19 15:16:37 -05:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Marius Posta
51c67d7eaa gradle: remove airbyteDocker.outputs dependencies (#30314) 2023-09-11 17:16:27 -07:00
Jonathan Pearlin
9ca8cdece6 🐛 Source Mongo: Revert recent changes (#28815) 
* revert changes

* revert changes

* revert changes

* revert changes

* revert changes

* Prepare release

* Add pull request to release notes

* Automated Commit - Formatting Changes

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-07-27 14:47:47 -05:00
Jonathan Pearlin
b017ee67f9 Test MongoDB Replica Set/Oplog via Testcontainers (#28400) 
* Update dependencies

* Use RawBsonDocument

* Add tests

* Formatting

* Incremental refresh test

* Formatting

* Fix NPE

* Test replica set/oplog via testcontainers

* Formatting

* PR feedback

* Automated Commit - Formatting Changes

* Add logging to debug integration test failures

* Fix failing test

* Remove credentials conversion logic

* Update documentation to include correct config format

* Use database from credentials

* Make expected catalog match test data

* Fix formatting

* Cleanup

* Improve logging

* Revert test catalog change

* Use test database

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
 2023-07-20 11:27:07 -05:00
Peter Hu
e5eac0a4cc use published protocol models jar (#22498) 
* add airbyte-protocol to deps.toml

* use published protocol jar for platform

* use published protocol jar for connectors

* point at published jar

* fix dep

* bump gcs storage

* fix build failures in standard-source-test

* fix deps

* downgrade alloy db because it is missing strictness tests

* Revert "downgrade alloy db because it is missing strictness tests"

This reverts commit cc6089d053.

---------

Co-authored-by: cgardens <charles@airbyte.io>
 2023-02-13 12:50:43 -06:00
Joe Reuter
6a10ae3e05 Rename source acceptance test to connector acceptance test (#21846) 
Rename source acceptance test to connector acceptance test
 2023-02-02 11:38:19 +01:00
Andrii Leonets
58e4ef6912 Enable SPEC SAT for Java sources (#18779) 
* enable py spec compatibility tests

* add missing properties

* use expected spec file instead of the source spec.json + use dummy config file

* add missing files for the clickhouse

* move test files to the integration-test from the unit test folder

* add missing files to cockroachdb

* add missing files to db2 + fix spec format

* add missing files to elasticsearch + fix spec format

* add missing files to jdbc

* add missing files to mongodb_v2 + fix spec format

* add missing files to mssql + fix spec format

* add missing files to mysql + fix spec format

* add missing files to postgres + fix spec format

* add missing files to oracle + fix spec format

* add missing files to redshift

* add missing files to sftp

* add missing files to snowflake + fix spec format

* add missing files to tidb

* add missing files to kafka - fix spec format

* airbyte-source-acceptance-test added

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* add missing import

* Delete acceptance-test-config.yml

* Delete acceptance-test-docker.sh

* Update build.gradle

* Update build.gradle

* format

* revert changes

* manual .sh files format

* upd expected spec

* format

* fix SAT after master merge

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
 2022-11-09 21:45:46 +02:00
Jonathan Pearlin
2b31011bce Separate platform and connector testcontainer versions (#13642) 
* Separate platform and connector testcontainer versions

* Fix dependency

* Fix dependency

* Fix dependency usage

* Prevent leaking testcontainer dependencies
 2022-06-10 09:34:31 -04:00
Davin Chia
83a89aa843 Fat Jar: Rename Dir Part 1 (#13476) 
Part 1 of #13122.

Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.

Explanation for what is happening:

Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
 2022-06-06 00:35:43 +08:00