1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

98 Commits

Author SHA1 Message Date
Alexandre Cuoci
f9bfd6df50 MongoDB Oplog Troubleshooting (#34122) 2024-01-10 13:56:58 -05:00
Rodi Reich Zilberman
ca33d3099c Update doc with schema less mode (#33550) 2023-12-19 14:41:41 -08:00
Akash Kulkarni
279dd98e9a [Source-mongo] : Op log logging (#33549) 2023-12-18 17:40:04 -04:00
Rodi Reich Zilberman
743ab29b36 doc update to reflect minimum supported atlas tier. (#32186) 2023-12-18 11:54:47 -08:00
Subodh Kant Chaturvedi
c1c2a31920 mongo-db-source: upgrade to latest CDK (#33438) 2023-12-19 00:24:24 +05:30
Rodi Reich Zilberman
aee153a24a mongodb schema-less mode (#32328)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-12-14 20:05:16 -04:00
Akash Kulkarni
f2e0585508 [Source-MongoDb] : Better logging for MongoDB (#33430) 2023-12-13 10:13:48 -08:00
Akash Kulkarni
4c9908496b Add debugger drivers for MongoDB & MySQL (#33356)
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-12-12 20:07:46 -04:00
Akash Kulkarni
1a09927d3e 🐛 Source MongoDB : Improve our discovery phase to not create large BSON objects (#32886)
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-11-30 10:49:15 -04:00
Akash Kulkarni
e2e0aba489 [source-mongodb-v2] : Fix support to read UUIDs (#32285)
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-11-13 23:51:05 -04:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00
Akash Kulkarni
ce415452cb [source-mongodb-v2] : Add support to read UUIDs (#32250) 2023-11-07 09:08:45 -08:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Rodi Reich Zilberman
f5c87968ca URL encode username and password fields (#32028)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-11-01 15:22:27 -05:00
Rodi Reich Zilberman
62eeaee648 🐛 Allow mongodb-v2 discover operation to use temp disk file (#31629) 2023-10-19 18:49:32 -05:00
Rodi Reich Zilberman
1d4ee5b7b7 🐛 Allow disk use in case a sort goes over mem limit (#31596) 2023-10-19 14:20:28 -07:00
Jonathan Pearlin
5d6597e5b6 Source MongoDB V2: Fix nested object filtering (#31034) 2023-10-03 18:31:00 -05:00
Jonathan Pearlin
4c251c11b3 Source MongoDB V2: Fix formatting (#31031) 2023-10-03 18:13:12 -05:00
Jonathan Pearlin
26787a82e5 Source MongoDB V2: Support incremental syncs via Debezium (#29969)
Co-authored-by: Cole Snodgrass <cole@airbyte.io>
Co-authored-by: colesnodgrass <colesnodgrass@users.noreply.github.com>
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
2023-10-03 12:37:10 -05:00
Jonathan Pearlin
9ca8cdece6 🐛 Source Mongo: Revert recent changes (#28815)
* revert changes

* revert changes

* revert changes

* revert changes

* revert changes

* Prepare release

* Add pull request to release notes

* Automated Commit - Formatting Changes

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-27 14:47:47 -05:00
Cole Snodgrass
0622860baf 🐛 Source Mongo: fix potential npe when collecting stats (#28760)
* avoid NPE in getCollectionStats

* remove accidental whitespace

* update mongo changelog
2023-07-27 08:18:58 -05:00
Jonathan Pearlin
3eb222fc89 Source MongoDB V2: Clean fieldPath before use in projection (#28733)
* Clean fieldPath before use in projection

* Update release notes

* Automated Commit - Format and Process Resources Changes

* Remove prefix $ from field name

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
2023-07-26 12:11:46 -05:00
Jonathan Pearlin
28bedf9675 Source MongoDB V2: Handle statistics collection failure gracefully (#28692)
* Fix bug introduced by statistics collection

* Fix syntax

* Formatting

* Fix failing test

* Fix failing test

* Update release notes
2023-07-25 15:22:02 -05:00
Jonathan Pearlin
496678df5f Source MongoDB V2: Record statistics about MongoDB source connection (#28527)
* Record statistics about source database/collection

* Formatting

* Remove unnecessary log message

* Only record statistics during sync

* Emit structured Google log data

* Revert use of System.out.println

* Try using Log4j ObjectMessage

* Revert "Try using Log4j ObjectMessage"

This reverts commit d518c87aa8.

* Prepare release

* Update changelog

* Correct release version
2023-07-24 12:41:40 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Mickaël Andrieu
e38baf4997 Docs: highlight the requirement on MongoDB user access levels (#21462) 2023-01-17 07:34:02 +00:00
VitaliiMaltsev
84f02995ba MongoDb Source: Increase performance of discover (#17614)
* MongoDb Source: Increase performance of discover

* bump version

* fixed tests

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-07 14:18:29 +03:00
Akash Kulkarni
4b990d0797 🚨 Add ability to enforce SSL in MongoDB source connector (#17590)
* Check and test for if we are disabling SSL/TLS in source-mongodb-strict-encrypt

* Update docs

* Address comments

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-06 11:57:36 -07:00
VitaliiMaltsev
022b8db345 Destination: BigQuery (denormalized typed struct) - Could not evaluate union, field is expected to be one of these: NULL, STRING. (#16401)
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Denormalized: Could not evaluate union on Mongo formatted type

* add test

* fixed BigQuery tests

* updated changelog

* replaced star import

* bump version

* auto-bump connector version [ci skip]

* fixed compilation error

* bump mongodb-strict-encrypt

* auto-bump connector version [ci skip]

* removed unused method

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-13 19:10:09 +03:00
mkhokh-33
5776dcc949 Mkhokh/8858 cursor fields for incremental update (#14356)
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor

* added tests for cursor fields incremental updates

* remove not used methods

* fix tests

* fix formatting

* revert RedshiftInsertDestinationAcceptanceTest

* fix ssl clickhouse test

* fix codereview comments

* fix clickhouse strict encrypt test

* fix postgres source after merge from master

* optimize ssl clickhouse test logic

* fix db2 test

* added check for cursor field should be not nullable, fix formatting

* fixed tests, formatting, issue after merge from master

* fix merge issues

* fix test for source-jdbc and remove some code used for local testing

* added description for cockroach db tests

* test changes

* refactoring for cursor field tests after code review comments

* refactoring for cursor field tests after code review comments

* fix code review comments

* remove unused imports

* fix formatting

* fix compilation error

* fix postgres and postgress ssl test

* Fixed bucket naming for S3

* remove CHAR, NCHAR from possible cursor types

* remove SYMBOL from possible cursor types

* removed redundant configs

* fixed mysql-strict-encrypt tests

* fixed mongodb-v2 tests

* minor format changes

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: vmaltsev <ihbectop@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-18 19:48:22 +03:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
Yurii Bidiuk
b3194b2200 🎉🐛: Source mongoDB: implement building JsonSchema with 'properties' for fields with type 'object' (#12428)
* mongodb: build JsonSchema with 'properties'

* add tests

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-05 12:56:07 +03:00
Nahid
fbad2b8c06 Source MongoDb: fix parsing of DBRef objects (#10276) 2022-02-22 18:51:46 -03:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
VitaliiMaltsev
2b0d0bdef6 Source MongoDB fetch authorized collections only (#9238)
* fix for jdk 17

* Source MongoDB show authorized collections

* add javadoc

* fixed checkstyle

* add CHANGELOG

* fix checkstyle

* refactoring

* bump version anf fix checkstyle

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
2022-01-13 15:41:34 +02:00
Iryna Grankova
b8c33bdfed Update fields in source-connectors specifications: mongodb-strict-encrypt, mongodb-v2 (#9202) 2021-12-30 17:30:26 +02:00
Joël Luijmes
ad55ab01d1 MongoDB limit 10k records during Discovery(#8491)
Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
2021-12-07 16:27:40 -03:00
andriikorotkov
9e087c6f9c 🐛 Source MongoDB - Added Milliseconds to Cursor for mongo date type (#8306)
* Added milliseconds for date format for cursor

* Added milliseconds for date format for cursor

* Added milliseconds for date format for cursor

* updated documentation
2021-11-29 18:36:49 +02:00
andriikorotkov
6bf9ebacb2 🐛 Updated source-mondodb-v2 performance and updated cursor for timestamp type (#8161)
* updated source-mongodb-v2 performance

* updated code style

* fixed remarks

* fixed remarks

* fixed remarks

* updated strict encrypt source mongodb version

* updated source mongodb work with empty collections

* updated source mongodb timestamp cursor

* updated mongodb source perfomance

* fix code style

* fix code style

* updated tests and documentation

* updated tests and documentation

* updated tests and documentation

* added vudangngoc changes

* updated code style

* updated code style
2021-11-23 13:56:25 +02:00
andriikorotkov
46e1f65905 🐛 Source MongoDB - Added milliseconds to convert timestamp to datetime format (#8046)
* Added milliseconds to convert timestamp to datetime format

* updated docs

* updated docs
2021-11-18 14:30:15 +02:00
andriikorotkov
8e0e0b4f79 🐛 Updated source-mongodb-v2 performance (#7982)
* updated source-mongodb-v2 performance

* updated code style

* fixed remarks

* fixed remarks

* fixed remarks

* updated strict encrypt source mongodb version
2021-11-18 12:05:58 +02:00
irynakruk
dbfd6cae88 Source MongoDB v2: Fixed nested document parsing (#7160)
* Fixed nested document parsing: added parsing of nested documents and arrays, added data types test
2021-10-25 09:16:34 -04:00
irynakruk
22c3a9e043 🐛 MongoDb Source: Added filter to avoid MongoDb system collections (#6860)
* Added filter to avoid MongoDb system collections
2021-10-11 13:41:24 -04:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
irynakruk
5551698002 Source MongoDb: added support via TLS/SSL (#6364)
* updated ssl configs;
fixed replica connection;
fixed incremental read;

* added MongoDbSourceAtlasAcceptanceTest with ssl enabled

* updated docs, moved TLS option for standalone instance type, enabled it by default for other types

* Update mongodb-v2.md

* updated README.md

* updated spec.json

* Code review changes

* updated ci_credentials.sh
2021-09-24 09:22:54 -04:00
irynakruk
ffecc1c9a2 🎉 New source: MongoDb ported to java (#5530)
* New abstraction for NoSql database sources

* New MongoDbSource: partial impl

* Added MongoDataType

* Improved MongoDatabase and fixed read method

* code review changes;

* merge clean up;

* Renamed NoSqlDatabase to AbstractDatabase

* formatter changes;

* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs

* code review changes: changed mongodb-new to mongodb-v2; left only new connector info in all docs

* updated spec.json and toDatabaseConfig() method

* updated doc accordingly to spec.json changes

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-09-09 08:14:16 -04:00