jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-21 02:51:29 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

317 Commits

Author SHA1 Message Date
Akash Kulkarni
ae39cbe82f Fix error message for saved offset not valid. (#35675) 2024-02-27 17:54:30 -04:00
Akash Kulkarni
40a5edc5fe [Source-mysql] : Add config to throw an error on invalid CDC position (#35338) 2024-02-20 14:09:05 -08:00
Marius Posta
e3ab5b6aab source-mysql: add and adopt TestDatabaseWithInvalidDatabaseName (#35210) 2024-02-15 09:30:48 -08:00
Akash Kulkarni
5bab3df367 Add airbyte trace utility to emit analytics messages & emit messages for MongoDB, Postgres & MySQL (#35036) 2024-02-13 19:10:38 -04:00
Xiaohan Song
82d9197587 🐛 source-mysql Support special chars in dbname (#34580) 2024-02-12 11:06:16 -08:00
Xiaohan Song
b85f9994f4 [Source-mysql] Add soft link in mysql test db (#35045) 2024-02-09 14:28:05 -08:00
Marius Posta
796c0ee494 source-mysql: adopt cleaned-up cdk (#34750) 2024-02-09 14:04:41 -06:00
Marius Posta
6f0fd7a2b9 refactor debezium harness in CDK, push connector-specific logic down to connectors (#34573) 2024-01-27 14:05:47 -06:00
Xiaohan Song
b290208391 Add count in state message for incremental syncs (#33005) 
Co-authored-by: xiaohansong <xiaohansong@users.noreply.github.com>
 2024-01-09 21:05:57 -04:00
Xiaohan Song
18e0e7741a ✨ Introduce StateIteratorProcessor in CDK (#33312) 2024-01-03 15:29:12 -08:00
Subodh Kant Chaturvedi
5b915c6837 mysql-source: remove LEGACY state flag (#33436) 2023-12-19 23:45:51 +05:30
Akash Kulkarni
4c9908496b Add debugger drivers for MongoDB & MySQL (#33356) 
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-12-12 20:07:46 -04:00
Rodi Reich Zilberman
f5ae28fc0c 🐛 Rodi/port zero date time behavior fix (#33210) 
Co-authored-by: Robert Tomczak <36033856+robertomczak@users.noreply.github.com>
Co-authored-by: robertomczak <robertomczak@users.noreply.github.com>
 2023-12-11 11:09:14 -08:00
Stephane Geneix
3cc5188e93 rename getImageModifiers into getContainerModifiers everywhere (#32958) 
simple refactor
 2023-12-07 20:08:47 -08:00
Stephane Geneix
d648ec5125 introduce ImageModifier to MySql source (#32900) 
introduce imageModifier to MySql so we don't use strings everywhere
 2023-12-07 15:50:38 -08:00
Stephane Geneix
0b8b11b3e5 introduce BaseImage to MySql source (#32899) 
introduce baseImage to MySql so we don't use strings everywhere
 2023-12-07 15:38:42 -08:00
Duy Nguyen
665e4f73ad source-mysql: merge strict encrypt variant into standard main mysql source (#31062) 
Co-authored-by: erohmensing <erohmensing@gmail.com>
Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
Co-authored-by: erohmensing <erohmensing@users.noreply.github.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: alafanechere <alafanechere@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
 2023-11-29 10:45:28 -04:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-11-27 12:49:06 -06:00
Stephane Geneix
49043b570f 🐛 📝 re-update all connectors for new CDK version and more stringent javac checks (#32125) 2023-11-08 18:21:14 -04:00
Stephane Geneix
9c6ed93a9d Revert "update all connectors for new CDK version and more stringent javac checks" 
This reverts commit 18ad87bbd1.
 2023-11-02 15:42:28 -07:00
Stephane Geneix
18ad87bbd1 update all connectors for new CDK version and more stringent javac checks 2023-11-02 15:24:58 -07:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960) 
Co-authored-by: postamar <postamar@users.noreply.github.com>
 2023-10-30 12:03:06 -07:00
Rodi Reich Zilberman
79080c88f5 Update tests to current Jackson lib behavior (#31197) 
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
 2023-10-11 08:23:50 -07:00
Aaron ("AJ") Steers
da32fc86bf 🚀 Launch Java CDK v0.1.0 (PR #30383) 
- Launch Java CDK `v0.1.0`. 🚀
- Remove `airbyte-docker` gradle plugin from java connectors.
- Add `airbyte-java-connector` gradle plugin.
- Remove `airbyte-commons` from global build.gradle.
- Add `/publish-java-cdk` slash command.
- Initial launch of `/airbyte-ci` slash command to test connectors.
 2023-09-27 15:46:02 -07:00
Akash Kulkarni
2097c0e4e4 Update MySQL streaming config (#30744) 2023-09-26 15:45:13 -07:00
Ben Church
5d8278900f Github Action: Add format.yml workflow (#30604) 
Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>
Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
 2023-09-21 18:11:41 -05:00
Duy Nguyen
11ad938152 [Source-MySQL] Enable Enhanced Standard sync with PK -> Cursor switch (#30556) 
Enable enhanced standard sync
 2023-09-21 13:38:05 -07:00
Duy Nguyen
f226503bd1 [Source-MySQL] Enhanced Standard sync with PK initial load -> Cursor based switch over (#30270) 2023-09-18 17:17:00 -05:00
Marius Posta
a5eb24e2fc bump testcontainers version to latest (#30493) 2023-09-18 03:50:43 -05:00
Duy Nguyen
fca0b6f62e [Source-MySQL] Correctly set Debezium MySQL connector database timezone property (#30333) 
Co-authored-by: subodh <subodh1810@gmail.com>
 2023-09-14 16:10:14 -05:00
Rodi Reich Zilberman
0936143f1e 🐛 Only take table size info for streams syncing via primary key (#30375) 
Co-authored-by: subodh <subodh1810@gmail.com>
 2023-09-13 14:35:10 -05:00
Subodh Kant Chaturvedi
6f0c4e2594 mysql-source-cdc: enable compression logic for schema history (#30308) 
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
 2023-09-13 13:14:01 -05:00
Subodh Kant Chaturvedi
5a4cc0d8b0 source-mysql-cdc: introduce logic to compress schema history if greater than 3MB (#30289) 
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
 2023-09-13 12:41:28 -05:00
Subodh Kant Chaturvedi
ad47884297 postgres+mysql sources: use constants to keep checkpointing behaviour same (#30197) 
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
 2023-09-12 02:43:23 -05:00
Akash Kulkarni
09df4f4875 Enable MySQL CDC + PK 100% (#30213) 2023-09-07 15:32:29 -05:00
Akash Kulkarni
86b4e03092 Add escape quoting to selecting max (pk_field) query (#30059) 2023-08-31 18:01:35 -05:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821) 
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
 2023-08-31 12:19:02 -05:00
Akash Kulkarni
da80408a53 Log storage engines used in MySQL (#30015) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
 2023-08-30 18:35:46 -05:00
Jonathan Pearlin
13ef239ccc ✨ Source MongoDB Internal POC: CDC Metadata Injector (#29730) 
Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Jose Pefaur <jose.pefaur@gmail.com>
Co-authored-by: subodh <subodh1810@gmail.com>
 2023-08-28 14:01:32 -05:00
Jonathan Pearlin
0724132a1a ✨ Source MongoDB Internal POC: Debezium Configuration Properties (#29727) 2023-08-28 08:44:39 -05:00
Marius Posta
f5c7c1c0b8 chore: get ./gradlew format to pass for the whole repo (same java style) (#29786) 2023-08-24 05:09:42 -05:00
Akash Kulkarni
cb3a4e0dc8 Initial load stopping condition (#29731) 2023-08-23 11:46:35 -05:00
Akash Kulkarni
e321b9e1ae source-mysql : Initial load. Handle composite pk case w/multiple repeated values (#29634) 
Co-authored-by: subodh <subodh1810@gmail.com>
 2023-08-22 16:12:00 -05:00
Duy Nguyen
b7dc462c4c [Source-MySQL]CDC: Enable frequent state emission during incremental runs (#29308) 2023-08-21 20:13:31 -05:00
Akash Kulkarni
2b18864f3b source-mysql : chunking queries impl (#29109) 2023-08-18 14:26:40 -05:00
Duy Nguyen
6ee426a261 [Source-Mysql] Set default cursor value for cdc mode (#28756) 2023-08-17 13:13:39 -05:00
Subodh Kant Chaturvedi
cc85da9a78 db-sources:sort primary key based on key-seq (#29346) 
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
 2023-08-15 17:47:11 -05:00
Lake Mossman
cb863bb552 🎨 Source MySQL: Add 'Recommended' before CDC description (#29220) 
* add 'Recommended' before CDC description

* bump version
 2023-08-08 16:58:50 -05:00
Subodh Kant Chaturvedi
c9d6eb7833 mysql-cdc:implementation to validate that saved offset still exists on server (#29013) 
* mysql-cdc:implementation to validate that saved offset still exists on server

* Automated Commit - Format and Process Resources Changes

* add the check to the main code flow + test

* address review comments

---------

Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
 2023-08-08 12:43:46 -05:00
Akash Kulkarni
807f104414 source-mysql : CDC PK snapshotting (#28757) 
* Initial Commit of CDC PK snapshotting

* Added unit tests

* Flip to true

* Refactor flags

* Re-add flag

* Comment cleanup

* Code review comments

* Automated Commit - Format and Process Resources Changes

* minor refactor

* Address comments

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
 2023-08-04 13:57:51 -05:00