jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 18:39:31 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

317 Commits

Author SHA1 Message Date
sophiecuiy
4846a52f60 add table filtering to mysql (#69228) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-11-14 10:35:55 -08:00
Rodi Reich Zilberman
55bb5d8dbe Set lower bound included flag in snapshot partitions (#69177) 2025-11-05 15:19:25 -08:00
Rodi Reich Zilberman
e8721949a8 9851 partition mysql source cdc syncs and refreshes are both taking a very long time (#69104) 2025-11-03 16:35:15 -08:00
Rodi Reich Zilberman
4afd43d9a6 Mysql extract proto temporal representation (#68094) 
Co-authored-by: subodhchaturvedi <subodh1810@gmail.com>
 2025-10-21 10:24:13 -07:00
Rodi Reich Zilberman
79b0c93291 14652 source mysql cdc meta fields encoded incorrectly to protobuf (#67151) 
## What
Among other CDC_* fields, CDC_UPDATED_AT and CDC_DELETED_AT fields are
sent as part of CDC snapshot and incremental records.
These two are unique because their catalog definition
(AirbyteSchemaType) is a String, but the values they hold are date time.
Json encoding relied on the fact that all dates encode to string in json
but in protobuf we encoded these values using
`.setTimestampWithTimezone` which prevented destination from pulling the
value out of protobuf. Destination relies on the catalog type to know
how to decode messages.
We need to ensure that these fields are encoded to protobuf using
`.setString` instead in order to match the catalog type definition.


#### Note: The CDK portion of this PR was separated to
https://github.com/airbytehq/airbyte/pull/67152. Once the CDK PR is
merged this PR will only include connector changes.

*** Please ignore the CDK changes on this PR ***

## How
Updated CDC Meta field type to support a unique type that accepts an
`OffsetDateTime` but encodes the value in protobuf as a string.

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
Users who previously used Mysql CDC in speed mode will need to refresh
those streams.

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [x] YES 💚
- [ ] NO ❌


Resolves https://github.com/airbytehq/airbyte/issues/66735
 2025-10-07 13:12:12 -07:00
Yarden Carmeli
63d68d7cf6 Source-MySQL: Prevent division by zero when split boundaries are empty (#66515) 2025-09-24 09:03:13 -07:00
Jim Ruppert
a61f616127 moved db version logging from mysql connector to CDK for all JDBC sou… (#64569) 
moved db version logging from mysql connector to CDK , will handle all JDBC sources
logs only during check operation
 2025-08-12 17:08:22 -07:00
Rodi Reich Zilberman
8ed6f3e14e CDC state fixes (#63377) 2025-07-30 18:01:08 -07:00
Jim Ruppert
b793d217e4 log of db version (#64134) 2025-07-30 17:23:12 -07:00
Rodi Reich Zilberman
8724d83401 Migrate concurrency setting (#63349) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-07-22 15:06:37 -07:00
Charles
50a97db804 chore: migrate connectors from MIT to ELv2 (#63723) 2025-07-22 14:26:58 -07:00
Rodi Reich Zilberman
4dc9bbce6e Rodi/cdc state socket mode (#62932) 2025-07-17 11:28:15 -07:00
Rodi Reich Zilberman
cbf42c3333 Socket mode in bulk extract CDK (#60993) 2025-07-07 12:54:08 -07:00
Rodi Reich Zilberman
cfc1243bd2 concurrent mysql read (#60218) 2025-05-29 13:38:46 -07:00
Rodi Reich Zilberman
a6a52d6429 Fix partition creation to avoid a loop when primary key is a string (#59732) 2025-05-09 11:29:41 -07:00
Rodi Reich Zilberman
dbd76645a3 CDK version bump (#59683) 2025-05-06 12:36:10 -07:00
Rodi Reich Zilberman
e97ed2611e Improve BigDecimal to json conversion to prevent scientific notation (#59223) 2025-05-06 10:35:31 -07:00
Matt Bayley
b486f268d2 Mitigate MySQL state with sci notation issue (#59150) 2025-04-29 23:13:22 -07:00
Yarden Carmeli
e4ddfdc4dd source-mysql: Update default ssl mode to required (#59144) 2025-04-29 12:46:25 -07:00
Matt Bayley
c21c460301 fix(cdc) no streams means include no tables (#55891) 2025-03-20 20:36:39 -07:00
Przemysław Dąbek
7c51ec118d [source-mysql] More meaningful errors for variables used in extra checks for CDC (#55731) 2025-03-14 13:09:42 -03:00
Yue Li
c26d2ae755 [source-mysql] don't do sampling for source-mysql (#55761) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-03-14 00:33:22 +00:00
Wenqi Hu
ae6db5ad2c Fix reversed assertions in MySQL source tests (#55756) 2025-03-13 13:28:54 -07:00
Matt Bayley
e52d188761 pass streams to debezium sources on cold start (#55734) 2025-03-13 10:58:37 -07:00
Matt Bayley
43858a9b08 [source-mysql] Fix fetching binlog status for version >=8.4 (#55237) 
Co-authored-by: matt.bayley <matt.bayley@matt.bayley--MacBook-Pro---G4WV0GW6J3>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yue.li@airbyte.io <yue.li@airbyte.io>
 2025-03-06 15:01:24 -08:00
Yarden Carmeli
d74e8a5794 [source-mysql] Set 'required' as the default encryption method in spec. (#55214) 2025-03-05 17:12:31 -08:00
Yarden Carmeli
eeda92acf3 [source-mysql] Improve Error Handling and Regex Detection (#54156) 2025-02-24 09:32:55 -08:00
Wenqi Hu
fc80e61358 [source mysql] Fix integration test (#54183) 2025-02-20 12:36:11 -08:00
Wenqi Hu
f0a96b3c1c [source volcano] Decorate stream with cursor field for trigger based CDC stream (#53152) 2025-02-12 22:31:22 +02:00
Yue Li
c288e1276a [source-mysql] fix typo in CDC configruation (#53195) 2025-02-06 20:09:30 +02:00
Marius Posta
8c2ed63cf4 source-mysql: adopt bulk-cdk-toolkit-extract-cdc API changes (#52039) 2025-01-30 23:06:08 +02:00
Rodi Reich Zilberman
b3f56fb349 Set cdc metadata fields types to match legacy connectors (#51029) 2025-01-13 22:24:18 +02:00
Marius Posta
ac054a641e source-mysql: fix converter bug (#50965) 2025-01-07 22:27:17 +02:00
Rodi Reich Zilberman
14aee8b1f5 Make sample query compatible with views (#50437) 
Co-authored-by: Marius Posta <marius@airbyte.io>
 2025-01-07 21:57:08 +02:00
Marius Posta
e01e6a21fa source-mysql: fix indentation level for rules defs (#50868) 2025-01-03 15:23:21 -08:00
Marius Posta
1d6b34cfba source-mysql: datatype fixes (#49918) 2024-12-20 19:44:03 +02:00
Marius Posta
c5a12bada2 source-mysql: clean up config (#49950) 2024-12-20 09:44:54 -05:00
Marius Posta
2910475443 source-mysql: simplify MySqlSelectQuerier (#49948) 2024-12-20 09:24:46 -05:00
Marius Posta
cc383cf360 source-mysql: normalize class names (#49939) 2024-12-19 15:44:33 -05:00
Rodi Reich Zilberman
8819bd143e [source-mysql] 7084 source mysql regression in 39x date parsing errors (#49932) 2024-12-19 17:26:10 +00:00
Marius Posta
426cabccd9 bulk-cdk: fix loss of numerical precision when deserializing JSON (#49920) 2024-12-18 15:43:20 -05:00
Marius Posta
b9ea48e137 source-mysql: better datatype integration tests (#49844) 2024-12-18 02:29:28 +02:00
Rodi Reich Zilberman
dfcdea8ade Send a correct fallback value for autogenerated boolean (#49830) 2024-12-17 06:36:51 -08:00
Rodi Reich Zilberman
61c57771c9 CDC partition reader should be allowed to complete its work with no t… (#48889) 2024-12-11 13:33:44 -08:00
Rodi Reich Zilberman
29a9e986b5 Set fetch size on a jdbc resultset rather than on a statement (#48840) 2024-12-09 08:23:43 -08:00
Rodi Reich Zilberman
c7f0eb7004 Nullify failed column conversion and mark with meta change in record (#48791) 2024-12-04 14:55:00 -08:00
Yarden Carmeli
cad4c1d923 Implement New Error Handler Based on RegexExceptionClassifierRule and JdbcExceptionClassifierRule (#48798) 
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
 2024-12-04 14:42:03 -08:00
Rodi Reich Zilberman
5f9d290800 Handle empty table states (#48629) 2024-12-03 01:10:23 +02:00
Rodi Reich Zilberman
552b952d5f Read schema history with a clean instance of FileSchemaHistory to pre… (#48754) 2024-12-02 02:51:27 -08:00
Xiaohan Song
b8faac7ba6 [source-mysql] do not check binlog if we have gtid validated (#48688) 2024-11-25 13:57:25 -08:00