jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-18 06:04:45 -05:00
Code Issues Packages Projects Releases Wiki Activity
18,317 Commits 5,337 Branches 518 Tags
ed271dba0737e910e04c19cd6972a4b748b68e15
Commit Graph

351 Commits

Author SHA1 Message Date
Edward Gao
355ac15a2e Destination Redshift: Implement refreshes (#40567) 2024-07-12 14:23:04 -04:00
Marius Posta
10a261489c java-cdk: better debezium logging (#41212) 
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
 2024-07-11 12:20:42 -07:00
Akash Kulkarni
310e6bd315 [Source-mysql] : Implement WASS algo (#38240) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-11 11:54:54 -04:00
Edward Gao
b2e7895ed3 Destination Bigquery+Snowflake: Fix resuming truncate refresh (#41041) 2024-07-08 21:37:52 -04:00
Xiaohan Song
acb9ce1f86 [source-postgres] Fix duplicate streams in postgres (#40719) 2024-07-05 12:05:52 -07:00
Gireesh Sreepathi
127c5015cd Destination Snowflake: Storage ops to support refreshes (#39473) 2024-07-01 16:54:50 -07:00
Edward Gao
779d363b35 CDK: Destinations: Backport CDK fixes for redshift (#40499) 2024-07-01 13:26:12 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Edward Gao
d7cadc4ccf CDK: Destinations: Backport to 0.35.15 (#40517) 
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
 2024-06-26 16:26:52 -07:00
Johnny Schmidt
a50847af39 Fail without enqueueing iff the airbyte message type is unrecognized (#40254) 2024-06-26 11:06:01 -07:00
Edward Gao
9458fd22fb Destination bigquery: Throw INCOMPLETE stream status as TRANSIENT error (#39526) 2024-06-18 16:03:13 -04:00
Edward Gao
1955c7c096 Destinations: throw error on empty catalog (#39552) 2024-06-18 09:58:59 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Edward Gao
cb6f6ec40a Destinations CDK: refreshes logic (#38622) 2024-06-17 08:59:49 -07:00
Edward Gao
2432cc8d01 Destinations: Refreshes: CDK updates (#38067) 2024-06-17 08:45:22 -07:00
Evan Tahler
b88a41d339 postgres source/destination - bump postgres driver version (#39460) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-06-14 15:35:17 -04:00
Akash Kulkarni
0759c84886 [WASS] : Create a transient error emitter (#39445) 2024-06-13 17:27:57 -04:00
Akash Kulkarni
fcc2940d20 [DB sources] : Debezium properties accept list of streams (#39405) 2024-06-13 15:11:10 -04:00
Yue Li
7508ffb2b8 CDK: translate java EOF exception's message to a more customer-friendly message (#39417) 
fixes airbytehq/oncall#4283
 2024-06-13 04:48:36 -07:00
Xiaohan Song
3c9a6121c4 Upgrade mongodb to use dbz 2.6.2 (#38238) 2024-06-11 13:25:24 -04:00
Edward Gao
05fd09b014 Destinations: Refreshes: Track stream statuses in async framework (#38075) 2024-06-10 10:01:50 -07:00
Edward Gao
a78647eb59 Destinations CDK: CatalogParser sets default namespace (#38121) 2024-06-10 09:40:30 -07:00
Edward Gao
c30119b641 Destination CDK: Increase json deserialization limit (#38763) 2024-06-07 22:50:23 +00:00
Yue Li
fd679fc07f cdk: handle null messages in ConnectorExceptionUtil (#39220) 
fixes airbytehq/airbyte-internal-issues#7974
 2024-06-06 10:33:21 -07:00
oyeliseiev-ua
76573f026c 🎉 New Source: SingleStore (#37337) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2024-06-06 09:35:50 -03:00
Gireesh Sreepathi
3d8f7ca602 Destination CDK: Skip flush of zero byte records. (#39106) 2024-06-05 17:09:21 +00:00
Yue Li
9a1663a251 Publish CDK for postgres source connector (#38834) 2024-06-03 14:57:00 -07:00
Yue Li
06b1b55228 CDK: throw config exception if no selectable table exists in user provided schemas (#38792) 
fixes airbytehq/airbyte-internal-issues#2252

"When we don't have permission to access the schema, the check succeeds, but refreshing the schema returns no tables. We should probably throw an error if the user configures the source schema as, e.g., public but doesn't grant our user permission to access that schema."

This patch includes a check that will throw a configuration error if a user does not have selectable tables from any of the provided schemas (in UI).If schemas actually contain no table, we would let the user proceed.
 2024-05-31 13:43:51 -07:00
Gireesh Sreepathi
492fdd94b5 Cdk forked branch (#38824) 2024-05-31 12:18:21 -07:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Akash Kulkarni
a2bcd4904c [DB sources] : Exit when encountering a config error (#38538) 2024-05-29 17:44:21 -04:00
Xiaohan Song
c63d40998a Publish cdk for postgres (#38771) 2024-05-29 13:47:23 -07:00
Xiaohan Song
b104667a0c Stream status on postgres (#38716) 
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
 2024-05-29 13:05:20 -07:00
Edward Gao
f368646da1 Destinations CDK: generation_id/sync_id plumbing (#38358) 2024-05-29 18:26:15 +00:00
Yue Li
f2560a850f CDK: make postgres temp_file_limit error a transient error (#38734) 
Fixes https://github.com/airbytehq/airbyte/issues/27090

Postgres throws a temp_file_limit error is translated to a system error for Airbyte. I believe this is true for both Postgres source and destination connectors.

This change makes this error transient so that it won't trigger sentry. It also fixes a bug that should use a lower-case version of a transient error message.
 2024-05-28 18:23:14 -07:00
Gireesh Sreepathi
21e29a665b Destinations cdk: ThreadCreationInfo cast as nullable (#38738) 2024-05-28 16:05:04 -07:00
Stephane Geneix
c884f195d8 destination-snowflake: convert all tests to kotlin (#38632) 2024-05-28 11:47:04 -07:00
Gireesh Sreepathi
503b819b84 Destination CDK: changes for snowflake kotlin conversion (#38638) 2024-05-24 00:29:49 +00:00
Stephane Geneix
94663440a2 bump CDK version for destination-snowflake, Remove nullability of IntegrationRunner.getThreadCreationInfo (#38586) 2024-05-23 16:53:10 -07:00
Rodi Reich Zilberman
7bd9423216 mysql stream statuses (#38198) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-23 19:25:51 -04:00
Stephane Geneix
f74f5d9a58 remove log4j from java cdk (#38583) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-05-23 12:02:52 -07:00
Stephane Geneix
d82639cb86 remove nullable from generics (#37555) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-05-23 11:46:54 -07:00
Stephane Geneix
b488213db4 remove unnecessary calls to toList() (#37540) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-05-23 11:36:46 -07:00
Stephane Geneix
460cdca771 remove calls to java stream() (#37539) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-05-23 11:20:40 -07:00
Stephane Geneix
0805f9e94c replace all java collectors.toSet with kotlin construct (#37538) 
## What
<!--
* Describe what the change is solving. Link all GitHub issues related to this change.
-->

## How
<!--
* Describe how code changes achieve the solution.
-->

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
 2024-05-23 11:09:48 -07:00
Stephane Geneix
b0ab1483af replace all java Collectors.toList with kotlin construct (#37537) 
more kotlin cleanup
 2024-05-23 11:01:45 -07:00
Stephane Geneix
c304df327e cleanup question marks in CDK code (#37518) 
just some kotlin cleanup
 2024-05-23 10:53:58 -07:00
Gireesh Sreepathi
f519db1dba Destinations CDK: Minor cleanup for snowflake (#38572) 2024-05-22 22:56:22 +00:00
Gireesh Sreepathi
9e373ddf5d Destinations CDK: Avoid issuing multiple create schema calls per stream. (#38357) 2024-05-20 19:18:11 -07:00
Gireesh Sreepathi
b47f80feab Destinations CDK: Add interfaces for operations by responsibility (#38107) 2024-05-17 13:40:28 -07:00