jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 10:32:35 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

139 Commits

Author SHA1 Message Date
Jason T Brown
afabbdf1c0 [source-postgres] Azure client auth for postgres (#57511) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2025-08-12 20:08:32 -07:00
Yarden Carmeli
8394ff5b22 source-postgres: Add error handling cases and bump CDK version (#61527) 2025-06-18 03:59:00 -07:00
Rodi Reich Zilberman
0b4a874976 Cache the cdc initial state once constructed to prevent race conditio… (#60876) 2025-05-29 10:37:47 -07:00
Davin Chia
112cb7b909 chore: move source postgres and mysql to new build flow. (#60214) 2025-05-13 11:13:13 -07:00
Davin Chia
6eb6f1ebac Migrate to the same Json Schema Generation Plugin. (#59722) 
These two sources today use different json schema generation plugins from:

The general Airbyte Gradle Plugins.
The old CDK
This is causing issues when using the Plugins in the new flow here: #59716.

[Incubating] Problems report is available at: file:///home/runner/work/airbyte/airbyte/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-***:generateJsonSchema2Pojo'.

> Method org/jsonschema2pojo/gradle/JsonSchemaExtension.getTargetLanguage()Lorg/jsonschema2pojo/Language; is abstract
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

Instead migrate both of these sources onto the same plugin provider and version.

- Migrate both over from org.jsonschema2pojo to com.github.eirnym.js2p.
- I also added what seems to be a missing task dependency. I'm not sure why this was never caught before. This is showing up in master as well.
 2025-05-08 13:41:53 -04:00
Yue Li
ad7aa7af4b [cdk, source-postgres, source-mysql, source-mongodb-v2] Fixing vulnerabilities in dependencies (#58132) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-04-24 08:28:30 -07:00
Yue Li
22390331ca [source-postgres] make source-postgres work with non-root base image (#50410) 
This patch starts using non-root base image for postgres. To do so, we have to save the SSL key to "/tmp" instead of ""
 2024-12-23 22:33:19 -08:00
Yue Li
d02c1ea2d8 [source-postgres] use Debezium 3.0.1 and increase MAX_FIRST_RECORD_WAIT_TIME (#48495) 2024-12-20 19:41:20 -08:00
Yue Li
ffde35c9e3 [source-postgres + CDK] use BigInteger instead of BigInt (#48482) 2024-11-13 15:04:39 -08:00
Yue Li
a71250d265 [source-postgres] make postgres source compile and use the latest CDK (#46299) 2024-10-02 14:06:59 -07:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Akash Kulkarni
a4544c106f [Source-postgres] : Adopt latest CDK (#43418) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-09 10:03:55 -07:00
Xiaohan Song
97d822f820 [cdk, source-mysql/postgres/mssql/mongodb-v2] log record count to state message (#42869) 
Co-authored-by: Yue Li <yue.li@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-07 04:35:22 +03:00
Xiaohan Song
ab005c6465 [source-mssql/postgres/mysql] fix rfr losing stream state (#42550) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-08-02 20:02:19 +03:00
Yue Li
d34d7e627a [source-postgres/mssql/mysql] use the latest CDK to use the latest fix of error translation (#42852) 2024-07-29 10:42:44 -07:00
Yue Li
81cd8f2005 [cdk] handle null message inside exception (#42417) 
Fixes airbytehq/oncall#5904, error message inside an exception could be null.
 2024-07-23 17:27:33 -07:00
Rodi Reich Zilberman
2a6d6e8a5f Discard op code m change events (#42431) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-23 09:35:49 -07:00
Rodi Reich Zilberman
250c02dbab null safety in debezium to airbyte message generation (#41622) 2024-07-22 23:37:37 -07:00
Xiaohan Song
af28105efa [source-mssql/mysql/postgres] Fix and cleanup oc map (#42024) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
 2024-07-22 11:32:51 -07:00
Akash Kulkarni
000dc477cf [WASS] : Enhancements in logging & analytics (#42122) 2024-07-21 23:07:18 -04:00
Yue Li
c7a1213508 [source-postgres] Disable incremental sync for view in Postgres xmin replication mode (#42108) 
Fixes airbytehq/airbyte-internal-issues#8559
Fixes airbytehq/oncall#5581

This patch disables incremental sync when a stream is found to be a view. This is done only for Postgres sources configured in xmin mode. This is done by adding a check in discovery() to only allow FULL_REFRESH sync mode.
 2024-07-19 14:38:14 -07:00
Akash Kulkarni
e53b8e245b Toggle useLocalCdk = false (#42120) 2024-07-19 16:08:59 -04:00
Yue Li
4417769038 [cdk, source-postgres, source-mysql] a new error handling and translation framework (#40208) 
Fixes airbytehq/airbyte-internal-issues#8516

This set of changes mainly moves error translation to be part of each connector.

In general, each connector needs to implement its own error translation class that inherits from the abstract class ConnectorExceptionTranslator, which is part of the CDK. By implementing, it means the connector developer or our support will populate the error dictionary with error samples with matching rules (e.g., regex). See the example we created for the Postgres source.
 2024-07-17 17:19:13 -07:00
Akash Kulkarni
8e5318044a [DB Sources] : Add analytics message to understand when WASS occurs (not just when we timeout) (#42068) 2024-07-17 15:26:57 -04:00
Akash Kulkarni
27d1437c4c [DB sources] : Add back Debezium heartbeat timeout error (#42055) 2024-07-17 13:55:44 -04:00
Akash Kulkarni
cb68670d67 [Source-Postgres] : Implement WASS algorithm (#41649) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-17 08:17:55 -07:00
Xiaohan Song
5f42d38712 Revert "Revert "[source-postgres] Fix duplicate streams in postgres"" (#41067) 2024-07-09 14:09:33 -07:00
Xiaohan Song
5573c3f77e Revert "[source-postgres] Fix duplicate streams in postgres" (#40757) 2024-07-05 14:51:23 -07:00
Xiaohan Song
acb9ce1f86 [source-postgres] Fix duplicate streams in postgres (#40719) 2024-07-05 12:05:52 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Evan Tahler
b88a41d339 postgres source/destination - bump postgres driver version (#39460) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-06-14 15:35:17 -04:00
Akash Kulkarni
6db30d6a51 [Source-postgres] : Bump cdk to latest (#39474) 2024-06-14 09:37:58 -07:00
Yue Li
3b3392317f source-postgres: upgrade Debezium to 2.6.2 (#39353) 
fixes airbytehq/airbyte-internal-issues#7739
 2024-06-08 20:45:29 -07:00
Yue Li
9a1663a251 Publish CDK for postgres source connector (#38834) 2024-06-03 14:57:00 -07:00
Yue Li
06b1b55228 CDK: throw config exception if no selectable table exists in user provided schemas (#38792) 
fixes airbytehq/airbyte-internal-issues#2252

"When we don't have permission to access the schema, the check succeeds, but refreshing the schema returns no tables. We should probably throw an error if the user configures the source schema as, e.g., public but doesn't grant our user permission to access that schema."

This patch includes a check that will throw a configuration error if a user does not have selectable tables from any of the provided schemas (in UI).If schemas actually contain no table, we would let the user proceed.
 2024-05-31 13:43:51 -07:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Akash Kulkarni
2bf0bf8d07 Bump versions for SQL server, Postgres (#38775) 2024-05-29 18:33:50 -04:00
Xiaohan Song
c63d40998a Publish cdk for postgres (#38771) 2024-05-29 13:47:23 -07:00
Xiaohan Song
b104667a0c Stream status on postgres (#38716) 
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
 2024-05-29 13:05:20 -07:00
Subodh Kant Chaturvedi
073b94062c db-sources: disable counts for state messages for FULL_REFRESH streams (#38208) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-15 11:15:26 -07:00
Akash Kulkarni
fcbfaa376f [DB source errors] : Handle common transient errors (#38104) 2024-05-13 20:27:07 -04:00
Xiaohan Song
59cdc36db6 [source-postgres] Fix final state message on state (#38171) 2024-05-13 19:17:03 -04:00
Xiaohan Song
80920d1394 Postgres on Resumable full refresh (#37112) 2024-05-10 12:13:37 -04:00
Akash Kulkarni
d26bd10c87 [DB sources] : Add plumbing for adding transient errors (#38030) 2024-05-09 10:21:58 -07:00
Rodi Reich Zilberman
f7f7f7ef2e Set debezium num retries property to zero (#37758) 2024-05-02 03:04:03 -04:00
Rodi Reich Zilberman
deabdbb16a Add debezium engine logs (#37754) 2024-05-01 20:25:57 -04:00
Akash Kulkarni
201e7e3630 [Source-postgres] : Remove debezium retries (#37726) 2024-05-01 08:26:29 -07:00
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00