jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 10:32:35 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

544 Commits

Author SHA1 Message Date
Edward Gao
3991e28549 Destination Clickhouse: increase test timeout (#69717) 2025-11-19 15:17:28 -08:00
Aaron ("AJ") Steers
a7be6e6a0f chore: Sort externalDocumentationUrls by type then title (do not merge) (#69742) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 21:51:00 -08:00
Aaron ("AJ") Steers
d9b08c06d9 feat: Add API release history URLs to 44 certified connectors (#69741) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 20:20:34 -08:00
Aaron ("AJ") Steers
4cde75568c feat: Add external documentation URLs to Group I source connectors (source-outbrain-amplify through source-pypi) (#69736) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
 2025-11-18 18:10:01 -08:00
Jason T Brown
afabbdf1c0 [source-postgres] Azure client auth for postgres (#57511) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2025-08-12 20:08:32 -07:00
Yarden Carmeli
8394ff5b22 source-postgres: Add error handling cases and bump CDK version (#61527) 2025-06-18 03:59:00 -07:00
Rodi Reich Zilberman
0b4a874976 Cache the cdc initial state once constructed to prevent race conditio… (#60876) 2025-05-29 10:37:47 -07:00
Davin Chia
112cb7b909 chore: move source postgres and mysql to new build flow. (#60214) 2025-05-13 11:13:13 -07:00
Davin Chia
6eb6f1ebac Migrate to the same Json Schema Generation Plugin. (#59722) 
These two sources today use different json schema generation plugins from:

The general Airbyte Gradle Plugins.
The old CDK
This is causing issues when using the Plugins in the new flow here: #59716.

[Incubating] Problems report is available at: file:///home/runner/work/airbyte/airbyte/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-***:generateJsonSchema2Pojo'.

> Method org/jsonschema2pojo/gradle/JsonSchemaExtension.getTargetLanguage()Lorg/jsonschema2pojo/Language; is abstract
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

Instead migrate both of these sources onto the same plugin provider and version.

- Migrate both over from org.jsonschema2pojo to com.github.eirnym.js2p.
- I also added what seems to be a missing task dependency. I'm not sure why this was never caught before. This is showing up in master as well.
 2025-05-08 13:41:53 -04:00
Aaron ("AJ") Steers
05965c781e chore: add poe_tasks.toml for all java connectors (#59670) 2025-05-05 17:28:27 -07:00
Yue Li
ad7aa7af4b [cdk, source-postgres, source-mysql, source-mongodb-v2] Fixing vulnerabilities in dependencies (#58132) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-04-24 08:28:30 -07:00
devin-ai-integration[bot]
3cf3c3d2fa [source-mysql, mssql, postgres, mongodb-v2] chore: update base image version for certified DB source connectors (#55234) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yue.li@airbyte.io <yue.li@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-03-06 12:15:07 -08:00
Wenqi Hu
5e81182bf2 [source-postgres] Fix issue #6857 (#53649) 2025-02-18 23:27:24 -08:00
Augustin
25cf90d3f6 source-postgres: bump base image version to 2.0.0 (#50870) 2025-01-06 09:05:31 +01:00
Yue Li
22390331ca [source-postgres] make source-postgres work with non-root base image (#50410) 
This patch starts using non-root base image for postgres. To do so, we have to save the SSL key to "/tmp" instead of ""
 2024-12-23 22:33:19 -08:00
Yue Li
d02c1ea2d8 [source-postgres] use Debezium 3.0.1 and increase MAX_FIRST_RECORD_WAIT_TIME (#48495) 2024-12-20 19:41:20 -08:00
Aaron ("AJ") Steers
83ecbe0fc3 CI: apply pre-commit format fix from #49806 (#49852) 2024-12-18 14:05:43 -08:00
Augustin
1d54d2a20c source-postgres: Use airbyte/java-connector-base:1.0.0 (#49838) 2024-12-18 08:34:37 +01:00
Rodi Reich Zilberman
3af71abb19 Simplify CTID_TABLE_BLOCK_SIZE query for Postgres integration (#48126) (#49469) 
Co-authored-by: Tom <tom@tom.kiwi>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2024-12-16 08:39:46 -08:00
Yue Li
ffde35c9e3 [source-postgres + CDK] use BigInteger instead of BigInt (#48482) 2024-11-13 15:04:39 -08:00
Natik Gadzhi
943e07061f fix(source-postgres): replace path with field in source docs (#46900) 2024-10-15 00:45:02 +03:00
Yue Li
f9aedb240f [source-postgres] support cdc against a read-replica (continuation) (#46322) 2024-10-02 23:26:32 +00:00
Yue Li
a71250d265 [source-postgres] make postgres source compile and use the latest CDK (#46299) 2024-10-02 14:06:59 -07:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Yue Li
f502315136 [source-postgres] use tcpAlive=true to make JDBC connection more robust upon backpressure (#44878) 2024-08-30 11:18:06 -07:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Xiaohan Song
cdedd93b56 [source-postgres] fix on final state (#44119) 2024-08-16 08:30:41 -07:00
Xiaohan Song
c31dab84e9 [source-postgres] Reenable cat test (#44121) 2024-08-15 11:33:41 -07:00
Augustin
601c3c455c Rename registries to registryOverrides in connector metadata (#43757) 2024-08-13 16:10:56 +02:00
Akash Kulkarni
421c7b34b1 [Source-Postgres] : Add invalid replication slot config error (#43945) 2024-08-12 15:50:57 -07:00
Akash Kulkarni
a4544c106f [Source-postgres] : Adopt latest CDK (#43418) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-09 10:03:55 -07:00
Xiaohan Song
97d822f820 [cdk, source-mysql/postgres/mssql/mongodb-v2] log record count to state message (#42869) 
Co-authored-by: Yue Li <yue.li@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-07 04:35:22 +03:00
Xiaohan Song
ab005c6465 [source-mssql/postgres/mysql] fix rfr losing stream state (#42550) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-08-02 20:02:19 +03:00
Xiaohan Song
57a2623283 add final teardown in postgres CAT test (#42916) 2024-08-01 13:28:48 -07:00
Xiaohan Song
224db75b22 [source-postgres] CAT test for CDC postgres (#42107) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Jeroen de Vries <jeroen@maildevries.eu>
Co-authored-by: prashant-mittal9 <168704883+prashant-mittal9@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: lazebnyi <lazebnyi@users.noreply.github.com>
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
Co-authored-by: Aldo Gonzalez <168454423+aldogonzalez8@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: Anton Karpets <karpets.as@gmail.com>
Co-authored-by: askarpets <anton.karpets@globallogic.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Tim Werner <t.werner@comventure.de>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: James Truty <jtruty@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Matias Samuel Miranda <matias.miranda@merantix-momentum.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
 2024-07-29 15:01:46 -07:00
Yue Li
d34d7e627a [source-postgres/mssql/mysql] use the latest CDK to use the latest fix of error translation (#42852) 2024-07-29 10:42:44 -07:00
Xiaohan Song
8bf87e807b [source-postgres]Non CDC CAT test for postgres (#41650) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: bgroff <bgroff@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Dan Aronson <dan.ar27@gmail.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
 2024-07-24 13:01:52 -07:00
Yue Li
81cd8f2005 [cdk] handle null message inside exception (#42417) 
Fixes airbytehq/oncall#5904, error message inside an exception could be null.
 2024-07-23 17:27:33 -07:00
Akash Kulkarni
af0394150f [WASS] : Remove final transient error emitter iterators (#42421) 2024-07-23 14:27:15 -04:00
Rodi Reich Zilberman
2a6d6e8a5f Discard op code m change events (#42431) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-23 09:35:49 -07:00
Yue Li
4cadb340b9 [source-postgres/mssql] Hide the "initial load timeout in hours" field by default in UI for postgres and mssql (#42411) 
Hide the field as the other advanced files were also hidden by default.
 2024-07-23 02:10:37 -07:00
Rodi Reich Zilberman
250c02dbab null safety in debezium to airbyte message generation (#41622) 2024-07-22 23:37:37 -07:00
Xiaohan Song
af28105efa [source-mssql/mysql/postgres] Fix and cleanup oc map (#42024) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
 2024-07-22 11:32:51 -07:00
Yue Li
03c57ea472 [source-postgres, source-mysql] Translating top opaque and unfriendly error messages (#42087) 
This patch includes multiple translation rules for additional top opaque and system error messages that can be converted to transient or config errors. Translation/error profiles are added to the exception handler class for Postgres and MySQL sources.
 2024-07-21 23:24:17 -07:00
Akash Kulkarni
000dc477cf [WASS] : Enhancements in logging & analytics (#42122) 2024-07-21 23:07:18 -04:00
Yue Li
c7a1213508 [source-postgres] Disable incremental sync for view in Postgres xmin replication mode (#42108) 
Fixes airbytehq/airbyte-internal-issues#8559
Fixes airbytehq/oncall#5581

This patch disables incremental sync when a stream is found to be a view. This is done only for Postgres sources configured in xmin mode. This is done by adding a check in discovery() to only allow FULL_REFRESH sync mode.
 2024-07-19 14:38:14 -07:00
Akash Kulkarni
e53b8e245b Toggle useLocalCdk = false (#42120) 2024-07-19 16:08:59 -04:00
Hongbo Miao
f4ae8e48b4 docs(postgres): fix initial time and heartbeat links (#40716) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2024-07-18 11:58:39 -03:00
Yue Li
4417769038 [cdk, source-postgres, source-mysql] a new error handling and translation framework (#40208) 
Fixes airbytehq/airbyte-internal-issues#8516

This set of changes mainly moves error translation to be part of each connector.

In general, each connector needs to implement its own error translation class that inherits from the abstract class ConnectorExceptionTranslator, which is part of the CDK. By implementing, it means the connector developer or our support will populate the error dictionary with error samples with matching rules (e.g., regex). See the example we created for the Postgres source.
 2024-07-17 17:19:13 -07:00
Akash Kulkarni
8e5318044a [DB Sources] : Add analytics message to understand when WASS occurs (not just when we timeout) (#42068) 2024-07-17 15:26:57 -04:00