jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 11:57:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
31,842 Commits 5,118 Branches 518 Tags
ad7aa7af4b88bc2c56bcbff37b7529a5b45fe74c
Commit Graph

534 Commits

Author SHA1 Message Date
Yue Li
ad7aa7af4b [cdk, source-postgres, source-mysql, source-mongodb-v2] Fixing vulnerabilities in dependencies (#58132) 
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-04-24 08:28:30 -07:00
devin-ai-integration[bot]
3cf3c3d2fa [source-mysql, mssql, postgres, mongodb-v2] chore: update base image version for certified DB source connectors (#55234) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yue.li@airbyte.io <yue.li@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2025-03-06 12:15:07 -08:00
Wenqi Hu
5e81182bf2 [source-postgres] Fix issue #6857 (#53649) 2025-02-18 23:27:24 -08:00
Augustin
25cf90d3f6 source-postgres: bump base image version to 2.0.0 (#50870) 2025-01-06 09:05:31 +01:00
Yue Li
22390331ca [source-postgres] make source-postgres work with non-root base image (#50410) 
This patch starts using non-root base image for postgres. To do so, we have to save the SSL key to "/tmp" instead of ""
 2024-12-23 22:33:19 -08:00
Yue Li
d02c1ea2d8 [source-postgres] use Debezium 3.0.1 and increase MAX_FIRST_RECORD_WAIT_TIME (#48495) 2024-12-20 19:41:20 -08:00
Aaron ("AJ") Steers
83ecbe0fc3 CI: apply pre-commit format fix from #49806 (#49852) 2024-12-18 14:05:43 -08:00
Augustin
1d54d2a20c source-postgres: Use airbyte/java-connector-base:1.0.0 (#49838) 2024-12-18 08:34:37 +01:00
Rodi Reich Zilberman
3af71abb19 Simplify CTID_TABLE_BLOCK_SIZE query for Postgres integration (#48126) (#49469) 
Co-authored-by: Tom <tom@tom.kiwi>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2024-12-16 08:39:46 -08:00
Yue Li
ffde35c9e3 [source-postgres + CDK] use BigInteger instead of BigInt (#48482) 2024-11-13 15:04:39 -08:00
Natik Gadzhi
943e07061f fix(source-postgres): replace path with field in source docs (#46900) 2024-10-15 00:45:02 +03:00
Yue Li
f9aedb240f [source-postgres] support cdc against a read-replica (continuation) (#46322) 2024-10-02 23:26:32 +00:00
Yue Li
a71250d265 [source-postgres] make postgres source compile and use the latest CDK (#46299) 2024-10-02 14:06:59 -07:00
Yue Li
2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 2024-09-18 14:46:10 +03:00
Yue Li
f502315136 [source-postgres] use tcpAlive=true to make JDBC connection more robust upon backpressure (#44878) 2024-08-30 11:18:06 -07:00
Yue Li
af98a810b5 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841) 2024-08-28 22:35:16 +03:00
Xiaohan Song
cdedd93b56 [source-postgres] fix on final state (#44119) 2024-08-16 08:30:41 -07:00
Xiaohan Song
c31dab84e9 [source-postgres] Reenable cat test (#44121) 2024-08-15 11:33:41 -07:00
Augustin
601c3c455c Rename registries to registryOverrides in connector metadata (#43757) 2024-08-13 16:10:56 +02:00
Akash Kulkarni
421c7b34b1 [Source-Postgres] : Add invalid replication slot config error (#43945) 2024-08-12 15:50:57 -07:00
Akash Kulkarni
a4544c106f [Source-postgres] : Adopt latest CDK (#43418) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-09 10:03:55 -07:00
Xiaohan Song
97d822f820 [cdk, source-mysql/postgres/mssql/mongodb-v2] log record count to state message (#42869) 
Co-authored-by: Yue Li <yue.li@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-08-07 04:35:22 +03:00
Xiaohan Song
ab005c6465 [source-mssql/postgres/mysql] fix rfr losing stream state (#42550) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-08-02 20:02:19 +03:00
Xiaohan Song
57a2623283 add final teardown in postgres CAT test (#42916) 2024-08-01 13:28:48 -07:00
Xiaohan Song
224db75b22 [source-postgres] CAT test for CDC postgres (#42107) 
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Evan Tahler <evan@airbyte.io>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: maxi297 <maxime@airbyte.io>
Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Jeroen de Vries <jeroen@maildevries.eu>
Co-authored-by: prashant-mittal9 <168704883+prashant-mittal9@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: lazebnyi <lazebnyi@users.noreply.github.com>
Co-authored-by: pmossman <pmossman@users.noreply.github.com>
Co-authored-by: Aldo Gonzalez <168454423+aldogonzalez8@users.noreply.github.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>
Co-authored-by: Anton Karpets <karpets.as@gmail.com>
Co-authored-by: askarpets <anton.karpets@globallogic.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Tim Werner <t.werner@comventure.de>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: James Truty <jtruty@gmail.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Matias Samuel Miranda <matias.miranda@merantix-momentum.com>
Co-authored-by: Akash Kulkarni <akash@airbyte.io>
 2024-07-29 15:01:46 -07:00
Yue Li
d34d7e627a [source-postgres/mssql/mysql] use the latest CDK to use the latest fix of error translation (#42852) 2024-07-29 10:42:44 -07:00
Xiaohan Song
8bf87e807b [source-postgres]Non CDC CAT test for postgres (#41650) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: bgroff <bgroff@users.noreply.github.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Dan Aronson <dan.ar27@gmail.com>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Catherine Noll <noll.catherine@gmail.com>
 2024-07-24 13:01:52 -07:00
Yue Li
81cd8f2005 [cdk] handle null message inside exception (#42417) 
Fixes airbytehq/oncall#5904, error message inside an exception could be null.
 2024-07-23 17:27:33 -07:00
Akash Kulkarni
af0394150f [WASS] : Remove final transient error emitter iterators (#42421) 2024-07-23 14:27:15 -04:00
Rodi Reich Zilberman
2a6d6e8a5f Discard op code m change events (#42431) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-23 09:35:49 -07:00
Yue Li
4cadb340b9 [source-postgres/mssql] Hide the "initial load timeout in hours" field by default in UI for postgres and mssql (#42411) 
Hide the field as the other advanced files were also hidden by default.
 2024-07-23 02:10:37 -07:00
Rodi Reich Zilberman
250c02dbab null safety in debezium to airbyte message generation (#41622) 2024-07-22 23:37:37 -07:00
Xiaohan Song
af28105efa [source-mssql/mysql/postgres] Fix and cleanup oc map (#42024) 
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: artem1205 <artem1205@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Antonio Papa <antoniogpapa@gmail.com>
Co-authored-by: Adam Marcus <marcua@marcua.net>
Co-authored-by: Bryce Groff <bgroff@hawaii.edu>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Patrick Nilan <nilan.patrick@gmail.com>
 2024-07-22 11:32:51 -07:00
Yue Li
03c57ea472 [source-postgres, source-mysql] Translating top opaque and unfriendly error messages (#42087) 
This patch includes multiple translation rules for additional top opaque and system error messages that can be converted to transient or config errors. Translation/error profiles are added to the exception handler class for Postgres and MySQL sources.
 2024-07-21 23:24:17 -07:00
Akash Kulkarni
000dc477cf [WASS] : Enhancements in logging & analytics (#42122) 2024-07-21 23:07:18 -04:00
Yue Li
c7a1213508 [source-postgres] Disable incremental sync for view in Postgres xmin replication mode (#42108) 
Fixes airbytehq/airbyte-internal-issues#8559
Fixes airbytehq/oncall#5581

This patch disables incremental sync when a stream is found to be a view. This is done only for Postgres sources configured in xmin mode. This is done by adding a check in discovery() to only allow FULL_REFRESH sync mode.
 2024-07-19 14:38:14 -07:00
Akash Kulkarni
e53b8e245b Toggle useLocalCdk = false (#42120) 2024-07-19 16:08:59 -04:00
Hongbo Miao
f4ae8e48b4 docs(postgres): fix initial time and heartbeat links (#40716) 
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
 2024-07-18 11:58:39 -03:00
Yue Li
4417769038 [cdk, source-postgres, source-mysql] a new error handling and translation framework (#40208) 
Fixes airbytehq/airbyte-internal-issues#8516

This set of changes mainly moves error translation to be part of each connector.

In general, each connector needs to implement its own error translation class that inherits from the abstract class ConnectorExceptionTranslator, which is part of the CDK. By implementing, it means the connector developer or our support will populate the error dictionary with error samples with matching rules (e.g., regex). See the example we created for the Postgres source.
 2024-07-17 17:19:13 -07:00
Akash Kulkarni
8e5318044a [DB Sources] : Add analytics message to understand when WASS occurs (not just when we timeout) (#42068) 2024-07-17 15:26:57 -04:00
Akash Kulkarni
27d1437c4c [DB sources] : Add back Debezium heartbeat timeout error (#42055) 2024-07-17 13:55:44 -04:00
Akash Kulkarni
cb68670d67 [Source-Postgres] : Implement WASS algorithm (#41649) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-17 08:17:55 -07:00
Xiaohan Song
e1c1944e05 [source-postgres] add optional for all array schemabuilder (#41654) 2024-07-15 18:13:16 +00:00
Akash Kulkarni
d8a6f438a7 [Source-postgres] : Throw exception if there are any streams acively undergoing vacuum (#41651) 2024-07-12 13:34:58 -07:00
Xiaohan Song
5f42d38712 Revert "Revert "[source-postgres] Fix duplicate streams in postgres"" (#41067) 2024-07-09 14:09:33 -07:00
Xiaohan Song
5573c3f77e Revert "[source-postgres] Fix duplicate streams in postgres" (#40757) 2024-07-05 14:51:23 -07:00
Xiaohan Song
acb9ce1f86 [source-postgres] Fix duplicate streams in postgres (#40719) 2024-07-05 12:05:52 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Evan Tahler
0cfaf79ab2 source-postgres: Remove strict check for stream states of unknown types (#40559) 2024-06-26 16:14:29 -07:00
Marius Posta
d5e447bf49 source-postgres: remove adaptiveFetch=true (#40223) 2024-06-23 12:46:07 -07:00