jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 18:39:31 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,109 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

544 Commits

Author SHA1 Message Date
Akash Kulkarni
27d1437c4c [DB sources] : Add back Debezium heartbeat timeout error (#42055) 2024-07-17 13:55:44 -04:00
Akash Kulkarni
cb68670d67 [Source-Postgres] : Implement WASS algorithm (#41649) 
Co-authored-by: Evan Tahler <evan@airbyte.io>
 2024-07-17 08:17:55 -07:00
Xiaohan Song
e1c1944e05 [source-postgres] add optional for all array schemabuilder (#41654) 2024-07-15 18:13:16 +00:00
Akash Kulkarni
d8a6f438a7 [Source-postgres] : Throw exception if there are any streams acively undergoing vacuum (#41651) 2024-07-12 13:34:58 -07:00
Xiaohan Song
5f42d38712 Revert "Revert "[source-postgres] Fix duplicate streams in postgres"" (#41067) 2024-07-09 14:09:33 -07:00
Xiaohan Song
5573c3f77e Revert "[source-postgres] Fix duplicate streams in postgres" (#40757) 2024-07-05 14:51:23 -07:00
Xiaohan Song
acb9ce1f86 [source-postgres] Fix duplicate streams in postgres (#40719) 2024-07-05 12:05:52 -07:00
Akash Kulkarni
3a3e058ae8 [DB-sources] : Improve heartbeat logic (#40516) 2024-07-01 12:51:33 -07:00
Evan Tahler
0cfaf79ab2 source-postgres: Remove strict check for stream states of unknown types (#40559) 2024-06-26 16:14:29 -07:00
Marius Posta
d5e447bf49 source-postgres: remove adaptiveFetch=true (#40223) 2024-06-23 12:46:07 -07:00
Xiaohan Song
b80a72888c [source-postgres/mssql/mysql] Send state and count for full refresh (#39349) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Anjay Goel <anjay.g@dashtoon.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: gosusnp <gosusnp@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Abdul Rahman Zantout <abed-zantout@live.com>
Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com>
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: Justin Flannery <juftin@juftin.com>
Co-authored-by: Mal Hancock <mallory@archangelic.space>
Co-authored-by: Enrique Alcázar Garzás <enrique.alcazar@omniloy.com>
Co-authored-by: Daniela García Nistor <58893405+gndaniela@users.noreply.github.com>
Co-authored-by: Carlo Nuccio <carlonuccio91@gmail.com>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Charlie Duong <charlie.duong@hey.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Cristina Mariscal <166420606+cmm-airbyte@users.noreply.github.com>
Co-authored-by: cristina.mariscal <cristina.mariscal@cristina.mariscal--MacBook-Pro---DFJ27FJFXX>
Co-authored-by: Gonzalo Villafañe Tapia <gvillafanetapia@gmail.com>
Co-authored-by: Jérémy Denquin <jdenquin@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: williammcguinness <wmcguinness@findoctave.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
Co-authored-by: Stephane Geneix <147216312+stephane-airbyte@users.noreply.github.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
 2024-06-17 13:30:56 -04:00
Evan Tahler
b88a41d339 postgres source/destination - bump postgres driver version (#39460) 
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
 2024-06-14 15:35:17 -04:00
Akash Kulkarni
6db30d6a51 [Source-postgres] : Bump cdk to latest (#39474) 2024-06-14 09:37:58 -07:00
Marius Posta
32f78307c8 source-postgres: connect with adaptiveFetch=true, take 2 (#38773) 2024-06-13 11:31:09 -04:00
Yue Li
3b3392317f source-postgres: upgrade Debezium to 2.6.2 (#39353) 
fixes airbytehq/airbyte-internal-issues#7739
 2024-06-08 20:45:29 -07:00
Yue Li
859b596383 source-postgres: read() throws config exception upon xmin wraparound (#38875) 
fixes airbytehq/airbyte-internal-issues#8006
 2024-06-05 12:17:14 -07:00
Yue Li
2791dd6565 source-postgres: check() throws config error upon detecting transaction ID wraparound (#38836) 
fixes airbytehq/airbyte-internal-issues#8008

This PR adds one more checkOperation for check(), which throws a config exception if Postgres has experienced one or more xmin wraparounds.
 2024-06-04 17:46:08 -07:00
Yue Li
b2846c03ec source-postgres: improve UI message and doc on xmin (#38848) 
The current recommendation for using xmin on the UI of the Postgres source connector is misleading; we want to say that the database is not experiencing a high transaction rate such that it is less likely to trigger xmin wraparound soon.

Added sections on the doc and troubleshooting page regarding Xmin as well.
 2024-06-04 16:06:23 -07:00
Yue Li
9a1663a251 Publish CDK for postgres source connector (#38834) 2024-06-03 14:57:00 -07:00
Yue Li
06b1b55228 CDK: throw config exception if no selectable table exists in user provided schemas (#38792) 
fixes airbytehq/airbyte-internal-issues#2252

"When we don't have permission to access the schema, the check succeeds, but refreshing the schema returns no tables. We should probably throw an error if the user configures the source schema as, e.g., public but doesn't grant our user permission to access that schema."

This patch includes a check that will throw a configuration error if a user does not have selectable tables from any of the provided schemas (in UI).If schemas actually contain no table, we would let the user proceed.
 2024-05-31 13:43:51 -07:00
Xiaohan Song
2d194fa359 Set isResumeable flag in catalog (#38584) 
Co-authored-by: Dhroov Makwana <pabloescoder@gmail.com>
Co-authored-by: Alexandre Girard <alexandre@airbyte.io>
Co-authored-by: btkcodedev <btk.codedev@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Yue Li <61070669+theyueli@users.noreply.github.com>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Natik Gadzhi <natik@respawn.io>
Co-authored-by: Danylo Jablonski <150933663+DanyloGL@users.noreply.github.com>
Co-authored-by: Maxime Carbonneau-Leclerc <3360483+maxi297@users.noreply.github.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Christo Grabowski <108154848+ChristoGrab@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>
Co-authored-by: Audrey Maldonado <audrey.maldonado@gmail.com>
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
Co-authored-by: Serhii Lazebnyi <serhii.lazebnyi@globallogic.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
 2024-05-30 10:23:09 -07:00
Akash Kulkarni
2bf0bf8d07 Bump versions for SQL server, Postgres (#38775) 2024-05-29 18:33:50 -04:00
Xiaohan Song
c63d40998a Publish cdk for postgres (#38771) 2024-05-29 13:47:23 -07:00
Xiaohan Song
b104667a0c Stream status on postgres (#38716) 
Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>
 2024-05-29 13:05:20 -07:00
Xiaohan Song
0db33b8494 Revert "source-postgres: connect with adaptiveFetch=true" (#38365) 2024-05-20 19:23:09 -04:00
Marius Posta
c97bbfc4ed source-postgres: connect with adaptiveFetch=true (#38289) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-20 10:49:19 -07:00
Evan Tahler
49fc60dea2 source-postgres - Streams not in the CDC publication still have a cursor and PK (#38303) 2024-05-16 16:18:26 -07:00
Augustin
31c95da499 [skip ci] Add connectorTestSuitesOptions to metadata (#38188) 2024-05-15 15:04:10 -05:00
Subodh Kant Chaturvedi
073b94062c db-sources: disable counts for state messages for FULL_REFRESH streams (#38208) 
Co-authored-by: Xiaohan Song <xiaohan@airbyte.io>
 2024-05-15 11:15:26 -07:00
Akash Kulkarni
fcbfaa376f [DB source errors] : Handle common transient errors (#38104) 2024-05-13 20:27:07 -04:00
Xiaohan Song
59cdc36db6 [source-postgres] Fix final state message on state (#38171) 2024-05-13 19:17:03 -04:00
Xiaohan Song
4b92f751cd [source-postgres] bugfix: Exclude views in incremental iterator (#38130) 2024-05-10 18:55:03 -04:00
Xiaohan Song
80920d1394 Postgres on Resumable full refresh (#37112) 2024-05-10 12:13:37 -04:00
Akash Kulkarni
d26bd10c87 [DB sources] : Add plumbing for adding transient errors (#38030) 2024-05-09 10:21:58 -07:00
Natik Gadzhi
cb80e6922a [tools] prettier rules for .md + formatting cleanup 2024-05-07 08:19:33 -07:00
Rodi Reich Zilberman
f7f7f7ef2e Set debezium num retries property to zero (#37758) 2024-05-02 03:04:03 -04:00
Rodi Reich Zilberman
deabdbb16a Add debezium engine logs (#37754) 2024-05-01 20:25:57 -04:00
Rodi Reich Zilberman
a9ae97af9b bump metadata (#37755) 2024-05-01 19:58:24 -04:00
Akash Kulkarni
201e7e3630 [Source-postgres] : Remove debezium retries (#37726) 2024-05-01 08:26:29 -07:00
Xiaohan Song
587e874eb5 Remove excessive logs in counter (#37541) 2024-04-24 13:31:46 -04:00
Akash Kulkarni
d790b7d5df [Source-postgres] : Better error messages on switching between sync m… (#37509) 2024-04-24 12:39:07 -04:00
Akash Kulkarni
9b9ec1cecd [source-postgres] : Remove legacy bad values handling code (#37445) 2024-04-22 13:47:57 -04:00
Akash Kulkarni
fda082966f [Source-postgres] : Populate airyte_meta.changes for xmin path (#36982) 2024-04-10 16:25:23 -07:00
Akash Kulkarni
0296c43169 [Db analytics] : add message for data type serialization error (#36981) 2024-04-10 15:46:15 -07:00
Akash Kulkarni
1815e38dd8 [Source-Postgres] : Bump cdk, reduce timeout (#36865) 2024-04-10 15:36:17 -04:00
Marius Posta
2495575795 java-cdk: re-export airbyte-api dependency (#36759) 2024-04-03 10:43:05 -07:00
Akash Kulkarni
2825239050 [Source-Postgres] : Toggle useLocalCdk = false (#36739) 2024-04-01 19:35:38 -04:00
Akash Kulkarni
e3f3379120 [Source-postgres] : Move to new Kotlin CDK (#36584) 2024-04-01 16:51:04 -04:00
Akash Kulkarni
9dee83729e [Source-postgres] : Source operations suport for meta column (#36432) 2024-03-26 10:27:20 -07:00
Xiaohan Song
1b99f1ab89 DBZ iterator migration to use SourceStateIterator (#36333) 2024-03-25 21:50:33 -07:00