1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

255 Commits

Author SHA1 Message Date
Akash Kulkarni
e11d765cb8 Better attribution of DBZ closing reason (#32686) 2023-11-28 21:04:00 -04:00
Marius Posta
23f3b6c570 source-postgres: fix useLocalCdk = true (#32891) 2023-11-28 12:38:21 -08:00
Marius Posta
d97a399a24 source-mysql, source-mssql: parallelize test execution (#32772)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-11-27 12:49:06 -06:00
Tim Roes
c5db1e947b 📚 Documentation November 2023 overhaul (#32811)
Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com>
Co-authored-by: timroes <timroes@users.noreply.github.com>
Co-authored-by: nataliekwong <nataliekwong@users.noreply.github.com>
2023-11-26 11:57:38 -08:00
Stephane Geneix
3f42418c15 🐛 📝 don't missparse +/-infinity in postgres-source (#31856) 2023-11-09 18:33:44 +00:00
Stephane Geneix
a2f8eec7f8 update CDK to 0.4.1 (#32193) 2023-11-04 09:53:08 -04:00
Marius Posta
61c24c97ed source-postgres: use CDK 0.4.0 (#32173) 2023-11-03 14:29:26 -07:00
Marcos Marx
fbf391c87c Fix typo (Airbtye => Airbyte) (#29038)
Co-authored-by: Przemysław Dąbek <373530+szemek@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
2023-11-03 14:10:57 +00:00
Akash Kulkarni
b08b09d7ea source-postgres : Bump deebezium version 2.2.0 -> 2.4.0 (#32068)
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: Marius Posta <marius@airbyte.io>
2023-11-01 13:15:43 -07:00
Marius Posta
bc78d2138c java CDK, source-postgres: speed up tests involving debezium (#31976)
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
2023-10-31 09:04:41 -07:00
Marius Posta
7cd8020ac8 java CDK: hoist top-level gradle projects into CDK (#31960)
Co-authored-by: postamar <postamar@users.noreply.github.com>
2023-10-30 12:03:06 -07:00
Stephane Geneix
412da024ac 🐛 📝 actually fix postgres standby error message by using the latest CDK (#31804)
Co-authored-by: Akash Kulkarni <113392464+akashkulk@users.noreply.github.com>
2023-10-25 15:01:12 -07:00
Akash Kulkarni
52520d01ce Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image (#31486)
Co-authored-by: Ben Church <ben@airbyte.io>
Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
2023-10-18 12:39:08 -07:00
Duy Nguyen
896fa4bafd [Source-Postgres] Update PostgresConverter to correctly stringify BigDecimals without scientific notation (#31309) 2023-10-13 10:11:35 -07:00
Rodi Reich Zilberman
12692cefbe 🐛 Update initial load query for old postgres to return a defined order … (#31328)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-10-12 16:40:31 -07:00
Rodi Reich Zilberman
7e92375438 bump cdk version for postgres, mysql sources (#31322) 2023-10-12 10:38:54 -07:00
Rodi Reich Zilberman
3d9578fdac Cap each log field (message, stacktrace) to max 32K characters (#30806)
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-10-11 13:34:53 -07:00
Marius Posta
8a22cf9082 🐛 source-postgres: fix jsonb[] handling bug (#30534) 2023-09-26 09:59:59 -07:00
Rodi Reich Zilberman
a876e60d08 29779 source postgres slow ctid read seen on customer connection (#30125)
Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
Co-authored-by: rodireich <rodireich@users.noreply.github.com>
2023-09-20 18:16:01 -05:00
Subodh Kant Chaturvedi
47e01208be postgres-source: validate filenode before processing each ctid chunk (#29672)
Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com>
2023-09-06 11:30:49 -05:00
Lake Mossman
b19ed6d8b3 Support display_type property for oneOf fields and adjust CDC connectors to use it (#29821)
Co-authored-by: lmossman <lmossman@users.noreply.github.com>
2023-08-31 12:19:02 -05:00
Tim Roes
9fd433b857 Cleanup IP addresses in docs (#29975)
Co-authored-by: timroes <timroes@users.noreply.github.com>
2023-08-31 15:59:05 +02:00
Tyler B
65e23ca627 Update postgres.md with detail about xmin (#29792) 2023-08-25 08:14:06 -04:00
Marius Posta
a26a4add26 Allow configuring "options=" in source-postgres jdbc url (#29534) 2023-08-23 13:12:21 -05:00
Aaron ("AJ") Steers
d094b152c9 Java CDK 'no-op': v0.0.1 (#28687)
Co-authored-by: aaronsteers <aaronsteers@users.noreply.github.com>
Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
Co-authored-by: cpdeethree <conor@airbyte.io>
Co-authored-by: Augustin <augustin@airbyte.io>
Co-authored-by: Joe Bell <joseph.bell@airbyte.io>
2023-08-21 14:01:32 -05:00
Alexandre Cuoci
86ba823234 improvements to postgres + cloudsql docs (#29415) 2023-08-14 17:20:50 -04:00
Alexandre Cuoci
8aa66790d5 fix postgres docs broken links (#29317)
* fixes to postgres docs

* no relative links

* Update docs/integrations/sources/postgres.md
2023-08-10 11:20:53 -04:00
Alexandre Cuoci
cfc6834a2e postgres docs update & improvements (#29295)
* postgres docs update & improvements

* remove extra postgres entry

* Updated xmin point to link.

---------

Co-authored-by: Prateek Mukhedkar <prateek@airbyte.io>
2023-08-09 16:11:20 -04:00
Evan Tahler
9210547af5 [Docs] No Deduped + Hostory, Append + Deduped is the future! (#29114)
* [Docs] No `Deduped + Hostory`, `Append + Deduped` is the future!

* fix links
2023-08-08 15:07:49 -07:00
Rodi Reich Zilberman
2d2cddd9f3 Rollout ctid cdc (#28708)
* source-postgres: enable ctid+cdc implementation

* 100% ctid rollout for cdc

* remove CtidFeatureFlags

* fix CdcPostgresSourceAcceptanceTest

* Bump versions and release notes

* Fix compilation error due to previous merge

---------

Co-authored-by: subodh <subodh1810@gmail.com>
2023-08-03 15:36:04 -05:00
Rodi Reich Zilberman
3365015db6 Use full enquoted table name for max cursor query (#28954)
* Use full enquoted table name for max cursor query

* Bump versions
2023-08-01 20:11:40 -05:00
Rodi Reich Zilberman
d37d2e271a Quote cursor name to support uppercase letters (required by postgres) (#28892)
* quote cursor name to support uppercase letters (required by postgres)

* sanity

* sanity
2023-07-31 22:16:52 -05:00
Rodi Reich Zilberman
fb7cacb321 Pre release ctid (#28339)
* enable ctid feature flag for testing.
bump versions

* Removed PostgresSourceTest pre-ctid and turn CtidEnabledPostgresSourceTest back into PostgresJdbcSourceAcceptanceTest (#28341)

Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com>

* Update PostgresSourceTest to no longer use Legacy AirbyteStateType

* Fix test.
Uncovered a bug in handling of money[] and float[]

* revert version change

* Orderly removal of ctid flag for cursor

* Handle correctly full refresh streams. (#28369)

* Handle correctly full refresh streams.
Handle xmin full refresh (currently blocked in ui).
remove unnecessary debug logs

* Test catalog with a mix of incremental and full refresh streams

* Filter out full refresh streams since they are not expected to be retuned by getIncrementalIterators()

* Filter out full refresh existing streams as well

* revert name to reflect incremental only streams

* 28505 xmin + ctid reclassify connection when ctid is not possible (#28548)

* reclassify to xmin stream we cannot read with ctid

* fix reclassify

* improve test

* Add initial load using xmin flow

* typo

* Added test for xmin initial load flow

* Improve testing for initial load with xmin flow

* sanity

* sanity

* bump versions and release notes

---------

Co-authored-by: Duy Nguyen <duy@airbyte.io>
2023-07-25 16:56:29 -05:00
Rodi Reich Zilberman
ff9fd07675 fix changelog table (#28649) 2023-07-24 19:53:17 -05:00
Akash Kulkarni
4c54e34bd7 Add back full_refresh to xmin syncs (#28336)
* Add back full_refresh to xmin syncs

* Dockerfile bumps + fix tests
2023-07-18 18:46:50 -05:00
Ben Church
6fa755f81d Java Pipeline Bump patch bump all java connectors in july (#28345)
* patch bump all java connectors in july

* Bump changelog
2023-07-14 21:01:39 -05:00
Duy Nguyen
633c939d46 [Source-postgres] Set default cursor value for cdc mode (#27442)
* use LSN as default cursor for postgres CDC

* Fixed static constant

* Set lsn default cursor value for postgres sync

* Bumped metadata and dockerfile versions

* Disable acceptance backwards compatibility discovery test as this is a breaking change

---------

Co-authored-by: Conor <cpdeethree@users.noreply.github.com>
2023-07-13 11:12:34 -05:00
Akash Kulkarni
cc8081d512 🎉 source-postgres : Enable xmin replication (#26723)
* Add logging functionality

- Add logging for DB version (all JDBC sources)
- Add logging for Xmin values

* Automated Change

* Renaming + comments

* More comments

* Initial commit for xmin prototype

- Add spec.json
- Catalog
- Try to change incremental query

* Skeleton for sync

Decision is to use a whole new sync method

* With cursor

* Working catalog + attempt at xmin record iterator

* Unit tested code

* Fix build break

* Fixing state

* more xmin

* Revert unchanged files

* Addressing code review comments

* use generated pojo

* Add iniital XminStateManagerTest

* Additional unit tests

* Added unit test

* Update xmin state manager test

* Remove spec.json xmin option

* Fixed bug where full refresh streams were not being filtered out for Xmin

* Fix latest safe change boundary issue

* Xmin spec update

* Fix

* Automated Commit - Format and Process Resources Changes

* Remove full_refresh as an option for xmin syncs and always include incremental syncs

* Fix integ test

* revert unused files

* revert

* Bump docker + update change log

* Modify order of CDC

* Update expected_spec.json

* Update expected_json

Fix acceptance tests

* Reverse order for xmin replication

* Update expected strict encrypt json

---------

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>
Co-authored-by: subodh <subodh1810@gmail.com>
2023-07-10 12:58:46 -05:00
Evan Tahler
9ce3cc6e79 Java DB Source connector licenses to Elv2 (#27737)
* Java DB Source connector licenses to Elv2

* Bump PR Ids
2023-06-28 18:52:02 -07:00
Akash Kulkarni
2f807440da Fix issue in streaming JDBC database (#27212)
* Fix issue in streaming JDBC database

* Bump + publish

* 🤖 Auto format source-db2 code [skip ci]

* 🤖 Auto format source-mssql code [skip ci]

* 🤖 Auto format source-postgres-strict-encrypt code [skip ci]

* 🤖 Auto format source-postgres code [skip ci]

* 🤖 Auto format source-mssql-strict-encrypt code [skip ci]

* 🤖 Auto format source-oracle code [skip ci]

* 🤖 Auto format source-tidb code [skip ci]

* 🤖 Auto format source-snowflake code [skip ci]

* 🤖 Auto format source-redshift code [skip ci]

* 🤖 Auto format source-mysql code [skip ci]

* 🤖 Auto format source-mysql-strict-encrypt code [skip ci]

---------

Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com>
2023-06-20 12:08:47 -05:00
Evan Tahler
9cbb7f267a Update postgres.md (#27325)
* Update postgres.md

* Update postgres.md

* Update postgres.md
2023-06-20 15:16:47 +00:00
Duy Nguyen
65d26bbe4d Source-postgres: Add prepareThreshold=0 to JDBC url to mitigate PGBouncer prepared statement [X] already exists. (#26873)
* Append PrepareThreshold=0 to JDBC param
2023-06-01 17:56:36 -07:00
Akash Kulkarni
a8ef7718a4 Remove incremental estimate query from Postgres (#26810)
* Remove incremental estimate query from Postgres

* Bump dockerfile + documentation
2023-05-31 17:16:24 -04:00
Rodi Reich Zilberman
c1ea190bfc Log full vacuum info (#26633)
* Log full vacuum info

* sanity

* commit change to force CI check

* improve queries

* Improve warning

* bump version and release note
2023-05-30 18:48:57 -04:00
Subodh Kant Chaturvedi
9b9809b006 fix(cdc): limit queue size to lower memory consumption (#26473)
* fix(cdc): limit queue size to lower memory consumption

* add queue size attribute in spec

* disable retries

* fix log

* review comments

* add validation test for queue size

* update expected spec

* bump version + changelog

* update metadata files

---------

Co-authored-by: Ben Church <ben@airbyte.io>
2023-05-25 22:34:40 +05:30
Duy Nguyen
1a45a66133 Source Postgres: Number(38,0) is incorrectly translated to a destination Double instead of BigInt (#25898)
* Translate Numeric with no decimals to BigInt + relevant tests
* Update CDC to handle numeric/decimal arrays correctly
* Revert spacing
* Preserve Decimal scale of numeric columns
* Bumped versions to 2.0.29 + docs
* Bumped versions in metadata.yaml
* Automated Change

---------

Co-authored-by: nguyenaiden <nguyenaiden@users.noreply.github.com>
2023-05-18 13:02:27 -07:00
Deepansh Mathur
ee2ca15016 Update postgres.md (#25950)
The numbering of the steps was wrong.

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Ben Church <ben@airbyte.io>
2023-05-11 18:44:52 -07:00
Subodh Kant Chaturvedi
94ed6554ea upgrade debezium version to 2.2.0.Final (#25401)
* upgrade debezium version to 2.2.0.Final

* bump version + changelog

* auto-bump connector version

* manual bump version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-27 22:12:20 +05:30
Jonathan Pearlin
a38af089e9 Track stream status in source (#24971)
* WIP Track stream status in source

* Revert formatting

* Revert formatting changes

* Remove unnecessary file

* Automated Change

* Automated Change

* Use new stream status trace message

* Rename class

* Remove unnecessary import

* Formatting

* Add tests

* Fix compile issues

* Automated Commit - Formatting Changes

* Remove TODO

* Fix compilation error

* Split STOPPED into INCOMPLETE and COMPLETE

* Remove unused import

* Changelog updates for source-postgres

* Remove unused import

* auto-bump connector version

---------

Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-26 15:14:25 -05:00
Akash Kulkarni
4164cd900e Revert "Better formatting/logging for pre-sync data" (#25560)
* Revert "Better formatting/logging for pre-sync data (#25459)"

This reverts commit 0d9b7b4f7b.

* Fix dockerfile + docs

* auto-bump connector version

---------

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2023-04-26 13:16:31 -05:00