1
0
mirror of synced 2025-12-21 02:51:29 -05:00
Commit Graph

375 Commits

Author SHA1 Message Date
Subodh Kant Chaturvedi
0160da7a61 update changelog to include debezium version upgrade (#13844) 2022-06-22 18:15:36 +05:30
Adam
a14fbda19f 📄 Postgres source: fix CDC setup order in docs (#13949)
* postgres source: fix CDC setup order docs

* Update docs/integrations/sources/postgres.md

Co-authored-by: Liren Tu <tuliren@gmail.com>
2022-06-21 11:52:06 -07:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
LiRen Tu
26f462327b 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823)
* Bump postgres version

* Update doc

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-15 18:44:53 -07:00
LiRen Tu
6964fe2740 📝 Postgres source: add troubleshooting for postgres hot standby (#13784)
* Add troubleshooting section in postgres doc

* Update doc

* Add one more solution
2022-06-14 22:02:42 -07:00
VitaliiMaltsev
f5a6a28211 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549)
* Postgres Source: fixed truncated precision if the value of the millisecond or second is 0

* check CI with 1.15.3 testcontainer

* check CI with 1.15.3 testcontainer

* returned latest version of testcontainer

* fixed checkstyle

* fixed checkstyle

* returned latest testcontainer version

* updated CHANGELOG

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-14 23:39:01 +03:00
Greg Solovyev
2cd62002be Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745)
* Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails
* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-13 23:28:46 -07:00
VitaliiMaltsev
a3ca3ab6b2 🐛 Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655)
* Postgres Source: fixed unsupposted date-time datatypes during incremental sync

* updated CHANGELOG

* add tests for incremental cursor check

* removed star import

* Postgres Source: fixed unsupposted date-time datatypes during incremental sync

* updated CHANGELOG

* add tests for incremental cursor check

* removed star import

* add timestamp datatype test

* Bump version in Dockerfile

* auto-bump connector version

Co-authored-by: grishick <greg@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-13 13:58:26 -07:00
LiRen Tu
fd2048da53 🎉 Postgres source: publish new version with JDBC fetch size v2 (#13576)
* Update postgres doc

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 19:01:50 -07:00
LiRen Tu
545a7a3eb6 🎉 JDBC source: adjust fetch size based on max memory and max row size (#13435)
* Switch to measure max row byte size

* Reduce fetch size change logs

* Update unit tests

* Determine jdbc buffer size based on max memory

* Bump postgres version

* Bump postgres version

* Bump mysql version

* Bump mssql version

* Format java code

* Increase hikari connection timeout

* Update data source default parameters

* auto-bump connector version

* Mark postgres 0.4.21 as not published

* Revert "Bump mysql version"

This reverts commit ad9135258c.

* Fix unit test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 14:29:34 -07:00
Yurii Bidiuk
4b266c1b7d 🐛 Source Postgres: write hstore as json (#13367)
* src-postgres: write hstore as json

* fix PR remarks

* bump version

* fix PR comments

* add text blocks
2022-06-06 22:25:29 +03:00
LiRen Tu
9648a5ceb6 Update postgres doc about cdc publication (#13433)
* Update postgres doc about cdc publication

* Update wording
2022-06-02 09:25:22 -07:00
VitaliiMaltsev
0a56b1d43f Postgres Source: add timezone awareness and handle BC dates (#13166)
* Postgres Source: add timezone awareness and handle BC dates

* fixed checkstyle

* add tests

* updated changelog

* removed star import

* fixed tests

* refactoring

* removed star import

* fixed bytea type

* created final static constants

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-01 10:46:39 +03:00
Yurii Bidiuk
16de42fde3 🎉Source Postgres (CDC) - add support of tsquery type (#13083)
* cdc - add support of tsquery type

* rebase on master

* Add changes made by the build

* Update version in Dockerfile

* auto-bump connector version

Co-authored-by: grishick <greg@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-28 22:03:59 -07:00
Yurii Bidiuk
fed651ba7c Source Posgres: [CDC] allow null values for columns (#13016)
* [cdc] source postgres: add optional to schema fields

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-27 13:45:18 -07:00
Eugene
c85e131d92 🎉Source-postgres: added custom JDBC parameters (jdbc_url_params) field (#12840)
* [10729] source-postgres: added custom JDBC parameters field
2022-05-17 15:51:13 +03:00
LiRen Tu
7d16fdafa1 🐞 Postgres source: fix azure database catalog retrieval (#12834)
* Remove host in azure username

* Bump version

* Fix test

* Replace host with jdbc_url

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-13 01:35:00 -07:00
LiRen Tu
3e6151a35a 🎉 Source postgres: publish fix of missing streams (#12689)
* Add logging

* Bump connector version

* Mute fb errors

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-08 21:06:06 -07:00
oneshcheret
cc6e18320b Postgres source: skipping columns with nullable values (#11760)
* Postgres source: skipping nullable columns during

* Postgres source: handling null value

* Postgres source: bump versions

* Postgres source: revert bump versions in definition yaml

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-06 16:49:06 -07:00
LiRen Tu
35f2aa9aed 🎉 Jdbc sources: publish new version with adaptive fetch size (#12480)
* Default scaffold to use adaptive streaming config

* Switch more connectors to use adaptive streaming config

* Bump version for cockroach db

* Bump version for db2

* Bump mssql version

* Bump mysql version

* Bump oracle version

* Bump postgres version

* Bump redshift version

* Bump snowflake version

* Bump tidb version

* auto-bump connector version

* Fix db2 findbug issue

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix more findbug issues

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix findbug issue for mysql-strict-encrypt

* Fix findbugs issue for oracle source

* auto-bump connector version

* Remove suppress warnings annotation

* Fix oracle encrypt tests

* Fix oracle encrypt acceptance test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 23:14:58 -07:00
Greg Solovyev
597d6ccbb8 Reconcile versions in git with versions in docker hub (#12368) 2022-04-26 11:41:59 -07:00
Eugene
d84df02f26 🎉Source-postgres: fixed roles for fetching materialized view processing (#11798)
* [10389] source-postgres: fixed roles for fetching materialized view processing
2022-04-08 20:05:41 +03:00
andriikorotkov
1d66672fad 🐛 Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats (#10508)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style

* updated cursor for jdbc sources

* updated cursor for jdbc sources

* updated cursor for jdbc sources
2022-02-22 13:26:10 +02:00
andriikorotkov
9d401fc9f7 🎉 Updated timestamp transformation with microseconds (#10242)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style
2022-02-18 14:51:58 +02:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
oneshcheret
84d1456fcf JDBC sources: update for sources that not have permissions issues (#10173)
* JDBC sources: update for sources that not have permissions issues

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: bump versions

* Postgres, CockroachDB and DB2 sources: bump source specs

* Postgres, CockroachDB and DB2 sources: revert changes for postgres-strict-encrypt
2022-02-09 19:23:23 +02:00
LiRen Tu
dc20baf9d7 Remove redundant github issue links in postgres doc (#10027)
* Remove redundant github issue links

* Remove redundant word

* Update wording

* Update wording
2022-02-02 23:46:42 -08:00
LiRen Tu
fa6ebdf3e3 📝 Source postgres: add workaround to sync a subset of columns (#10024)
* Add workaround for partial table permission issue in doc

* Update doc

* Update doc

* Update doc

* Remove dedundant sentence

* Update doc
2022-02-02 21:36:02 -08:00
Serhii Chvaliuk
abf383f896 Update fields in source-connectors specifications: postgres (#9807)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-01-31 16:57:48 +02:00
Andrii Leonets
4a290fa9ff [#9554 PR] Postgres/Mssql Source: Increase version (#9753)
* #9554 PR

* format

* incr version for Postgres

* Update docs/integrations/sources/postgres.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* test upd

* test upd

* incr ver for mssql as effected source

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-01-25 22:39:33 +02:00
Eugene
4534703589 🎉Source Postgres: Set up connection - add schema selection (#9360)
* [1435] Source Postgres: Set up connection - added schema selection
2022-01-13 16:24:38 +02:00
Eugene
d791972278 🐛Source-postgres: added materialized views processing (#9116)
* [9012] Source-postgres: added materialized views processing
2022-01-05 21:33:58 +02:00
Augustin
06fc680e4b fix anchors in postgres doc (#9219) 2022-01-03 17:59:48 +01:00
Yurii Bidiuk
cd30cf4fca bump version for affected sources from #8749 (#8958)
* bump versions

* fix tests for cockroachdb

* update changelog
2021-12-24 14:41:22 +02:00
LiRen Tu
ff4b83bb1f 🎉 Source Postgres: support all Postgres 14 types (#8726)
* Add skeleton to support all postgres types

* Consolidate type tests

* Fix corner cases

* Bump postgres version

* Add tests for time and timetz

* Format code

* Revert date to timestamp

* Update comment

* Fix unit tests

* 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (#8749)

* support array for jdbc sources

* fixed PR comments, added test cases

* added more elements for test case

* Fix test case

* add array test case for JdbcSourceOperations

Co-authored-by: Liren Tu <tuliren.git@outlook.com>

* Revert changes to support special number values

Postgres source cannot handle these special values yet
See https://github.com/airbytehq/airbyte/issues/8902

* Revert infinity and nan assertion in unit tests

This reverts commit 3bee7d19ea.

* Update documentation

* Bump postgres source version in seed

Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
2021-12-19 23:59:43 -08:00
oneshcheret
fcf7491fee 🐛 Validate incorrect handling '\n' symbols in ssh key (#8371)
* 🐛 Validate \n in ssh key

* bump versions for ssh key connectors

* update versions for ssh key connectors

* temporal fix for checking failed tests

* revert temp changes and destination oracle version

* bump versions in config for ssh key related connectors
2021-12-03 20:11:51 +02:00
Yurii Bidiuk
8726d39450 #7727 source postgres: add several types to integration tests (#7913) 2021-12-02 13:59:51 +02:00
LiRen Tu
aeb15cea8f 🐛 Source postgres: fix money type with amount > 1000 (#7995)
* Fix postgres money type > 1000

* Convert money to double

* Bump source postgres version

* Do not bump version in seed yet

* Check type ignoring case

* Test insertion with currency symbol

* Test parsing money string method

* Add test case with currency symbol and negative values

* Update version date

* Bump seed version
2021-11-28 23:46:43 -08:00
Alexander Tsukanov
6e47a8f990 🐛 Source Postgres: views are not listed during schema discovery (#8266)
* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Remove GRANTEE and PRIVILEGE constants from AbstractJdbcSource.java

* airbyte-8254 Version bumped.

* airbyte-8254 Version bumped.

* airbyte-8254 Version bumped.
2021-11-26 18:22:20 +02:00
Alexander Tsukanov
dc04654456 🎉 Source PostgreSQL: Handle the case when DB user doesn't has privileges to select the table (#8010)
* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Added docs, bump dockerfiles

* airbyte-5172 Fix PR comments.

* airbyte-5172 Fixed PR comments.

* airbyte-5172 bumped connector version.
2021-11-19 18:02:40 +02:00
oneshcheret
7cf31ea50c 🐛 Fix data type tests in CdcPostgresSourceDatatypeTest (#7339)
* Fix data type tests in CdcPostgresSourceComprehensiveTest

* update style format

* bump version for postgres source

* bump version for postgres source in json definition

* remove unnecessary comments from test and bump version for postgres strict encrypt source

* resolved potential conflicts with debezium utils in mssql converter implementation

* resolved potential conflicts with debezium utils in mssql converter implementation

* Update notes for money type in postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* added test cases for converting data values for postgres cdc, remove time zone utc from test container

* remove redundant void message from test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update style format

* fix time zone in DebeziumConverterUtilsTest

* set utc time zone in DataTypeUtils

* set utc time zone for date format

* revert changes regarding timezone in date format, disable tests with number and duration

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-02 17:46:42 +02:00
Sherif A. Nada
d9261c6290 Remove outdated block saying postgres doesn't support GCP CDC 2021-10-28 14:59:01 -07:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Jenny Brown
2e5fbba434 Clarify ssh private key format for ssh tunnels (#6585)
* Clarify ssh private key format for ssh tunnels
* Improved SSH Tunnel key generation steps, fixed formatting
* Modified wording 'app' to 'connector' for consistency
* Ran format
2021-10-04 13:59:47 -05:00
Abhi Vaidyanatha
ee05b314e8 Add Schema Selection to Postgres feature table (#6703) 2021-10-04 00:29:23 -07:00
Abhi Vaidyanatha
e584685ddb 📚 Docs Refresh: Postgres Source (#6664)
* Docs Refresh: Postgres Source

* Add CDC back to the steps.

* Add note about TLS.

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update SSH Tunnel feature.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-03 22:49:48 -07:00
julianopiovezan
911998baf8 Fix typo in CREATE PUBLICATION statement. (#6476) 2021-09-28 16:43:51 -03:00
Charles
8ad43afb07 SSH for Postgres Destination (#5743)
Co-authored-by: George Claireaux <phlair@users.noreply.github.com>
2021-09-07 17:06:25 -07:00
Charles
7bf531a967 SSH for Postgres Source (#5742) 2021-09-02 11:32:04 -07:00
irynakruk
954d7ccbb2 🐛 Source Postgres: fix CDC OOM issue (#5304)
* Added functionality for pgoutput plugin

* Updated docs and tests

* refactoring

* bumped version;
fixed test

* updated description;

* bumped version;

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-08-19 14:49:57 +03:00