1
0
mirror of synced 2025-12-22 11:31:02 -05:00
Commit Graph

138 Commits

Author SHA1 Message Date
Liren Tu
cf7698e003 🎉 Postgres source: make initial cdc waiting time configurable (#14451)
* Add initial waiting time param

* Pass in first record wait time from config

* Set initial waiting time to 10s in tests

* Log first record waiting time

* Fix expected_spec.json

* Reduce initial waiting seconds in test more

* Add tests

* Complete tests

* Refactor code

* Bump version

* Fix pr id in changelog

* Set shorter initial waiting time in one more test

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-21 17:31:09 -07:00
VitaliiMaltsev
df445713aa Remove additionalProperties from JDBC source connectors (#14574)
* Remove additional properties from JDBC specs

* fixed SnowflakeSourceAuthAcceptanceTest

* fixed OracleStrictEncryptJdbcSourceAcceptanceTest

* fixed Oracle Strict encrypt expected spec

* updated Changelog

* fixed CockroachDbSourceTest

* fixed CockroachDb Tests

* fixed CockroachDb Tests for Linux OS

* bump versions

* bump clickhouse-source-strict-encrypt

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed dependencies for Clickhouse Source

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* fixed OracleSourceDatatypeTest

* auto-bump connector version [ci skip]

* rollback ClickHouse Source changelog

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-21 11:01:34 +03:00
andriikorotkov
2dc4759264 Source postgres: Add SSL certificates to source postgres (#13840)
* added ssl certificates for postgres source

* added command for remove client private key after transformation to encrypted key with .pk8 extension

* added connection with CA and client certificates for postgres destination

* updated code style

* moved common methods to the common class

* moved common methods to the common class

* fixed remarks

* updated postgres source tests

* added minor changes to spec and added fixes to password mechanism

* updated postgres source tests

* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors

* fixed check style

* updated documentation and versions of connectors

* updated ordrs in test spec

* fixed minor remarks in specs and expected_specs

* fixed minor remarks in specs and expected_specs

* fixed Dockerfile

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* rollback changes for postgres destination and move them to a new pull request

* rollback changes for postgres destination and move them to a new pull request

* fixed code style

* fixed code style

* fixed code style

* fixed code style

* fixed code style

* auto-bump connector version

* updated version of postgres strict-encrypt source

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-18 12:58:53 +03:00
Ryan Fu
5edf3fed71 🚨 Validate source JDBC url parameters 🚨 (#14586)
* Moved toDatabaseConfigStatic subfunction into the main toDatabaseConfig function

* Moved Postgres JDBC_URL_PARAMS_KEY into JdbcUtils to reduce redundancy

* Migrated useSsl method into JdbcUtils to remove duplication

* Created constants for config parameters to match PostgresDestination

* Add validation check for overwritten connection properties

* Added test coverage for useSsl and clarifying javadoc comments

* Updated useSsl tests to match linter and javadoc comments for useSsl

* Fixed isCDC method with proper PostgresUtil function and corresponding imports

* Bumping Dockerfile and adding changelog description

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-15 11:24:59 -07:00
Benoit Moriceau
939afb7a07 Force legacy if feature flag not set (#14694)
* Force legacy if feature flag not set

* Fix acceptance test

* bump version

* PR comments

* Revert Integration test

* Update some integration test

* asset legacy type and data is correct in source tests

* better diff

* disable test cause its not valid for LEGACY state (#14703)

* auto-bump connector version

Co-authored-by: lmossman <lake@airbyte.io>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-14 10:53:53 -07:00
Tuhai Maksym
4738160efa 14230: postgres metadata calculations (#14602)
* 14230: calculations added

* 14230: text rephrased
2022-07-14 13:35:29 +03:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
Tuhai Maksym
dcf4650189 14230: doc updated (#14490) 2022-07-07 14:23:06 +03:00
Liren Tu
baebe1619c Release new postgres strict encrypt version (#14331)
* Bump postgres strict encrypt version

* Update changelogs

* Update doc
2022-06-30 21:50:15 -07:00
Andrii Leonets
8b19a70b25 Source Postgres : use more simple and comprehensive query to get selectable tables (#14251)
* use more simple and comprehensive query to get selectable tables

* cover case when schema is not specified

* add test to check discover with different ways of grants

* format

* incr ver

* incr ver

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-30 23:58:30 +03:00
Subodh Kant Chaturvedi
4fd3cb1f11 release new postgres source connector version 0.4.29 (#14265)
* release new postgres source connector version 0.4.29

* add changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-30 01:16:39 +05:30
Greg Solovyev
8ebd5a0ab0 Improve documentation for Postgres Source (#13830)
* Improve documentation for Postgres Source
 * add information about additional JDBC params
 * add anchors for doc sections
 * fix link to CDC on Bare Metal
 * add more details about parsing date/time values
 * add doc link to SSH fields
2022-06-27 15:05:16 -07:00
Subodh Kant Chaturvedi
0160da7a61 update changelog to include debezium version upgrade (#13844) 2022-06-22 18:15:36 +05:30
Adam
a14fbda19f 📄 Postgres source: fix CDC setup order in docs (#13949)
* postgres source: fix CDC setup order docs

* Update docs/integrations/sources/postgres.md

Co-authored-by: Liren Tu <tuliren@gmail.com>
2022-06-21 11:52:06 -07:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
LiRen Tu
26f462327b 🎉 Postgres source: prepare for removing strict-encrypt variant (#13823)
* Bump postgres version

* Update doc

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-15 18:44:53 -07:00
LiRen Tu
6964fe2740 📝 Postgres source: add troubleshooting for postgres hot standby (#13784)
* Add troubleshooting section in postgres doc

* Update doc

* Add one more solution
2022-06-14 22:02:42 -07:00
VitaliiMaltsev
f5a6a28211 🐛 Postgres Source: fixed truncated precision if the value of the milliseconds or seconds is 0 (#13549)
* Postgres Source: fixed truncated precision if the value of the millisecond or second is 0

* check CI with 1.15.3 testcontainer

* check CI with 1.15.3 testcontainer

* returned latest version of testcontainer

* fixed checkstyle

* fixed checkstyle

* returned latest testcontainer version

* updated CHANGELOG

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-14 23:39:01 +03:00
Greg Solovyev
2cd62002be Fallback to parsing datetime and time strings w/ and w/o timezones in case DateTimeParseException is thrown (#13745)
* Fall back to parsing w/ or w/o TZ if parsing a date or a time string fails
* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-13 23:28:46 -07:00
VitaliiMaltsev
a3ca3ab6b2 🐛 Postgres Source: fixed unsupported date-time datatypes during incremental sync (#13655)
* Postgres Source: fixed unsupposted date-time datatypes during incremental sync

* updated CHANGELOG

* add tests for incremental cursor check

* removed star import

* Postgres Source: fixed unsupposted date-time datatypes during incremental sync

* updated CHANGELOG

* add tests for incremental cursor check

* removed star import

* add timestamp datatype test

* Bump version in Dockerfile

* auto-bump connector version

Co-authored-by: grishick <greg@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-13 13:58:26 -07:00
LiRen Tu
fd2048da53 🎉 Postgres source: publish new version with JDBC fetch size v2 (#13576)
* Update postgres doc

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 19:01:50 -07:00
LiRen Tu
545a7a3eb6 🎉 JDBC source: adjust fetch size based on max memory and max row size (#13435)
* Switch to measure max row byte size

* Reduce fetch size change logs

* Update unit tests

* Determine jdbc buffer size based on max memory

* Bump postgres version

* Bump postgres version

* Bump mysql version

* Bump mssql version

* Format java code

* Increase hikari connection timeout

* Update data source default parameters

* auto-bump connector version

* Mark postgres 0.4.21 as not published

* Revert "Bump mysql version"

This reverts commit ad9135258c.

* Fix unit test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 14:29:34 -07:00
Yurii Bidiuk
4b266c1b7d 🐛 Source Postgres: write hstore as json (#13367)
* src-postgres: write hstore as json

* fix PR remarks

* bump version

* fix PR comments

* add text blocks
2022-06-06 22:25:29 +03:00
LiRen Tu
9648a5ceb6 Update postgres doc about cdc publication (#13433)
* Update postgres doc about cdc publication

* Update wording
2022-06-02 09:25:22 -07:00
VitaliiMaltsev
0a56b1d43f Postgres Source: add timezone awareness and handle BC dates (#13166)
* Postgres Source: add timezone awareness and handle BC dates

* fixed checkstyle

* add tests

* updated changelog

* removed star import

* fixed tests

* refactoring

* removed star import

* fixed bytea type

* created final static constants

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-01 10:46:39 +03:00
Yurii Bidiuk
16de42fde3 🎉Source Postgres (CDC) - add support of tsquery type (#13083)
* cdc - add support of tsquery type

* rebase on master

* Add changes made by the build

* Update version in Dockerfile

* auto-bump connector version

Co-authored-by: grishick <greg@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-28 22:03:59 -07:00
Yurii Bidiuk
fed651ba7c Source Posgres: [CDC] allow null values for columns (#13016)
* [cdc] source postgres: add optional to schema fields

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-27 13:45:18 -07:00
Eugene
c85e131d92 🎉Source-postgres: added custom JDBC parameters (jdbc_url_params) field (#12840)
* [10729] source-postgres: added custom JDBC parameters field
2022-05-17 15:51:13 +03:00
LiRen Tu
7d16fdafa1 🐞 Postgres source: fix azure database catalog retrieval (#12834)
* Remove host in azure username

* Bump version

* Fix test

* Replace host with jdbc_url

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-13 01:35:00 -07:00
LiRen Tu
3e6151a35a 🎉 Source postgres: publish fix of missing streams (#12689)
* Add logging

* Bump connector version

* Mute fb errors

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-08 21:06:06 -07:00
oneshcheret
cc6e18320b Postgres source: skipping columns with nullable values (#11760)
* Postgres source: skipping nullable columns during

* Postgres source: handling null value

* Postgres source: bump versions

* Postgres source: revert bump versions in definition yaml

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-06 16:49:06 -07:00
LiRen Tu
35f2aa9aed 🎉 Jdbc sources: publish new version with adaptive fetch size (#12480)
* Default scaffold to use adaptive streaming config

* Switch more connectors to use adaptive streaming config

* Bump version for cockroach db

* Bump version for db2

* Bump mssql version

* Bump mysql version

* Bump oracle version

* Bump postgres version

* Bump redshift version

* Bump snowflake version

* Bump tidb version

* auto-bump connector version

* Fix db2 findbug issue

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix more findbug issues

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* Fix findbug issue for mysql-strict-encrypt

* Fix findbugs issue for oracle source

* auto-bump connector version

* Remove suppress warnings annotation

* Fix oracle encrypt tests

* Fix oracle encrypt acceptance test

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 23:14:58 -07:00
Greg Solovyev
597d6ccbb8 Reconcile versions in git with versions in docker hub (#12368) 2022-04-26 11:41:59 -07:00
Eugene
d84df02f26 🎉Source-postgres: fixed roles for fetching materialized view processing (#11798)
* [10389] source-postgres: fixed roles for fetching materialized view processing
2022-04-08 20:05:41 +03:00
andriikorotkov
1d66672fad 🐛 Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats (#10508)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style

* updated cursor for jdbc sources

* updated cursor for jdbc sources

* updated cursor for jdbc sources
2022-02-22 13:26:10 +02:00
andriikorotkov
9d401fc9f7 🎉 Updated timestamp transformation with microseconds (#10242)
* updated timestamp transformation with microseconds

* updated timestamp transformation with microseconds

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql datatype tests

* updated mysql tests

* removed extra logs

* fixed cursor for timestamp

* updated test

* updated type transformation

* updated type transformation

* updated type transformation

* updated tests

* updated oracle tests

* updated documentations and connectors versions

* updated documentations and connectors versions

* fix code style
2022-02-18 14:51:58 +02:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
oneshcheret
84d1456fcf JDBC sources: update for sources that not have permissions issues (#10173)
* JDBC sources: update for sources that not have permissions issues

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: updated case when user has none permitted table

* Postgres, CockroachDB and DB2 sources: bump versions

* Postgres, CockroachDB and DB2 sources: bump source specs

* Postgres, CockroachDB and DB2 sources: revert changes for postgres-strict-encrypt
2022-02-09 19:23:23 +02:00
LiRen Tu
dc20baf9d7 Remove redundant github issue links in postgres doc (#10027)
* Remove redundant github issue links

* Remove redundant word

* Update wording

* Update wording
2022-02-02 23:46:42 -08:00
LiRen Tu
fa6ebdf3e3 📝 Source postgres: add workaround to sync a subset of columns (#10024)
* Add workaround for partial table permission issue in doc

* Update doc

* Update doc

* Update doc

* Remove dedundant sentence

* Update doc
2022-02-02 21:36:02 -08:00
Serhii Chvaliuk
abf383f896 Update fields in source-connectors specifications: postgres (#9807)
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
2022-01-31 16:57:48 +02:00
Andrii Leonets
4a290fa9ff [#9554 PR] Postgres/Mssql Source: Increase version (#9753)
* #9554 PR

* format

* incr version for Postgres

* Update docs/integrations/sources/postgres.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* test upd

* test upd

* incr ver for mssql as effected source

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-01-25 22:39:33 +02:00
Eugene
4534703589 🎉Source Postgres: Set up connection - add schema selection (#9360)
* [1435] Source Postgres: Set up connection - added schema selection
2022-01-13 16:24:38 +02:00
Eugene
d791972278 🐛Source-postgres: added materialized views processing (#9116)
* [9012] Source-postgres: added materialized views processing
2022-01-05 21:33:58 +02:00
Augustin
06fc680e4b fix anchors in postgres doc (#9219) 2022-01-03 17:59:48 +01:00
Yurii Bidiuk
cd30cf4fca bump version for affected sources from #8749 (#8958)
* bump versions

* fix tests for cockroachdb

* update changelog
2021-12-24 14:41:22 +02:00
LiRen Tu
ff4b83bb1f 🎉 Source Postgres: support all Postgres 14 types (#8726)
* Add skeleton to support all postgres types

* Consolidate type tests

* Fix corner cases

* Bump postgres version

* Add tests for time and timetz

* Format code

* Revert date to timestamp

* Update comment

* Fix unit tests

* 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (#8749)

* support array for jdbc sources

* fixed PR comments, added test cases

* added more elements for test case

* Fix test case

* add array test case for JdbcSourceOperations

Co-authored-by: Liren Tu <tuliren.git@outlook.com>

* Revert changes to support special number values

Postgres source cannot handle these special values yet
See https://github.com/airbytehq/airbyte/issues/8902

* Revert infinity and nan assertion in unit tests

This reverts commit 3bee7d19ea.

* Update documentation

* Bump postgres source version in seed

Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
2021-12-19 23:59:43 -08:00
oneshcheret
fcf7491fee 🐛 Validate incorrect handling '\n' symbols in ssh key (#8371)
* 🐛 Validate \n in ssh key

* bump versions for ssh key connectors

* update versions for ssh key connectors

* temporal fix for checking failed tests

* revert temp changes and destination oracle version

* bump versions in config for ssh key related connectors
2021-12-03 20:11:51 +02:00
Yurii Bidiuk
8726d39450 #7727 source postgres: add several types to integration tests (#7913) 2021-12-02 13:59:51 +02:00