1
0
mirror of synced 2025-12-21 11:01:41 -05:00
Commit Graph

244 Commits

Author SHA1 Message Date
Eugene
d791972278 🐛Source-postgres: added materialized views processing (#9116)
* [9012] Source-postgres: added materialized views processing
2022-01-05 21:33:58 +02:00
Augustin
06fc680e4b fix anchors in postgres doc (#9219) 2022-01-03 17:59:48 +01:00
Yurii Bidiuk
cd30cf4fca bump version for affected sources from #8749 (#8958)
* bump versions

* fix tests for cockroachdb

* update changelog
2021-12-24 14:41:22 +02:00
LiRen Tu
ff4b83bb1f 🎉 Source Postgres: support all Postgres 14 types (#8726)
* Add skeleton to support all postgres types

* Consolidate type tests

* Fix corner cases

* Bump postgres version

* Add tests for time and timetz

* Format code

* Revert date to timestamp

* Update comment

* Fix unit tests

* 🐛 Jdbc sources: switch from "string" to "array" schema type for columns with JDBCType.ARRAY (#8749)

* support array for jdbc sources

* fixed PR comments, added test cases

* added more elements for test case

* Fix test case

* add array test case for JdbcSourceOperations

Co-authored-by: Liren Tu <tuliren.git@outlook.com>

* Revert changes to support special number values

Postgres source cannot handle these special values yet
See https://github.com/airbytehq/airbyte/issues/8902

* Revert infinity and nan assertion in unit tests

This reverts commit 3bee7d19ea.

* Update documentation

* Bump postgres source version in seed

Co-authored-by: Yurii Bidiuk <35812734+yurii-bidiuk@users.noreply.github.com>
2021-12-19 23:59:43 -08:00
oneshcheret
fcf7491fee 🐛 Validate incorrect handling '\n' symbols in ssh key (#8371)
* 🐛 Validate \n in ssh key

* bump versions for ssh key connectors

* update versions for ssh key connectors

* temporal fix for checking failed tests

* revert temp changes and destination oracle version

* bump versions in config for ssh key related connectors
2021-12-03 20:11:51 +02:00
Yurii Bidiuk
8726d39450 #7727 source postgres: add several types to integration tests (#7913) 2021-12-02 13:59:51 +02:00
LiRen Tu
aeb15cea8f 🐛 Source postgres: fix money type with amount > 1000 (#7995)
* Fix postgres money type > 1000

* Convert money to double

* Bump source postgres version

* Do not bump version in seed yet

* Check type ignoring case

* Test insertion with currency symbol

* Test parsing money string method

* Add test case with currency symbol and negative values

* Update version date

* Bump seed version
2021-11-28 23:46:43 -08:00
Alexander Tsukanov
6e47a8f990 🐛 Source Postgres: views are not listed during schema discovery (#8266)
* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Source Postgres: views are not listed during schema discovery

* airbyte-8254 Remove GRANTEE and PRIVILEGE constants from AbstractJdbcSource.java

* airbyte-8254 Version bumped.

* airbyte-8254 Version bumped.

* airbyte-8254 Version bumped.
2021-11-26 18:22:20 +02:00
Alexander Tsukanov
dc04654456 🎉 Source PostgreSQL: Handle the case when DB user doesn't has privileges to select the table (#8010)
* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Do not list tables we do not have access to in the connection settings

* airbyte-5172 Added docs, bump dockerfiles

* airbyte-5172 Fix PR comments.

* airbyte-5172 Fixed PR comments.

* airbyte-5172 bumped connector version.
2021-11-19 18:02:40 +02:00
oneshcheret
7cf31ea50c 🐛 Fix data type tests in CdcPostgresSourceDatatypeTest (#7339)
* Fix data type tests in CdcPostgresSourceComprehensiveTest

* update style format

* bump version for postgres source

* bump version for postgres source in json definition

* remove unnecessary comments from test and bump version for postgres strict encrypt source

* resolved potential conflicts with debezium utils in mssql converter implementation

* resolved potential conflicts with debezium utils in mssql converter implementation

* Update notes for money type in postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* added test cases for converting data values for postgres cdc, remove time zone utc from test container

* remove redundant void message from test

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* update style format

* fix time zone in DebeziumConverterUtilsTest

* set utc time zone in DataTypeUtils

* set utc time zone for date format

* revert changes regarding timezone in date format, disable tests with number and duration

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-11-02 17:46:42 +02:00
Sherif A. Nada
d9261c6290 Remove outdated block saying postgres doesn't support GCP CDC 2021-10-28 14:59:01 -07:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Jenny Brown
2e5fbba434 Clarify ssh private key format for ssh tunnels (#6585)
* Clarify ssh private key format for ssh tunnels
* Improved SSH Tunnel key generation steps, fixed formatting
* Modified wording 'app' to 'connector' for consistency
* Ran format
2021-10-04 13:59:47 -05:00
Abhi Vaidyanatha
ee05b314e8 Add Schema Selection to Postgres feature table (#6703) 2021-10-04 00:29:23 -07:00
Abhi Vaidyanatha
e584685ddb 📚 Docs Refresh: Postgres Source (#6664)
* Docs Refresh: Postgres Source

* Add CDC back to the steps.

* Add note about TLS.

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/sources/postgres.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update SSH Tunnel feature.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2021-10-03 22:49:48 -07:00
julianopiovezan
911998baf8 Fix typo in CREATE PUBLICATION statement. (#6476) 2021-09-28 16:43:51 -03:00
Charles
8ad43afb07 SSH for Postgres Destination (#5743)
Co-authored-by: George Claireaux <phlair@users.noreply.github.com>
2021-09-07 17:06:25 -07:00
Charles
7bf531a967 SSH for Postgres Source (#5742) 2021-09-02 11:32:04 -07:00
irynakruk
954d7ccbb2 🐛 Source Postgres: fix CDC OOM issue (#5304)
* Added functionality for pgoutput plugin

* Updated docs and tests

* refactoring

* bumped version;
fixed test

* updated description;

* bumped version;

Co-authored-by: Iryna Kruk <iryna.o.kruk@globallogic.com>
2021-08-19 14:49:57 +03:00
Eugene
a78efe090b 🎉 JAVA-Based connectors: Bumped version for some javabased connector to start using Config Validator from core module (#5398)
* Updated some java-based connectors version to start using new json config validator from java core
2021-08-17 22:30:16 +03:00
Marcos Marx
d91ff7dcf1 Documentation add data types to databases (#5008)
* add postgres data types

* add mysql data types

* add mssql data types

* add oracle data type
2021-07-26 23:02:46 -03:00
Abhi Vaidyanatha
1f3acff869 Remove statement about Postgres connector being based on Singer (#4740)
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
2021-07-13 23:22:21 -07:00
Sherif A. Nada
71c7cecd74 Source postgres: document that replicating views is supported (#4008) 2021-06-28 00:11:37 -05:00
Marcos Marx
88d7953a18 Add CHANGELOG for Sources (#4215)
* add changelog for postres source

* add changelog for mysql source

* add changelog for mssql source

* add changelog for file source

* add changelog for google sheets

* add full changelog for file source

* add full changelog for google sheet source

* add full changelog for mssql source

* add changelog for http requests

* add changelog for salesforce source

* add changelog for facebook marketing source

* add changelog for google analytics source

* add changelog for shopify native source connector
2021-06-18 17:45:09 -03:00
Marcos Marx
a18b0e80d0 GitBook: [master] 161 pages and 75 assets modified 2021-05-16 22:51:02 +00:00
Jared Rhizor
beeee785f2 clarify publication and replication identity handling in cdc docs (#3258)
* clarify publication and replication identity handling in cdc docs

* remove character
2021-05-06 17:34:42 -07:00
Marcos Marx
7c70eb02cd GitBook: [master] 84 pages and 72 assets modified 2021-05-03 12:14:16 +00:00
Abhi Vaidyanatha
e378d40236 Documentation Reorganization (#3124)
* First reorganization pass.

* Stop auto-generating api docs html file.

* Update spelling

* Final cleanup.

* Final changes_REAL_actual_2_thisone

* fix path for generating api docs

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: jrhizor <me@jaredrhizor.com>
2021-04-29 15:18:52 -07:00
Davin Chia
42686add8a Release connectors with namespace change. (#2990)
Release all connectors affected by namespace change. Includes all JDBC sources and destinations.

Also add documentation for normalisation. Prerequisite to actually releasing 0.21.0-alpha.
2021-04-21 11:35:08 +08:00
Michel Tricot
e4326aa21e Revert "GitBook: [master] 25 pages and 46 assets modified"
This reverts commit 0b4eddc7cf.
2021-04-16 14:20:42 -07:00
Michel Tricot
0b4eddc7cf GitBook: [master] 25 pages and 46 assets modified 2021-04-16 20:29:16 +00:00
Jared Rhizor
2b19da82d8 postgres cdc (#2548)
* spike

* more

* debezium wip

* use oneof for configuration

* iterator wrapping structure

* push current

* working loop

* move capability into source

* hack it into a sharable state

* debezium test runner (#2617)

* CDC Wait for Values (#2618)

* output actual AirbyteMessages for cdc (#2631)

* message conversion

* fmt

* add lsn extraction and comparison (#2613)

* postgres cdc catalog (#2673)

* update cdc catalog

* A

* table selection for cdc (#2690)

* table selection for cdc

* fix broken merge

* also test double quote in name

* Add state management to CDC (#2718)

* CDC: Fix Producer/Consumer State Machine (#2721)

* CDC Postgres Tests (#2777)

* fix postgres cdc image name and run check before reading data (#2785)

* minor postgres cdc fixes

* add test and fix check behavior

* fix

* improve comment

* remove unused props, remove todos, add some more sanity tests (#2791)

* cdc: add offset store tests (#2793)

* clean (#2798)

* postgres cdc docs (#2784)

* cdc docs

* Update docs/integrations/sources/postgres.md

Co-authored-by: Charles <giardina.charles@gmail.com>

* address gcp

* learn too english

* add link

* add more disk space warnings

* add additional cdc use case

* add information on how to find postgresql.conf

* add how to find the file

Co-authored-by: Charles <giardina.charles@gmail.com>

* various merge conflict fixes (#2799)

* cdc standard tests (#2813)

* require cdc users to create publications & update docs (#2818)

* postgres cdc race condition

* working? but different process

* add additional logging to help debug in the future

* everything done except working config

* remove unintended change

* Use oneOf in PG CDC spec (#2827)

* add oneOf configuration for postgres cdc  (#2831)

* add oneof configuration for cdc postgres

* fmt

Co-authored-by: Charles <giardina.charles@gmail.com>

* fix test (#2834)

* fix test

* bump version

* add docs on creating replica identities (#2838)

* add docs on creating replica identities

* emphasize danger

* grammar

* bump pg version in source catalog

* generate seed files

Co-authored-by: cgardens <giardina.charles@gmail.com>
2021-04-09 16:36:58 -07:00
Shrif Nada
ff458bbd9d GitBook: [master] 111 pages modified 2021-01-26 04:14:55 +00:00
Jared Rhizor
4ab0464274 add postgres troubleshooting docs (#1653)
* add postgres troubleshooting docs

* update headings

* add to other db sources

* move connection troubleshooting to faq
2021-01-25 10:56:09 -08:00
John Lafleur
2acc3840d0 GitBook: [docs_feature_incremental] 33 pages modified (#1683)
Co-authored-by: John Lafleur <john@dataline.io>
2021-01-17 07:32:24 +11:00
Shrif Nada
8ffefbb79e GitBook: [master] 5 pages modified 2021-01-16 17:11:48 +00:00
John Lafleur
5532aa370a Updating Incremental within docs pages for all sources to make it more explicit (#1381)
* GitBook: [incremental_update_docs] 79 pages modified

* GitBook: [incremental_update_docs] one page modified

Co-authored-by: John Lafleur <john@dataline.io>
2020-12-18 21:03:34 +11:00
Charles
01cdeafdb1 release v0.8.0-alpha (#1339) 2020-12-16 14:01:57 -08:00
Jared Rhizor
e3297eb60c Add Stripe source docs (#373)
* GitBook: [jrhizor/stripe-docs] 8 pages and 15 assets modified

* Update stripe.md
2020-09-22 16:35:26 -07:00
johnlafleur
92777616f9 GitBook: [john_09/20/2020] 13 pages and 3 assets modified (#356) 2020-09-20 21:37:39 -07:00
Michel Tricot
e3737a2c06 Introducing Airbyte (#348) 2020-09-18 20:41:09 -07:00
Shrif Nada
fb0e89ad3e GitBook: [master] 29 pages modified 2020-09-18 17:48:35 +00:00
Sherif A. Nada
2aca6380d9 Add postgres source docs and template (#331) 2020-09-17 14:50:45 -07:00
Michel Tricot
1c1cc474fd GitBook: [master] 20 pages modified 2020-08-11 23:18:06 +00:00