1
0
mirror of synced 2025-12-22 03:21:25 -05:00
Commit Graph

2421 Commits

Author SHA1 Message Date
Yurii Bidiuk
bb39b36016 Destiantion Kinesis: add required fields (#16952)
* Destiantion Kinesis: add required fields

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-22 18:03:26 +03:00
Akash Kulkarni
c83a5953bd 🎉 Destination Snowflake : Update JDBC driver for Snowflake to 3.13.19 (#16839)
Destination Snowflake : Update JDBC driver for Snowflake to 3.13.19 (#16839)

Co-authored-by: Liren Tu <tuliren.git@outlook.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-21 16:27:51 -07:00
Amruta Ranade
a6af9611a6 added SSL Tunnel method rec (#16782) 2022-09-15 17:36:32 -04:00
Yurii Bidiuk
10e4aae891 Use LOG4J2 to wrap connectors logs to JSON format (#15668)
* Use LOG4J2 to wrap connectors logs to JSON format

* log connector's stack trace directly as a message

* add stack_trace field to json template

* bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* revert versions for destinations: postgres, mssql, mysql

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-14 23:27:13 +03:00
VitaliiMaltsev
022b8db345 Destination: BigQuery (denormalized typed struct) - Could not evaluate union, field is expected to be one of these: NULL, STRING. (#16401)
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Denormalized: Could not evaluate union on Mongo formatted type

* add test

* fixed BigQuery tests

* updated changelog

* replaced star import

* bump version

* auto-bump connector version [ci skip]

* fixed compilation error

* bump mongodb-strict-encrypt

* auto-bump connector version [ci skip]

* removed unused method

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-13 19:10:09 +03:00
Yurii Bidiuk
73653b3b02 Destiantion Mariadb: add custom JDBC params (#16391)
* Destiantion Mariadb: add custom JDBC params

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-12 12:07:00 +03:00
Yurii Bidiuk
8c56eeb02f Destination RabbitMQ: Make password marked as airbyte_secret (#16528)
* Destination RabbitMQ make password as airbyte_secret

* update changelog

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-09 13:54:27 +03:00
Andrii Leonets
40c84476d7 Destination BigQuery : Increase version for BQ PR #14023 (#16494)
* Increase version for BQ PR #14023

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-08 19:03:27 -07:00
Adam Fletcher
80397adbe1 🐛 Destination Postgres: fix missing database name URL Encoding (#16046)
* Fix missing database name URL Encoding in Postgres destination

* add unit test

* bump connector version and update doc

* auto-bump connector version [ci skip]

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-08 21:17:29 -03:00
Eugene
e121ceeb01 🎉Destination-clickhouse: added custom jdbc parameters field (#16444)
* [10717] Destination-clickhouse: added custom jdbc parameters field
2022-09-08 21:36:54 +03:00
Riley Brook
01723602d4 Update typo (Google Authentication) (#16458)
Typo reads "Instagram" where it should read "Google"
2022-09-08 15:18:33 -03:00
Oleksandr Sheheda
d105177e76 MQTT Destination: Password is not marked as a secret field in spec (#16263)
* [16219] MQTT Destination: Password is not marked as a secret field in spec

* [16219] MQTT Destination: Password is not marked as a secret field in spec

* [16219] MQTT Destination: Password is not marked as a secret field in spec
added logs for troubleshooting

* [16219] MQTT Destination: Password is not marked as a secret field in spec
updated maxinflight and connect_timeout for test only

* [16219] MQTT Destination: Password is not marked as a secret field in spec
updated maxinflight for test only

* [16219] MQTT Destination: Password is not marked as a secret field in spec
added config param max_in_flight for test

* [16219] MQTT Destination: Password is not marked as a secret field in spec
made refactoring

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-07 13:36:27 +03:00
Yurii Bidiuk
31215cb5c8 Destination MongoDB: Remove additionalProperties:false (#16025)
* Remove additionalProperties:false from destination-mongodb

* bumb version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-02 13:25:24 +03:00
Rodi Reich Zilberman
fc13413c9b Enforce secure connection in destination-postgres-strict-encrypt (#16260)
* Enforce secure connection in destination-postgres-strict-encrypt

* bump docker image values and update changelog.

* Typo

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-02 00:47:19 -07:00
Liren Tu
fd547c64d7 🐛 Avro related destinations: fix field name clashing with combined restrict (#16243)
* Add test case for allOf and oneOf

* Bump version

* Add pr id

* Bump gcs version

* Bump dest jdbc

* Bump redshift

* Bump snowflake

* Bump databricks

* Bump bigquery

* Revert "Bump dest jdbc"

This reverts commit f10497e96a.

* Use a new pat to avoid api rate limit

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* Revert databricks bump

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-01 18:06:27 -07:00
Yevhen Sukhomud
5773c224bc Destination Redshift: custom JDBC parameters (#16208) 2022-09-01 18:09:04 +07:00
Daemonxiao
d4524032ae 🎉 New Destination: TiDB (#15592)
* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* Add new destination-tidb

* support sync

* Add normalization-tidb

* fix failed tests

* Add unnest marco

* fmt

* fmt

* fix integration test

* Update docs/integrations/destinations/tidb.md

Co-authored-by: Xiang Zhang <angwerzx@126.com>

* Update doc

* Update doc

* Update doc

* bump normalization version

* update normalization changelog

* run format

* add dest def

* generat spec

Co-authored-by: Xiang Zhang <angwerzx@126.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-08-31 16:50:27 -03:00
Haithem SOUALA
7834eefb96 :fix: Destination S3: connector check ignores the s3_bucket_path (#15207)
* fix: use correct var for s3 bucket path

* typo

* fix - use bucket path instead of bucket name

* fix: use correct var for s3 bucket path

* typo

* fix - use bucket path instead of bucket name

* add fix to changelog

* solve conflict

* solve conflict

* fix - use bucket path instead of bucket name

* solve md file conflict

* solve dockerfile conflict

* auto-bump connector version [ci skip]

* add eof

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-24 06:46:29 -03:00
Jagruti Tiwari
8288c16485 fix: replace airbyte oss with airbyte open source (#15885)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-08-24 01:01:53 -03:00
VitaliiMaltsev
3aefd3308f Bigquery Destinatioin throw exception if job failed (#15787)
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Destination throw exception if job failed

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-23 12:46:27 +03:00
Nataly Merezhuk
e1f44a13c0 Corrects MySQL destination documantion: deduped history is not supported. (#14854)
Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
2022-08-22 11:36:35 -04:00
andriikorotkov
d63a24ed41 🎉 Destination postgres: Add SSL certificates and update normalization (#14743)
* added ssl certificates for postgres source

* added command for remove client private key after transformation to encrypted key with .pk8 extension

* added connection with CA and client certificates for postgres destination

* updated code style

* moved common methods to the common class

* moved common methods to the common class

* fixed remarks

* updated postgres source tests

* added minor changes to spec and added fixes to password mechanism

* updated postgres source tests

* updated strict-encrypt postgres source and destination and added tests for SSL certificates for all postgres connectors

* fixed check style

* updated documentation and versions of connectors

* updated ordrs in test spec

* fixed minor remarks in specs and expected_specs

* fixed minor remarks in specs and expected_specs

* fixed Dockerfile

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed remarks

* fixed code style

* fixed connectors version in definition file

* updated postgres destination normalization

* updated postgres destination tests

* fixed code style for postgres source and destination

* pulled master changes

* removed allow mode for destination-postgres-strect-encrypt

* updated connectors version

* fixed custom DBT transformation and enabled test for it

* updated normalization version

* updated keystore password generation method

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-15 16:32:37 +03:00
Bhupesh Varshney
4b4b4993c1 Docs: fix broken link to parquet docs (#15632) 2022-08-15 08:02:38 -03:00
Andy Jih
e8a0d5c851 Docs updates for destinations affected by per-stream state (#15578)
* Docs updates for destinations affected by per-stream state

* adding -alpha tag
2022-08-11 15:37:22 -07:00
VitaliiMaltsev
a280113cce Destination S3: add LZO compression support (#15394)
* Fixed bucket naming for S3

* Destination S3: add LZO compression support for parquet files

* Destination S3: add LZO compression support for parquet files

* implemented logic for aarch64

* removed redundant logging

* updated changelog

* moved intstall of native-lzo lib to Dockerfile

* removed redundant logging

* add unit test for aarch64

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-10 20:44:51 +03:00
Tuhai Maksym
aaa3aaed01 15310: Destination Scylla: Handle per-stream state (#15399)
* 15310: Destination Scylla: Handle per-stream state

* 15399: test fix

* 15318: test fix

* 15318: updating version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-10 12:43:55 +03:00
Yuhui Shi
474e8c40f4 🎉 New Destination: SQLite (#15018)
* sqlite destination skeleton

* Add doc

* Add destination to  general docs

* clean unused files

* add seed and icon

* sqlite destination skeleton

* Add doc

* Add destination to  general docs

* add seed and icon

* correct config file

* remove unused import

* run seed spec file

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-08-09 14:00:38 -03:00
Liren Tu
69972d606d Bigquery destination: fix changelog and publish new version (#15409)
* Fix bigquery changelog

* Bump bigquery denormalized

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-08 11:05:23 -07:00
Liren Tu
df99de0a28 Destination azure: bump version (#15412)
* Bump azure version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-08 10:55:49 -07:00
Ilya Pantsyr
28b3db683f Docs: fix misspelled name in docs (#15392) 2022-08-08 11:52:33 -03:00
Greg Solovyev
95aeb39325 Fix typo in change log (#15343)
* Fix typo in several change logs

Co-authored-by: andrii.leonets <aleonets@gmail.com>
2022-08-05 09:13:24 -07:00
oneshcheret
823a79b69a S3, Databricks and Gcs destinations fix test and publish (#15360)
* Postgres source added items for array data type

* Postgres source updated tests for array data type

* S3 destination fix key pair oauth test

* S3 destination clean code

* S3 destination bump version

* S3 destination bump version

* Databricks and gcs destinations bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-05 17:21:56 +03:00
Oleksandr Sheheda
ac33e193f1 🎉 Destination DynamoDB: Handle per-stream state (#15350)
* [15304] 🎉 Destination DynamoDB: Handle per-stream state

* [15304] 🎉 Destination DynamoDB: Handle per-stream state

* [15304] 🎉 Destination DynamoDB: Handle per-stream state

* [15304] 🎉 Destination DynamoDB: Handle per-stream state

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-05 12:24:31 +03:00
Benoit Moriceau
a4a5662a0a Fix Mongo dest (#15211)
* Fix Mongo dest

* Update changelog

* auto-bump connector version [ci skip]

* bump strict encrypt version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-03 17:38:18 -07:00
Greg Solovyev
d3d60c1067 Align strict encrypt and regular connector versions for destination-mysql (#15258)
* Align strict endrypt and regular connector versions

* Retain changelog for strict encrypt
2022-08-03 13:47:35 -07:00
Greg Solovyev
67aab69a3c Align MS SQL regular and strict encrypt versions (#15260)
* Align MS SQL regular and strict encrypt versions

* Update changelog with the PR number
2022-08-03 13:08:46 -07:00
Marcelo Castro
76c4053282 Destination BigQuery: Enabling Application Default Credentials (#14784)
* Enabling different bigquery authentications

Current implementation only accepts Service Accounts. For developers
it is very desirable that we can login with Application Default Credentials
(ADC) https://cloud.google.com/sdk/gcloud/reference/auth/application-default

* bump version and update doc

* auto-bump connector version [ci skip]

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-03 15:18:34 -03:00
Andrii Leonets
e7b9e00949 Fix multiply log bindings (#14801)
* Fix multiply log bindings

* Exclude slf4j-reload4j

* Exclude slf4j-log4j12 for debezium

* Increase version for debezium related sources and json converter related destinations

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* rebump bigquery versions

* mark destinations s3, gcs as unpublishd

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2022-08-03 09:36:30 -07:00
Amruta Ranade
cc7e850a42 Update bigquery.md 2022-08-03 12:08:53 -04:00
Benoit Moriceau
8bf5986a0b Fix bq standard mode (#15180)
* Fix

* Add version and changelog

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-02 10:00:12 -07:00
oneshcheret
6b35b237a1 Snowflake destination: support key pair authentication (#14388)
* Snowflake destination: support key pair authentication.

* Snowflake destination: update docs

* Snowflake destination: support key pair authentication for normalization, added tests

* Snowflake destination: update normalization

* Snowflake destination: update way of read secrets for test

* Snowflake destination: moved test to another class for test purpose

* Snowflake destination: update secrets for test purpose

* Snowflake destination: revert changes added for test purpose

* Snowflake destination: changes added for test purpose

* Snowflake destination: updated required fields in specs

* Snowflake destination: clean up

* Snowflake destination: support encrypted key pair authentication (#14589)

* Snowflake destination: add support for ecrypted private key

* Snowflake destination: temp reverting for test purpose

* Revert "Snowflake destination: temp reverting for test purpose"

This reverts commit 260ecce6da.

* Snowflake destination: add passphrase and remove auth_type from required fields

* Snowflake destination: format code

* Snowflake destination: clean up

* Snowflake destination: update docs

* Normalization for Snowflake destination: added unit tests and change file creating process

* Normalization for Snowflake destination: renamed property passphrase to password

* Snowflake destination: apply changes from normalization

* Snowflake destination: clean code

* Snowflake destination: clean up

* auto-bump connector version [ci skip]

Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-28 18:40:54 +03:00
Troy
bb66bc724d Destination MongoDB: use SHA256 instead of MD5 (#14561)
* fix: use sha256 instead of md5

* bump connector version

* match strict-encrypt version with oss

* auto-bump connector version [ci skip]

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-27 18:47:09 -03:00
Stéphane Kattoor
5b6a2256bb Destination AWS Datalake: documentation update to match Airbyte template (#13716)
* Documentation update to match Airbyte template

* Update AWS Datalake doc
2022-07-27 17:29:05 -03:00
Oleksandr Sheheda
3751be47cb [10719] Destination Oracle: custom JDBC parameters (#13841)
* [10719] Destination Oracle: custom JDBC parameters

* [10719] Destination Oracle: custom JDBC parameters
fixed tests

* [10719] Destination Oracle: custom JDBC parameters
fixed tests

* [10719] Destination Oracle: custom JDBC parameters
fix for SSH oracle tests

* [10719] Destination Oracle: custom JDBC parameters
fixed test

* [10719] Destination Oracle: custom JDBC parameters
updated image tag

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-27 13:37:44 +03:00
Greg Solovyev
b4063172ff change jdbc clickhoue verstion into 0.3.2-patch9 (#14820)
* bump versions of Destination Clickhouse
* Align Destination Clickhouse  and strict encrypt versions

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> and @mzitnik
2022-07-19 15:32:04 -07:00
Evan Tahler
b0f559dfca Google Sheets Destination flushes on every state message and flush when buffer gets too large (#14751)
* Google Sheets Destination flushes on every state message

* add PR number to readme

* suggestions

* reverted clean change

* added memory allocation check for records_buffer(stream)

* Update airbyte-integrations/connectors/destination-google-sheets/destination_google_sheets/writer.py

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* auto-bump connector version

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-15 16:15:39 -07:00
Benoit Moriceau
b1ec1ed494 Bump mqtt connector (#14648)
* Bump mqtt connector

* match PR

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-15 10:06:45 -07:00
Benoit Moriceau
1b733182ea Bump elasticsearch version (#14640)
* Bump elasticsearch version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-15 10:01:04 -07:00
Ryan Fu
71ab5ff216 Ryan/sync oracle version number (#14736)
* Bump oracle version number to match #14618

* Removed additionalProperties: false from oracle

* remove additionalProperties:true from destination oracle specs

* Mark 0.1.17 as unpublished, because it was never pushed to Docker Hub

Co-authored-by: grishick <greg@airbyte.io>
2022-07-15 08:49:49 -07:00
Alexander Tsukanov
82297a9b18 🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (#14494)
airbyte-4184: Make S3 output filename configurable
2022-07-15 15:56:21 +03:00