1
0
mirror of synced 2026-01-02 21:02:43 -05:00
Commit Graph

353 Commits

Author SHA1 Message Date
Tuhai Maksym
aa28d448d8 12708: Add an option to use encryption with staging in Redshift Destination (#13675)
* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated
2022-06-22 14:31:04 +03:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753)
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-20 14:56:33 +03:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
George Claireaux
e63458fabb Publishing java connectors with better formatted stacktrace in Trace messages (#13852)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Fix md and lg spacing scss vars (#13886)

* Delete TEMP-testing-command.yml

* auto-bump connector version

* Delete publish-command.yml

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* Update _variables.scss

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
2022-06-17 16:15:24 +01:00
Edward Gao
5f843415db 📚 update MSSQL docs to reflect that we support normalization (#13826) 2022-06-16 08:16:00 -07:00
oneshcheret
8e54f4fd6e S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483)
* S3 destination: Updating processing data types for Avro/Parquet formats

* S3 destination: handle comparing data types

* S3 destination: clean code

* S3 destination: clean code

* S3 destination: handle case with unexpected json schema type

* S3 destination: clean code

* S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class

* S3 destination: clean code

* S3 destination: clean code

* GCS destination: Update data types processing for Avro/Parquet formats

* GCS destination: clean redundant code

* S3 destination: handle case with numbers inside array

* S3 destination: clean code

* S3 destination: add unit test

* S3 destination: update unit test cases with number types.

* S3 destination: update unit tests.

* S3 destination: bump version for s3 and gcs

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-14 14:15:58 +03:00
Andy
fe6eda5b37 🎉 Destination databricks: rename to databricks lakehouse (#13722)
* Update Databricks naming

* Update destination_spec

* Update BOOTSTRAP.md

* Update Dockerfile

* Update README.md

* Update spec.json

* Update databricks.md

* Update databricks.md

* Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>
2022-06-13 17:56:01 -07:00
Amruta Ranade
7ecfa13ee0 Updated BigQuery, Google Sheets, Facebook Marketing, Salesforce docs (#13717)
* initial changes

* Edited google Sheets doc

* More edits

* edited the intro and prereqs for BigQuery

* edited the data loading section

* more edits

* Grammatical edits

* Formatting edits
2022-06-13 14:25:41 -04:00
Simon Späti
93c413c0ed 📚 Destination Databricks: Clarify on Destination Format with Delta-Table (#13396) 2022-06-10 18:06:27 +02:00
Marcos Marx
862bf36f26 Docs: Destination ElasticSearch remove port from doc requirements (#13665) 2022-06-10 10:29:41 -03:00
Alexander Tsukanov
b45014da6b 🎉 Destination Redshift: Add "Loading Method" option to Redshift Destination spec and UI (#13415)
* airbyte-12709 Add "Loading Method" option to Redshift Destination spec.

* airbyte-12709: Fixed unit tests.

* airbyte-12709: Updated README.md.

* airbyte-12709: change the number of PR in redshift.md.

* airbyte-12709: Added backward compatibility for old json schema.

* airbyte-12709: Fix PR comments.

* airbyte-12709: Removed throwing an exception. Fixed PR comments.

* airbyte-12709: Bump the airbyte version.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-10 14:47:03 +03:00
LiRen Tu
bb46941867 🎉 Databricks destination: rename to Databricks delta lake (#13630)
* Rename databricks connector

* Rename connector in the seed

* Update changelog with pr id

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-08 18:22:55 -07:00
pcsoni89
b4e43c2ddd 🎉 Destination MSSQL: added custom JDBC parameters support. (#13054)
* 🎉 Destination MSSQL: added custom JDBC parameters support.

* 🎉  Destination MSSQL: added custom JDBC parameters support.

* 🎉  Destination MSSQL: added custom JDBC parameters support.

* airbyte-101718: Bumped version.

* auto-bump connector version

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Alexander Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-08 14:55:45 +03:00
LiRen Tu
e996c2c14f 🐛 BigQuery destination: always run GCS destination check for GCS staging method (#13579)
* Run gcs destination check all the time

* Add missing permissions to the status message

* Add more error messages

* Bump bigquery version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 17:52:49 -07:00
Yurii Bidiuk
d6b50c1ce0 Destination BigQuery: refactored spec and doc (#13424)
* change spec

* refactored readme file for bigquery

* change spec order

* bump version
2022-06-07 18:36:40 +03:00
Marcos Marx
9aa0220820 revert redshift and json operations (#13465) 2022-06-03 19:44:40 -03:00
Adam
071a8e9dc8 Destination Redshift: fixed array contents verification for SUPER (#13069)
* add array handling to json flatten

* bump version, add changelog entry

* test: add flatten unit tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-03 15:46:16 -03:00
VitaliiMaltsev
748291bb0c Snowflake Destination apply buffering strategy on GCS staging (#12952)
* Destination Snowflake apply buffering strategy for GCS

* Snowflake Destination apply buffering strategy on GCS staging

* updated CHANGELOG

* fixed checkstyle

* fixed compile error

* refactoring

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-01 10:28:02 +03:00
Misko Lee
721b5b69c7 Downgrade ElasticSearch batch size from 256mib to 32mib (#11752)
* Update ElasticsearchAirbyteMessageConsumerFactory.java

* add changleog and bump connector version seed file

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-05-30 14:37:32 -03:00
Alexander Tsukanov
5371f87ee6 🐛 Destination Redshift: Fix DATs blind spot - connection check (#13090)
* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Merged master.

* airbyte-12459: Merged master.

* auto-bump connector version

Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-25 19:14:10 +03:00
Baz
7b0a541040 🎉 Destination Google Sheets: update docs (#13151) 2022-05-24 22:44:23 +03:00
Eugene
d6f4ef4d38 [9964] destination-mqtt: fixed build tests failures (#13099)
* [9964] destination-mqtt: fixed build tests failures
2022-05-24 16:44:20 +03:00
Oleksandr Sheheda
44a43f0338 Improve check connection performance for all JDBC-related DBs (#12820)
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
2022-05-24 16:06:01 +03:00
Tuhai Maksym
70d48fa874 Destination-S3: Remove configurable part size (#13043)
* Destination-S3: Remove configurable part size

* Changelog updated

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-20 22:48:55 +03:00
Adam
9e31d026c8 Destination Redshift: Fixed maximum record size for SUPER type (#12940)
* fix #12680: use correct max size for SUPER type

* address review comments

* docs: add changelog entry

* bump version

* test: make large record actually too big

* fix: use a method instead of member variable for correct override

* fix: if overall object is invalid, don't check individual elements

* feat: add unit test for isValidData

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-20 10:32:21 -04:00
Chun-Sheng, Li
2c409ffaad Docs: Remove useless column (#13013) 2022-05-19 11:29:18 -04:00
LiRen Tu
5ddef8639a 📝 Databricks destination: clarify server hostname in the doc (#12943) 2022-05-17 20:37:07 -07:00
Mohit Reddy
58f20c455b Destination Kafka: Allow configuring PLAIN SASL (#12876)
* airbyte: Allow configuring PLAIN SASL mechanism for Kafka dest connector through WebApp

* bump up the version of kafka dest connector

* bump up the image tag for kafka destination connector in defs

* update _specs for kafka destination connector to match defs; SeedConnectorSpecs did not update it automatically
2022-05-17 12:27:52 -04:00
LiRen Tu
d6f8c2627b 📝 Update bigquery doc about hmac key troubleshooting (#12895) 2022-05-16 15:37:07 -07:00
LiRen Tu
7c35b03595 🎉 Databricks destination: use new jdbc driver and open source the connector (#12861)
* Migrate to public databricks jdbc driver

* Update documentation

* Bump version

* Format code

* Check in databricks in seed file

* Check in databricks 0.2.0

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-16 14:48:52 -07:00
VitaliiMaltsev
4804f746dc Destination Redshift fixed NPE in S3 staging (#12869)
* Destination Snowflake fixed npe in S3 staging

* removed star imports

* updated changelog

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-16 16:37:28 +03:00
LiRen Tu
e00f487d98 📝 Destination bigquery: mark service account as required for cloud (#12768)
* Update spec

* Update doc

* Bump version and update changelog

* Modify wording

* Add sample service account key json

* Add screenshots and common permission issues

* Refactor service account helper method

* Update log message

* Update version date in changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-15 01:28:36 -07:00
Andy
a3658ba13a Update spec to replace markdown (#12797)
* Update spec to replace markdown

* Update version

* S3 Destination: Update change log

* auto-bump connector version

Co-authored-by: oneshcheret <sashkaliife@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-12 19:16:00 +03:00
George Claireaux
440f3db8bb Destinations BigQuery & Snowflake: use latest base-java to emit AirbyteTraceMessage on errros (#12805)
* bumping BQ and Snowflake dests to get AirbyteTraceMessage from base-java

* add PR link to changelogs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-12 14:23:29 +01:00
VitaliiMaltsev
4b5bd2b5a4 Redshift Destination Apply buffering strategy (#12601)
* Redshift Destination: Apply buffering strategy

* add manifest uploading

* refactoring

* fixed checkstyle

* updated CHANGELOG

* removed redundant static

* airbyte-12265: Added  stagingOperations.onDestinationCloseOperations() to StagingConsumerFactory.java.

* airbyte-12265: Created operations and copiers java packages.

* safe delete of RedshiftCopyS3Destination.java

* rename tests

* bump version

* auto-bump connector version

Co-authored-by: Oleksandr Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-12 12:30:54 +03:00
Baz
feb0d2f378 🎉 New Destination: Implement Destination Google Sheets using CDK (#12135) 2022-05-12 01:10:28 +03:00
Stéphane Kattoor
66d760955d Fix typo in spec.json for destination AWS Datalake & fix python license plugin setup (#11811) 2022-05-06 01:28:00 -07:00
LiRen Tu
bd59726668 Publish new connectors to log offending json fields in json to avro conversion for debugging (#12578)
* Publish new connectors to log debugging info in json to avro conversion

* Add pull request id

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-04 11:47:07 -07:00
Edward Gao
43470a2a87 🎉 Destination Snowflake: Add option to stage encrypted files via S3; fix purge_staging_data (#12452) 2022-05-04 07:31:53 -07:00
Alexandre Girard
1021428998 Destination Bigquery: Update Dataset location field description (#12528)
* Update field description

* reset

* Bump

* reset

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-02 17:45:36 -07:00
Alexandre Girard
553024780f Destination Bigquery: Dataset location is a required field (#12477)
* Dataset location is required

* Update doc

* Bump versions

* reset

* Bump

* downgrade

* Revert "downgrade"

This reverts commit b482c54d6f.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 15:34:43 -07:00
oneshcheret
fd7f21e071 S3 destination: updating docs regarding certification (#11966)
* S3 destination: updating docs regarding certification

* S3 destination: updating docs by new template

* Apply suggestions from code review

Co-authored-by: Andy <andy@airbyte.io>

* S3 destination: updating docs by new template

Co-authored-by: Andy <andy@airbyte.io>
2022-04-29 18:17:13 +03:00
Eugene
6565315a04 Destination-Postgres: updated postgres.md documentation as per BETA template (#12279)
* Updated postgres.md documentation as per BETA template
2022-04-29 09:55:11 +03:00
VitaliiMaltsev
b16e13e9cf Redshift Destination: update spec (#12100)
* Redshift Destination: update spec

* update spec.json

* update links in spec.json

* added more links to spec.json | refactoring

* updated docs with stadard connector template

* added hyperlink to documentation for part_size field
2022-04-27 14:01:26 +03:00
Greg Solovyev
597d6ccbb8 Reconcile versions in git with versions in docker hub (#12368) 2022-04-26 11:41:59 -07:00
Greg Solovyev
cce3b21506 Manually bump destination-mssql version (#12291) 2022-04-26 08:52:46 -07:00
Greg Solovyev
bcf4f89ae8 Manually bump destination-oracle version (#12292) 2022-04-26 08:50:50 -07:00
noahkawasaki-airbyte
d46058e456 🎉 Destination Postgres: Add jdbc_url_params input (#12195)
* Add jdbc_url_params input for Destination Postgres
2022-04-25 08:12:31 -07:00
Greg Solovyev
63f6fc5712 Update destination-postgres version in Dockerfile and docs (#12290)
* Update destination-postgres version in Dockerfile and docs

* Manually bump destination-postgres version
2022-04-22 18:44:52 -07:00