1
0
mirror of synced 2025-12-22 03:21:25 -05:00
Commit Graph

2421 Commits

Author SHA1 Message Date
Jimmy Ma
6653cd30a3 Bump destination-postgres to 0.3.21 (#14479)
* Bump destination-postgres to 0.3.21

Goal is to trigger a publish of the arm64 build

* Update destination/postgres changelog

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-14 16:58:46 -07:00
VitaliiMaltsev
095b403c7b Remove additionalProperties: false from JDBC destination connectors (#14618)
* Remove additional properties from JDBC destinations spec

* Remove additional properties from JDBC destinations spec

* removed additionalProperties from Databricks destination and MSSql Destination

* updated Changelog

* bump version

* bump Oracle Strict encrypt

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-15 01:00:41 +03:00
Benoit Moriceau
8dc558d496 Bump local json connector (#14641)
* Bump local json connector

* update doc

* Fix publish

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-13 08:06:08 -07:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
Amruta Ranade
d4071a0ae5 source-salesforce,source-google-sheet,destination-snowflake: Updated field descriptions(#13729) 2022-07-07 15:55:52 +02:00
mtyoleg
9ac9e27ca7 Destination Clickhouse: Added socket_timeout property (#13516)
* Added socket_timeout property for destination-clickhouse, jdbc default of 30 sec is to low

* correct unit test

* Added socket_timeout property for destination-clickhouse, jdbc default of 30 sec is to low

* correct unit test

* bump connector version and udpate doc

* auto-bump connector version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-07-06 07:53:20 -03:00
VitaliiMaltsev
671039e03e Snowflake Source and Destination: updated docs for using network policies (#14409) 2022-07-06 12:59:39 +03:00
Liren Tu
98d087ba3d Release new destination s3 version to pick up latest change (#14332)
* Bump s3 version

* Update pr id

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-30 21:58:53 -07:00
Sophia Wiley
1db8e99072 Add Metrics section to Scaling Airbyte doc (#14224)
* Added metrics section to scaling airbyte doc

* Updated URL in doc

* Deleted link

* Added link

* Added backslashes before brackets that aren't links

* Edited note about tagged metrics

* Changed list

* Changed spacing

* Changed spacing

* Changed spacing

* Deleted period

* Fixed broken firebolt link

* Added tables

* Cleaned up wording in tables
2022-06-29 10:24:03 -07:00
VitaliiMaltsev
d3f4fdd8fc BigQuery Denormalized "airbyte_type": "big_integer" to INT64 (#14079)
* BigQuery Denormalized "airbyte_type": "big_integer" to INT64

* updated changelog

* added unit test

* removed star import

* fixed checkstyle

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-29 19:47:15 +03:00
Adam
79f7d060db 🐛 destination-redshift: use s3 bucket path for s3 cleanup (#14190) 2022-06-27 23:19:26 +02:00
Harshith Mullapudi
9c289516ee Harshith/test pr 13118 (#14192)
* Firebolt destination

* feat: Write method dropdown

* feat: Use future-proof Auth in SDK

* refactor: Move writer instantiation

* fix: tests are failing

* fix: tests are failing

* fix: tests are failing

* chore: added connector to definitions

* fix: formatting and spec

* fix: formatting for orbit

Co-authored-by: ptiurin <petro.tiurin@firebolt.io>
2022-06-28 02:16:30 +05:30
Adam
16d99b9b0c 🐛 Destination Redshift: use s3 bucket path for s3 staging operations (#13916) 2022-06-27 20:02:45 +02:00
VitaliiMaltsev
11ea066f47 Remove "additionalProperties": false from specs for connectors with staging (#14114)
* Remove "additionalProperties": false from spec for connectors with staging

* Remove "additionalProperties": false from spec for Redshift destination

* bump versions

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-24 20:04:04 +03:00
Alexander Tsukanov
01a366ca57 🐛 Destination Redshift: Improved discovery for redshift-destination not SUPER streams (#13690)
airbyte-12843: Improved discovery for redshift-destination not SUPER tables, excluded views from discovery.
2022-06-24 17:50:17 +03:00
Tuhai Maksym
6ca0e53469 12708: Add an option to use encryption with staging in Redshift Desti… (#14013)
* 12708: Add an option to use encryption with staging in Redshift Destination (#13675)

* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated

* 12708: specs updated

* 12708: specs updated

* 12708: removing autogenerated files from PR

* 12708: changelog updated

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-24 16:07:28 +03:00
Tuhai Maksym
3a179a822b Revert "12708: Add an option to use encryption with staging in Redshift Destination (#13675)" (#14010)
This reverts commit aa28d448d8.
2022-06-22 17:02:18 +03:00
Tuhai Maksym
aa28d448d8 12708: Add an option to use encryption with staging in Redshift Destination (#13675)
* 12708: Add an option to use encryption with staging in Redshift Destination

* 12708: docs/docker configs updated

* 12708: merge with master

* 12708: merge fix

* 12708: code review implementation

* 12708: fix for older configs

* 12708: fix for older configs in check

* 12708: merge from master (consolidation issue)

* 12708: versions updated
2022-06-22 14:31:04 +03:00
George Claireaux
e7dae06168 Enhance /publish to allow for multiple connectors and parallel execution (#13864)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Delete TEMP-testing-command.yml

* auto-bump connector version

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* destinations

* + snowflake

* saved

* auto-bump connector version

* auto-bump connector version

* java source bumps

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove twice-defined methods

* label things

* revert action

* using the new test action

* point at action

* wrong tag on action

* update pool label

* update to use new ec2-github-runner fork

* this needs to be more generic than publisher

* change publish to run on pool

* add comment about runner-pool usage

* updated publish command docs for multi & parallel connector runs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* unbump failed publish versions

* missed dockerfiles

* remove failed docs

* mssql fix

* overhauled the git comment output

* bumping a test connector that should work

* slight order switcheroo

* output connectors properly in first message

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-21 18:40:33 +01:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753)
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-20 14:56:33 +03:00
George Claireaux
c5783aa799 connectors that published (#13932) 2022-06-20 12:42:56 +01:00
George Claireaux
e63458fabb Publishing java connectors with better formatted stacktrace in Trace messages (#13852)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Fix md and lg spacing scss vars (#13886)

* Delete TEMP-testing-command.yml

* auto-bump connector version

* Delete publish-command.yml

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* Update _variables.scss

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
2022-06-17 16:15:24 +01:00
Edward Gao
5f843415db 📚 update MSSQL docs to reflect that we support normalization (#13826) 2022-06-16 08:16:00 -07:00
oneshcheret
8e54f4fd6e S3 and GCS destinations: Updating processing data types for Avro/Parquet formats (#13483)
* S3 destination: Updating processing data types for Avro/Parquet formats

* S3 destination: handle comparing data types

* S3 destination: clean code

* S3 destination: clean code

* S3 destination: handle case with unexpected json schema type

* S3 destination: clean code

* S3 destination: Extract the same logic for Avro/Parquet formats to separate parent class

* S3 destination: clean code

* S3 destination: clean code

* GCS destination: Update data types processing for Avro/Parquet formats

* GCS destination: clean redundant code

* S3 destination: handle case with numbers inside array

* S3 destination: clean code

* S3 destination: add unit test

* S3 destination: update unit test cases with number types.

* S3 destination: update unit tests.

* S3 destination: bump version for s3 and gcs

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-14 14:15:58 +03:00
Andy
fe6eda5b37 🎉 Destination databricks: rename to databricks lakehouse (#13722)
* Update Databricks naming

* Update destination_spec

* Update BOOTSTRAP.md

* Update Dockerfile

* Update README.md

* Update spec.json

* Update databricks.md

* Update databricks.md

* Update airbyte-integrations/connectors/destination-databricks/BOOTSTRAP.md

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>

Co-authored-by: LiRen Tu <tuliren.git@outlook.com>
2022-06-13 17:56:01 -07:00
Amruta Ranade
7ecfa13ee0 Updated BigQuery, Google Sheets, Facebook Marketing, Salesforce docs (#13717)
* initial changes

* Edited google Sheets doc

* More edits

* edited the intro and prereqs for BigQuery

* edited the data loading section

* more edits

* Grammatical edits

* Formatting edits
2022-06-13 14:25:41 -04:00
Simon Späti
93c413c0ed 📚 Destination Databricks: Clarify on Destination Format with Delta-Table (#13396) 2022-06-10 18:06:27 +02:00
Marcos Marx
862bf36f26 Docs: Destination ElasticSearch remove port from doc requirements (#13665) 2022-06-10 10:29:41 -03:00
Alexander Tsukanov
b45014da6b 🎉 Destination Redshift: Add "Loading Method" option to Redshift Destination spec and UI (#13415)
* airbyte-12709 Add "Loading Method" option to Redshift Destination spec.

* airbyte-12709: Fixed unit tests.

* airbyte-12709: Updated README.md.

* airbyte-12709: change the number of PR in redshift.md.

* airbyte-12709: Added backward compatibility for old json schema.

* airbyte-12709: Fix PR comments.

* airbyte-12709: Removed throwing an exception. Fixed PR comments.

* airbyte-12709: Bump the airbyte version.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-10 14:47:03 +03:00
LiRen Tu
bb46941867 🎉 Databricks destination: rename to Databricks delta lake (#13630)
* Rename databricks connector

* Rename connector in the seed

* Update changelog with pr id

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-08 18:22:55 -07:00
pcsoni89
b4e43c2ddd 🎉 Destination MSSQL: added custom JDBC parameters support. (#13054)
* 🎉 Destination MSSQL: added custom JDBC parameters support.

* 🎉  Destination MSSQL: added custom JDBC parameters support.

* 🎉  Destination MSSQL: added custom JDBC parameters support.

* airbyte-101718: Bumped version.

* auto-bump connector version

Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
Co-authored-by: Alexander Tsukanov <alexander.tsukanovvv@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-08 14:55:45 +03:00
LiRen Tu
e996c2c14f 🐛 BigQuery destination: always run GCS destination check for GCS staging method (#13579)
* Run gcs destination check all the time

* Add missing permissions to the status message

* Add more error messages

* Bump bigquery version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 17:52:49 -07:00
Yurii Bidiuk
d6b50c1ce0 Destination BigQuery: refactored spec and doc (#13424)
* change spec

* refactored readme file for bigquery

* change spec order

* bump version
2022-06-07 18:36:40 +03:00
Marcos Marx
9aa0220820 revert redshift and json operations (#13465) 2022-06-03 19:44:40 -03:00
Adam
071a8e9dc8 Destination Redshift: fixed array contents verification for SUPER (#13069)
* add array handling to json flatten

* bump version, add changelog entry

* test: add flatten unit tests

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-03 15:46:16 -03:00
VitaliiMaltsev
748291bb0c Snowflake Destination apply buffering strategy on GCS staging (#12952)
* Destination Snowflake apply buffering strategy for GCS

* Snowflake Destination apply buffering strategy on GCS staging

* updated CHANGELOG

* fixed checkstyle

* fixed compile error

* refactoring

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-01 10:28:02 +03:00
Misko Lee
721b5b69c7 Downgrade ElasticSearch batch size from 256mib to 32mib (#11752)
* Update ElasticsearchAirbyteMessageConsumerFactory.java

* add changleog and bump connector version seed file

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
2022-05-30 14:37:32 -03:00
Alexander Tsukanov
5371f87ee6 🐛 Destination Redshift: Fix DATs blind spot - connection check (#13090)
* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up.

* airbyte-12459: Merged master.

* airbyte-12459: Merged master.

* auto-bump connector version

Co-authored-by: Oleksandr Sheheda <alexandr-shegeda@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-25 19:14:10 +03:00
Baz
7b0a541040 🎉 Destination Google Sheets: update docs (#13151) 2022-05-24 22:44:23 +03:00
Eugene
d6f4ef4d38 [9964] destination-mqtt: fixed build tests failures (#13099)
* [9964] destination-mqtt: fixed build tests failures
2022-05-24 16:44:20 +03:00
Oleksandr Sheheda
44a43f0338 Improve check connection performance for all JDBC-related DBs (#12820)
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
2022-05-24 16:06:01 +03:00
Tuhai Maksym
70d48fa874 Destination-S3: Remove configurable part size (#13043)
* Destination-S3: Remove configurable part size

* Changelog updated

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-20 22:48:55 +03:00
Adam
9e31d026c8 Destination Redshift: Fixed maximum record size for SUPER type (#12940)
* fix #12680: use correct max size for SUPER type

* address review comments

* docs: add changelog entry

* bump version

* test: make large record actually too big

* fix: use a method instead of member variable for correct override

* fix: if overall object is invalid, don't check individual elements

* feat: add unit test for isValidData

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-20 10:32:21 -04:00
Chun-Sheng, Li
2c409ffaad Docs: Remove useless column (#13013) 2022-05-19 11:29:18 -04:00
LiRen Tu
5ddef8639a 📝 Databricks destination: clarify server hostname in the doc (#12943) 2022-05-17 20:37:07 -07:00
Mohit Reddy
58f20c455b Destination Kafka: Allow configuring PLAIN SASL (#12876)
* airbyte: Allow configuring PLAIN SASL mechanism for Kafka dest connector through WebApp

* bump up the version of kafka dest connector

* bump up the image tag for kafka destination connector in defs

* update _specs for kafka destination connector to match defs; SeedConnectorSpecs did not update it automatically
2022-05-17 12:27:52 -04:00
LiRen Tu
d6f8c2627b 📝 Update bigquery doc about hmac key troubleshooting (#12895) 2022-05-16 15:37:07 -07:00
LiRen Tu
7c35b03595 🎉 Databricks destination: use new jdbc driver and open source the connector (#12861)
* Migrate to public databricks jdbc driver

* Update documentation

* Bump version

* Format code

* Check in databricks in seed file

* Check in databricks 0.2.0

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-16 14:48:52 -07:00
VitaliiMaltsev
4804f746dc Destination Redshift fixed NPE in S3 staging (#12869)
* Destination Snowflake fixed npe in S3 staging

* removed star imports

* updated changelog

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-16 16:37:28 +03:00