1
0
mirror of synced 2026-01-08 03:06:34 -05:00
Commit Graph

99 Commits

Author SHA1 Message Date
Eugene
6640a47c02 [888]Destination-Bigquery: fixed check method to capture mismatch dataset location (#19358)
* [888]Destination-Bigquery: fixed check method to capture mismatch dataset location
2022-11-11 23:07:03 +02:00
Eugene
df2b061095 🐛Destination-bigquery: added additional verifications to Check method (#18554)
* [16992] Destination-bigquery: added additional verifications to Check connection method
2022-11-10 15:44:40 +02:00
Edward Gao
85e93cb006 Destinations BigQuery and GCS: update docs about bucket encryption (#18315) 2022-10-21 13:34:48 -07:00
Edward Gao
e21d54fbab Destination Bigquery: log the full stacktrace rather than just message (#18162)
* log the full stacktrace rather than just message

* version bumps + changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-10-19 09:23:49 -07:00
Yurii Bidiuk
e07bd6bee3 Destination BigQuery: add user-agent header (#16890)
* Destiantion BigQuery: add user-agent header

* fixed comments

* fix tests

* refactor code

* refactoring

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-26 20:31:27 +03:00
Edward Gao
23c8efa2b2 Destination Bigquery / Bigquery-denormalized: Only override dataset ID if stream namespace is null/empty (#17054)
* stop overriding namespace?

* set namespace if needed

* also check for empty namespace

* version bump + changelog

* auto-bump connector version [ci skip]

* sanitize dataset id

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-22 12:41:47 -07:00
Yurii Bidiuk
10e4aae891 Use LOG4J2 to wrap connectors logs to JSON format (#15668)
* Use LOG4J2 to wrap connectors logs to JSON format

* log connector's stack trace directly as a message

* add stack_trace field to json template

* bump versions

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* revert versions for destinations: postgres, mssql, mysql

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-14 23:27:13 +03:00
VitaliiMaltsev
022b8db345 Destination: BigQuery (denormalized typed struct) - Could not evaluate union, field is expected to be one of these: NULL, STRING. (#16401)
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Denormalized: Could not evaluate union on Mongo formatted type

* add test

* fixed BigQuery tests

* updated changelog

* replaced star import

* bump version

* auto-bump connector version [ci skip]

* fixed compilation error

* bump mongodb-strict-encrypt

* auto-bump connector version [ci skip]

* removed unused method

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-13 19:10:09 +03:00
Andrii Leonets
40c84476d7 Destination BigQuery : Increase version for BQ PR #14023 (#16494)
* Increase version for BQ PR #14023

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-08 19:03:27 -07:00
Liren Tu
fd547c64d7 🐛 Avro related destinations: fix field name clashing with combined restrict (#16243)
* Add test case for allOf and oneOf

* Bump version

* Add pr id

* Bump gcs version

* Bump dest jdbc

* Bump redshift

* Bump snowflake

* Bump databricks

* Bump bigquery

* Revert "Bump dest jdbc"

This reverts commit f10497e96a.

* Use a new pat to avoid api rate limit

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* Revert databricks bump

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-09-01 18:06:27 -07:00
Jagruti Tiwari
8288c16485 fix: replace airbyte oss with airbyte open source (#15885)
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
2022-08-24 01:01:53 -03:00
VitaliiMaltsev
3aefd3308f Bigquery Destinatioin throw exception if job failed (#15787)
* Fixed bucket naming for S3

* removed redundant configs

* BigQuery Destination throw exception if job failed

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-23 12:46:27 +03:00
Andy Jih
e8a0d5c851 Docs updates for destinations affected by per-stream state (#15578)
* Docs updates for destinations affected by per-stream state

* adding -alpha tag
2022-08-11 15:37:22 -07:00
Liren Tu
69972d606d Bigquery destination: fix changelog and publish new version (#15409)
* Fix bigquery changelog

* Bump bigquery denormalized

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-08 11:05:23 -07:00
Greg Solovyev
95aeb39325 Fix typo in change log (#15343)
* Fix typo in several change logs

Co-authored-by: andrii.leonets <aleonets@gmail.com>
2022-08-05 09:13:24 -07:00
Marcelo Castro
76c4053282 Destination BigQuery: Enabling Application Default Credentials (#14784)
* Enabling different bigquery authentications

Current implementation only accepts Service Accounts. For developers
it is very desirable that we can login with Application Default Credentials
(ADC) https://cloud.google.com/sdk/gcloud/reference/auth/application-default

* bump version and update doc

* auto-bump connector version [ci skip]

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-03 15:18:34 -03:00
Andrii Leonets
e7b9e00949 Fix multiply log bindings (#14801)
* Fix multiply log bindings

* Exclude slf4j-reload4j

* Exclude slf4j-log4j12 for debezium

* Increase version for debezium related sources and json converter related destinations

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* rebump bigquery versions

* mark destinations s3, gcs as unpublishd

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: subodh <subodh1810@gmail.com>
Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2022-08-03 09:36:30 -07:00
Amruta Ranade
cc7e850a42 Update bigquery.md 2022-08-03 12:08:53 -04:00
Benoit Moriceau
8bf5986a0b Fix bq standard mode (#15180)
* Fix

* Add version and changelog

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-08-02 10:00:12 -07:00
Topher Lubaway
9c6c092a22 Revert "Improving docusaurus sidebar generation (#1927) (#14369)" (#14596)
This reverts commit a2c194a11f.
2022-07-11 15:27:14 -05:00
Mykyta Serbynevskiy
a2c194a11f Improving docusaurus sidebar generation (#1927) (#14369)
* Improving docusaurus sidebar generation (#1927)

* Added "Career & open positions" folder to sidebar, adjusted "Project overview" folder

* Deleted "career-and-open-positions" folder from sidebar
2022-07-08 14:18:27 -05:00
VitaliiMaltsev
d3f4fdd8fc BigQuery Denormalized "airbyte_type": "big_integer" to INT64 (#14079)
* BigQuery Denormalized "airbyte_type": "big_integer" to INT64

* updated changelog

* added unit test

* removed star import

* fixed checkstyle

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-29 19:47:15 +03:00
VitaliiMaltsev
11ea066f47 Remove "additionalProperties": false from specs for connectors with staging (#14114)
* Remove "additionalProperties": false from spec for connectors with staging

* Remove "additionalProperties": false from spec for Redshift destination

* bump versions

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-24 20:04:04 +03:00
VitaliiMaltsev
c283d9d159 Deprecate PART_SIZE_MB in connectors using S3/GCS storage (#13753)
* Removed part_size from connectors that use StreamTransferManager

* fixed S3DestinationConfigTest

* fixed S3JsonlFormatConfigTest

* upadate changelog and bump version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* upadate changelog and bump version for Redshift and Snowflake destinations

* auto-bump connector version

* fix GCS staging test

* fix GCS staging test

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-20 14:56:33 +03:00
George Claireaux
e63458fabb Publishing java connectors with better formatted stacktrace in Trace messages (#13852)
* start

* revert

* azblob

* bq

* bq denorm

* megapublish baaaabyyyy

* fix needs

* matrix connectors

* auto-bump connector version

* dont failfast and max parallel 5

* multi runno

* minor

* testing matrix agents

* name

* testing multi agents

* tmp fix

* new multi agents

* multi test

* tryy

* let's do this

* magico

* fix

* label test

* couple more connector bumps

* temp

* things

* check this

* lets gooo

* more connectors

* Fix md and lg spacing scss vars (#13886)

* Delete TEMP-testing-command.yml

* auto-bump connector version

* Delete publish-command.yml

* added comment describing bash part

* running single thread

* catch sentry cli

* auto-bump connector version

* Update _variables.scss

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
2022-06-17 16:15:24 +01:00
Amruta Ranade
7ecfa13ee0 Updated BigQuery, Google Sheets, Facebook Marketing, Salesforce docs (#13717)
* initial changes

* Edited google Sheets doc

* More edits

* edited the intro and prereqs for BigQuery

* edited the data loading section

* more edits

* Grammatical edits

* Formatting edits
2022-06-13 14:25:41 -04:00
LiRen Tu
e996c2c14f 🐛 BigQuery destination: always run GCS destination check for GCS staging method (#13579)
* Run gcs destination check all the time

* Add missing permissions to the status message

* Add more error messages

* Bump bigquery version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-06-07 17:52:49 -07:00
Yurii Bidiuk
d6b50c1ce0 Destination BigQuery: refactored spec and doc (#13424)
* change spec

* refactored readme file for bigquery

* change spec order

* bump version
2022-06-07 18:36:40 +03:00
LiRen Tu
d6f8c2627b 📝 Update bigquery doc about hmac key troubleshooting (#12895) 2022-05-16 15:37:07 -07:00
LiRen Tu
e00f487d98 📝 Destination bigquery: mark service account as required for cloud (#12768)
* Update spec

* Update doc

* Bump version and update changelog

* Modify wording

* Add sample service account key json

* Add screenshots and common permission issues

* Refactor service account helper method

* Update log message

* Update version date in changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-15 01:28:36 -07:00
George Claireaux
440f3db8bb Destinations BigQuery & Snowflake: use latest base-java to emit AirbyteTraceMessage on errros (#12805)
* bumping BQ and Snowflake dests to get AirbyteTraceMessage from base-java

* add PR link to changelogs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-12 14:23:29 +01:00
LiRen Tu
bd59726668 Publish new connectors to log offending json fields in json to avro conversion for debugging (#12578)
* Publish new connectors to log debugging info in json to avro conversion

* Add pull request id

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-04 11:47:07 -07:00
Alexandre Girard
1021428998 Destination Bigquery: Update Dataset location field description (#12528)
* Update field description

* reset

* Bump

* reset

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-05-02 17:45:36 -07:00
Alexandre Girard
553024780f Destination Bigquery: Dataset location is a required field (#12477)
* Dataset location is required

* Update doc

* Bump versions

* reset

* Bump

* downgrade

* Revert "downgrade"

This reverts commit b482c54d6f.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-29 15:34:43 -07:00
VitaliiMaltsev
0f8e11e243 🐛 BigQuery Destination failed conditional binding (#12068)
* BigQuery Destination check conditional binding

* fixed checkstyle

* BigQuery Destination fixed bug with contitional binding

* update CHANGELOG

* update CHANGELOG

* add explaining comment

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-15 19:17:46 +03:00
Eugene
281e0eec35 🎉Destination-bigquery-denorm: fixed emittedAt timestamp (#11978)
* [11067] Destination-bigquery-denorm: fixed emittedAt timestamp
2022-04-15 17:39:51 +03:00
LiRen Tu
8bd2d9b518 🎉 BigQuery destination: use serialized buffer for gcs staging (#11776)
* Rebase bigquery changes to master

* Add comments

* Uncomment test code

* Format code

* Bump versions

* Fix denormalized destination target table name

* Fix avro schema for denormalized destination

* Remove unnecessary params from consumer factory

* Add back previous version

* Add warning about standard mode

* auto-bump connector version

* Bump version for bigquery in seed

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
2022-04-07 16:59:19 -07:00
andriikorotkov
72dcd8126a 🐛 Destination bigquery denormalize "allOf" and "anyOf" fix (#11166)
* fix bug with anyOf and allOf json blocks

* updated tests and formatter

* clean code

* updated tests

* updated destination_specs yaml file

* updated version of connector and docs
2022-04-05 21:52:13 +03:00
Oleksandr Sheheda
24d7ab1f58 10148 BigQuery Destination/Denormalized Destination spec updates (#11620)
* 10148 BigQuery Destination/Denormalized Destination spec updates

* 10148 updated changelog

* 10148 updated docker image version

* 10148 updated docker image version
2022-04-02 20:24:39 +03:00
Eugene
2008970c7c 🎉Destination-Bigquery-denormalized: added tests for schema selection (#11636)
* [8895] Destination-Bigquery-denormalized: added tests for schema selection
2022-04-01 20:34:30 +03:00
Eugene
69841cee73 🎉BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas (#11454)
* [8870] BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas
2022-03-30 21:35:37 +03:00
VitaliiMaltsev
2ed4c05a19 🎉 BigQuery Destination: improve check performance (#11350)
* fix for jdk 17

* add CHANGELOG

* bump redshift version

* add CHANGELOG

* Update airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java

* bump version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
2022-03-24 20:36:17 +02:00
andriikorotkov
8771ba99fb 🎉 BigQuery destination mlp (#11238)
* fix bug with anyOf and allOf json blocks

* updated spec and documentation

* rollback last changes

* updated spec

* updated spec

* updated bigquery permissions in doc

* updated spec

* updated spec

* updated spec

* updated tests

* fixed remarks

* updated tests

* updated tests

* added new bigquery version

* updated spec

* updated version

* updated definitions
2022-03-23 13:46:40 +02:00
LiRen Tu
21ec23cc31 🐞 Fix invalid char in snowflake & bigquery namespace (#10793)
* Add namespace test for snowflake

* Enable namespace test for bigquery

* Format code

* Capitalize test case id

* Update exception message to point to test case file

* Update snowflake name transformer to prepend underscore

* Override convertStreamName instead of getIdentifier

* Add missing state message

* Remove unused import

* Disable more namespace test cases

We don't want to introduce changes that will affect existing connections for now.

* Dry method that mutates namespace

* Pass through null

* Normalize namespace

* Fix test case

* Revert consumer factory changes

* Normalize namespace in catalog

* Revert catalog normalization

* Enable namespace test for all snowflake destination tests

* Test namespace for both bigquery destination tests

* Add unit test for bigquery name transformer

* Transform bigquery schema name

* Fix avro name transformer

* Normalize avro namespace

* Standardize namespace in gcs utils

* Bump version for snowflake and bigquery

* Enable namespace test for bigquery denormalized

* Dry bigquery denormalized acceptance test

* Revert some of the variable scope change

* Fix unit test

* Bump version

* Introduce getNamespace method

* Implement getNamespace method for bigquery

* Switch to getNamespace methods

* Update comments

* Fix bigquery denormalized acceptance test

* Format code

* Dry bigquery destination test

* Skip partition test for gcs mode

* Bump version
2022-03-19 17:47:24 -07:00
Christophe Duong
b333f4fc75 🐛 Handle try/catch in BigQuery destination consumers (#10755)
* Handle try/catch in BigQuery destination consumers

* Remove parallelStream

* Bumpversion of connector

* update changelogs

* update seeds
2022-03-03 16:31:42 +01:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
LiRen Tu
5133ce6f4c 🐛 Destination snowflake & bigquery: fix null pointer exception (#9959)
* Prevent null exception

* Check nullable schema name

* Bump version

* Bump version in seed
2022-02-01 04:27:07 -08:00
LiRen Tu
8e8f402b8a 🎉 Destination snowflake & bigquery: integrate with sentry (#9945)
* Update doc

* Use empty dsn when sentry is not enabled

* Bump version in seed
2022-01-31 20:27:52 -08:00
Alexander Tsukanov
479f0d7c8d [MVP] Integrate sentry to all java-based connectors (#9745)
* airbyte-9328: Added Sentry integration to BigQuery and BigQuery denormalized connector.

* airbyte-5050: Added strategy for INSERT ROW.

* airbyte-9328: Added Sentry integration to Snowflake.

* airbyte-9328: Fix Sentry config.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fix PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Small changes.

* airbyte-9328: Small changes.

* airbyte-9328: Move SENTRY DSN keys to Dockerfiles.

* Use new dsn

* Revert format

* Remove sentry dsn from compose temporarily

* Log sentry event id

* Move sentry to java base

* Remove sentry code from bigquery

* Update dockerfiles

* Fix build

* Update release tag format

* Bump version

* Add env to dockerfiles

* Fix e2e test connector dockerfil

* Fix snowflake bigquery dockerfile

* Mark new versions as unpublished

Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
2022-01-29 16:58:35 -08:00
Yurii Bidiuk
0187efd030 Destination BigQuery: update description for some fields (#9573)
* update desrcription for bq destination setup

* bump version
2022-01-18 21:59:01 +02:00