jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-23 03:47:05 -05:00
Code Issues Packages Projects Releases Wiki Activity
41,480 Commits 5,116 Branches 518 Tags
yarden/source-mongodb-support-dynamic-arrays
Commit Graph

321 Commits

Author SHA1 Message Date
LiRen Tu
d6f8c2627b 📝 Update bigquery doc about hmac key troubleshooting (#12895) 2022-05-16 15:37:07 -07:00
LiRen Tu
e00f487d98 📝 Destination bigquery: mark service account as required for cloud (#12768) 
* Update spec

* Update doc

* Bump version and update changelog

* Modify wording

* Add sample service account key json

* Add screenshots and common permission issues

* Refactor service account helper method

* Update log message

* Update version date in changelog

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-15 01:28:36 -07:00
George Claireaux
440f3db8bb Destinations BigQuery & Snowflake: use latest base-java to emit AirbyteTraceMessage on errros (#12805) 
* bumping BQ and Snowflake dests to get AirbyteTraceMessage from base-java

* add PR link to changelogs

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-12 14:23:29 +01:00
LiRen Tu
bd59726668 Publish new connectors to log offending json fields in json to avro conversion for debugging (#12578) 
* Publish new connectors to log debugging info in json to avro conversion

* Add pull request id

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-04 11:47:07 -07:00
Alexandre Girard
1021428998 Destination Bigquery: Update Dataset location field description (#12528) 
* Update field description

* reset

* Bump

* reset

* auto-bump connector version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-05-02 17:45:36 -07:00
Alexandre Girard
553024780f Destination Bigquery: Dataset location is a required field (#12477) 
* Dataset location is required

* Update doc

* Bump versions

* reset

* Bump

* downgrade

* Revert "downgrade"

This reverts commit b482c54d6f.

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-29 15:34:43 -07:00
VitaliiMaltsev
0f8e11e243 🐛 BigQuery Destination failed conditional binding (#12068) 
* BigQuery Destination check conditional binding

* fixed checkstyle

* BigQuery Destination fixed bug with contitional binding

* update CHANGELOG

* update CHANGELOG

* add explaining comment

* bump version

* auto-bump connector version

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-15 19:17:46 +03:00
Eugene
281e0eec35 🎉Destination-bigquery-denorm: fixed emittedAt timestamp (#11978) 
* [11067] Destination-bigquery-denorm: fixed emittedAt timestamp
 2022-04-15 17:39:51 +03:00
LiRen Tu
8bd2d9b518 🎉 BigQuery destination: use serialized buffer for gcs staging (#11776) 
* Rebase bigquery changes to master

* Add comments

* Uncomment test code

* Format code

* Bump versions

* Fix denormalized destination target table name

* Fix avro schema for denormalized destination

* Remove unnecessary params from consumer factory

* Add back previous version

* Add warning about standard mode

* auto-bump connector version

* Bump version for bigquery in seed

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
 2022-04-07 16:59:19 -07:00
andriikorotkov
72dcd8126a 🐛 Destination bigquery denormalize "allOf" and "anyOf" fix (#11166) 
* fix bug with anyOf and allOf json blocks

* updated tests and formatter

* clean code

* updated tests

* updated destination_specs yaml file

* updated version of connector and docs
 2022-04-05 21:52:13 +03:00
Oleksandr Sheheda
24d7ab1f58 10148 BigQuery Destination/Denormalized Destination spec updates (#11620) 
* 10148 BigQuery Destination/Denormalized Destination spec updates

* 10148 updated changelog

* 10148 updated docker image version

* 10148 updated docker image version
 2022-04-02 20:24:39 +03:00
Eugene
2008970c7c 🎉Destination-Bigquery-denormalized: added tests for schema selection (#11636) 
* [8895] Destination-Bigquery-denormalized: added tests for schema selection
 2022-04-01 20:34:30 +03:00
Eugene
69841cee73 🎉BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas (#11454) 
* [8870] BigQuery denormalized Destination: Integration test enhancement for picking test-data and schemas
 2022-03-30 21:35:37 +03:00
VitaliiMaltsev
2ed4c05a19 🎉 BigQuery Destination: improve check performance (#11350) 
* fix for jdk 17

* add CHANGELOG

* bump redshift version

* add CHANGELOG

* Update airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java

* bump version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2022-03-24 20:36:17 +02:00
andriikorotkov
8771ba99fb 🎉 BigQuery destination mlp (#11238) 
* fix bug with anyOf and allOf json blocks

* updated spec and documentation

* rollback last changes

* updated spec

* updated spec

* updated bigquery permissions in doc

* updated spec

* updated spec

* updated spec

* updated tests

* fixed remarks

* updated tests

* updated tests

* added new bigquery version

* updated spec

* updated version

* updated definitions
 2022-03-23 13:46:40 +02:00
LiRen Tu
21ec23cc31 🐞 Fix invalid char in snowflake & bigquery namespace (#10793) 
* Add namespace test for snowflake

* Enable namespace test for bigquery

* Format code

* Capitalize test case id

* Update exception message to point to test case file

* Update snowflake name transformer to prepend underscore

* Override convertStreamName instead of getIdentifier

* Add missing state message

* Remove unused import

* Disable more namespace test cases

We don't want to introduce changes that will affect existing connections for now.

* Dry method that mutates namespace

* Pass through null

* Normalize namespace

* Fix test case

* Revert consumer factory changes

* Normalize namespace in catalog

* Revert catalog normalization

* Enable namespace test for all snowflake destination tests

* Test namespace for both bigquery destination tests

* Add unit test for bigquery name transformer

* Transform bigquery schema name

* Fix avro name transformer

* Normalize avro namespace

* Standardize namespace in gcs utils

* Bump version for snowflake and bigquery

* Enable namespace test for bigquery denormalized

* Dry bigquery denormalized acceptance test

* Revert some of the variable scope change

* Fix unit test

* Bump version

* Introduce getNamespace method

* Implement getNamespace method for bigquery

* Switch to getNamespace methods

* Update comments

* Fix bigquery denormalized acceptance test

* Format code

* Dry bigquery destination test

* Skip partition test for gcs mode

* Bump version
 2022-03-19 17:47:24 -07:00
Christophe Duong
b333f4fc75 🐛 Handle try/catch in BigQuery destination consumers (#10755) 
* Handle try/catch in BigQuery destination consumers

* Remove parallelStream

* Bumpversion of connector

* update changelogs

* update seeds
 2022-03-03 16:31:42 +01:00
Lake Mossman
3d8a0dc048 Add ExitOnOutOfMemoryError to java connectors and bump versions (#10256) 2022-02-14 15:49:15 -08:00
LiRen Tu
5133ce6f4c 🐛 Destination snowflake & bigquery: fix null pointer exception (#9959) 
* Prevent null exception

* Check nullable schema name

* Bump version

* Bump version in seed
 2022-02-01 04:27:07 -08:00
LiRen Tu
8e8f402b8a 🎉 Destination snowflake & bigquery: integrate with sentry (#9945) 
* Update doc

* Use empty dsn when sentry is not enabled

* Bump version in seed
 2022-01-31 20:27:52 -08:00
Alexander Tsukanov
479f0d7c8d [MVP] Integrate sentry to all java-based connectors (#9745) 
* airbyte-9328: Added Sentry integration to BigQuery and BigQuery denormalized connector.

* airbyte-5050: Added strategy for INSERT ROW.

* airbyte-9328: Added Sentry integration to Snowflake.

* airbyte-9328: Fix Sentry config.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fix PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Fixed PR comments.

* airbyte-9328: Small changes.

* airbyte-9328: Small changes.

* airbyte-9328: Move SENTRY DSN keys to Dockerfiles.

* Use new dsn

* Revert format

* Remove sentry dsn from compose temporarily

* Log sentry event id

* Move sentry to java base

* Remove sentry code from bigquery

* Update dockerfiles

* Fix build

* Update release tag format

* Bump version

* Add env to dockerfiles

* Fix e2e test connector dockerfil

* Fix snowflake bigquery dockerfile

* Mark new versions as unpublished

Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Liren Tu <tuliren.git@outlook.com>
 2022-01-29 16:58:35 -08:00
Yurii Bidiuk
0187efd030 Destination BigQuery: update description for some fields (#9573) 
* update desrcription for bq destination setup

* bump version
 2022-01-18 21:59:01 +02:00
Koji Matsumoto
3f9cbecbaf Destination BigQuery: Accept Dataset ID field prefixed by Project ID (#8383) 
* add Dataset ID parse method

* add BigQuery Destination unit test

* update change log

* fit to the latest code base

* update change log

* change var name to const name

* change public method to private

* add test cases for testGetDatasetIdFail

* add integration test for dataset-id prefixed with project-id

* fix getDatasetId

* add comment to parameterized test provider

* update docker image versions

* update docker image versions again
 2022-01-18 14:41:12 +02:00
Andrii Leonets
c904b44998 BigQuery Destination : Fix GCS processing of Facebook data (#9415) 
* Fix GCS Avro file processing with invalid "-" character

* Extend test data to cover the case

* incr ver

* s3 ver upd

* add dependency

* add dependency
 2022-01-13 19:16:20 +02:00
Eugene
44cb30aca5 🐛Destination-gcs\destination-bigquery(gcs) - updated check() method to handle that user has both storage.objects.create and storage.multipartUploads.create roles (#9121) 
* [9044] Destination-gcs\destination-bigquery(gcs) - updated check() method to handle that user has both storage.objects.create and storage.multipartUploads.create roles
 2022-01-10 22:00:16 +02:00
Charles
e80d6140d2 📖 Clarify staging setup guide for bq & gcs destination (#9255) 
* clarify confusing parts of setting up staging for bq destination

* Added Storage Admin

* update gcs destination docs too

* fix indentation

* Update required permission list

Co-authored-by: Liren Tu <tuliren.git@outlook.com>
 2022-01-06 14:34:49 -08:00
Yurii Bidiuk
14c3536d2f 🐛 Destination Snowflake | BigQuery: add part_size config to UI (#9039) 
* add part_size to Snowflake-s3, Bihquery-gcs

* add part size for bigquery-denormalized

* update changelog md

* add limints for part size

* update bigquery version
 2021-12-23 22:23:26 +02:00
Sherif A. Nada
6316cb920c Update bigquery.md 2021-12-21 15:22:36 -08:00
Sherif A. Nada
0210071599 Update Bigquery docs page 2021-12-21 15:18:59 -08:00
VitaliiMaltsev
a3100cba8f 🐛 Destination S3: avro and parquet formats have issues with JsonToAvroSchemaConverter (#8574) 
* add namespace to avro record type

* refactoring

* Add unit tests

* added CHANGELOG

* fix typo in method name

* fix typo in method name

* fix for jdk 17

* created recursive keys adding

* refactoring

* format code

* cleanup Dockerfile

* refactoring

* removed unneded tests case

* updated namespace generation

* removed unneeded method from AvroNameTransformer

* resolved merge conflicts

* removed unused imports

* reformat the code

* bump version

* bump Bigquery Denormalized version

Co-authored-by: vmaltsev <vitalii.maltsev@globallogic.com>
Co-authored-by: Oleksandr Sheheda <alexandrshegeda@gmail.com>
 2021-12-21 17:51:48 +02:00
Andrii Leonets
637fac3483 BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files (#8788) 
* impl BigQuery uploader

* rework BigQuery destination in order to use Uploaders

* fix constructor

* fix test

* add job waiting

* fix correct closing

* Throw exception if something goes wrong.

* Move data format logic to the formatters.

* remove duplication from tests

* JsonToAvro schema transformation

* destination-bigquery: print BQ error when sync fails

* add uploader config

* uncommit table drop

* switch to S3 json converter + disable invalid tests

* fix test stuck

* move getSchema to formatters

* replace Date-time to Timestamp for the Denormalized Avro processing. BigQuery doesn't support direct transformation from file.

* make test in line with GCS impl

* add string type for Ref fields (GCS impl)

* fix format root of NULL with nested date-time fields

* adopt array processing

* minor fix

* fix incorrect array filtering

* format

* Connector throw error after close no more.

* uncommit fixed tests

* format

* Add GCS to denormalized spec

* Increase version

* fix s3 tests

* review upd

* fix gcs

* format + minor beautifier

* revert docker file change

* correct condition

* Update airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/AvroRecordFactory.java

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* Update airbyte-integrations/connectors/destination-bigquery-denormalized/src/main/resources/spec.json

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* Update airbyte-integrations/connectors/destination-bigquery-denormalized/src/main/resources/spec.json

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* Update airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/formatter/BigQueryRecordFormatter.java

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* don't throw warns during record formatting

* add back adding String to all logical avro S3 types controlled by param.

* add docs

* fix internal call.

* fix GCS AVRO integration test

* Update airbyte-integrations/connectors/destination-bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/uploader/AbstractGscBigQueryUploader.java

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* Update airbyte-integrations/connectors/destination-gcs/src/main/java/io/airbyte/integrations/destination/gcs/util/GcsUtils.java

Co-authored-by: LiRen Tu <tuliren@gmail.com>

* remove unused code

* remove duplication in BigQuery formatters

* replace group file delete to specific file delete to avoid deleting unfinished loads

* restore docker files

* Revert "restore docker files"

This reverts commit 0a06d81852.

* resolved conflicts

* Update airbyte-integrations/connectors/destination-bigquery/Dockerfile

Co-authored-by: mkhokh <mariia.khokh@globallogic.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: Oleksandr Sheheda <alexandr.shegeda@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
 2021-12-20 19:00:04 +02:00
Christophe Duong
3740f25a2c BigQuery locations update (#8816) 
* Update bigquery dataset locations

* Update BQ denormalized locations too

* Bumpversion bq destinations
 2021-12-16 10:48:16 +01:00
Sherif A. Nada
cf126ced9b 🐛 Destination BQ Denormalized: handle null values in fields described by a $ref schema (#7804) 2021-11-09 21:53:41 -08:00
Alexander Tsukanov
48d825046f 🐛 Destination BigQuery-Denormalized: Fix JSON with $ref Definition keys (#7736) 
* BUG-6638: Fix fields with $ref definition.

* BUG-6638: Added integration tests.

* BUG-6638: Added integration tests.

* BUG-6638: Added integration tests.

* BUG-6638: Added doc and pumped Dockerfile version.

* BUG-6638: Added doc and pumped Dockerfile version.

* BUG-6638: replaced for with forEach

* BUG-6638: Bumped specification
 2021-11-09 13:39:39 +02:00
Alexander Tsukanov
a915034954 🐛 Destination BigQuery-denormalized: Added conversion from JSON Datetime to BigQuery format (#7413) 2021-11-03 13:29:45 +02:00
Christophe Duong
27df558a89 🎉 BigQuery destinations with partitionned/clustered keys (#7240) 
* [ #5959 ][ #2579 ] Add support of partitioned tables by _airbyte_emitted_at field (#7141)

Co-authored-by: Andrés Bravo <andresbravog@gmail.com>
 2021-10-25 12:41:04 +02:00
Abhi Vaidyanatha
ae32ecbb27 GitBook: [master] 186 pages and 77 assets modified 2021-10-08 21:17:47 +00:00
Abhi Vaidyanatha
276d914a57 Direct to Discourse tagged list instead of specific issues on Troubleshooting. (#6848) 2021-10-07 00:05:56 -07:00
Marcos Marx
118492071c Destination BigQuery: add support to dataset starting with numbers (#6733) 
* change check function

* make test harder

* change tests

* add tests˜
 2021-10-06 13:00:33 -03:00
Abhi Vaidyanatha
ecbc3f78ed Add emitted field error Troubleshooting for BigQuery Destination (#6702) 2021-10-04 00:26:18 -07:00
Abhi Vaidyanatha
770bbf9e14 📚 Docs Refresh: Jira Source (#6699) 
* Clean up MSSQL Troubleshooting

* Docs Refresh: Jira Source

* Update BQ troubleshooting

* Add discussion
 2021-10-04 00:05:52 -07:00
Abhi Vaidyanatha
68f1b2cf13 Add Troubleshooting for BigQuery Destination. (#6698) 2021-10-03 23:55:42 -07:00
Abhi Vaidyanatha
bee03dc50f 📚 Docs Refresh: BigQuery Destination (#6665) 
* Docs Refresh: BigQuery Destination

* Add language for Cloud.

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Update docs/integrations/destinations/bigquery.md

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

* Add link to GCS.

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
 2021-10-03 22:54:48 -07:00
Nicolas MOREAU
0105ca91b4 🎉 BigQuery Denormalized Destination: Support for more bigquery types through the format annotation (#6145) 
* destination-bigquery-denormalized: introduce json spec format key handling

* destination-bigquery-denormalized: Bump version
 2021-09-28 21:06:14 -03:00
Sherif A. Nada
909dfa5f10 Reorganize BQ docs 2021-09-21 11:57:43 -07:00
Abhi Vaidyanatha
d732eaf80e Add information on which destinations support Incremental - Deduped History in their docs (#6031) 
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
 2021-09-16 16:40:12 -07:00
irynakruk
895430edb6 BigQuery Denormalized NPE fix (#5881) 
* Simple NPE fix

* Update bigquery.md

* Added test with null object and array
 2021-09-08 09:09:55 -04:00
Eugene
f32b14e536 🎉 Destination Bigquery: added gcs upload option (#5614) 
destination-bigquery: added GCS uploading type
 2021-09-08 15:21:38 +03:00
Eugene
e2c593e1ab 🐛 Destination BigQueryDenormalized : fixed stackoverflow and array type parsing when source forget to send "items" details for it (#5813) 
* Fixed (StackOverflow) destination bigquery denormalized destination to handle the case when received schema doesn't contain a data type for Array type
 2021-09-06 17:27:19 +03:00
Andrii Leonets
634aa3d026 🐛 Destination BigQuery(Denormalized): Fix processing arrays of records (#5261) 
* [4487] Fix processing arrays of records
 2021-08-10 16:22:53 +03:00