jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-20 18:39:31 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
yarden/source-mongodb-support-dynamic-arrays
airbyte/docs/integrations/destinations/bigquery.md
devin-ai-integration[bot] b25da68e14 fix(destination-bigquery): Add backticks to column names in SQL generation to prevent syntax errors (#67401) 
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
2025-11-25 17:18:01 +00:00

76 KiB
Raw Permalink Blame History

BigQuery

Setting up the BigQuery destination connector involves setting up the data loading method and configuring the BigQuery destination connector using the Airbyte UI.

This page guides you through setting up the BigQuery destination connector.

Prerequisites

Setup guide

Step 1: Set up a data loading method

Using Batched Standard Inserts

You can use the BigQuery driver's built-in conversion to take INSERT statements and convert that to file uploads which are then loaded into BigQuery in batches. This is the simplest way to load data into BigQuery in a performant way. These staging files are managed by BigQuery and deleted automatically after the load is complete.

Using a Google Cloud Storage bucket

If you want more control of how and where your staging files are stored, you can opt to use a GCS bucket.

To use a Google Cloud Storage bucket:

  1. Create a Cloud Storage bucket with the Protection Tools set to none or Object versioning. Make sure the bucket does not have a retention policy.
  2. Create an HMAC key and access ID.
  3. Grant the Storage Object Admin role to the Google Cloud Service Account. This must be the same service account as the one you configure for BigQuery access in the BigQuery connector setup step.
  4. Make sure your Cloud Storage bucket is accessible from the machine running Airbyte. The easiest way to verify if Airbyte is able to connect to your bucket is via the check connection tool in the UI.

Your bucket must be encrypted using a Google-managed encryption key (this is the default setting when creating a new bucket). We currently do not support buckets using customer-managed encryption keys (CMEK). You can view this setting under the "Configuration" tab of your GCS bucket, in the Encryption type row.

Step 2: Set up the BigQuery connector

  1. Log into your Airbyte Cloud or Airbyte Open Source account.
  2. Click Destinations and then click + New destination.
  3. On the Set up the destination page, select BigQuery from the Destination type dropdown.
  4. Enter the name for the BigQuery connector.
  5. For Project ID, enter your Google Cloud project ID.
  6. For Dataset Location, select the location of your BigQuery dataset.

:::warning You cannot change the location later. :::

  1. For Default Dataset ID, enter the BigQuery Dataset ID.
  2. For Loading Method, select Batched Standard Inserts or GCS Staging.
  3. For Service Account Key JSON (Required for cloud, optional for open-source), enter the Google Cloud Service Account Key in JSON format.

:::note Be sure to copy all contents in the Account Key JSON file including the brackets. :::

  1. For Transformation Query Run Type (Optional), select interactive to have BigQuery run interactive query jobs or batch to have BigQuery run batch queries.

:::note Interactive queries are executed as soon as possible and count towards daily concurrent quotas and limits, while batch queries are executed as soon as idle resources are available in the BigQuery shared resource pool. If BigQuery hasn't started the query within 24 hours, BigQuery changes the job priority to interactive. Batch queries don't count towards your concurrent rate limit, making it easier to start many queries at once. :::

  1. For Google BigQuery Client Chunk Size (Optional), use the default value of 15 MiB. Later, if you see networking or memory management problems with the sync (specifically on the destination), try decreasing the chunk size. In that case, the sync will be slower but more likely to succeed.

Supported sync modes

The BigQuery destination connector supports the following sync modes:

  • Full Refresh Sync
  • Incremental - Append Sync
  • Incremental - Append + Deduped

Output schema

The final table contains these fields, in addition to the columns declared in your stream schema:

  • airbyte_raw_id
  • _airbyte_generation_id
  • airbyte_extracted_at
  • _airbyte_meta

Again, see here for more information about these fields.

The output tables in BigQuery are partitioned by the Time-unit column airbyte_extracted_at at a daily granularity and clustered by airbyte_extracted_at and the table Primary Keys. Partitions boundaries are based on UTC time. This is useful to limit the number of partitions scanned when querying these partitioned tables, by using a predicate filter (a WHERE clause). Filters on the partitioning column are used to prune the partitions and reduce the query cost. (The parameter Require partition filter is not enabled by Airbyte, but you may toggle it by updating the produced tables.)

Legacy Raw Tables schema

If you enable the Legacy raw tables option, the connector will write tables in this format.

Airbyte outputs each stream into its own raw table in airbyte_internal dataset by default (you can override this via the Airbyte Internal Table Dataset Name option). Contents in the raw table are NOT deduplicated.

The raw table contains these fields:

  • _airbyte_raw_id
  • _airbyte_generation_id
  • _airbyte_extracted_at
  • _airbyte_loaded_at
  • _airbyte_meta
  • _airbyte_data

_airbyte_data is a JSON blob with the record's data. See here for more information about the other fields.

BigQuery Naming Conventions

Follow BigQuery Datasets Naming conventions.

Airbyte converts any invalid characters into _ characters when writing data. However, since datasets that begin with _ are hidden on the BigQuery Explorer panel, Airbyte prepends the namespace with n for converted namespaces.

Data type map

Airbyte type BigQuery type
STRING STRING
STRING (BASE64) STRING
STRING (BIG_NUMBER) STRING
STRING (BIG_INTEGER) STRING
NUMBER NUMERIC
INTEGER INT64
BOOLEAN BOOL
STRING (TIMESTAMP_WITH_TIMEZONE) TIMESTAMP
STRING (TIMESTAMP_WITHOUT_TIMEZONE) DATETIME
STRING (TIME_WITH_TIMEZONE) STRING
STRING (TIME_WITHOUT_TIMEZONE) TIME
DATE DATE
OBJECT JSON
ARRAY JSON

Troubleshooting permission issues

The service account does not have the proper permissions.

  • Make sure the BigQuery service account has BigQuery User and BigQuery Data Editor roles or equivalent permissions as those two roles.
  • If the GCS staging mode is selected, ensure the BigQuery service account has the right permissions to the GCS bucket and path or the Cloud Storage Admin role, which includes a superset of the required permissions.

The HMAC key is wrong.

  • Make sure the HMAC key is created for the BigQuery service account, and the service account has permission to access the GCS bucket and path.

Tutorials

Now that you have set up the BigQuery destination connector, check out the following BigQuery tutorials:

Changelog

Expand to review
Version Date Pull Request Subject
3.0.16 2025-11-25 67401 Add backticks to column names in SQL generation to prevent syntax errors.
3.0.15 2025-11-12 69307 Handle out-of-range timestamps/times.
3.0.14 2025-11-11 69231 Upgrade to Bulk CDK 0.1.74.
3.0.13 2025-11-01 69126 Upgrade to Bulk CDK 0.1.61.
3.0.12 2025-10-29 69083 Fail loudly if Bigquery detects bad records.
3.0.11 2025-10-27 68671 Log record count per load job.
3.0.10 2025-10-21 67153 Implement new proto schema implementation
3.0.9 2025-10-16 68152 Update to new TableOperationsClient interface.
3.0.8 2025-10-05 67078 Remove memory limit for sync jobs to improve performance and resource utilization.
3.0.7 2025-09-02 65905 Promoting release candidate 3.0.7-rc.1 to a main version.
3.0.7-rc.1 2025-08-27 65114 Implement SOCKET+PROTO mode support.
3.0.6 2025-07-21 63700 Improve error reporting for Billing errors.
3.0.5 2025-07-15 63312 Pull in upstream fix to support null chars in GCS staging mode.
3.0.4 2025-07-15 63327 Improve error reporting for Billing errors.
3.0.3 2025-07-02 62495 Improve error reporting for misconfigured connections; improve support for complex types.
3.0.2 2025-06-26 62106 Improve error reporting during schema evolution.
3.0.1 2025-06-26 62085 Correctly handle stream names/namespaces and column names which start with a digit.
3.0.0 2025-06-25 59752 Upgrade to direct-load tables; add option for soft CDC deletes.
2.12.4 2025-06-24 62045 Promoting release candidate 2.12.4-rc.6 to a main version.
2.12.4-rc.6 2025-06-24 62041 2.12.4 RC 6 More retries is better retries
2.12.4-rc.5 2025-06-23 62016 2.12.4 RC 5 Handle records up to 20MiB
2.12.4-rc.4 2025-06-23 62014 2.12.4 RC 4 Improve JSON schema parser's behavior on invalid source schemas
2.12.4-rc.3 2025-06-18 61702 2.12.4 RC 3 Fix an issue with streams with special chars on bulk loader
2.12.4-rc.2 2025-06-18 61700 2.12.4 RC 2 (throw more informative error on invalid catalog)
2.12.4-rc.1 2025-06-16 61637 2.12.4 RC 1 (theoretically equivalent to 2.12.0, but with fixed global state handling in CDK)
2.12.3 2025-06-16 61648 This is not the release you are looking for. identical to 2.10.2
2.12.1 2025-06-13 61588 Publish version to account for possible duplicate publishing in pipeline. Noop change. WARNING: THIS HAS A BUG. DO NOT USE.
2.12.0 2025-06-06 61432 Improve performance in GCS staging mode by writing GZIP-compressed files.
2.11.4 2025-05-30 61018 Always emit a useful error message when erroring during sync startup.
2.11.3 2025-06-02 61321 CHECK operation doesn't need to drop the dataset anymore.
2.11.2 2025-05-29 60986 CHECK operation uses configured default dataset instead of airbyte_internal_test.
2.11.1 2025-05-29 60973 Improve error recovery when a stream's sync mode and schema are both changed and the previous sync exited uncleanly.
2.11.0 2025-05-27 60922 Promoting release candidate 2.11.0-rc.7 to a main version.
2.11.0-rc.7 2025-05-22 60865 Add a random element to the check sync namespace
2.11.0-rc.6 2025-05-22 60859 Improve bigquery quota usage in standard inserts for streams with heavy load
2.11.0-rc.5 2025-05-21 60841 Improve behavior in syncs with many state messages
2.11.0-rc.4 2025-05-19 60791 Fix bug in detecting schema change when stream has no columns
2.11.0-rc.3 2025-05-16 60324 Fix bug in syncs with many state message
2.11.0-rc.2 2025-05-15 60302 Fix bugs in handling special characters / uppercase characters in column names
2.11.0-rc.1 2025-05-08 59729 Version 2.11.0 RC 1
2.10.2 2025-04-29 59142 Add europe-north2 region.
2.10.1 2025-04-14 57569 Remove StringSubstitutor usage
2.10.0 2025-04-02 56982 Change default raw table partitioning scheme to be on _airbyte_extracted_at; upgrade CDK
2.9.3 2025-03-24 56355 Upgrade to airbyte/java-connector-base:2.0.1 to be M4 compatible.
2.9.2 2025-01-10 51482 Use a non root base image
2.9.1 2024-12-18 49902 Use a base image: airbyte/java-connector-base:1.0.0
2.9.0 2024-09-18 45433 upgrade all dependencies
2.8.9 2024-08-20 44476 Increase message parsing limit to 100mb
2.8.8 2024-08-22 44526 Revert protocol compliance fix
2.8.7 2024-08-15 42504 Fix bug in refreshes logic (already mitigated in platform, just fixing protocol compliance)
2.8.6 2024-07-30 42511 Added a copy operation to validate copy permissions in the check function
2.8.5 2024-07-22 42407 Batched Standard Inserts is default loading mode
2.8.4 2024-07-15 41968 Don't hang forever on empty stream list; shorten error message on INCOMPLETE stream status
2.8.3 2024-07-12 41674 Upgrade to latest CDK
2.8.2 2024-07-08 41041 Fix resume logic in truncate refreshes to prevent data loss
2.8.1 2024-06-25 39379 Removing requirement of a redundant permission bigquery.datasets.create permission
2.8.0 2024-06-21 39904 Convert all production code to kotlin
2.7.1 2024-06-17 39526 Internal code change for improved error reporting in case of source/platform failure (INCOMPLETE stream status / empty ConfiguredCatalog).
2.7.0 2024-06-17 38713 Support for refreshes and resumable full refresh. WARNING: You must upgrade to platform 0.63.7 before upgrading to this connector version.
2.6.3 2024-06-10 38331 Internal code changes in preparation for future feature release
2.6.2 2024-06-07 38764 Increase message length limit to 50MiB
2.6.1 2024-05-29 38770 Internal code change (switch to CDK artifact)
2.6.0 2024-05-28 38359 Propagate airbyte_meta from sources; add generation_id column
2.5.1 2024-05-22 38591 Bugfix to include forward-slash when cleaning up stage
2.5.0 2024-05-22 38132 Major rewrite of existing code, Adapting to CDK changes introduced in 38107
2.4.20 2024-05-13 38131 Cleanup BigQueryWriteConfig and reuse StreamConfig; Adapt to StreamConfig signature changes
2.4.19 2024-05-10 38125 adopt latest CDK code
2.4.18 2024-05-10 38111 No functional changes, deleting unused code
2.4.17 2024-05-09 38098 Internal build structure change
2.4.16 2024-05-08 37714 Adopt CDK 0.34.0
2.4.15 2024-05-07 34611 Adopt CDK 0.33.2
2.4.14 2024-02-25 37584 Remove unused insecure dependencies from CDK
2.4.13 2024-02-25 36899 adopt latest CDK
2.4.12 2024-03-04 35315 Adopt CDK 0.23.11
2.4.11 2024-02-22 35569 Fix logging bug.
2.4.10 2024-02-15 35240 Adopt CDK 0.20.9
2.4.9 2024-02-15 35285 Adopt CDK 0.20.8
2.4.8 2024-02-12 35144 Adopt CDK 0.20.2
2.4.7 2024-02-12 35111 Adopt CDK 0.20.1
2.4.6 2024-02-09 34575 Adopt CDK 0.20.0
2.4.5 2024-02-08 34745 Adopt CDK 0.19.0
2.4.4 2024-02-08 35027 Upgrade CDK to 0.17.1
2.4.3 2024-02-01 34728 Upgrade CDK to 0.16.4; Notable changes from 0.14.2, 0.15.1 and 0.16.3
2.4.2 2024-01-24 34451 Improve logging for unparseable input
2.4.1 2024-01-24 34458 Improve error reporting
2.4.0 2024-01-24 34468 Upgrade CDK to 0.14.0
2.3.31 2024-01-22 #34023 Combine DDL operations into a single execution
2.3.30 2024-01-12 #34226 Upgrade CDK to 0.12.0; Cleanup dependencies
2.3.29 2024-01-09 #34003 Fix loading credentials from GCP Env
2.3.28 2024-01-08 #34021 Add idempotency ids in dummy insert for check call
2.3.27 2024-01-05 #33948 Skip retrieving initial table state when setup fails
2.3.26 2024-01-04 #33730 Internal code structure changes
2.3.25 2023-12-20 #33704 Update to java CDK 0.10.0 (no changes)
2.3.24 2023-12-20 #33697 Stop creating unnecessary tmp tables
2.3.23 2023-12-18 #33124 Make Schema Creation Separate from Table Creation
2.3.22 2023-12-14 #33451 Remove old spec option
2.3.21 2023-12-13 #33232 Only run typing+deduping for a stream if the stream had any records
2.3.20 2023-12-08 #33263 Adopt java CDK version 0.7.0
2.3.19 2023-12-07 #32326 Update common T&D interfaces
2.3.18 2023-12-04 #33084 T&D SQL statements moved to debug log level
2.3.17 2023-12-04 #33078 Further increase gcs COPY timeout
2.3.16 2023-11-14 #32526 Clean up memory manager logs.
2.3.15 2023-11-13 #32468 Further error grouping enhancements
2.3.14 2023-11-06 #32234 Remove unused config option.
2.3.13 2023-11-08 #32125 fix compiler warnings
2.3.12 2023-11-08 #32309 Revert: Use Typed object for connection config
2.3.11 2023-11-07 #32147 Use Typed object for connection config
2.3.10 2023-11-07 #32261 Further improve error reporting
2.3.9 2023-11-07 #32112 GCS staging mode: reduce flush frequency to use rate limit more efficiently
2.3.8 2023-11-06 #32026 Move SAFE_CAST transaction to separate transactions
2.3.7 2023-11-06 #32190 Further improve error reporting
2.3.6 2023-11-06 #32193 Adopt java CDK version 0.4.1.
2.3.5 2023-11-02 #31983 Improve error reporting
2.3.4 2023-10-31 #32010 Add additional data centers.
2.3.3 2023-10-30 #31985 Delay upgrade deadline to Nov 7
2.3.2 2023-10-30 #31960 Adopt java CDK version 0.2.0.
2.3.1 2023-10-27 #31529 Performance enhancement (switch to a merge statement for incremental-dedup syncs)
2.3.0 2023-10-25 #31686 Opt out flag for typed and deduped tables
2.2.0 2023-10-25 #31520 Stop deduping raw table
2.1.6 2023-10-23 #31717 Remove inadvertent Destination v2 check
2.1.5 2023-10-17 #30069 Staging destination async
2.1.4 2023-10-17 #31191 Improve typing+deduping performance by filtering new raw records on extracted_at
2.1.3 2023-10-10 #31358 Stringify array and object types for type:string column in final table
2.1.2 2023-10-10 #31194 Deallocate unused per stream buffer memory when empty
2.1.1 2023-10-10 #31083 Fix precision of numeric values in async destinations
2.1.0 2023-10-09 #31149 No longer fail syncs when PKs are null - try do dedupe anyway
2.0.26 2023-10-09 #31198 Clarify configuration groups
2.0.25 2023-10-09 #31185 Increase staging file upload timeout to 5 minutes
2.0.24 2023-10-06 #31139 Bump CDK version
2.0.23 2023-10-06 #31129 Reduce async buffer size
2.0.22 2023-10-04 #31082 Revert null PK checks
2.0.21 2023-10-03 #31028 Update timeout
2.0.20 2023-09-26 #30779 Final table PK columns become non-null and skip check for null PKs in raw records (performance)
2.0.19 2023-09-26 #30775 Increase async block size
2.0.18 2023-09-27 #30739 Fix column name collision detection
2.0.17 2023-09-26 #30696 Attempt unsafe typing operations with an exception clause
2.0.16 2023-09-22 #30697 Improve resiliency to unclean exit during schema change
2.0.15 2023-09-21 #30640 Handle streams with identical name and namespace
2.0.14 2023-09-20 #30069 Staging destination async
2.0.13 2023-09-19 #30592 Internal code changes
2.0.12 2023-09-19 #30319 Improved testing
2.0.11 2023-09-18 #30551 GCS Staging is first loading method option
2.0.10 2023-09-15 #30491 Improve error message display
2.0.9 2023-09-14 #30439 Fix a transient error
2.0.8 2023-09-12 #30364 Add log message
2.0.7 2023-08-29 #29878 Internal code changes
2.0.6 2023-09-05 #29917 Improve performance by changing metadata error array construction from ARRAY_CONCAT to ARRAY_AGG
2.0.5 2023-08-31 #30020 Run typing and deduping tasks in parallel
2.0.4 2023-09-05 #30117 Type and Dedupe at sync start and then every 6 hours
2.0.3 2023-09-01 #30056 Internal refactor, no behavior change
2.0.2 2023-09-01 #30120 Improve performance on very wide streams by skipping SAFE_CAST on strings
2.0.1 2023-08-29 #29972 Publish a new version to supersede old v2.0.0
2.0.0 2023-08-27 #29783 Destinations V2
1.10.2 2023-08-24 #29805 Destinations v2: Don't soft reset in migration
1.10.1 2023-08-23 #29774 Destinations v2: Don't soft reset overwrite syncs
1.10.0 2023-08-21 #29636 Destinations v2: Several Critical Bug Fixes (cursorless dedup, improved floating-point handling, improved special characters handling; improved error handling)
1.9.1 2023-08-21 #28687 Under the hood: Add dependency on Java CDK v0.0.1.
1.9.0 2023-08-17 #29560 Destinations v2: throw an error on disallowed column name prefixes
1.8.1 2023-08-17 #29522 Migration BugFix - ensure raw dataset created
1.8.0 2023-08-17 #29498 Fix checkpointing logic in GCS staging mode
1.7.8 2023-08-15 #29461 Migration BugFix - ensure migration happens before table creation for GCS staging.
1.7.7 2023-08-11 #29381 Destinations v2: Add support for streams with no columns
1.7.6 2023-08-04 #28894 Destinations v2: Add v1 -> v2 migration Logic
1.7.5 2023-08-04 #29106 Destinations v2: handle unusual CDC deletion edge case
1.7.4 2023-08-04 #29089 Destinations v2: improve special character handling in column names
1.7.3 2023-08-03 #28890 Internal code updates; improved testing
1.7.2 2023-08-02 #28976 Fix composite PK handling in v1 mode
1.7.1 2023-08-02 #28959 Destinations v2: Fix CDC syncs in non-dedup mode
1.7.0 2023-08-01 #28894 Destinations v2: Open up early access program opt-in
1.6.0 2023-07-26 #28723 Destinations v2: Change raw table dataset and naming convention
1.5.8 2023-07-25 #28721 Destinations v2: Handle cursor change across syncs
1.5.7 2023-07-24 #28625 Destinations v2: Limit Clustering Columns to 4
1.5.6 2023-07-21 #28580 Destinations v2: Create dataset in user-specified location
1.5.5 2023-07-20 #28490 Destinations v2: Fix schema change detection in OVERWRITE mode when existing table is empty; other code refactoring
1.5.4 2023-07-17 #28382 Destinations v2: Schema Change Detection
1.5.3 2023-07-14 #28345 Increment patch to trigger a rebuild
1.5.2 2023-07-05 #27936 Internal scaffolding change for future development
1.5.1 2023-06-30 #27891 Revert bugged update
1.5.0 2023-06-27 #27781 License Update: Elv2
1.4.6 2023-06-28 #27268 Internal scaffolding change for future development
1.4.5 2023-06-21 #27555 Reduce image size
1.4.4 2023-05-25 #26585 Small tweak in logs for clarity
1.4.3 2023-05-17 #26213 Fix bug in parsing file buffer config count
1.4.2 2023-05-10 #25925 Testing update. Normalization tests are now done in the destination container.
1.4.1 2023-05-11 #25993 Internal library update
1.4.0 2023-04-29 #25570 Internal library update. Bumping version to stay in sync with BigQuery-denormalized.
1.3.4 2023-04-28 #25588 Internal scaffolding change for future development
1.3.3 2023-04-27 #25346 Internal code cleanup
1.3.1 2023-04-20 #25097 Internal scaffolding change for future development
1.3.0 2023-04-19 #25287 Add parameter to configure the number of file buffers when GCS is used as the loading method
1.2.20 2023-04-12 #25122 Add additional data centers
1.2.19 2023-03-29 #24671 Fail faster in certain error cases
1.2.18 2023-03-23 #24447 Set the Service Account Key JSON field to always_show: true so that it isn't collapsed into an optional fields section
1.2.17 2023-03-17 #23788 S3-Parquet: added handler to process null values in arrays
1.2.16 2023-03-10 #23931 Added support for periodic buffer flush
1.2.15 2023-03-10 #23466 Changed S3 Avro type from Int to Long
1.2.14 2023-02-08 #22497 Fixed table already exists error
1.2.13 2023-01-26 #20631 Added support for destination checkpointing with staging
1.2.12 2023-01-18 #21087 Wrap Authentication Errors as Config Exceptions
1.2.11 2023-01-18 #21144 Added explicit error message if sync fails due to a config issue
1.2.9 2022-12-14 #20501 Report GCS staging failures that occur during connection check
1.2.8 2022-11-22 #19489 Added non-billable projects handle to check connection stage
1.2.7 2022-11-11 #19358 Fixed check method to capture mismatch dataset location
1.2.6 2022-11-10 #18554 Improve check connection method to handle more errors
1.2.5 2022-10-19 #18162 Improve error logs
1.2.4 2022-09-26 #16890 Add user-agent header
1.2.3 2022-09-22 #17054 Respect stream namespace
1.2.1 2022-09-14 #15668 (bugged, do not use) Wrap logs in AirbyteLogMessage
1.2.0 2022-09-09 #14023 (bugged, do not use) Cover arrays only if they are nested
1.1.16 2022-09-01 #16243 Fix Json to Avro conversion when there is field name clash from combined restrictions (anyOf, oneOf, allOf fields)
1.1.15 2022-08-22 #15787 Throw exception if job failed
1.1.14 2022-08-03 #14784 Enabling Application Default Credentials
1.1.13 2022-08-02 #14801 Fix multiple log bindings
1.1.12 2022-08-02 #15180 Fix standard loading mode
1.1.11 2022-06-24 #14114 Remove "additionalProperties": false from specs for connectors with staging
1.1.10 2022-06-16 #13852 Updated stacktrace format for any trace message errors
1.1.9 2022-06-17 #13753 Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager
1.1.8 2022-06-07 #13579 Always check GCS bucket for GCS loading method to catch invalid HMAC keys.
1.1.7 2022-06-07 #13424 Reordered fields for specification.
1.1.6 2022-05-15 #12768 Clarify that the service account key json field is required on cloud.
1.1.5 2022-05-12 #12805 Updated to latest base-java to emit AirbyteTraceMessage on error.
1.1.4 2022-05-04 #12578 In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging.
1.1.3 2022-05-02 #12528 Update Dataset location field description
1.1.2 2022-04-29 #12477 Dataset location is a required field
1.1.1 2022-04-15 #12068 Fixed bug with GCS bucket conditional binding
1.1.0 2022-04-06 #11776 Use serialized buffering strategy to reduce memory consumption.
1.0.2 2022-03-30 #11620 Updated spec
1.0.1 2022-03-24 #11350 Improve check performance
1.0.0 2022-03-18 #11238 Updated spec and documentation
0.6.12 2022-03-18 #10793 Fix namespace with invalid characters
0.6.11 2022-03-03 #10755 Make sure to kill children threads and stop JVM
0.6.8 2022-02-14 #10256 Add -XX:+ExitOnOutOfMemoryError JVM option
0.6.6 2022-02-01 #9959 Fix null pointer exception from buffered stream consumer.
0.6.6 2022-01-29 #9745 Integrate with Sentry.
0.6.5 2022-01-18 #9573 BigQuery Destination : update description for some input fields
0.6.4 2022-01-17 #8383 Support dataset-id prefixed by project-id
0.6.3 2022-01-12 #9415 BigQuery Destination : Fix GCS processing of Facebook data
0.6.2 2022-01-10 #9121 Fixed check method for GCS mode to verify if all roles assigned to user
0.6.1 2021-12-22 #9039 Added part_size configuration to UI for GCS staging
0.6.0 2021-12-17 #8788 BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files
0.5.1 2021-12-16 #8816 Update dataset locations
0.5.0 2021-10-26 #7240 Output partitioned/clustered tables
0.4.1 2021-10-04 #6733 Support dataset starting with numbers
0.4.0 2021-08-26 #5296 Added GCS Staging uploading option
0.3.12 2021-08-03 #3549 Add optional arg to make a possibility to change the BigQuery client's chunk\buffer size
0.3.11 2021-07-30 #5125 Enable additionalPropertities in spec.json
0.3.10 2021-07-28 #3549 Add extended logs and made JobId filled with region and projectId
0.3.9 2021-07-28 #5026 Add sanitized json fields in raw tables to handle quotes in column names
0.3.6 2021-06-18 #3947 Service account credentials are now optional.
0.3.4 2021-06-07 #3277 Add dataset location option
Reference in New Issue View Git Blame Copy Permalink