jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

🐛Destination S3 and GCS - Fixed connector's bug that prevent writing streams with more than 50GB (#5890)

Browse Source 
Co-authored-by: ievgeniit <ievgeniit>
This commit is contained in:
Eugene
2021-09-18 00:21:07 +03:00
committed by GitHub
parent 69b2453108
commit 64ce6fd484
32 changed files with 938 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/integrations/destinations/gcs.md
View File

@@ -25,6 +25,7 @@ The Airbyte GCS destination allows you to sync data to cloud storage buckets. Ea
| HMAC Key Access ID | string | HMAC key access ID . The access ID for the GCS bucket. When linked to a service account, this ID is 61 characters long; when linked to a user account, it is 24 characters long. See [HMAC key](https://cloud.google.com/storage/docs/authentication/hmackeys) for details. |
| HMAC Key Secret | string | The corresponding secret for the access ID. It is a 40-character base-64 encoded string. |
| Format | object | Format specific configuration. See below for details. |
| Part Size | integer | Arg to configure a block size. Max allowed blocks by GCS = 10,000, i.e. max stream size = blockSize * 10,000 blocks. |
Currently, only the [HMAC key](https://cloud.google.com/storage/docs/authentication/hmackeys) is supported. More credential types will be added in the future.
@@ -373,5 +374,6 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A
| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.2 | 2021-09-12 | [#5720](https://github.com/airbytehq/airbyte/issues/5720) | Added configurable block size for stream. Each stream is limited to 10,000 by GCS |
| 0.1.1 | 2021-08-26 | [#5296](https://github.com/airbytehq/airbyte/issues/5296) | Added storing gcsCsvFileLocation property for CSV format. This is used by destination-bigquery (GCS Staging upload type) |
| 0.1.0 | 2021-07-16 | [#4329](https://github.com/airbytehq/airbyte/pull/4784) | Initial release. |

6
docs/integrations/destinations/s3.md
View File

@@ -26,6 +26,7 @@ The Airbyte S3 destination allows you to sync data to AWS S3/ Minio S3. Each str
| Access Key ID | string | AWS/Minio credential. |
| Secret Access Key | string | AWS/Minio credential. |
| Format | object | Format specific configuration. See below for details. |
| Part Size | integer | Arg to configure a block size. Max allowed blocks by S3 = 10,000, i.e. max stream size = blockSize * 10,000 blocks. |
⚠️ Please note that under "Full Refresh Sync" mode, data in the configured bucket and path will be wiped out before each sync. We recommend you to provision a dedicated S3 resource for this sync to prevent unexpected data deletion from misconfiguration. ⚠️
@@ -376,8 +377,9 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A
| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.11 | 2021-10-08 | [#5729](https://github.com/airbytehq/airbyte/pull/5729) | For field names that start with a digit, a `_` will be appended at the beginning for the` Parquet` and `Avro` formats. |
| 0.1.10 | 2021-09-13 | [#4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator |
| 0.1.12 | 2021-09-13 | [#5720](https://github.com/airbytehq/airbyte/issues/5720) | Added configurable block size for stream. Each stream is limited to 10,000 by S3 |
| 0.1.11 | 2021-09-10 | [#5729](https://github.com/airbytehq/airbyte/pull/5729) | For field names that start with a digit, a `_` will be appended at the beginning for the` Parquet` and `Avro` formats. |
| 0.1.10 | 2021-08-17 | [#4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator |
| 0.1.9 | 2021-07-12 | [#4666](https://github.com/airbytehq/airbyte/pull/4666) | Fix MinIO output for Parquet format. |
| 0.1.8 | 2021-07-07 | [#4613](https://github.com/airbytehq/airbyte/pull/4613) | Patched schema converter to support combined restrictions. |
| 0.1.7 | 2021-06-23 | [#4227](https://github.com/airbytehq/airbyte/pull/4227) | Added Avro and JSONL output. |