jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

Destination S3 Data Lake: certify (#54724)

Browse Source 
Co-authored-by: Francis Genet <francis.genet@airbyte.io>
This commit is contained in:
Edward Gao
2025-03-03 09:39:17 -08:00
committed by GitHub
parent 1645341f3b
commit 04d153eb38
5 changed files with 18 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
airbyte-integrations/connectors/destination-s3-data-lake/build.gradle
View File

@@ -6,7 +6,7 @@ plugins {
airbyteBulkConnector {
core = 'load'
toolkits = ['load-iceberg-parquet', 'load-aws']
cdk = 'local'
cdk = '0.334'
}
application {

4
airbyte-integrations/connectors/destination-s3-data-lake/metadata.yaml
View File

@@ -26,7 +26,7 @@ data:
alias: airbyte-connector-testing-secret-store
connectorType: destination
definitionId: 716ca874-520b-4902-9f80-9fad66754b89
dockerImageTag: 0.3.14
dockerImageTag: 0.3.15
dockerRepository: airbyte/destination-s3-data-lake
documentationUrl: https://docs.airbyte.com/integrations/destinations/s3-data-lake
githubIssueLabel: destination-s3-data-lake
@@ -39,7 +39,7 @@ data:
oss:
enabled: true
releaseStage: alpha
supportLevel: community
supportLevel: certified
supportsRefreshes: true
tags:
- language:java

5
docs/integrations/destinations/iceberg.md
View File

@@ -1,5 +1,10 @@
# Iceberg
:::info
You should switch to the [S3 Data Lake](./s3-data-lake.md) destination, which offers
superior implementations of core Airbyte features and significantly better performance.
:::
<!-- env:cloud -->
:::caution

16
docs/integrations/destinations/s3-data-lake.md
View File

@@ -1,15 +1,5 @@
# S3 Data Lake
:::caution
This connector is in early access and still evolving.
Future updates may introduce breaking changes.
We're interested in hearing about your experience! See [Github](https://github.com/airbytehq/airbyte/discussions/50404)
for more information on joining the beta.
:::
This page guides you through the process of setting up the S3 Data Lake destination connector.
This connector writes the Iceberg table format to S3, or an S3-compatible storage backend.
@@ -127,6 +117,11 @@ If these conditions are not met, you may see inaccurate data in the destination
taking precendence over newer records). If this happens, you should use the `append` or `overwrite`
sync mode.
:::caution
Certain API sources are known to have streams which do not meet these conditions, including
Stripe and Monday.
:::
## Branching
Iceberg supports [Git-like semantics](https://iceberg.apache.org/docs/latest/branching/) over your data.
@@ -157,6 +152,7 @@ drop all table versions.
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------|
| 0.3.15 | 2025-02-28 | [\#54724](https://github.com/airbytehq/airbyte/pull/54724) | Certify connector |
| 0.3.14 | 2025-02-14 | [\#53241](https://github.com/airbytehq/airbyte/pull/53241) | New CDK interface; perf improvements, skip initial record staging |
| 0.3.13 | 2025-02-14 | [\#53697](https://github.com/airbytehq/airbyte/pull/53697) | Internal refactor |
| 0.3.12 | 2025-02-12 | [\#53170](https://github.com/airbytehq/airbyte/pull/53170) | Improve documentation, tweak error handling of invalid schema evolution |

5
docs/integrations/destinations/s3-glue.md
View File

@@ -2,6 +2,9 @@
This page guides you through the process of setting up the S3 destination connector with Glue.
You should switch to the [S3 Data Lake](./s3-data-lake.md) destination, which offers
superior implementations of core Airbyte features and significantly better performance.
## Prerequisites
List of required fields:
@@ -318,4 +321,4 @@ the output filename will have an extra extension (GZIP: `.jsonl.gz`).
| 0.1.1 | 2022-12-13 | [19907](https://github.com/airbytehq/airbyte/pull/19907) | Fix parsing empty object in schema |
| 0.1.0 | 2022-11-17 | [18695](https://github.com/airbytehq/airbyte/pull/18695) | Initial Commit |
</details>
</details>