jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs: Add edit schema page (#44396)

Browse Source
This commit is contained in:
Natalie Kwong
2024-08-26 08:26:36 -07:00
committed by GitHub
parent b034e7d444
commit c54d1a6fe5
10 changed files with 163 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/operator-guides/refreshes.md
View File

@@ -4,15 +4,9 @@ products: all
# Refreshes
:::info
## Overview
This page describes a coming-soon feature. Refreshes will be available as of Airbyte v1.0.0.
:::
## Refreshes Overview
As of Airbyte 1.0, Refresh Syncs have been added to the Airbyte platform. A Refresh Sync erases the current cursor you have set for any incremental streams, effectively ‘rewinding’ the sync back to the beginning. This is helpful in the following scenarios:
A Refresh Sync erases the current cursor you have set for any incremental streams, effectively ‘rewinding’ the sync back to the beginning. This is helpful in the following scenarios:
- The source you're syncing from does not sync record deletions/removals, and you wish to mirror the source stream, which would include removing deleted records
- The source you are syncing from is unreliable and you wish to re-attempt the sync
@@ -105,20 +99,20 @@ Notice that user #2’s latest entry doesn’t belong to the current (e.g. `max(
## Frequently Asked Questions (FAQ)
### How are full refresh append/overwrite different from merge/truncate refresh?
### How is a Full Refresh sync that uses Append + Overwrite to write to the destination different from a refresh?
They're completely identical! A full refresh append sync is just running a merge refresh on every sync; a full refresh overwrite sync is just running a truncate refresh on every sync. Notably, this means that `_airbyte_generation_id` will increment on every sync.
They're completely identical! A Full Refresh | Append + Overwite sync is running a refresh on every sync and retaining all records. A Full Refresh | Overwrite sync is running a refresh on every sync and removing any reecords that no longer appear in the source. Notably, this means that `_airbyte_generation_id` will increment on every sync.
### Does the generation ID reset to 0 after running a clear+resync? What about a truncate refresh?
### Does the generation ID reset to 0 after running a Clear and sync again?
The generation ID will be incremented when you run a clear or refresh. Airbyte will never decrease the generation ID.
The generation ID will be incremented whenever you run a clear or refresh. Airbyte will never decrease the generation ID.
Notably, for clear syncs: if you run a refresh immediately after running a clear (including syncing a full refresh stream normally, as noted in the [previous question](#how-are-full-refresh-appendoverwrite-different-from-mergetruncate-refresh)), that will _also_ increment the generation ID. This means the generation ID will increment twice, even though one of those "generations" never emitted any records.
Iif you run a Refresh immediately after running a Clear (including syncing a full refresh stream normally, as noted in the above question, that will _also_ increment the generation ID. This means the generation ID will increment twice, even though one of those "generations" never emitted any records.
### For DV2 destinations, how do clears / truncate refreshes interact with the raw and final tables?
### For DV2 destinations, how do clears or Refreshes that remove records interact with the raw and final tables?
All preeexisting data will be deleted from both the raw and final tables. If you want to retain that data, you should run a _merge_ refresh.
All pre-existing data will be deleted from both the raw and final tables. If you want to retain that data, you should run a Refresh that retains records.
### If I refresh/clear a single stream in a connection, does the generation ID increment for other streams in that connection?
### If I Refresh or Clear a single stream in a connection, does the generation ID increment for other streams in that connection?
No. Streams within a connection have independent generation IDs. Clearing/refreshing a single stream will only increment that stream's generation; other streams are unaffected.
No. Streams within a connection have independent generation IDs. Clearing/Refreshing a single stream will only increment that stream's generation. Other streams are unaffected.