jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

Docs: streams table (#24326)

Browse Source 
* Added updated text

* edits

* edits to text

* removed header

* edited links

* edited some text

* added new gifs to doc

* typo fixes

* Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md

Co-authored-by: Andy Jih <andyjih@users.noreply.github.com>

* Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md

Co-authored-by: Andy Jih <andyjih@users.noreply.github.com>

* Update edit-stream-configuration.md

* Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md

Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>

* Update edit-stream-configuration.md

edited based on Edmundo's suggestions

---------

Co-authored-by: Andy Jih <andyjih@users.noreply.github.com>
Co-authored-by: Edmundo Ruiz Ghanem <168664+edmundito@users.noreply.github.com>
This commit is contained in:
Sophia Wiley
2023-03-28 18:35:14 +02:00
committed by GitHub
parent 25069a3913
commit 3ba5a3c644
2 changed files with 124 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

152
docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
View File

@@ -1,55 +1,77 @@
# Edit stream configuration
# Manage syncs
By editing stream configurations, you can customize how your data syncs to the destination. This allows you to manage aspects of the sync, such as replication frequency, destination namespace, and ensuring your data is written to the correct location.
After you have created a connection, you can change how your data syncs to the destination by modifying the [configuration settings](#configure-connection-settings) and the [stream settings](#modify-streams-in-your-connection).
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
## Configure connection settings
Configuring the connection settings allows you to manage various aspects of the sync, such as how often data syncs and where data is written.
To configure these settings:
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
2. Click the **Replication** tab.
The **Transfer** and **Streams** settings include the following parameters:
3. Click the **Configuration** dropdown.
| Parameter | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Replication frequency | How often the data syncs |
| [Non-breaking schema updates](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes/#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
| Destination Namespace | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |
You can configure the following settings:
:::note
These parameters apply to all streams in the connection.
:::note
These settings apply to all streams in the connection.
:::
If you need to use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html):
| Setting | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Replication frequency | How often the data syncs |
| Destination namespace | Where the replicated data is written |
| Destination stream prefix | How you identify streams from different connectors |
| [Non-breaking schema updates](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes/#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
To use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html):
1. In the **Replication Frequency** dropdown, click **Cron**.
2. Enter a cron expression and choose a time zone to create a sync schedule.
:::note
* Only one sync per connection can run at a time.
* If cron schedules a sync to run before the last one finishes, the scheduled sync will start after the last sync completes.
* Airbyte Cloud does not allow schedules that sync more than once per hour.
* If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync.
* Airbyte Cloud does not support schedules that sync more frequently than once per hour.
:::
In the **Activate the streams you want to sync section**, you can make changes to any stream you choose.
## Modify streams in your connection
To search for a stream:
In the **Activate the streams you want to sync** table, you can choose which streams to sync and how they are loaded to the destination.
1. Click the **Search stream name** search box.
:::note
A connection's schema consists of one or many streams. Each stream is most commonly associated with a database table or an API endpoint. Within a stream, there can be one or many fields or columns.
:::
2. Type the name of the stream you want to find.
To modify streams:
3. Streams matching your search are displayed in the list.
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change.
To change individual stream configuration:
2. Click the **Replication** tab.
![Single Edit Gif 7](https://user-images.githubusercontent.com/106352739/187313088-85c61a6d-1025-45fa-b14e-a7fe86defea4.gif)
3. Scroll down to the **Activate the streams you want to sync** table.
1. In the **Sync** column of the stream, toggle the sync on or off.
:::note
2. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply.
You can modify a single stream, or you can modify multiple streams at once.
:::
Modify an individual stream:
![gif-single-edit-march-2023](https://user-images.githubusercontent.com/106352739/226917693-068256da-c948-4f22-bdce-49f5bad95bf6.gif)
1. In the **Activate the streams you want to sync** table, toggle **Sync** on or off for your selected stream.
2. Click the **Sync mode** dropdown and select the sync mode you want to apply.
:::note
@@ -57,28 +79,33 @@ Depending on the sync mode you select, you may need to choose a cursor or primar
:::
3. If there is a dropdown arrow in the **Cursor** or **Primary key** fields, click the dropdown arrow and choose the cursor or primary key.
3. Select the **Cursor** or **Primary keys** when they are required by the stream sync mode.
To change multiple stream configurations:
:::note
![Batch Edit gif 5](https://user-images.githubusercontent.com/106352739/187312110-d16b4f9a-9d43-4b23-b644-b64004f33b58.gif)
Source-defined cursors and primary keys are selected automatically and cannot be changed in the table.
1. Click the first checkbox in the table header to select all streams in the connection.
2. Deselect the checkboxes of streams you do not want to apply these changes to.
:::
3. In the highlighted header of the table, toggle the sync on or off.
4. Click on a stream to display the stream details panel.
4. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply to these streams.
5. Toggle individual fields to include or exclude them in the sync, or use the toggle in the table header to select all fields at once.
5. If there is a dropdown arrow in the **Cursor** or **Primary key** fields of the highlighted table header, click the dropdown arrow and choose the cursor or primary key.
:::note
6. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes.
* You can only deselect top-level fields. You cannot deselect nested fields.
* The Airbyte platform may read all data from the source (depending on the source), but it will only write data to the destination from fields you selected. Deselecting fields will not prevent the Airbyte platform from reading them.
* When you refresh the schema, newly added fields will be selected by default, even if you have previously deselected fields in that stream.
To save the changes:
1. Click **Save changes**, or click **Cancel** to discard the changes.
:::
2. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
6. Depending on the sync mode you chose for your connection, you can select the **Cursor** or **Primary keys** for individual fields in this table.
7. Click the **X** to close the stream details panel.
8. Click **Save changes**, or click **Cancel** to discard the changes.
9. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
:::caution
@@ -86,9 +113,50 @@ Airbyte recommends that you reset streams. A reset will delete data in the desti
:::
3. Click **Save connection**, or click **Cancel** to close the dialog.
10. Click **Save connection**.
To refresh the source schema:
1. Click **Refresh source schema** to fetch the schema of your data source.
Modify multiple streams:
2. If the schema has changed, the **Refreshed source schema** dialog displays them.
![gif-batch-edit-march-2023](https://user-images.githubusercontent.com/106352739/226917994-c43941db-bb54-4a12-8270-f24fc4e2e6a7.gif)
1. In the **Activate the streams you want to sync** table, select the checkboxes of streams that you want to apply changes to.
:::note
To select or deselect all streams, click the checkbox in the table header. To deselect an individual stream, deselect its checkbox in the table.
:::
* In the highlighted footer of the table:
1. Toggle **Sync** on or off.
2. Click the **Sync mode** dropdown and select the sync mode you want to apply.
:::note
Depending on the sync mode you select, you may need to choose a cursor or primary key.
:::
3. Select the **Cursor** and **Primary keys** if there are dropdowns in those fields.
:::note
Source-defined cursors and primary keys cannot be changed while configuring multiple streams.
:::
4. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes.
2. Click **Save changes**, or click **Cancel** to discard the changes.
3. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes.
:::caution
Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior.
:::
4. Click **Save connection**.

14
docs/cloud/managing-airbyte-cloud/manage-schema-changes.md
View File

@@ -55,3 +55,17 @@ To review and fix breaking schema changes:
If a connection’s source schema has breaking changes, it will stop syncing. You must review and fix the changes before editing the connection or resuming syncs.
:::
### Manually refresh the source schema
In addition to Airbyte Cloud’s automatic schema change detection, you can manually refresh the source schema to stay up to date with changes in your schema.
To manually refresh the source schema:
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to refresh.
2. Click the **Replication** tab.
3. In the **Activate the streams you want to sync** table, click **Refresh source schema** to fetch the schema of your data source.
2. If there are changes to the schema, you can review them in the **Refreshed source schema** dialog.