jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

GitBook: [master] 10 pages modified

Browse Source
This commit is contained in:
Shrif Nada
2020-12-18 00:49:56 +00:00
committed by gitbook-bot
parent 222413104a
commit b8f2b33dfa
10 changed files with 30 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/architecture/incremental.md
View File

@@ -1,4 +1,4 @@
# Incremental - Append
# Incremental
## Overview
@@ -74,7 +74,7 @@ Some sources are able to determine the cursor that they use without any user inp
![](../.gitbook/assets/incremental_source_defined.png)
(You can find a more technical details about the configuration data model [here](catalog.md)).
\(You can find a more technical details about the configuration data model [here](catalog.md)\).
## User-Defined Cursor
@@ -82,4 +82,5 @@ Some sources cannot define the cursor without user input. For example, in the [p
![](../.gitbook/assets/incremental_user_defined.png)
(You can find a more technical details about the configuration data model [here](catalog.md)).
\(You can find a more technical details about the configuration data model [here](catalog.md)\).

1
docs/changelog.md
View File

@@ -77,3 +77,4 @@ This is our very first release after 2 months of work.
* **New blog:** we published a few articles on our startup journey, but also about our vision to making data integrations a commodity.
Stay tuned, we will have new sources and destinations very soon! Don't hesitate to subscribe to our [newsletter](https://airbyte.io/#subscribe-newsletter) to receive our product updates and community news.

27
docs/contributing-to-airbyte/building-new-connector/standard-source-tests.md
View File

@@ -1,23 +1,32 @@
# Standard Source Test Suite
Test methods start with `test`. Other methods are internal helpers in the java class implementing the test suite.
### testGetSpec
## testGetSpec
Verify that a spec operation issued to the connector returns a valid spec.
### testCheckConnection
## testCheckConnection
Verify that a check operation issued to the connector with the input config file returns a success response.
### testDiscover
## testDiscover
Verifies when a discover operation is run on the connector using the given config file, a valid catalog is output by the connector.
### testFullRefreshRead
## testFullRefreshRead
Configuring all streams in the input catalog to full refresh mode, verifies that a read operation produces some RECORD messages.
### testIdenticalFullRefreshes
## testIdenticalFullRefreshes
Configuring all streams in the input catalog to full refresh mode, performs two read operations on all streams which support full refresh syncs. It then verifies that the RECORD messages output from both were identical.
### testIncrementalSyncWithState
This test verifies that all streams in the input catalog which support incremental sync can do so correctly. It does this by running two read operations on the connector's Docker image: the first takes the configured catalog and config provided to this test as input. It then verifies that the sync produced a non-zero number of RECORD and STATE messages. The second read takes the same catalog and config used in the first test, plus the last STATE message output by the first read operation as the input state file. It verifies that no records are produced (since we read all records in the first sync). This test is performed only for streams which support incremental. Streams which do not support incremental sync are ignored. If no streams in the input catalog support incremental sync, this test is skipped.
## testIncrementalSyncWithState
### testEmptyStateIncrementalIdenticalToFullRefresh
If the source does not support incremental sync, this test is skipped. Otherwise, this test runs two syncs: one where all streams provided in the input catalog sync in full refresh mode, and another where all the streams which in the input catalog which support incremental, sync in incremental mode (streams which don't support incremental sync in full refresh mode). Then, the test asserts that the two syncs produced the same RECORD messages. Any other type of message is disregarded.
This test verifies that all streams in the input catalog which support incremental sync can do so correctly. It does this by running two read operations on the connector's Docker image: the first takes the configured catalog and config provided to this test as input. It then verifies that the sync produced a non-zero number of RECORD and STATE messages. The second read takes the same catalog and config used in the first test, plus the last STATE message output by the first read operation as the input state file. It verifies that no records are produced \(since we read all records in the first sync\). This test is performed only for streams which support incremental. Streams which do not support incremental sync are ignored. If no streams in the input catalog support incremental sync, this test is skipped.
## testEmptyStateIncrementalIdenticalToFullRefresh
If the source does not support incremental sync, this test is skipped. Otherwise, this test runs two syncs: one where all streams provided in the input catalog sync in full refresh mode, and another where all the streams which in the input catalog which support incremental, sync in incremental mode \(streams which don't support incremental sync in full refresh mode\). Then, the test asserts that the two syncs produced the same RECORD messages. Any other type of message is disregarded.

2
docs/contributing-to-airbyte/building-new-connector/testing-connectors.md
View File

@@ -22,7 +22,7 @@ The test suite then runs its test cases, which include:
* Running a `read` operation should produce at least one record.
* Running two consecutive full refresh reads should produce identical records.
See all the test cases and their description in [Standard Source Tests](./standard-source-tests.md).
See all the test cases and their description in [Standard Source Tests](standard-source-tests.md).
### Setting up standard tests for your connector

2
docs/integrations/sources/exchangeratesapi-io.md
View File

@@ -24,7 +24,7 @@ Currencies are `number` and the date is a `string`.
| Feature | Supported? |
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental Sync | No |
| Incremental - Append Sync | Yes |
### Getting started

2
docs/integrations/sources/github.md
View File

@@ -15,7 +15,7 @@ Several output streams are available from this source \(commits, issues, pull\_r
| Feature | Supported? |
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental Sync | No |
| Incremental - Append Sync | yes |
| Replicate Incremental Deletes | No |
| SSL connection | Yes |

2
docs/integrations/sources/marketo.md
View File

@@ -33,7 +33,7 @@ Feature
| Supported?\(Yes/No\) | Notes | |
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental Sync | No | |
| Incremental - Append SyncY | Yes | |
### Performance considerations

2
docs/integrations/sources/shopify.md
View File

@@ -33,7 +33,7 @@ This Source is capable of syncing the following core Streams:
| Feature | Supported?\(Yes/No\) | Notes |
| :--- | :--- | :--- |
| Full Refresh Sync | yes | |
| Incremental Sync | no | |
| Incremental - Append Sync | yes | |
### Performance considerations

2
docs/integrations/sources/stripe.md
View File

@@ -19,7 +19,7 @@ The [Stripe API](https://stripe.com/docs/api) uses the same [JSONSchema](https:/
| Feature | Supported? |
| :--- | :--- |
| Full Refresh Sync | Yes |
| Incremental Sync | No |
| Incremental - Append Sync | Yes |
| Replicate Incremental Deletes | No |
| SSL connection | Yes |

2
docs/integrations/sources/zendesk-support.md
View File

@@ -39,7 +39,7 @@ This Source is capable of syncing the following core Streams:
| Feature | Supported?\(Yes/No\) | Notes |
| :--- | :--- | :--- |
| Full Refresh Sync | yes | |
| Incremental Sync | no | |
| Incremental - Append Sync | yes | |
### Performance considerations