jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

GitBook: [master] 94 pages modified

Browse Source
This commit is contained in:
Shrif Nada
2021-01-06 17:06:10 +00:00
committed by gitbook-bot
parent d161ce45dd
commit 220157137d
7 changed files with 48 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/SUMMARY.md
View File

@@ -73,9 +73,10 @@
* [Contributing to Airbyte](contributing-to-airbyte/README.md)
* [Code of Conduct](contributing-to-airbyte/code-of-conduct.md)
* [Developing Locally](contributing-to-airbyte/developing-locally.md)
* [Building New Connectors](contributing-to-airbyte/building-new-connector/README.md)
* [Python Connectors](contributing-to-airbyte/building-new-connector/python-connectors.md)
* [Developing Connectors](contributing-to-airbyte/building-new-connector/README.md)
* [Best Practices for Connector Development](contributing-to-airbyte/building-new-connector/best-practices-for-connector-development.md)
* [Java Connectors](contributing-to-airbyte/building-new-connector/java-connectors.md)
* [Python Connectors](contributing-to-airbyte/building-new-connector/python-connectors.md)
* [Testing Connectors](contributing-to-airbyte/building-new-connector/testing-connectors.md)
* [Standard Source Test Suite](contributing-to-airbyte/building-new-connector/standard-source-tests.md)
* [Code Style](contributing-to-airbyte/code-style.md)

10
docs/architecture/connections.md
View File

@@ -1,21 +1,21 @@
# Connections
A connection is a configuration for syncing data between a source and a destination. To setup a connection, a user must configure things such such as:
A connection is a configuration for syncing data between a source and a destination. To setup a connection, a user must configure things such such as:
* A catalog selection: which [streams and fields](catalog.md) to replicate from the source
* Sync mode: Whether streams should be replicated using [Full Refresh](full-refresh.md) or [Incremental](incremental.md) syncs
* Sync schedule: when to trigger a sync of the data
For information about catalog selections and sync modes, see [AirbyteCatalog & ConfiguredAirbyteCatalog](catalog.md) and [Full Refresh](full-refresh.md) and [Incremental](incremental.md). Sync schedules are explained below.
For information about catalog selections and sync modes, see [AirbyteCatalog & ConfiguredAirbyteCatalog](catalog.md) and [Full Refresh](full-refresh.md) and [Incremental](incremental.md). Sync schedules are explained below.
#### Sync schedules
## Sync schedules
Syncs will be triggered by either:
Syncs will be triggered by either:
* A manual request \(i.e: clicking the "Sync Now" button in the UI\)
* A schedule
When a scheduled connection is first created, a sync is executed as soon as possible. After that, a sync is run once the time since the last sync \(whether it was triggered manually or due to a schedule\) has exceeded the schedule interval. For example, consider the following illustrative scenario:
When a scheduled connection is first created, a sync is executed as soon as possible. After that, a sync is run once the time since the last sync \(whether it was triggered manually or due to a schedule\) has exceeded the schedule interval. For example, consider the following illustrative scenario:
* **October 1st, 2pm**, a user sets up a connection to sync data every 24 hours.
* **October 1st, 2:01pm**: sync job runs

4
docs/architecture/jobs.md
View File

@@ -1,6 +1,6 @@
# Workers & Jobs
In Airbyte, all interactions with connectors are run as jobs performed by a Worker. Examples of workers are:
In Airbyte, all interactions with connectors are run as jobs performed by a Worker. Examples of workers are:
* Spec worker: retrieves the specification of a connector \(the inputs needed to run this connector\)
* Check connection worker: verifies that the inputs to a connector are valid and can be used to run a sync
@@ -9,7 +9,7 @@ In Airbyte, all interactions with connectors are run as jobs performed by a Work
See the [architecture overview](high-level-view.md) for more information about workers.
### Job State Machine
## Job State Machine
Jobs in the worker follow the following state machine.

2
docs/contributing-to-airbyte/building-new-connector/README.md
View File

@@ -1,4 +1,4 @@
# Building New Connectors
# Developing Connectors
Airbyte supports two types of connectors: Sources and Destinations. A connector takes the form of a Docker image which follows the [Airbyte specification](../../architecture/airbyte-specification.md).

2
docs/contributing-to-airbyte/building-new-connector/best-practices-for-connector-development.md Normal file
View File

@@ -0,0 +1,2 @@
# Best Practices for Connector Development

55
docs/integrations/sources/github.md
View File

@@ -8,30 +8,31 @@ This Github source wraps the [Singer Github Tap](https://github.com/singer-io/ta
### Output schema
This connector outputs the following streams:
* [Assignees](https://developer.github.com/v3/issues/assignees/#list-assignees)
* [Collaborators](https://developer.github.com/v3/repos/collaborators/#list-collaborators)
* [Comments](https://developer.github.com/v3/issues/comments/#list-comments-in-a-repository)
* [Commits](https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository)
* [Commit comments](https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-commit-comments-for-a-repository)
* [Events](https://docs.github.com/en/free-pro-team@latest/rest/reference/activity#list-repository-events)
* [Issues](https://developer.github.com/v3/issues/#list-issues-for-a-repository)
* [Issue events](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-issue-events-for-a-repository)
* [Issue labels](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-labels-for-a-repository)
* [Issue milestones](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-milestones)
* [Projects](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-repository-projects)
* [Project cards](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-project-cards)
* [Project columns](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-project-columns)
* [Pull requests](https://developer.github.com/v3/pulls/#list-pull-requests)
* [PR commits](https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-commits-on-a-pull-request)
* [Pull request reviews](https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews)
* [Releases](https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-releases)
* [Reviews](https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request)
* [Review comments](https://developer.github.com/v3/pulls/comments)
* [Stargazers](https://developer.github.com/v3/activity/starring/#list-stargazers)
* [Teams](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#list-teams)
* [Team members](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#list-team-members)
* [Team memberships](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#get-team-membership-for-a-user)
This connector outputs the following streams:
* [Assignees](https://developer.github.com/v3/issues/assignees/#list-assignees)
* [Collaborators](https://developer.github.com/v3/repos/collaborators/#list-collaborators)
* [Comments](https://developer.github.com/v3/issues/comments/#list-comments-in-a-repository)
* [Commits](https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository)
* [Commit comments](https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-commit-comments-for-a-repository)
* [Events](https://docs.github.com/en/free-pro-team@latest/rest/reference/activity#list-repository-events)
* [Issues](https://developer.github.com/v3/issues/#list-issues-for-a-repository)
* [Issue events](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-issue-events-for-a-repository)
* [Issue labels](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-labels-for-a-repository)
* [Issue milestones](https://docs.github.com/en/free-pro-team@latest/rest/reference/issues#list-milestones)
* [Projects](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-repository-projects)
* [Project cards](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-project-cards)
* [Project columns](https://docs.github.com/en/free-pro-team@latest/rest/reference/projects#list-project-columns)
* [Pull requests](https://developer.github.com/v3/pulls/#list-pull-requests)
* [PR commits](https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-commits-on-a-pull-request)
* [Pull request reviews](https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews)
* [Releases](https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-releases)
* [Reviews](https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request)
* [Review comments](https://developer.github.com/v3/pulls/comments)
* [Stargazers](https://developer.github.com/v3/activity/starring/#list-stargazers)
* [Teams](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#list-teams)
* [Team members](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#list-team-members)
* [Team memberships](https://docs.github.com/en/free-pro-team@latest/rest/reference/teams#get-team-membership-for-a-user)
### Features
@@ -51,15 +52,15 @@ The Github connector should not run into Github API limitations under normal usa
### Requirements
* Github Account
* Github Personal Access Token wih the necessary permissions (described below)
* Github Personal Access Token wih the necessary permissions \(described below\)
### Setup guide
Log into Github and then generate a [personal access token](https://github.com/settings/tokens).
Your token should have at least the `repo` scope. Depending on which streams you want to sync, the user generating the token needs more permissions:
Your token should have at least the `repo` scope. Depending on which streams you want to sync, the user generating the token needs more permissions:
* For syncing Collaborators, the user which generates the personal access token must be a collaborator. To become a collaborator, they must be invited by an owner. Read more about access permissions [here](https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/access-permissions-on-github).
* Syncing [Teams](https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-organizations-and-teams/about-teams) is only available to authenticated members of a team's [organization](https://docs.github.com/en/free-pro-team@latest/rest/reference/orgs). [Personal user accounts](https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/types-of-github-accounts) and repositories belonging to them don't have access to Teams features.
* To sync the Projects stream, the repository must have the Projects feature enabled.

14
docs/integrations/sources/mixpanel.md
View File

@@ -10,13 +10,13 @@ This Hubspot source wraps the [Singer Mixpanel Tap](https://github.com/singer-io
Several output streams are available from this source:
* [Export](https://developer.mixpanel.com/docs/exporting-raw-data#section-export-api-reference) (Incremental)
* [Engage](https://developer.mixpanel.com/docs/data-export-api#section-engage) (Full table)
* [Funnels](https://developer.mixpanel.com/docs/data-export-api#section-funnels) (Incremental)
* [Revenue](https://developer.mixpanel.com/docs/data-export-api#section-hr-span-style-font-family-courier-revenue-span) (Incremental)
* [Annotations](https://developer.mixpanel.com/docs/data-export-api#section-annotations) (Full table)
* [Cohorts](https://developer.mixpanel.com/docs/cohorts#section-list-cohorts) (Full table)
* [Cohort Members](https://developer.mixpanel.com/docs/data-export-api#section-engage) (Full table)
* [Export](https://developer.mixpanel.com/docs/exporting-raw-data#section-export-api-reference) \(Incremental\)
* [Engage](https://developer.mixpanel.com/docs/data-export-api#section-engage) \(Full table\)
* [Funnels](https://developer.mixpanel.com/docs/data-export-api#section-funnels) \(Incremental\)
* [Revenue](https://developer.mixpanel.com/docs/data-export-api#section-hr-span-style-font-family-courier-revenue-span) \(Incremental\)
* [Annotations](https://developer.mixpanel.com/docs/data-export-api#section-annotations) \(Full table\)
* [Cohorts](https://developer.mixpanel.com/docs/cohorts#section-list-cohorts) \(Full table\)
* [Cohort Members](https://developer.mixpanel.com/docs/data-export-api#section-engage) \(Full table\)
If there are more endpoints you'd like Airbyte to support, please [create an issue.](https://github.com/airbytehq/airbyte/issues/new/choose)