jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

GitBook: [master] 161 pages and 75 assets modified

Browse Source
This commit is contained in:
Marcos Marx
2021-05-16 22:51:02 +00:00
committed by gitbook-bot
parent 18c1d6eb1b
commit a18b0e80d0
77 changed files with 1322 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/integrations/README.md
View File

@@ -1,2 +1,2 @@
# Connectors
# Connector Catalog

5
docs/integrations/custom-connectors.md
View File

@@ -36,8 +36,9 @@ Note that this new connector could just be an updated version of an existing con
## Upgrading a connector
To upgrade your connector version, go to the admin panel in the left hand side of the UI, find this connector in the list, and input the latest connector version.
To upgrade your connector version, go to the admin panel in the left hand side of the UI, find this connector in the list, and input the latest connector version.
![](../.gitbook/assets/upgrading_connector_admin_panel.png)
To browse the available connector versions, simply click on the relevant link in the `Image` column to navigate to the connector's DockerHub page. From there, simply click on the `Tags` section in the top bar.
To browse the available connector versions, simply click on the relevant link in the `Image` column to navigate to the connector's DockerHub page. From there, simply click on the `Tags` section in the top bar.

7
docs/integrations/destinations/mysql.md
View File

@@ -46,9 +46,7 @@ MySQL doesn't differentiate between a database and schema. A database is essenti
### Setup the MySQL destination in Airbyte
Before setting up MySQL destination in Airbyte, you need to set the [local_infile](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_local_infile)
system variable to true. You can do this by running the query `SET GLOBAL local_infile = true` with a user with [SYSTEM_VARIABLES_ADMIN](https://dev.mysql.com/doc/refman/8.0/en/privileges-provided.html#priv_system-variables-admin) permission.
This is required cause Airbyte uses `LOAD DATA LOCAL INFILE` to load data into table.
Before setting up MySQL destination in Airbyte, you need to set the [local\_infile](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_local_infile) system variable to true. You can do this by running the query `SET GLOBAL local_infile = true` with a user with [SYSTEM\_VARIABLES\_ADMIN](https://dev.mysql.com/doc/refman/8.0/en/privileges-provided.html#priv_system-variables-admin) permission. This is required cause Airbyte uses `LOAD DATA LOCAL INFILE` to load data into table.
You should now have all the requirements needed to configure MySQL as a destination in the UI. You'll need the following information to configure the MySQL destination:
@@ -56,4 +54,5 @@ You should now have all the requirements needed to configure MySQL as a destinat
* **Port**
* **Username**
* **Password**
* **Database**
* **Database**

10
docs/integrations/destinations/snowflake.md
View File

@@ -144,17 +144,19 @@ When an identifier is double-quoted, it is stored and resolved exactly as entere
Therefore, Airbyte Snowflake destination will create tables and schemas using the Unquoted identifiers when possible or fallback to Quoted Identifiers if the names are containing special characters.
## Cloud Storage Staging
By default, Airbyte uses batches of `INSERT` commands to add data to a temporary table before copying it over to the final table in Snowflake. This is too slow for larger/multi-GB replications. For those larger replications we recommend configuring using cloud storage to allow batch writes and loading.
### AWS S3
For AWS S3, you will need to create a bucket and provide credentials to access the bucket. We recommend creating a bucket that is only used for Airbyte to stage data to Snowflake. Airbyte needs read/write access to interact with this bucket.
### Google Cloud Storage (GCS)
### Google Cloud Storage \(GCS\)
First you will need to create a GCS bucket.
Then you will need to run the script below:
* You must run the script as the account admin for Snowflake.
* You should replace `AIRBYTE_ROLE` with the role you used for Airbyte's Snowflake configuration.
* Replace `YOURBUCKETNAME` with your bucket name
@@ -162,7 +164,8 @@ Then you will need to run the script below:
* `gcs_airbyte_integration` must be used
The script:
```
```text
create storage INTEGRATION gcs_airbyte_integration
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = GCS
@@ -179,6 +182,7 @@ GRANT USAGE ON stage gcs_airbyte_stage TO ROLE AIRBYTE_ROLE;
DESC STORAGE INTEGRATION gcs_airbyte_integration;
```
The final query should show a `STORAGE_GCP_SERVICE_ACCOUNT` property with an email as the property value.
The final query should show a `STORAGE_GCP_SERVICE_ACCOUNT` property with an email as the property value.
Finally, you need to add read/write permissions to your bucket with that email.

3
docs/integrations/sources/clickhouse.md
View File

@@ -53,4 +53,5 @@ GRANT SELECT ON <database name>.* TO 'airbyte'@'%';
You can limit this grant down to specific tables instead of the whole database. Note that to replicate data from multiple ClickHouse databases, you can re-run the command above to grant access to all the relevant schemas, but you'll need to set up multiple sources connecting to the same db on multiple schemas.
Your database user should now be ready for use with Airbyte.
Your database user should now be ready for use with Airbyte.

2
docs/integrations/sources/facebook-marketing.md
View File

@@ -50,7 +50,7 @@ For more information, see the [Facebook Insights API documentation. ](https://de
Facebook heavily throttles API tokens generated from Facebook Apps by default, making it infeasible to use such a token for syncs with Airbyte. To be able to use this connector without your syncs taking days due to rate limiting follow the instructions in the Setup Guide below to access better rate limits.
{% endhint %}
See Facebook's [documentation on rate limiting](https://developers.facebook.com/docs/marketing-api/overview/authorization/#access-levels) for more information on requesting a quota upgrade.
See Facebook's [documentation on rate limiting](https://developers.facebook.com/docs/marketing-api/overview/authorization/#access-levels) for more information on requesting a quota upgrade.
## Getting started

14
docs/integrations/sources/google-search-console.md
View File

@@ -4,7 +4,7 @@
The Google Search Console source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run.
This source wraps the [Singer Google Search Console Tap](https://github.com/singer-io/tap-google-search-console).
This source wraps the [Singer Google Search Console Tap](https://github.com/singer-io/tap-google-search-console).
### Output schema
@@ -17,7 +17,7 @@ This Source is capable of syncing the following Streams:
* [Performance report date](https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query)
* [Performance report device](https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query)
* [Performance report page](https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query)
* [Performance report query (keyword)](https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query)
* [Performance report query \(keyword\)](https://developers.google.com/webmaster-tools/search-console-api-original/v3/searchanalytics/query)
### Data type mapping
@@ -44,15 +44,17 @@ This connector attempts to back off gracefully when it hits Reports API's rate l
## Getting started
### Requirements
* Credentials to a Google Service Account with delegated Domain Wide Authority
* Email address of the workspace admin which created the Service Account
### Create a Service Account with delegated domain wide authority
Follow the Google Documentation for performing [Delegating domain-wide authority](https://developers.google.com/identity/protocols/oauth2/service-account#delegatingauthority) to create a Service account with delegated domain wide authority. This account must be created by an administrator of the Google Workspace.
Please make sure to grant the following OAuth scopes to the service user:
Follow the Google Documentation for performing [Delegating domain-wide authority](https://developers.google.com/identity/protocols/oauth2/service-account#delegatingauthority) to create a Service account with delegated domain wide authority. This account must be created by an administrator of the Google Workspace. Please make sure to grant the following OAuth scopes to the service user:
1. `https://www.googleapis.com/auth/webmasters.readonly`
At the end of this process, you should have JSON credentials to this Google Service Account.
At the end of this process, you should have JSON credentials to this Google Service Account.
You should now be ready to use the Google Workspace Admin Reports API connector in Airbyte.
You should now be ready to use the Google Workspace Admin Reports API connector in Airbyte.

9
docs/integrations/sources/pokeapi.md
View File

@@ -2,12 +2,14 @@
## Overview
The PokéAPI source currently supports only Full Refresh syncs.
The PokéAPI source currently supports only Full Refresh syncs.
This source uses the fully open [PokéAPI](https://pokeapi.co/docs/v2#info) to serve and retrieve information about Pokémon. This connector should be primarily used for educational purposes or for getting a trial source up and running without needing any dependencies.
### Output schema
Currently, only one output stream is available from this source, which is the Pokémon output stream. This schema is defined [here](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-pokeapi/schemas/pokemon.json).
### Data type mapping
The PokéAPI uses the same [JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions happen as part of this source.
@@ -24,8 +26,9 @@ The PokéAPI uses the same [JSONSchema](https://json-schema.org/understanding-js
### Performance considerations
According to the API's [fair use policy](https://pokeapi.co/docs/v2#fairuse), please make sure to cache resources retrieved from the PokéAPI wherever possible.
According to the API's [fair use policy](https://pokeapi.co/docs/v2#fairuse), please make sure to cache resources retrieved from the PokéAPI wherever possible.
## Dependencies/Requirements
- As this API is fully open and is not rate-limited, no authentication or rate-limiting is performed, so you can use this connector right out of the box without any further configuration.
* As this API is fully open and is not rate-limited, no authentication or rate-limiting is performed, so you can use this connector right out of the box without any further configuration.

2
docs/integrations/sources/postgres.md
View File

@@ -158,7 +158,7 @@ This slot **must** use `pgoutput`.
#### Create publications and replication identities for tables
For each table you want to replicate with CDC, you should add the replication identity (the method of distinguishing between rows) first. We recommend using `ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;` to use primary keys to distinguish between rows. After setting the replication identity, you will need to run `CREATE PUBLICATION airbyte_publication FOR TABLES <tbl1, tbl2, tbl3>;`. This publication name is customizable. **You must add the replication identity before creating the publication. Otherwise, `ALTER`/`UPDATE`/`DELETE` statements may fail if Postgres cannot determine how to uniquely identify rows.** Please refer to the [Postgres docs](https://www.postgresql.org/docs/10/sql-alterpublication.html) if you need to add or remove tables from your publication in the future.
For each table you want to replicate with CDC, you should add the replication identity \(the method of distinguishing between rows\) first. We recommend using `ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;` to use primary keys to distinguish between rows. After setting the replication identity, you will need to run `CREATE PUBLICATION airbyte_publication FOR TABLES <tbl1, tbl2, tbl3>;`. This publication name is customizable. **You must add the replication identity before creating the publication. Otherwise, `ALTER`/`UPDATE`/`DELETE` statements may fail if Postgres cannot determine how to uniquely identify rows.** Please refer to the [Postgres docs](https://www.postgresql.org/docs/10/sql-alterpublication.html) if you need to add or remove tables from your publication in the future.
The UI currently allows selecting any tables for CDC. If a table is selected that is not part of the publication, it will not replicate even though it is selected. If a table is part of the publication but does not have a replication identity, that replication identity will be created automatically on the first run if the Airbyte user has the necessary permissions.

30
docs/integrations/sources/stripe.md
View File

@@ -8,24 +8,24 @@ The Stripe source supports both Full Refresh and Incremental syncs. You can choo
This Source is capable of syncing the following core Streams:
* [Balance Transactions](https://stripe.com/docs/api/balance_transactions/list) (Incremental)
* [Balance Transactions](https://stripe.com/docs/api/balance_transactions/list) \(Incremental\)
* [Bank accounts](https://stripe.com/docs/api/customer_bank_accounts/list)
* [Charges](https://stripe.com/docs/api/charges/list) (Incremental)
* [Coupons](https://stripe.com/docs/api/coupons/list) (Incremental)
* [Customers](https://stripe.com/docs/api/customers/list) (Incremental)
* [Customer Balance Transactions](https://stripe.com/docs/api/customer_balance_transactions/list) (Incremental)
* [Disputes](https://stripe.com/docs/api/disputes/list) (Incremental)
* [Events](https://stripe.com/docs/api/events/list) (Incremental)
* [Invoices](https://stripe.com/docs/api/invoices/list) (Incremental)
* [Invoice Items](https://stripe.com/docs/api/invoiceitems/list) (Incremental)
* [Charges](https://stripe.com/docs/api/charges/list) \(Incremental\)
* [Coupons](https://stripe.com/docs/api/coupons/list) \(Incremental\)
* [Customers](https://stripe.com/docs/api/customers/list) \(Incremental\)
* [Customer Balance Transactions](https://stripe.com/docs/api/customer_balance_transactions/list) \(Incremental\)
* [Disputes](https://stripe.com/docs/api/disputes/list) \(Incremental\)
* [Events](https://stripe.com/docs/api/events/list) \(Incremental\)
* [Invoices](https://stripe.com/docs/api/invoices/list) \(Incremental\)
* [Invoice Items](https://stripe.com/docs/api/invoiceitems/list) \(Incremental\)
* [Invoice Line Items](https://stripe.com/docs/api/invoices/invoice_lines)
* [Payouts](https://stripe.com/docs/api/payouts/list) (Incremental)
* [Plans](https://stripe.com/docs/api/plans/list) (Incremental)
* [Products](https://stripe.com/docs/api/products/list) (Incremental)
* [Refunds](https://stripe.com/docs/api/refunds/list) (Incremental)
* [Subscriptions](https://stripe.com/docs/api/subscriptions/list) (Incremental)
* [Payouts](https://stripe.com/docs/api/payouts/list) \(Incremental\)
* [Plans](https://stripe.com/docs/api/plans/list) \(Incremental\)
* [Products](https://stripe.com/docs/api/products/list) \(Incremental\)
* [Refunds](https://stripe.com/docs/api/refunds/list) \(Incremental\)
* [Subscriptions](https://stripe.com/docs/api/subscriptions/list) \(Incremental\)
* [Subscription Items](https://stripe.com/docs/api/subscription_items/list)
* [Transfers](https://stripe.com/docs/api/transfers/list) (Incremental)
* [Transfers](https://stripe.com/docs/api/transfers/list) \(Incremental\)
### Data type mapping

9
docs/integrations/sources/zendesk-chat.md
View File

@@ -11,12 +11,12 @@ This source can sync data for the [Zendesk Chat API](https://developer.zendesk.c
This Source is capable of syncing the following core Streams:
* [Accounts](https://developer.zendesk.com/rest_api/docs/chat/accounts#show-account)
* [Agents](https://developer.zendesk.com/rest_api/docs/chat/agents#list-agents) (Incremental)
* [Agent Timelines](https://developer.zendesk.com/rest_api/docs/chat/incremental_export#incremental-agent-timeline-export) (Incremental)
* [Agents](https://developer.zendesk.com/rest_api/docs/chat/agents#list-agents) \(Incremental\)
* [Agent Timelines](https://developer.zendesk.com/rest_api/docs/chat/incremental_export#incremental-agent-timeline-export) \(Incremental\)
* [Chats](https://developer.zendesk.com/rest_api/docs/chat/chats#list-chats)
* [Shortcuts](https://developer.zendesk.com/rest_api/docs/chat/shortcuts#list-shortcuts)
* [Triggers](https://developer.zendesk.com/rest_api/docs/chat/triggers#list-triggers)
* [Bans](https://developer.zendesk.com/rest_api/docs/chat/bans#list-bans) (Incremental)
* [Bans](https://developer.zendesk.com/rest_api/docs/chat/bans#list-bans) \(Incremental\)
* [Departments](https://developer.zendesk.com/rest_api/docs/chat/departments#list-departments)
* [Goals](https://developer.zendesk.com/rest_api/docs/chat/goals#list-goals)
* [Skills](https://developer.zendesk.com/rest_api/docs/chat/skills#list-skills)
@@ -38,7 +38,7 @@ This Source is capable of syncing the following core Streams:
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental Sync | Yes | |
| SSL connection | Yes | |
| SSL connection | Yes | |
### Performance considerations
@@ -57,3 +57,4 @@ The Zendesk connector should not run into Zendesk API limitations under normal u
Generate a Access Token as described in [Zendesk Chat docs](https://developer.zendesk.com/rest_api/docs/chat/auth)
We recommend creating a restricted, read-only key specifically for Airbyte access. This will allow you to control which resources Airbyte should be able to access.