diff --git a/README.md b/README.md index 3e5d104a037..46c2befe69c 100644 --- a/README.md +++ b/README.md @@ -2,15 +2,15 @@ ![GitHub Workflow Status](https://img.shields.io/github/workflow/status/airbytehq/airbyte/Airbyte%20CI) ![License](https://img.shields.io/github/license/airbytehq/airbyte) -![](docs/.gitbook/assets/airbyte_horizontal_dark%20%281%29%20%281%29.svg) +![](docs/.gitbook/assets/airbyte_horizontal_dark%20%281%29%20%281%29%20%282%29.svg) **Data integration made simple, secure and extensible.** The new open-source standard to sync data from applications, APIs & databases to warehouses. -[![](docs/.gitbook/assets/deploy-locally%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29.svg)](docs/deploying-airbyte/on-your-workstation.md) [![](docs/.gitbook/assets/deploy-on-aws%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29.svg)](docs/deploying-airbyte/on-aws-ec2.md) [![](docs/.gitbook/assets/deploy-on-gcp%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29.svg)](docs/deploying-airbyte/on-gcp-compute-engine.md) +[![](docs/.gitbook/assets/deploy-locally%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%282%29.svg)](docs/deploying-airbyte/on-your-workstation.md) [![](docs/.gitbook/assets/deploy-on-aws%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%282%29.svg)](docs/deploying-airbyte/on-aws-ec2.md) [![](docs/.gitbook/assets/deploy-on-gcp%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%282%29.svg)](docs/deploying-airbyte/on-gcp-compute-engine.md) -![](docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines%20%281%29%20%281%29.png) +![](docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines%20%281%29%20%281%29%20%282%29.png) Airbyte is on a mission to make data integration pipelines a commodity. @@ -64,7 +64,7 @@ For general help using Airbyte, please refer to the official Airbyte documentati ## Roadmap -Check out our [roadmap](docs/roadmap.md) to get informed on what we are currently working on, and what we have in mind for the next weeks, months and years. +Check out our [roadmap](docs/roadmap.md) to get informed on what we are currently working on, and what we have in mind for the next weeks, months and years. ## License diff --git a/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (1).png b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (1).png new file mode 100644 index 00000000000..a1cd2265698 Binary files /dev/null and b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (2).png b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (2).png new file mode 100644 index 00000000000..a1cd2265698 Binary files /dev/null and b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (3).png b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (3).png new file mode 100644 index 00000000000..a1cd2265698 Binary files /dev/null and b/docs/.gitbook/assets/02_set-up-sources (1) (1) (1) (1) (3).png differ diff --git a/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (1).png b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (1).png new file mode 100644 index 00000000000..ed8b8e4ca77 Binary files /dev/null and b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (2).png b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (2).png new file mode 100644 index 00000000000..ed8b8e4ca77 Binary files /dev/null and b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (3).png b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (3).png new file mode 100644 index 00000000000..ed8b8e4ca77 Binary files /dev/null and b/docs/.gitbook/assets/03_set-up-connection (1) (1) (1) (1) (3).png differ diff --git a/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (1).png b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (1).png new file mode 100644 index 00000000000..f17502c13eb Binary files /dev/null and b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (2).png b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (2).png new file mode 100644 index 00000000000..f17502c13eb Binary files /dev/null and b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (3).png b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (3).png new file mode 100644 index 00000000000..f17502c13eb Binary files /dev/null and b/docs/.gitbook/assets/04_source-details (1) (1) (1) (1) (3).png differ diff --git a/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..b6f1f635e22 Binary files /dev/null and b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (2).png b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (2).png new file mode 100644 index 00000000000..b6f1f635e22 Binary files /dev/null and b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (3).png b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (3).png new file mode 100644 index 00000000000..b6f1f635e22 Binary files /dev/null and b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (3).png differ diff --git a/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (4).png b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (4).png new file mode 100644 index 00000000000..b6f1f635e22 Binary files /dev/null and b/docs/.gitbook/assets/10-000-feet-view (1) (1) (1) (2) (1) (1) (1) (4).png differ diff --git a/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (1).png b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (1).png new file mode 100644 index 00000000000..136322db68f Binary files /dev/null and b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (2).png b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (2).png new file mode 100644 index 00000000000..136322db68f Binary files /dev/null and b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (2).png b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (2).png new file mode 100644 index 00000000000..136322db68f Binary files /dev/null and b/docs/.gitbook/assets/airbyte-ui-for-your-integration-pipelines (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (1).png b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (1).png new file mode 100644 index 00000000000..2c57dc71a7a Binary files /dev/null and b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (2).png b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (2).png new file mode 100644 index 00000000000..2c57dc71a7a Binary files /dev/null and b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (2).png differ diff --git a/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (3).png b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (3).png new file mode 100644 index 00000000000..2c57dc71a7a Binary files /dev/null and b/docs/.gitbook/assets/airbyte_get-started (1) (1) (1) (3).png differ diff --git a/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (1).svg b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (1).svg new file mode 100644 index 00000000000..c1ecb9fbf10 --- /dev/null +++ b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (1).svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (2).svg b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (2).svg new file mode 100644 index 00000000000..c1ecb9fbf10 --- /dev/null +++ b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (1) (2).svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (2).svg b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (2).svg new file mode 100644 index 00000000000..c1ecb9fbf10 --- /dev/null +++ b/docs/.gitbook/assets/airbyte_horizontal_dark (1) (1) (2).svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/docs/.gitbook/assets/aws_ec2_ami (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_ami (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..6bc3f2260e8 Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_ami (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/aws_ec2_instance_type (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_instance_type (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..c719d46722e Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_instance_type (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/aws_ec2_instance_view (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_instance_view (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..371b0ef5a2d Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_instance_view (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/aws_ec2_launch (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_launch (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..259366b6202 Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_launch (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/aws_ec2_security_group (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_security_group (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..f584064dad2 Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_security_group (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/aws_ec2_ssh_key (1) (1) (1) (2) (1) (1) (1).png b/docs/.gitbook/assets/aws_ec2_ssh_key (1) (1) (1) (2) (1) (1) (1).png new file mode 100644 index 00000000000..1776c280db7 Binary files /dev/null and b/docs/.gitbook/assets/aws_ec2_ssh_key (1) (1) (1) (2) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg new file mode 100644 index 00000000000..9540e40bcf1 --- /dev/null +++ b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg new file mode 100644 index 00000000000..9540e40bcf1 --- /dev/null +++ b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg new file mode 100644 index 00000000000..9540e40bcf1 --- /dev/null +++ b/docs/.gitbook/assets/deploy-locally (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg new file mode 100644 index 00000000000..1b90bdf3a1f --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg new file mode 100644 index 00000000000..1b90bdf3a1f --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg new file mode 100644 index 00000000000..1b90bdf3a1f --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-aws (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg new file mode 100644 index 00000000000..620a22bb70a --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (1).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg new file mode 100644 index 00000000000..620a22bb70a --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg new file mode 100644 index 00000000000..620a22bb70a --- /dev/null +++ b/docs/.gitbook/assets/deploy-on-gcp (1) (1) (1) (1) (2) (1) (1) (1) (2) (1) (2).svg @@ -0,0 +1,4 @@ + + + + diff --git a/docs/.gitbook/assets/gcp_ce_configure (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/gcp_ce_configure (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..5850c00302c Binary files /dev/null and b/docs/.gitbook/assets/gcp_ce_configure (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/gcp_ce_launch (1) (1) (1) (2) (1) (1) (1) (1).png b/docs/.gitbook/assets/gcp_ce_launch (1) (1) (1) (2) (1) (1) (1) (1).png new file mode 100644 index 00000000000..4d6ec00e7f3 Binary files /dev/null and b/docs/.gitbook/assets/gcp_ce_launch (1) (1) (1) (2) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/screen-shot-2020-10-30-at-2.44.55-pm (1) (1) (1) (1) (1) (1).png b/docs/.gitbook/assets/screen-shot-2020-10-30-at-2.44.55-pm (1) (1) (1) (1) (1) (1).png new file mode 100644 index 00000000000..747fcfaebac Binary files /dev/null and b/docs/.gitbook/assets/screen-shot-2020-10-30-at-2.44.55-pm (1) (1) (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.25.21-pm (1) (1) (1) (1).png b/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.25.21-pm (1) (1) (1) (1).png new file mode 100644 index 00000000000..54f0734338a Binary files /dev/null and b/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.25.21-pm (1) (1) (1) (1).png differ diff --git a/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.35.40-pm (1) (1) (1) (1).png b/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.35.40-pm (1) (1) (1) (1).png new file mode 100644 index 00000000000..996f72267ba Binary files /dev/null and b/docs/.gitbook/assets/screen-shot-2020-11-03-at-9.35.40-pm (1) (1) (1) (1).png differ diff --git a/docs/architecture/high-level-view.md b/docs/architecture/high-level-view.md index 2680c282186..a8851d5f2be 100644 --- a/docs/architecture/high-level-view.md +++ b/docs/architecture/high-level-view.md @@ -4,7 +4,7 @@ description: A high level view of Airbyte's components. # High-level View -![3.048-Kilometer view](../.gitbook/assets/10-000-feet-view%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29.png) +![3.048-Kilometer view](../.gitbook/assets/10-000-feet-view%20%281%29%20%281%29%20%281%29%20%282%29%20%281%29%20%281%29%20%281%29%20%283%29.png) * `UI`: Acts as the control center for Airbyte. From the UI, you can configure new integration connections. You can also track the different syncing jobs and view logs. * `Config Store`: Stores all the connections information \(credentials, frequency...\). diff --git a/docs/contributing-to-airbyte/building-new-connector/README.md b/docs/contributing-to-airbyte/building-new-connector/README.md index 9152b5228cd..25760a35f55 100644 --- a/docs/contributing-to-airbyte/building-new-connector/README.md +++ b/docs/contributing-to-airbyte/building-new-connector/README.md @@ -2,7 +2,7 @@ A connector takes the form of a Docker image which follows the [Airbyte specification](../../architecture/airbyte-specification.md). -We support 2 types of connectors: Sources and Destinations. +We support 2 types of connectors: Sources and Destinations. To build a new connector, we provide templates so you don't need to start everything from scratch. ## The Airbyte specification @@ -55,3 +55,4 @@ Once you've finished iterating on the changes to a connector as specified in its ``` 6. The new version of the connector is now available for everyone who uses it. Thank you! + diff --git a/docs/getting-started-tutorial.md b/docs/getting-started-tutorial.md index e5940cbd93f..ae19bd26e1c 100644 --- a/docs/getting-started-tutorial.md +++ b/docs/getting-started-tutorial.md @@ -24,13 +24,13 @@ Once you see an Airbyte banner, the UI is ready to go at [http://localhost:8000/ You should see an onboarding page. Enter your email if you want updates about Airbyte and continue. -![](.gitbook/assets/airbyte_get-started%20%281%29%20%281%29%20%281%29.png) +![](.gitbook/assets/airbyte_get-started%20%281%29%20%281%29%20%281%29%20%282%29.png) ## 2. Set up your first connection Now you will see a wizard that allows you choose the data you want to send through Airbyte. -![](.gitbook/assets/02_set-up-sources%20%281%29%20%281%29%20%281%29%20%281%29.png) +![](.gitbook/assets/02_set-up-sources%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29.png) As of our alpha launch, we have one database source \(Postgres\) and two API sources \(an exchange rate API and the Stripe API\). We're currently building an integration framework that makes it easy to create sources and destinations, so you should expect many more soon. Please reach out to us if you need a specific connector or would like to help build one. @@ -76,7 +76,7 @@ DB Name: postgres After adding the destination, you can choose what tables and columns you want to sync. -![](.gitbook/assets/03_set-up-connection%20%281%29%20%281%29%20%281%29%20%281%29.png) +![](.gitbook/assets/03_set-up-connection%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29.png) For this demo, we recommend leaving the defaults and selecting "Every 5 Minutes" as the frequency. Click `Set Up Connection` to finish setting up the sync. @@ -84,7 +84,7 @@ For this demo, we recommend leaving the defaults and selecting "Every 5 Minutes" You should now see a list of sources with the source you just added. Click on it to find more information about your connection. This is the page where you can update any settings about this source and how it syncs. There should be a `Completed` job under the history section. If you click on that run, it will show logs from that run. -![](.gitbook/assets/04_source-details%20%281%29%20%281%29%20%281%29%20%281%29.png) +![](.gitbook/assets/04_source-details%20%281%29%20%281%29%20%281%29%20%281%29%20%282%29.png) One of biggest problems we've seen in tools like Fivetran is the lack of visibility when debugging. In Airbyte, allowing full log access and the ability to debug and fix connector problems is one of our highest priorities. We'll be working hard to make these logs accessible and understandable. diff --git a/docs/integrations/destinations/README.md b/docs/integrations/destinations/README.md index 9c9c9e2a9e6..fcb4e36122e 100644 --- a/docs/integrations/destinations/README.md +++ b/docs/integrations/destinations/README.md @@ -3,3 +3,4 @@ description: 'Data warehouses, data lakes, databases...' --- # Destinations + diff --git a/docs/integrations/sources/file.md b/docs/integrations/sources/file.md index 42c81896d3f..ee12bf0bacf 100644 --- a/docs/integrations/sources/file.md +++ b/docs/integrations/sources/file.md @@ -1,14 +1,12 @@ -# File +# Files ## Overview -File are often exchanged or published in various remote locations. This source aims to support an expanding range of file formats and storage providers. -The File source supports Full Refresh syncs. That is, every time a sync is run, Airbyte will copy all rows in the file and columns you set up for replication into the destination in a new table. +File are often exchanged or published in various remote locations. This source aims to support an expanding range of file formats and storage providers. The File source supports Full Refresh syncs. That is, every time a sync is run, Airbyte will copy all rows in the file and columns you set up for replication into the destination in a new table. ### Output schema -At this time, this source produces only a single stream for the target file as it replicates only one file at a time for the moment. -We'll be considering to improve this behavior by globing folders or use patterns to capture more files in the next iterations as well as more file formats and storage providers. +At this time, this source produces only a single stream for the target file as it replicates only one file at a time for the moment. We'll be considering to improve this behavior by globing folders or use patterns to capture more files in the next iterations as well as more file formats and storage providers. ### Features @@ -17,8 +15,8 @@ We'll be considering to improve this behavior by globing folders or use patterns | Full Refresh Sync | Yes | | Incremental Sync | No | | Replicate Incremental Deletes | No | -| Replicate Folders (multiple Files) | No | -| Replicate Glob Patterns (multiple Files) | No | +| Replicate Folders \(multiple Files\) | No | +| Replicate Glob Patterns \(multiple Files\) | No | ### Storage Providers @@ -30,8 +28,8 @@ We'll be considering to improve this behavior by globing folders or use patterns | SFTP / SSH / SCP | Yes | | WebHDFS | Untested | | local filesystem | Untested | -| Azure Blob Storage | Untested (hidden) | -| HDFS | Untested (hidden) | +| Azure Blob Storage | Untested \(hidden\) | +| HDFS | Untested \(hidden\) | ### File Formats @@ -48,40 +46,37 @@ We'll be considering to improve this behavior by globing folders or use patterns ### Performance considerations -In order to read large files from a remote location, we are leveraging the capabilities of [smart_open](https://pypi.org/project/smart-open/). -However, for the `CSV` file format, it is possible to switch to either -[GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) -implementations as it is natively supported by the `pandas` library. -This choice is made possible through the optional `reader_impl` parameter. +In order to read large files from a remote location, we are leveraging the capabilities of [smart\_open](https://pypi.org/project/smart-open/). However, for the `CSV` file format, it is possible to switch to either [GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) implementations as it is natively supported by the `pandas` library. This choice is made possible through the optional `reader_impl` parameter. ### Limitations -- Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md). This may not be ideal as a Source and not well tested yet. -- The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting. +* Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md). This may not be ideal as a Source and not well tested yet. +* The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting. ## Getting started -- Once the File Source is selected, you should define both the storage provider along its URL and format of the file. -- Depending on the choice made previously, more options may be necessary, especially when accessing private data. -- In case of GCS, it is necessary to provide the content of the service account keyfile to access private buckets. See settings of [BigQuery Destination](../destinations/bigquery.md) -- In case of AWS S3, the pair of `aws_access_key_id` and `aws_secret_access_key` is necessary to access private S3 buckets. +* Once the File Source is selected, you should define both the storage provider along its URL and format of the file. +* Depending on the choice made previously, more options may be necessary, especially when accessing private data. +* In case of GCS, it is necessary to provide the content of the service account keyfile to access private buckets. See settings of [BigQuery Destination](../destinations/bigquery.md) +* In case of AWS S3, the pair of `aws_access_key_id` and `aws_secret_access_key` is necessary to access private S3 buckets. ### Reader Options -The Reader in charge of loading the file format is currently based on [Pandas IO Tools](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html). -It is possible to customize how to load the file into a Pandas DataFrame as part of this Source Connector. -This is doable in the `reader_options` that should be in JSON format and depends on the chosen file format (see panda's documentation, depending on the format). +The Reader in charge of loading the file format is currently based on [Pandas IO Tools](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html). It is possible to customize how to load the file into a Pandas DataFrame as part of this Source Connector. This is doable in the `reader_options` that should be in JSON format and depends on the chosen file format \(see panda's documentation, depending on the format\). -For example, if the format `CSV` is selected, then options from the [read_csv](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-read-csv-table) functions are available. -- It is therefore possible to customize the `delimiter` (or `sep`) to `\t` in case of tab separated files. -- Header line can be ignored with `header=0` and customized with `names` -- etc +For example, if the format `CSV` is selected, then options from the [read\_csv](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-read-csv-table) functions are available. -We would therefore provide in the `reader_options` the following json: -``` +* It is therefore possible to customize the `delimiter` \(or `sep`\) to `\t` in case of tab separated files. +* Header line can be ignored with `header=0` and customized with `names` +* etc + +We would therefore provide in the `reader_options` the following json: + +```text { "sep" : "\t", "header" : 0, "names": "column1, column2"} ``` -In case you select `JSON` format, then options from the [read_json](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-json-reader) reader are available. +In case you select `JSON` format, then options from the [read\_json](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-json-reader) reader are available. + +For example, you can use the `{"orient" : "records"}` to change how orientation of data is loaded \(if data is `[{column -> value}, … , {column -> value}]`\) -For example, you can use the `{"orient" : "records"}` to change how orientation of data is loaded (if data is `[{column -> value}, … , {column -> value}]`) \ No newline at end of file diff --git a/docs/integrations/sources/salesforce.md b/docs/integrations/sources/salesforce.md index 0056a618f54..5932a44e76a 100644 --- a/docs/integrations/sources/salesforce.md +++ b/docs/integrations/sources/salesforce.md @@ -34,9 +34,7 @@ The connector is restricted by normal Salesforce rate limiting. For large transf We recommend the following [walkthrough](https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b) **while keeping in mind the edits we suggest below** for setting up a Salesforce app that can pull data from Salesforce and locating the credentials you need to provide to Airbyte. -Suggested edits: -1. If your salesforce URL does not take the form `X.salesforce.com`, use your actual Salesforce domain name. For example, if your Salesforce URL is `awesomecompany.force.com` then use that instead of `awesomecompany.salesforce.com`. -1. When running a `curl` command, always run it with the `-L` option to follow any redirects. +Suggested edits: 1. If your salesforce URL does not take the form `X.salesforce.com`, use your actual Salesforce domain name. For example, if your Salesforce URL is `awesomecompany.force.com` then use that instead of `awesomecompany.salesforce.com`. 1. When running a `curl` command, always run it with the `-L` option to follow any redirects. #### is\_sandbox diff --git a/docs/roadmap.md b/docs/roadmap.md index 677adebd193..c776d6a52a8 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -6,7 +6,7 @@ description: 'Here''s what''s coming in the next few days, weeks, months, and ye ## Coming within a few days -Check out our [GitHub project](https://github.com/airbytehq/airbyte/projects/1) to see the features we're currently working on or about to. You may also give us insights, by adding your own issues and voting for specific features / integrations. +Check out our [GitHub project](https://github.com/airbytehq/airbyte/projects/1) to see the features we're currently working on or about to. You may also give us insights, by adding your own issues and voting for specific features / integrations. ## Coming within a few weeks