From 3765bcf3b85d11bcdbc27e6c877f00632c2ea9b3 Mon Sep 17 00:00:00 2001 From: Abhi Vaidyanatha Date: Wed, 28 Jul 2021 14:00:08 -0700 Subject: [PATCH] Refactor docs to improve clarity around connector development. (#5037) * Refactor docs to improve clarity around connector development. * Merge architecture and setup of source acceptance tests to reference * Address reviewsa * Address review comments Co-authored-by: Abhi Vaidyanatha --- .gitbook.yaml | 21 ++++++ docs/SUMMARY.md | 66 ++++++++--------- .../README.md | 21 +++--- docs/connector-development/airbyte101.md | 6 ++ .../best-practices.md | 2 +- .../cdk-python}/README.md | 20 +++++- .../cdk-python}/basic-concepts.md | 0 .../cdk-python}/full-refresh-stream.md | 0 .../cdk-python}/http-streams.md | 0 .../cdk-python}/incremental-stream.md | 0 .../cdk-python}/python-concepts.md | 0 .../cdk-python}/schemas.md | 0 .../cdk-python/stream-slices.md} | 0 .../testing-connectors/README.md} | 67 ----------------- .../legacy-standard-source-tests.md | 2 +- .../source-acceptance-tests-reference.md} | 72 +++++++++++++++++-- .../standard-source-tests.md | 0 .../tutorials/README.md | 0 .../tutorials/adding-incremental-sync.md | 2 +- .../build-a-connector-the-hard-way.md | 12 ++-- .../tutorials/building-a-java-destination.md | 2 +- .../building-a-python-destination.md | 0 .../tutorials/building-a-python-source.md | 6 +- .../tutorials/cdk-speedrun.md | 4 +- .../0-getting-started.md | 0 .../1-creating-the-source.md | 0 .../2-install-dependencies.md | 0 .../3-define-inputs.md | 0 .../4-connection-checking.md | 0 .../5-declare-schema.md | 2 +- .../cdk-tutorial-python-http/6-read-data.md | 4 +- .../7-use-connector-in-airbyte.md | 0 .../8-test-your-connector.md | 0 .../cdk-tutorial-python-http/README.md | 2 + docs/contributing-to-airbyte/README.md | 6 +- .../building-new-connector/java-connectors.md | 16 ----- .../monorepo-python-development.md | 0 .../python/concepts/README.md | 32 --------- .../cdk-tutorial-python-http/README.md | 2 - .../tutorials/README.md | 2 - docs/operator-guides/browsing-output-logs.md | 2 +- .../beginners-guide-to-catalog.md | 8 +-- docs/understanding-airbyte/catalog.md | 2 +- 43 files changed, 185 insertions(+), 196 deletions(-) rename docs/{contributing-to-airbyte/building-new-connector => connector-development}/README.md (90%) create mode 100644 docs/connector-development/airbyte101.md rename docs/{contributing-to-airbyte/building-new-connector => connector-development}/best-practices.md (93%) rename docs/{contributing-to-airbyte/python => connector-development/cdk-python}/README.md (78%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/basic-concepts.md (100%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/full-refresh-stream.md (100%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/http-streams.md (100%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/incremental-stream.md (100%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/python-concepts.md (100%) rename docs/{contributing-to-airbyte/python/concepts => connector-development/cdk-python}/schemas.md (100%) rename docs/{contributing-to-airbyte/python/concepts/stream_slices.md => connector-development/cdk-python/stream-slices.md} (100%) rename docs/{contributing-to-airbyte/building-new-connector/testing-connectors.md => connector-development/testing-connectors/README.md} (52%) rename docs/{contributing-to-airbyte/building-new-connector => connector-development/testing-connectors}/legacy-standard-source-tests.md (98%) rename docs/{contributing-to-airbyte/building-new-connector/source-acceptance-tests.md => connector-development/testing-connectors/source-acceptance-tests-reference.md} (64%) rename docs/{contributing-to-airbyte/building-new-connector => connector-development/testing-connectors}/standard-source-tests.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/README.md (100%) rename docs/{contributing-to-airbyte => connector-development}/tutorials/adding-incremental-sync.md (99%) rename docs/{contributing-to-airbyte => connector-development}/tutorials/build-a-connector-the-hard-way.md (98%) rename docs/{contributing-to-airbyte => connector-development}/tutorials/building-a-java-destination.md (99%) rename docs/{contributing-to-airbyte => connector-development}/tutorials/building-a-python-destination.md (100%) rename docs/{contributing-to-airbyte => connector-development}/tutorials/building-a-python-source.md (97%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-speedrun.md (98%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/0-getting-started.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/1-creating-the-source.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/2-install-dependencies.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/3-define-inputs.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/4-connection-checking.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/5-declare-schema.md (97%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/6-read-data.md (98%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md (100%) rename docs/{contributing-to-airbyte/python => connector-development}/tutorials/cdk-tutorial-python-http/8-test-your-connector.md (100%) create mode 100644 docs/connector-development/tutorials/cdk-tutorial-python-http/README.md delete mode 100644 docs/contributing-to-airbyte/building-new-connector/java-connectors.md rename docs/contributing-to-airbyte/{building-new-connector => }/monorepo-python-development.md (100%) delete mode 100644 docs/contributing-to-airbyte/python/concepts/README.md delete mode 100644 docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/README.md delete mode 100644 docs/contributing-to-airbyte/tutorials/README.md rename docs/{contributing-to-airbyte/tutorials => understanding-airbyte}/beginners-guide-to-catalog.md (95%) diff --git a/.gitbook.yaml b/.gitbook.yaml index 676ec9c8893..1174a110841 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -51,3 +51,24 @@ redirects: tutorials/tutorials/building-a-python-source: ./contributing-to-airbyte/building-new-connector/tutorials/building-a-python-source.md upgrading-airbyte: ./operator-guides/upgrading-airbyte.md tutorials/upgrading-airbyte: ./operator-guides/upgrading-airbyte.md + contributing-to-airbyte/python: ./connector-development/cdk-python.md + contributing-to-airbyte/python/concepts/basic-concepts: ./connector-development/cdk-python/basic-concepts.md + contributing-to-airbyte/python/concepts/schemas: ./connector-development/cdk-python/schemas.md + contributing-to-airbyte/python/concepts/full-refresh-stream: ./connector-development/cdk-python/full-refresh-stream.md + contributing-to-airbyte/python/concepts/incremental-stream: ./connector-development/cdk-python/incremental-stream.md + contributing-to-airbyte/python/concepts/http-streams: ./connector-development/cdk-python/http-streams.md + contributing-to-airbyte/python/concepts/python-concepts: ./connector-development/cdk-python/python-concepts.md + contributing-to-airbyte/python/concepts/stream_slices: ./connector-development/cdk-python/stream-slices.md + contributing-to-airbyte/python/tutorials: ./connector-development/tutorials.md + contributing-to-airbyte/python/tutorials/cdk-speedrun: ./connector-development/tutorials/cdk-speedrun.md + contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http: ./connector-development/tutorials/cdk-tutorial-python-http.md + contributing-to-airbyte/building-new-connector: ./connector-development.md + contributing-to-airbyte/building-new-connector/best-practices: ./connector-development.md/best-practices.md + contributing-to-airbyte/building-new-connector/monorepo-python-development: ./contributing-to-airbyte/monorepo-python-development.md + contributing-to-airbyte/building-new-connector/testing-connectors: ./connector-development/testing-connectors.md + contributing-to-airbyte/building-new-connector/source-acceptance-tests: ./connector-development/testing-connectors/source-acceptance-tests-reference.md + contributing-to-airbyte/building-new-connector/tutorials: ./connector-development/tutorials.md + contributing-to-airbyte/building-new-connector/tutorials/beginners-guide-to-catalog: ./understanding-airbyte/beginners-guide-to-catalog.md + contributing-to-airbyte/building-new-connector/tutorials/building-a-python-source: ./connector-development/tutorials/building-a-python-source.md + contributing-to-airbyte/building-new-connector/tutorials/building-a-python-destination: ./connector-development/tutorials/building-a-python-destination.md + contributing-to-airbyte/building-new-connector/tutorials/building-a-java-destination: ./connector-development/tutorials/building-a-java-destination.md \ No newline at end of file diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 639b2b61bf1..083b54386ed 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -132,51 +132,47 @@ * [S3](integrations/destinations/s3.md) * [Snowflake](integrations/destinations/snowflake.md) * [Custom or New Connector](integrations/custom-connectors.md) +* [Connector Development](connector-development/README.md) + * [Connector Development Kit \(Python)\)](connector-development/cdk-python/README.md) + * [Basic Concepts](connector-development/cdk-python/basic-concepts.md) + * [Defining Stream Schemas](connector-development/cdk-python/schemas.md) + * [Full Refresh Streams](connector-development/cdk-python/full-refresh-stream.md) + * [Incremental Streams](connector-development/cdk-python/incremental-stream.md) + * [HTTP-API-based Connectors](connector-development/cdk-python/http-streams.md) + * [Python Concepts](connector-development/cdk-python/python-concepts.md) + * [Stream Slices](connector-development/cdk-python/stream_slices.md) + * [Airbyte 101 for Connector Development](connector-development/airbyte101.md) + * [Testing Connectors](connector-development/testing-connectors/README.md) + * [Source Acceptance Tests Reference](connector-development/testing-connectors/source-acceptance-tests-reference.md) + * [Tutorials](connector-development/tutorials/README.md) + * [Python CDK Speedrun: Creating a Source](connector-development/tutorials/cdk-speedrun.md) + * [Python CDK: Creating a HTTP API Source](connector-development/tutorials/cdk-tutorial-python-http/README.md) + * [Getting Started](connector-development/tutorials/cdk-tutorial-python-http/0-getting-started.md) + * [Step 1: Creating the Source](connector-development/tutorials/cdk-tutorial-python-http/1-creating-the-source.md) + * [Step 2: Install Dependencies](connector-development/tutorials/cdk-tutorial-python-http/2-install-dependencies.md) + * [Step 3: Define Inputs](connector-development/tutorials/cdk-tutorial-python-http/3-define-inputs.md) + * [Step 4: Connection Checking](connector-development/tutorials/cdk-tutorial-python-http/4-connection-checking.md) + * [Step 5: Declare the Schema](connector-development/tutorials/cdk-tutorial-python-http/5-declare-schema.md) + * [Step 6: Read Data](connector-development/tutorials/cdk-tutorial-python-http/6-read-data.md) + * [Step 7: Use the Connector in Airbyte](connector-development/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md) + * [Step 8: Test Connector](connector-development/tutorials/cdk-tutorial-python-http/8-test-your-connector.md) + * [Building a Python Source](connector-development/tutorials/building-a-python-source.md) + * [Building a Python Destination](connector-development/tutorials/building-a-python-destination.md) + * [Building a Java Destination](connector-development/tutorials/building-a-java-destination.md) + * [Best Practices](connector-development/best-practices.md) * [Contributing to Airbyte](contributing-to-airbyte/README.md) * [Code of Conduct](contributing-to-airbyte/code-of-conduct.md) * [Developing Locally](contributing-to-airbyte/developing-locally.md) * [Developing on Kubernetes](contributing-to-airbyte/developing-on-kubernetes.md) - * [Connector Development Kit \(Python\)](contributing-to-airbyte/python/README.md) - * [Concepts](contributing-to-airbyte/python/concepts/README.md) - * [Basic Concepts](contributing-to-airbyte/python/concepts/basic-concepts.md) - * [Defining Stream Schemas](contributing-to-airbyte/python/concepts/schemas.md) - * [Full Refresh Streams](contributing-to-airbyte/python/concepts/full-refresh-stream.md) - * [Incremental Streams](contributing-to-airbyte/python/concepts/incremental-stream.md) - * [HTTP-API-based Connectors](contributing-to-airbyte/python/concepts/http-streams.md) - * [Python Concepts](contributing-to-airbyte/python/concepts/python-concepts.md) - * [Stream Slices](contributing-to-airbyte/python/concepts/stream_slices.md) - * [Tutorials](contributing-to-airbyte/python/tutorials/README.md) - * [Speedrun: Creating a Source with the CDK](contributing-to-airbyte/python/tutorials/cdk-speedrun.md) - * [Creating an HTTP API Source](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/README.md) - * [Getting Started](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/0-getting-started.md) - * [Step 1: Creating the Source](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/1-creating-the-source.md) - * [Step 2: Install Dependencies](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/2-install-dependencies.md) - * [Step 3: Define Inputs](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/3-define-inputs.md) - * [Step 4: Connection Checking](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/4-connection-checking.md) - * [Step 5: Declare the Schema](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/5-declare-schema.md) - * [Step 6: Read Data](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/6-read-data.md) - * [Step 7: Use the Connector in Airbyte](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md) - * [Step 8: Test Connector](contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/8-test-your-connector.md) - * [Developing Connectors](contributing-to-airbyte/building-new-connector/README.md) - * [Best Practices](contributing-to-airbyte/building-new-connector/best-practices.md) - * [Java Connectors](contributing-to-airbyte/building-new-connector/java-connectors.md) - * [Monorepo Python Development](contributing-to-airbyte/building-new-connector/monorepo-python-development.md) - * [Testing Connectors](contributing-to-airbyte/building-new-connector/testing-connectors.md) - * [Source Connector Acceptance Tests](contributing-to-airbyte/building-new-connector/source-acceptance-tests.md) - * [Tutorials](contributing-to-airbyte/tutorials/README.md) - * [A Beginner's Guide to the AirbyteCatalog](contributing-to-airbyte/tutorials/beginners-guide-to-catalog.md) - * [Building a Connector: The Hard Way](contributing-to-airbyte/tutorials/build-a-connector-the-hard-way.md) - * [Adding Incremental to a Source](contributing-to-airbyte/tutorials/adding-incremental-sync.md) - * [Building a Python Source](contributing-to-airbyte/tutorials/building-a-python-source.md) - * [Building a Python Destination](contributing-to-airbyte/tutorials/building-a-python-destination.md) - * [Building a Java Destination](contributing-to-airbyte/tutorials/building-a-java-destination.md) + * [Monorepo Python Development](contributing-to-airbyte/monorepo-python-development.md) * [Code Style](contributing-to-airbyte/code-style.md) * [Gradle Cheatsheet](contributing-to-airbyte/gradle-cheatsheet.md) * [Updating Documentation](contributing-to-airbyte/updating-documentation.md) * [Templates](contributing-to-airbyte/templates/README.md) * [Connector Doc Template](contributing-to-airbyte/templates/integration-documentation-template.md) * [Understanding Airbyte](understanding-airbyte/README.md) - * [AirbyteCatalog & ConfiguredAirbyteCatalog](understanding-airbyte/catalog.md) + * [A Beginner's Guide to the AirbyteCatalog](understanding-airbyte/beginners-guide-to-catalog.md) + * [AirbyteCatalog Reference](understanding-airbyte/catalog.md) * [Airbyte Specification](understanding-airbyte/airbyte-specification.md) * [Basic Normalization](understanding-airbyte/basic-normalization.md) * [Connections](understanding-airbyte/connections/README.md) diff --git a/docs/contributing-to-airbyte/building-new-connector/README.md b/docs/connector-development/README.md similarity index 90% rename from docs/contributing-to-airbyte/building-new-connector/README.md rename to docs/connector-development/README.md index 250d76e2b84..df35aed1a45 100644 --- a/docs/contributing-to-airbyte/building-new-connector/README.md +++ b/docs/connector-development/README.md @@ -1,6 +1,6 @@ -# Developing Connectors +# Connector Development -Airbyte supports two types of connectors: Sources and Destinations. A connector takes the form of a Docker image which follows the [Airbyte specification](../../understanding-airbyte/airbyte-specification.md). +Airbyte supports two types of connectors: Sources and Destinations. A connector takes the form of a Docker image which follows the [Airbyte specification](../understanding-airbyte/airbyte-specification.md). To build a new connector in Java or Python, we provide templates so you don't need to start everything from scratch. @@ -8,12 +8,11 @@ To build a new connector in Java or Python, we provide templates so you don't ne ## Connector-Development Kit (CDK) -You can build a connector very quickly with the [Airbyte CDK](../python/README.md), which generates 75% of the code required for you. - +You can build a connector very quickly with the [Airbyte CDK](cdk-python/README.md), which generates 75% of the code required for you. ## The Airbyte specification -Before building a new connector, review [Airbyte's data protocol specification](../../understanding-airbyte/airbyte-specification.md). +Before building a new connector, review [Airbyte's data protocol specification](../understanding-airbyte/airbyte-specification.md). ## Adding a new connector @@ -22,7 +21,7 @@ Before building a new connector, review [Airbyte's data protocol specification]( To add a new connector you need to: 1. Implement & Package your connector in an Airbyte Protocol compliant Docker image -2. Add integration tests for your connector. At a minimum, all connectors must pass [Airbyte's standard test suite](testing-connectors.md), but you can also add your own tests. +2. Add integration tests for your connector. At a minimum, all connectors must pass [Airbyte's standard test suite](testing-connectors/testing-connectors.md), but you can also add your own tests. 3. Document how to build & test your connector 4. Publish the Docker image containing the connector @@ -54,15 +53,17 @@ and choose the relevant template. This will generate a new connector in the `air Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial: -* [Building a Python source ](../tutorials/building-a-python-source.md) -* [Building a Python destination](../tutorials/building-a-python-destination.md) -* [Building a Java destination ](../tutorials/building-a-java-destination.md) +* [Speedrun: Building a HTTP source with the CDK](tutorials/cdk-speedrun.md) +* [Building a HTTP source with the CDK](tutorials/cdk-tutorial-python-http) +* [Building a Python source](tutorials/building-a-python-source.md) +* [Building a Python destination](tutorials/building-a-python-destination.md) +* [Building a Java destination](tutorials/building-a-java-destination.md) As you implement your connector, make sure to review the [Best Practices for Connector Development](best-practices.md) guide. Following best practices is not a requirement for merging your contribution to Airbyte, but it certainly doesn't hurt ;\) ### 2. Integration tests -At a minimum, your connector must implement the acceptance tests described in [Testing Connectors](testing-connectors.md) +At a minimum, your connector must implement the acceptance tests described in [Testing Connectors](testing-connectors/testing-connectors.md) **Note: Acceptance tests are not yet available for Python destination connectors. Coming [soon](https://github.com/airbytehq/airbyte/issues/4698)!** diff --git a/docs/connector-development/airbyte101.md b/docs/connector-development/airbyte101.md new file mode 100644 index 00000000000..a49f5fada8d --- /dev/null +++ b/docs/connector-development/airbyte101.md @@ -0,0 +1,6 @@ +# Airbyte 101 for Connector Development + +## The Airbyte Catalog + +The Airbyte catalog defines the relationship between your incoming data's schema and the schema of your output stream. This +is an incredibly important concept to understand as a connector dev, so check out the AirbyteCatalog [here](../understanding-airbyte/beginners-guide-to-catalog.md). \ No newline at end of file diff --git a/docs/contributing-to-airbyte/building-new-connector/best-practices.md b/docs/connector-development/best-practices.md similarity index 93% rename from docs/contributing-to-airbyte/building-new-connector/best-practices.md rename to docs/connector-development/best-practices.md index bcdf9cc9262..ed55448a086 100644 --- a/docs/contributing-to-airbyte/building-new-connector/best-practices.md +++ b/docs/connector-development/best-practices.md @@ -31,7 +31,7 @@ When reviewing connectors, we'll use the following "checklist" to verify whether * **API connectors** should validate records that every stream outputs data * If this causes rate limiting problems, there should be a periodic CI build which tests this on a less frequent cadence to avoid rate limiting -**Thoroughly test edge cases.** While Airbyte provides a [Standard Test Suite](testing-connectors.md#airbytes-standard-tests) that all connectors must pass, it's not possible for the standard test suite to cover all edge cases. When in doubt about whether the standard tests provide sufficient evidence of functionality, write a custom test case for your connector. +**Thoroughly test edge cases.** While Airbyte provides a [Standard Test Suite](testing-connectors/testing-connectors.md#airbytes-standard-tests) that all connectors must pass, it's not possible for the standard test suite to cover all edge cases. When in doubt about whether the standard tests provide sufficient evidence of functionality, write a custom test case for your connector. ### Check Connection diff --git a/docs/contributing-to-airbyte/python/README.md b/docs/connector-development/cdk-python/README.md similarity index 78% rename from docs/contributing-to-airbyte/python/README.md rename to docs/connector-development/cdk-python/README.md index 73885174649..0a1def505ab 100644 --- a/docs/contributing-to-airbyte/python/README.md +++ b/docs/connector-development/cdk-python/README.md @@ -25,7 +25,25 @@ Additionally, you can follow [this tutorial](https://github.com/airbytehq/airbyt ### Concepts & Documentation -See the [concepts docs](concepts/) for a tour through what the API offers. +#### Basic Concepts + +If you want to learn more about the classes required to implement an Airbyte Source, head to our [basic concepts doc](./basic-concepts.md). + +#### Full Refresh Streams + +If you have questions or are running into issues creating your first full refresh stream, head over to our [full refresh stream doc](./full-refresh-stream.md). If you have questions about implementing a `path` or `parse_response` function, this doc is for you. + +#### Incremental Streams + +Having trouble figuring out how to write a `stream_slices` function or aren't sure what a `cursor_field` is? Head to our [incremental stream doc](./incremental-stream.md). + +#### Practical Tips + +Airbyte recommends using the CDK template generator to develop with the CDK. The template generates created all the required scaffolding, with convenient TODOs, allowing developers to truly focus on implementing the API. + +For tips on useful Python knowledge, see the [Python Concepts](./python-concepts.md) page. + +You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](../tutorials/cdk-tutorial-python-http) ### Example Connectors diff --git a/docs/contributing-to-airbyte/python/concepts/basic-concepts.md b/docs/connector-development/cdk-python/basic-concepts.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/basic-concepts.md rename to docs/connector-development/cdk-python/basic-concepts.md diff --git a/docs/contributing-to-airbyte/python/concepts/full-refresh-stream.md b/docs/connector-development/cdk-python/full-refresh-stream.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/full-refresh-stream.md rename to docs/connector-development/cdk-python/full-refresh-stream.md diff --git a/docs/contributing-to-airbyte/python/concepts/http-streams.md b/docs/connector-development/cdk-python/http-streams.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/http-streams.md rename to docs/connector-development/cdk-python/http-streams.md diff --git a/docs/contributing-to-airbyte/python/concepts/incremental-stream.md b/docs/connector-development/cdk-python/incremental-stream.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/incremental-stream.md rename to docs/connector-development/cdk-python/incremental-stream.md diff --git a/docs/contributing-to-airbyte/python/concepts/python-concepts.md b/docs/connector-development/cdk-python/python-concepts.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/python-concepts.md rename to docs/connector-development/cdk-python/python-concepts.md diff --git a/docs/contributing-to-airbyte/python/concepts/schemas.md b/docs/connector-development/cdk-python/schemas.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/schemas.md rename to docs/connector-development/cdk-python/schemas.md diff --git a/docs/contributing-to-airbyte/python/concepts/stream_slices.md b/docs/connector-development/cdk-python/stream-slices.md similarity index 100% rename from docs/contributing-to-airbyte/python/concepts/stream_slices.md rename to docs/connector-development/cdk-python/stream-slices.md diff --git a/docs/contributing-to-airbyte/building-new-connector/testing-connectors.md b/docs/connector-development/testing-connectors/README.md similarity index 52% rename from docs/contributing-to-airbyte/building-new-connector/testing-connectors.md rename to docs/connector-development/testing-connectors/README.md index 4c226360c74..d439f0b1aa6 100644 --- a/docs/contributing-to-airbyte/building-new-connector/testing-connectors.md +++ b/docs/connector-development/testing-connectors/README.md @@ -1,72 +1,5 @@ # Testing Connectors -## Source Acceptance Tests - -To ensure a minimum quality bar, Airbyte runs all connectors against the same set of integration tests \(sources & destinations have two different test suites\). Those tests ensure that each connector adheres to the [Airbyte Specification](../../understanding-airbyte/airbyte-specification.md) and responds correctly to Airbyte commands when provided valid \(or invalid\) inputs. - -_Note: If you are looking for reference documentation for the deprecated first version of test suites, see_ [_Standard Tests \(Legacy\)_](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/contributing-to-airbyte/building-new-connector/legacy-standard-source-tests.md)_._ - -### Architecture of standard tests - -The Standard Test Suite runs its tests against the connector's Docker image. It takes as input the configuration file `acceptance-tests-config.yml`. - -![Standard test sequence diagram](../../.gitbook/assets/standard_tests_sequence_diagram.png) - -The Standard Test Suite use pytest as a test runner and was built as pytest plugin `source-acceptance-test`. This plugin adds a new configuration option `—acceptance-test-config` - it should points to the folder with `acceptance-tests-config.yml`. - -Each test suite has a timeout and will fail if the limit is exceeded. - -See all the test cases, their description, and inputs in [Source Acceptance Tests](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md). - -### Setting up standard tests for your connector - -Create `acceptance-test-config.yml`. In most cases, your connector already has this file in its root folder. Here is an example of the minimal `acceptance-test-config.yml`: - -```yaml -connector_image: airbyte/source-some-connector:dev -tests: - spec: - - spec_path: "some_folder/spec.json" -``` - -Build your connector image if needed. - -```text -docker build . -``` - -Run one of the two scripts in the root of the connector: - -* `python -m pytest -p integration_tests.acceptance` - to run tests inside virtual environment -* `./acceptance-test-docker.sh` - to run tests from a docker container - -If the test fails you will see detail about the test and where to find its inputs and outputs to reproduce it. You can also debug failed tests by adding `—pdb —last-failed`: - -```text -python -m pytest -p integration_tests.acceptance --pdb --last-failed -``` - -See other useful pytest options [here](https://docs.pytest.org/en/stable/usage.html) - -### Dynamically managing inputs & resources used in standard tests - -Since the inputs to standard tests are often static, the file-based runner is sufficient for most connectors. However, in some cases, you may need to run pre or post hooks to dynamically create or destroy resources for use in standard tests. For example, if we need to spin up a Redshift cluster to use in the test then tear it down afterwards, we need the ability to run code before and after the tests, as well as customize the Redshift cluster URL we pass to the standard tests. If you have need for this use case, please reach out to us via [Github](https://github.com/airbytehq/airbyte) or [Slack](https://slack.airbyte.io). We currently support it for Java & Python, and other languages can be made available upon request. - -#### Python - -Create pytest yield-fixture with your custom setup/teardown code and place it in `integration_tests/acceptance.py`, Example of fixture that starts a docker container before tests and stops before exit: - -```python -@pytest.fixture(scope="session", autouse=True) -def connector_setup(): - """ This fixture is a placeholder for external resources that acceptance test might require. - """ - client = docker.from_env() - container = client.containers.run("your/docker-image", detach=True) - yield - container.stop() -``` - ## Running Integration tests The GitHub `master` and branch builds will build the core Airbyte infrastructure \(scheduler, ui, etc\) as well as the images for all connectors. Integration tests \(tests that run a connector's image against an external resource\) can be run one of three ways. diff --git a/docs/contributing-to-airbyte/building-new-connector/legacy-standard-source-tests.md b/docs/connector-development/testing-connectors/legacy-standard-source-tests.md similarity index 98% rename from docs/contributing-to-airbyte/building-new-connector/legacy-standard-source-tests.md rename to docs/connector-development/testing-connectors/legacy-standard-source-tests.md index f881a1b1133..99069d4fd50 100644 --- a/docs/contributing-to-airbyte/building-new-connector/legacy-standard-source-tests.md +++ b/docs/connector-development/testing-connectors/legacy-standard-source-tests.md @@ -2,7 +2,7 @@ ## Airbyte's Standard Tests (v1) -This document describes the old version Standard Tests, please check the latest version [here](testing-connectors.md) +This document describes the old version Standard Tests, please check the latest version [here](../../connector-development/testing-connectors/testing-connectors.md) To ensure a minimum quality bar, Airbyte runs all connectors against the same set of integration tests \(sources & destinations have two different test suites\). Those tests ensure that each connector adheres to the [Airbyte Specification](../../architecture/airbyte-specification.md) and responds correctly to Airbyte commands when provided valid \(or invalid\) inputs. ### Architecture of standard tests diff --git a/docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md b/docs/connector-development/testing-connectors/source-acceptance-tests-reference.md similarity index 64% rename from docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md rename to docs/connector-development/testing-connectors/source-acceptance-tests-reference.md index 8052d535d26..6d7cf5558b3 100644 --- a/docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md +++ b/docs/connector-development/testing-connectors/source-acceptance-tests-reference.md @@ -1,6 +1,71 @@ -# Source Acceptance Test Suite +# Source Acceptance Tests Reference -Tests configurable via `acceptance-test-config.yml`. Each test has a number of inputs, +To ensure a minimum quality bar, Airbyte runs all connectors against the same set of integration tests \(sources & destinations have two different test suites\). Those tests ensure that each connector adheres to the [Airbyte Specification](../../understanding-airbyte/airbyte-specification.md) and responds correctly to Airbyte commands when provided valid \(or invalid\) inputs. + +_Note: If you are looking for reference documentation for the deprecated first version of test suites, see_ [_Standard Tests \(Legacy\)_](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/contributing-to-airbyte/building-new-connector/legacy-standard-source-tests.md)_._ + +## Architecture of standard tests + +The Standard Test Suite runs its tests against the connector's Docker image. It takes as input the configuration file `acceptance-tests-config.yml`. + +![Standard test sequence diagram](../../.gitbook/assets/standard_tests_sequence_diagram.png) + +The Standard Test Suite use pytest as a test runner and was built as pytest plugin `source-acceptance-test`. This plugin adds a new configuration option `—acceptance-test-config` - it should points to the folder with `acceptance-tests-config.yml`. + +Each test suite has a timeout and will fail if the limit is exceeded. + +See all the test cases, their description, and inputs in [Source Acceptance Tests](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md). + +## Setting up standard tests for your connector + +Create `acceptance-test-config.yml`. In most cases, your connector already has this file in its root folder. Here is an example of the minimal `acceptance-test-config.yml`: + +```yaml +connector_image: airbyte/source-some-connector:dev +tests: + spec: + - spec_path: "some_folder/spec.json" +``` + +Build your connector image if needed. + +```text +docker build . +``` + +Run one of the two scripts in the root of the connector: + +* `python -m pytest -p integration_tests.acceptance` - to run tests inside virtual environment +* `./acceptance-test-docker.sh` - to run tests from a docker container + +If the test fails you will see detail about the test and where to find its inputs and outputs to reproduce it. You can also debug failed tests by adding `—pdb —last-failed`: + +```text +python -m pytest -p integration_tests.acceptance --pdb --last-failed +``` + +See other useful pytest options [here](https://docs.pytest.org/en/stable/usage.html) + +## Dynamically managing inputs & resources used in standard tests + +Since the inputs to standard tests are often static, the file-based runner is sufficient for most connectors. However, in some cases, you may need to run pre or post hooks to dynamically create or destroy resources for use in standard tests. For example, if we need to spin up a Redshift cluster to use in the test then tear it down afterwards, we need the ability to run code before and after the tests, as well as customize the Redshift cluster URL we pass to the standard tests. If you have need for this use case, please reach out to us via [Github](https://github.com/airbytehq/airbyte) or [Slack](https://slack.airbyte.io). We currently support it for Java & Python, and other languages can be made available upon request. + +### Python + +Create pytest yield-fixture with your custom setup/teardown code and place it in `integration_tests/acceptance.py`, Example of fixture that starts a docker container before tests and stops before exit: + +```python +@pytest.fixture(scope="session", autouse=True) +def connector_setup(): + """ This fixture is a placeholder for external resources that acceptance test might require. + """ + client = docker.from_env() + container = client.containers.run("your/docker-image", detach=True) + yield + container.stop() +``` + +These tests are configurable via `acceptance-test-config.yml`. Each test has a number of inputs, you can provide multiple sets of inputs which will cause the same to run multiple times - one for each set of inputs. Example of `acceptance-test-config.yml`: @@ -17,8 +82,7 @@ tests: # Tests configuration # discovery: # skip this test incremental: [] # skip this test as well ``` -Ideally, the connector should pass all acceptance tests. If the test or its inputs omitted this test will be skipped. - + ## Test Spec Verify that a spec operation issued to the connector returns a valid spec. | Input | Type| Default | Note | diff --git a/docs/contributing-to-airbyte/building-new-connector/standard-source-tests.md b/docs/connector-development/testing-connectors/standard-source-tests.md similarity index 100% rename from docs/contributing-to-airbyte/building-new-connector/standard-source-tests.md rename to docs/connector-development/testing-connectors/standard-source-tests.md diff --git a/docs/contributing-to-airbyte/python/tutorials/README.md b/docs/connector-development/tutorials/README.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/README.md rename to docs/connector-development/tutorials/README.md diff --git a/docs/contributing-to-airbyte/tutorials/adding-incremental-sync.md b/docs/connector-development/tutorials/adding-incremental-sync.md similarity index 99% rename from docs/contributing-to-airbyte/tutorials/adding-incremental-sync.md rename to docs/connector-development/tutorials/adding-incremental-sync.md index 51bfe681967..8dbfc53a8da 100644 --- a/docs/contributing-to-airbyte/tutorials/adding-incremental-sync.md +++ b/docs/connector-development/tutorials/adding-incremental-sync.md @@ -1,4 +1,4 @@ -# Adding Incremental to a Source +# Adding Incremental Sync to a Source ## Overview diff --git a/docs/contributing-to-airbyte/tutorials/build-a-connector-the-hard-way.md b/docs/connector-development/tutorials/build-a-connector-the-hard-way.md similarity index 98% rename from docs/contributing-to-airbyte/tutorials/build-a-connector-the-hard-way.md rename to docs/connector-development/tutorials/build-a-connector-the-hard-way.md index 01e4c2036b0..2d9f2a7a104 100644 --- a/docs/contributing-to-airbyte/tutorials/build-a-connector-the-hard-way.md +++ b/docs/connector-development/tutorials/build-a-connector-the-hard-way.md @@ -7,9 +7,9 @@ description: Building a source connector without using any helpers to learn the This tutorial walks you through building a simple Airbyte source without using any helpers to demonstrate the following concepts in Action: * [The Airbyte Specification](../../understanding-airbyte/airbyte-specification.md) and the interface implemented by a source connector -* [The AirbyteCatalog](beginners-guide-to-catalog.md) +* [The AirbyteCatalog](../../understanding-airbyte/beginners-guide-to-catalog.md) * [Packaging your connector](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/contributing-to-airbyte/contributing-to-airbyte/building-new-connector/README.md#1-implement--package-the-connector) -* [Testing your connector](../building-new-connector/testing-connectors.md) +* [Testing your connector](../../connector-development/testing-connectors/testing-connectors.md) At the end of this tutorial, you will have a working source that you will be able to use in the Airbyte UI. @@ -359,13 +359,13 @@ Our connector is able to detect valid and invalid configs correctly. Two methods #### Implementing Discover -The `discover` command outputs a Catalog, a struct that declares the Streams and Fields \(Airbyte's equivalents of tables and columns\) output by the connector. It also includes metadata around which features a connector supports \(e.g. which sync modes\). In other words it describes what data is available in the source. If you'd like to read a bit more about this concept check out our [Beginner's Guide to the Airbyte Catalog](beginners-guide-to-catalog.md) or for a more detailed treatment read the [Airbyte Specification](../../understanding-airbyte/airbyte-specification.md). +The `discover` command outputs a Catalog, a struct that declares the Streams and Fields \(Airbyte's equivalents of tables and columns\) output by the connector. It also includes metadata around which features a connector supports \(e.g. which sync modes\). In other words it describes what data is available in the source. If you'd like to read a bit more about this concept check out our [Beginner's Guide to the Airbyte Catalog](../../understanding-airbyte/beginners-guide-to-catalog.md) or for a more detailed treatment read the [Airbyte Specification](../../understanding-airbyte/airbyte-specification.md). The data output by this connector will be structured in a very simple way. This connector outputs records belonging to exactly one Stream \(table\). Each record contains three Fields \(columns\): `date`, `price`, and `stock_ticker`, corresponding to the price of a stock on a given day. To implement `discover`, we'll: -1. Add a method `discover` in `source.py` which outputs the Catalog. To better understand what a catalog is, check out our [Beginner's Guide to the AirbyteCatalog](beginners-guide-to-catalog.md). +1. Add a method `discover` in `source.py` which outputs the Catalog. To better understand what a catalog is, check out our [Beginner's Guide to the AirbyteCatalog](../../understanding-airbyte/beginners-guide-to-catalog.md). 2. Extend the arguments parser to use detect the `discover --config ` command and call the `discover` method. Let's implement `discover` by adding the following in `source.py`: @@ -476,7 +476,7 @@ python source.py read --config --catalog :integrationTest`. Make sure to run this command from the Airbyte repository root. diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-speedrun.md b/docs/connector-development/tutorials/cdk-speedrun.md similarity index 98% rename from docs/contributing-to-airbyte/python/tutorials/cdk-speedrun.md rename to docs/connector-development/tutorials/cdk-speedrun.md index fbb4517ee3b..8d5820c60fc 100644 --- a/docs/contributing-to-airbyte/python/tutorials/cdk-speedrun.md +++ b/docs/connector-development/tutorials/cdk-speedrun.md @@ -1,8 +1,8 @@ -# Speedrun: Creating a Source with the CDK +# Python CDK Speedrun: Creating a Source ## CDK Speedrun \(HTTP API Source Creation [Any%](https://en.wikipedia.org/wiki/Speedrun#:~:text=Any%25%2C%20or%20fastest%20completion%2C,the%20game%20to%20its%20fullest.&text=Specific%20requirements%20for%20a%20100,different%20depending%20on%20the%20game.) Route\) -This is a blazing fast guide to building an HTTP source connector. Think of it as the TL;DR version of [this tutorial.](./cdk-tutorial-python-http/0-getting-started.md) +This is a blazing fast guide to building an HTTP source connector. Think of it as the TL;DR version of [this tutorial.](cdk-tutorial-python-http/0-getting-started.md) ## Dependencies diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/0-getting-started.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/0-getting-started.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/0-getting-started.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/0-getting-started.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/1-creating-the-source.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/1-creating-the-source.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/1-creating-the-source.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/1-creating-the-source.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/2-install-dependencies.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/2-install-dependencies.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/2-install-dependencies.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/2-install-dependencies.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/3-define-inputs.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/3-define-inputs.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/3-define-inputs.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/3-define-inputs.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/4-connection-checking.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/4-connection-checking.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/4-connection-checking.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/4-connection-checking.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/5-declare-schema.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/5-declare-schema.md similarity index 97% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/5-declare-schema.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/5-declare-schema.md index d032d5ae2ec..faad2660cf8 100644 --- a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/5-declare-schema.md +++ b/docs/connector-development/tutorials/cdk-tutorial-python-http/5-declare-schema.md @@ -71,5 +71,5 @@ you should see some output like: It's that simple! Now the connector knows how to declare your connector's stream's schema. We declare only one stream since our source is simple, but the principle is exactly the same if you had many streams. -You can also dynamically define schemas, but that's beyond the scope of this tutorial. See the [schema docs](../../concepts/full-refresh-stream.md#defining-the-streams-schema) for more information. +You can also dynamically define schemas, but that's beyond the scope of this tutorial. See the [schema docs](../../cdk-python/full-refresh-stream.md#defining-the-streams-schema) for more information. diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/6-read-data.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/6-read-data.md similarity index 98% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/6-read-data.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/6-read-data.md index bb5485abdc4..cd67d1cc5ac 100644 --- a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/6-read-data.md +++ b/docs/connector-development/tutorials/cdk-tutorial-python-http/6-read-data.md @@ -2,7 +2,7 @@ Describing schemas is good and all, but at some point we have to start reading data! So let's get to work. But before, let's describe what we're about to do: -The `HttpStream` superclass, like described in the [concepts documentation](../../concepts/http-streams.md), is facilitating reading data from HTTP endpoints. It contains built-in functions or helpers for: +The `HttpStream` superclass, like described in the [concepts documentation](../../cdk-python/http-streams.md), is facilitating reading data from HTTP endpoints. It contains built-in functions or helpers for: * authentication * pagination @@ -176,7 +176,7 @@ We'll implement the `stream_slices` method to return a list of the dates for whi return self._chunk_date_range(start_date) ``` -Each slice will cause an HTTP request to be made to the API. We can then use the information present in the `stream_slice` parameter \(a single element from the list we constructed in `stream_slices` above\) to set other configurations for the outgoing request like `path` or `request_params`. For more info about stream slicing, see [the slicing docs](../../concepts/stream_slices.md). +Each slice will cause an HTTP request to be made to the API. We can then use the information present in the `stream_slice` parameter \(a single element from the list we constructed in `stream_slices` above\) to set other configurations for the outgoing request like `path` or `request_params`. For more info about stream slicing, see [the slicing docs](../../stream_slices.md). In order to pull data for a specific date, the Exchange Rates API requires that we pass the date as the path component of the URL. Let's override the `path` method to achieve this: diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/7-use-connector-in-airbyte.md diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/8-test-your-connector.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/8-test-your-connector.md similarity index 100% rename from docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/8-test-your-connector.md rename to docs/connector-development/tutorials/cdk-tutorial-python-http/8-test-your-connector.md diff --git a/docs/connector-development/tutorials/cdk-tutorial-python-http/README.md b/docs/connector-development/tutorials/cdk-tutorial-python-http/README.md new file mode 100644 index 00000000000..1faf1a277c0 --- /dev/null +++ b/docs/connector-development/tutorials/cdk-tutorial-python-http/README.md @@ -0,0 +1,2 @@ +# Creating an HTTP API Source with the Python CDK + diff --git a/docs/contributing-to-airbyte/README.md b/docs/contributing-to-airbyte/README.md index e9797885c6b..3cb1326d9e8 100644 --- a/docs/contributing-to-airbyte/README.md +++ b/docs/contributing-to-airbyte/README.md @@ -28,15 +28,15 @@ Here is a list of easy [good first issues](https://github.com/airbytehq/airbyte/ It's easy to add your own connector to Airbyte! **Since Airbyte connectors are encapsulated within Docker containers, you can use any language you like.** Here are some links on how to add sources and destinations. We haven't built the documentation for all languages yet, so don't hesitate to reach out to us if you'd like help developing connectors in other languages. -For sources, simply head over to our [Python CDK](./python/README.md). +For sources, simply head over to our [Python CDK](../connector-development/cdk-python/README.md). {% hint style="info" %} The CDK currently does not support creating destinations, but it will very soon. {% endhint %} * See [Building new connectors](building-new-connector/) to get started. -* Since we frequently build connectors in Python, on top of Singer or in Java, we've created generator libraries to get you started quickly: [Build Python Source Connectors](tutorials/building-a-python-source.md) and [Build Java Connectors](building-new-connector/java-connectors.md) -* Integration tests \(tests that run a connector's image against an external resource\) can be run one of three ways, as detailed [here](building-new-connector/testing-connectors.md) +* Since we frequently build connectors in Python, on top of Singer or in Java, we've created generator libraries to get you started quickly: [Build Python Source Connectors](../connector-development/tutorials/building-a-python-source.md) and [Build Java Connectors](building-new-connector/java-connectors.md) +* Integration tests \(tests that run a connector's image against an external resource\) can be run one of three ways, as detailed [here](../connector-development/testing-connectors/testing-connectors.md) **Please note that, at no point in time, we will ask you to maintain your connector.** The goal is that the Airbyte team and the community helps maintain the connector. diff --git a/docs/contributing-to-airbyte/building-new-connector/java-connectors.md b/docs/contributing-to-airbyte/building-new-connector/java-connectors.md deleted file mode 100644 index 783d318cbf6..00000000000 --- a/docs/contributing-to-airbyte/building-new-connector/java-connectors.md +++ /dev/null @@ -1,16 +0,0 @@ -# Java Connectors - -For some connectors we use `java` as many data platforms, sources and destination have very mature java libraries. - -Java connectors are fully embedded in the `gradle` build and follow the normal development process \(more details on [developing locally](../developing-locally.md)\). - - -### Generate the Template -For the JDBC based sources the Code Generator maybe used to create a module's skeleton. - -```bash -$ cd airbyte-integrations/connector-templates/generator # start from repo root -$ ./generate.sh -``` - -In menu choose the `"Java JDBC Source"` \ No newline at end of file diff --git a/docs/contributing-to-airbyte/building-new-connector/monorepo-python-development.md b/docs/contributing-to-airbyte/monorepo-python-development.md similarity index 100% rename from docs/contributing-to-airbyte/building-new-connector/monorepo-python-development.md rename to docs/contributing-to-airbyte/monorepo-python-development.md diff --git a/docs/contributing-to-airbyte/python/concepts/README.md b/docs/contributing-to-airbyte/python/concepts/README.md deleted file mode 100644 index 14a34f632d8..00000000000 --- a/docs/contributing-to-airbyte/python/concepts/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# Concepts - -This concepts section serves as a general introduction to the Python CDK. Readers will certainly benefit from a deeper understanding of the [Airbyte Specification](https://docs.airbyte.io/architecture/airbyte-specification) before proceeding, but we do a quick overview of it in our basic concepts guide below. - -## Basic Concepts - -If you want to learn more about the classes required to implement an Airbyte Source, head to our [basic concepts doc](basic-concepts.md). - -## Full Refresh Streams - -If you have questions or are running into issues creating your first full refresh stream, head over to our [full refresh stream doc](full-refresh-stream.md). If you have questions about implementing a `path` or `parse_response` function, this doc is for you. - -## Incremental Streams - -Having trouble figuring out how to write a `stream_slices` function or aren't sure what a `cursor_field` is? Head to our [incremental stream doc](incremental-stream.md). - -## Practical Tips - -Airbyte recommends using the CDK template generator to develop with the CDK. The template generates created all the required scaffolding, with convenient TODOs, allowing developers to truly focus on implementing the API. - -For tips on useful Python knowledge, see the [Python Concepts](python-concepts.md) page. - -You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](https://github.com/airbytehq/airbyte/tree/4a397d25247db77a7b78783d26dae35bc3900f59/airbyte-cdk/python/docs/tutorials/http_api_source.md) - -## Examples - -Those interested in getting their hands dirty can check out implemented APIs: - -* [Exchange Rates API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-exchange-rates/source_exchange_rates/source.py) \(Incremental\) -* [Stripe API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py) \(Incremental and Full-Refresh\) -* [Slack API](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py) \(Incremental and Full-Refresh\) - diff --git a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/README.md b/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/README.md deleted file mode 100644 index 466943b269c..00000000000 --- a/docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# Creating an HTTP API Source - diff --git a/docs/contributing-to-airbyte/tutorials/README.md b/docs/contributing-to-airbyte/tutorials/README.md deleted file mode 100644 index c6d4782483f..00000000000 --- a/docs/contributing-to-airbyte/tutorials/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# Contributing to Airbyte - diff --git a/docs/operator-guides/browsing-output-logs.md b/docs/operator-guides/browsing-output-logs.md index 81339b9e6a9..5ccba396577 100644 --- a/docs/operator-guides/browsing-output-logs.md +++ b/docs/operator-guides/browsing-output-logs.md @@ -72,7 +72,7 @@ normalize target_config.json ### Reading the content of the catalog.json file -For example, it is often useful to inspect the content of the [catalog](../contributing-to-airbyte/tutorials/beginners-guide-to-catalog.md) file. You could do so by running a `cat` command: +For example, it is often useful to inspect the content of the [catalog](../understanding-airbyte/beginners-guide-to-catalog.md) file. You could do so by running a `cat` command: ```bash docker run -it --rm --volume airbyte_workspace:/data busybox cat /data/9/2/catalog.json diff --git a/docs/contributing-to-airbyte/tutorials/beginners-guide-to-catalog.md b/docs/understanding-airbyte/beginners-guide-to-catalog.md similarity index 95% rename from docs/contributing-to-airbyte/tutorials/beginners-guide-to-catalog.md rename to docs/understanding-airbyte/beginners-guide-to-catalog.md index 90142da4d91..e2d63efcd02 100644 --- a/docs/contributing-to-airbyte/tutorials/beginners-guide-to-catalog.md +++ b/docs/understanding-airbyte/beginners-guide-to-catalog.md @@ -2,7 +2,7 @@ ## Overview -The goal of this article is to make the `AirbyteCatalog` approachable to someone contributing to Airbyte for the first time. If you are looking to get deeper into the details of the catalog, you can read our technical specification on it [here](../../understanding-airbyte/catalog.md). +The goal of this article is to make the `AirbyteCatalog` approachable to someone contributing to Airbyte for the first time. If you are looking to get deeper into the details of the catalog, you can read our technical specification on it [here](catalog.md). The goal of the `AirbyteCatalog` is to describe _what_ data is available in a source. The goal of the `ConfiguredAirbyteCatalog` is to, based on an `AirbyteCatalog`, specify _how_ data from the source is replicated. @@ -16,7 +16,7 @@ This article will illustrate how to use `AirbyteCatalog` via a series of example * [Dynamic Streams Example](beginners-guide-to-catalog.md#Dynamic-Streams-Example) * [Nested Schema Example](beginners-guide-to-catalog.md#Nested-Schema-Example) -In order to understand in depth how to configure incremental data replication, head over to the [incremental replication docs](../../understanding-airbyte/connections/incremental-append.md). +In order to understand in depth how to configure incremental data replication, head over to the [incremental replication docs](connections/incremental-append.md). ## Database Example @@ -92,7 +92,7 @@ The catalog is structured as a list of `AirbyteStream`. In the case of a databas Let's walk through what each field in a stream means. * `name` - The name of the stream. -* `supported_sync_modes` - This field lists the type of data replication that this source supports. The possible values in this array include `FULL_REFRESH` \([docs](../../understanding-airbyte/connections/full-refresh-overwrite.md)\) and `INCREMENTAL` \([docs](../../understanding-airbyte/connections/incremental-append.md)\). +* `supported_sync_modes` - This field lists the type of data replication that this source supports. The possible values in this array include `FULL_REFRESH` \([docs](connections/full-refresh-overwrite.md)\) and `INCREMENTAL` \([docs](connections/incremental-append.md)\). * `source_defined_cursor` - If the stream supports `INCREMENTAL` replication, then this field signal whether the source can figure out how to detect new records on its own or not. * `json_schema` - This field is a [JsonSchema](https://json-schema.org/understanding-json-schema) object that describes the structure of the data. Notice that each key in the `properties` object corresponds to a column name in our database table. @@ -137,7 +137,7 @@ Let's walk through each field in the `ConfiguredAirbyteStream`: * `sync_mode` - This field must be one of the values that was in `supported_sync_modes` in the `AirbyteStream` - Configures which sync mode will be used when data is replicated. * `stream` - Hopefully this one looks familiar! This field contains an `AirbyteStream`. It should be _identical_ the one we saw in the `AirbyteCatalog`. -* `cursor_field` - When `sync_mode` is `INCREMENTAL` and `source_defined_cursor = false`, this field configures which field in the stream will be used to determine if a record should be replicated or not. Read more about this concept in our [documentation of incremental replication](../../understanding-airbyte/connections/incremental-append.md). +* `cursor_field` - When `sync_mode` is `INCREMENTAL` and `source_defined_cursor = false`, this field configures which field in the stream will be used to determine if a record should be replicated or not. Read more about this concept in our [documentation of incremental replication](connections/incremental-append.md). ### Summary of the Postgres Example diff --git a/docs/understanding-airbyte/catalog.md b/docs/understanding-airbyte/catalog.md index 3b20bd162cb..9841febb94b 100644 --- a/docs/understanding-airbyte/catalog.md +++ b/docs/understanding-airbyte/catalog.md @@ -1,4 +1,4 @@ -# AirbyteCatalog & ConfiguredAirbyteCatalog +# AirbyteCatalog & ConfiguredAirbyteCatalog Reference ## Overview