e029353b9f
* method yielding airbytemessage * move to Stream * update abstract source * reset * missing file * Yield request and response as log messages * only emit request and responses if the debug flag is on * add test docker image * script to run acceptance tests with local cdk * Update conftest to use a different image * extract to method * dont use a different image tag * Always install local cdk * break the cdk * get path from current working directory * or * ignore unit test * debug log * Revert "AMI change: ami-0f23be2f917510c26 -> ami-005924fb76f7477ce (#18689)" This reverts commitbf06decf73. * build from the top * Update source-acceptance-test * fix * copy setup * some work on the gradle plugin * reset to master * delete unused file * delete unused file * reset to master * optional argument * delete dead code * use latest cdk with sendgrid * fix sendgrid dockerfile * break the cdk * use local file * Revert "break the cdk" This reverts commit600c195541. * dont raise an exception * reset to master * unit tests * missing test * more unit tests * remove deprecated comment * newline * reset to master * remove files * reset * Update abstract source * remove method from stream * convert to airbytemessage * unittests * Update * unit test * remove debug logs * Revert "remove debug logs" This reverts commita1a139ef37. * Revert "Revert "remove debug logs"" This reverts commitb1d62cdb60. * Revert "reset to master" This reverts commit3fa6a004c1. * fix * slightly better test * typing * extract method * Revert "Revert "reset to master"" This reverts commit5dac7c2804. * reset to master * reset to master * Revert "reset to master" This reverts commit3fa6a004c1. * Comment * operate on the message * Revert "Revert "reset to master"" This reverts commit5833c84d0a. * comment * test * Revert "test" This reverts commit2f91b803b0. * test * Revert "test" This reverts commit62d95ebbb5. * test * Revert "test" This reverts commit27150ba341. * format * format * symlink * Update setup * update path * reset to master * update * Add local files * greenhouse * format * symlink * try reordering * better error message * better log message * reset to master * Revert "merge for qa" This reverts commitad7128f2c5, reversing changes made to7196c22a73. * reset to master * reset to master * reset to master * format * gradlew format * right type hints * reset to master * reset to master * gradlew format * a bunch of small fixes * Update output format * fixes from feedback * fixme comment * streams cannot return AirbyteRecordMessage * fix * format * only return logs when running on debug mode * move branching * update typing * remove dead code * fix simpleretriever name * i think this is better * log response.text * debug flag * comment * pass config * comments * run SATs * fix most of the unit tests * fix unit test * reset to master * runFromPath * Revert "runFromPath" This reverts commit85979a801a. * Revert "run SATs" This reverts commita8a8a2da95. * no need to convert to dict * fix test
Connector Development Kit (Python)
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
- HTTP APIs (REST APIs, GraphQL, etc..)
- Singer Taps
- Generic Python sources (anything not covered by the above)
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.
Getting Started
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
cd airbyte-integrations/connector-templates/generator
./generate.sh
then follow the interactive prompt. Next, find all TODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
Concepts & Documentation
See the concepts docs for a tour through what the API offers.
Example Connectors
HTTP Connectors:
Singer connectors:
Simple Python connectors using the barebones Source abstraction:
Contributing
First time setup
We assume python points to python >=3.9.
Setup a virtual env:
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies
Iteration
- Iterate on the code locally
- Run tests via
pytest -s unit_tests - Perform static type checks using
mypy airbyte_cdk.MyPyconfiguration is in.mypy.ini. - The
type_check_and_test.shscript bundles both type checking and testing in one convenient command. Feel free to use it!
Testing
All tests are located in the unit_tests directory. Run pytest --cov=airbyte_cdk unit_tests/ to run them. This also presents a test coverage report.
Publishing a new version to PyPi
- Bump the package version in
setup.py - Open a PR
- An Airbyte member must comment
/publish-cdk dry-run=trueto publish the package to test.pypi.org or/publish-cdk dry-run=falseto publish it to the real index of pypi.org.
Coming Soon
- Full OAuth 2.0 support (including refresh token issuing flow via UI or CLI)
- Airbyte Java HTTP CDK
- CDK for Async HTTP endpoints (request-poll-wait style endpoints)
- CDK for other protocols
- Don't see a feature you need? Create an issue and let us know how we can help!