44ec661b5a
* comment * comment * comments * fix * test for instantiating chain retrier * fix parsing * cleanup * fix * reset * never raise on http error * remove print * comment * comment * comment * comment * remove prints * add declarative stream to registry * start working on limit paginator * support for offset pagination * tests * move limit value * extract request option * boilerplate * page increment * delete offset paginator * update conditional paginator * refactor and fix test * fix test * small fix * Delete dead code * Add docstrings * quick fix * exponential backoff * fix test * fix * delete unused properties * fix * missing unit tests * uppercase * docstrings * rename to success * compare full request instead of just url * renmae module * rename test file * rename interface * rename default retrier * rename to compositeerrorhandler * fix missing renames * move action to filter * str -> minmaxdatetime * small fixes * plural * add example * handle header variations * also fix wait time from * allow using a regex to extract the value * group() * docstring * add docs * update comment * docstrings * fix tests * rename param * cleanup stop_condition * cleanup * Add examples * interpolated pagination strategy * dont need duplicate class * docstrings * more docstrings * docstrings * fix tests * first pass at substream * seems to work for a single stream * can also be defined in requester with stream_state * tmp update * update comment * Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * version: Update Parquet library to latest release (#14502) The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commitc72862b613* merge * 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Docs: Fixed broken links (#14622) * fixing broken links * more broken links * source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620) * Helm Chart: Add external temporal option (#14597) * conflict env configmap and chart lock * reverting lock * add eof lines and documentation on values yaml * conflict json file * rollback json * solve conflict * correct minio with new version Co-authored-by: Guy Feldman <gfeldman@86labs.com> * 🎉 Add YAML format to source-file reader (#14588) * Add yaml reader * Update docs * Bumpversion of connector * bump docs * Update pyarrow dependency * Upgrade pandas dependency * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🎉 Source Okta: add GroupMembers stream (#14380) * add Group_Members stream to okta source - Group_Members return a list of users, the same schema of Users stream. - Create a shared schema users, and both group_members and users sechema use it as a reference. - Add Group_Members stream to source connector * add tests and fix logs schema - fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validatorddb87afad8/jsonschema/_validators.py (L279-L285)- change grouop_members to use id as the cursor field since `filter` is not supported in the query string - fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value - remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api. * last polish before submit the PR - bump docker version - update changelog - add the right abnormal value for logs stream - correct the sample catalog * address comments:: - improve comments for until parameter under the logs stream - add use_cache on groupMembers * add use_cache to Group_Members * change configured_catalog to test * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * split test files * renames * missing unit test * add missing unit tests * rename * assert isinstance * start extracting to their own files * use final instead of classmethod * assert we retry 429 errors * Add log * replace asserts with valueexceptions * delete superfluous print statement * only accept minmaxdatetime * fix factory so we don't need to union everything with strings * get class_name from type * remove from class types registry * process error handlers one at a time * sort * delete print statement * comment * comment * format * delete unused file * comment * interpolatedboolean * comment * not optional * not optional * unit tests * fix request body data * add test * move file to right module * update * reset to master * format * rename to pass_by * rename to page size * fix * fix some tests * reset * fix * fix some of the tests * fix test * fix more tests * all tests pass * path is not optional * reset * reset * reset * delete print * remove prints * delete duplicate method * add test * fix body data * delete extra newlines * move to subpackage * fix imports * handle str body data * simplify * Update tests * filter dates before stream state * Revert "Update tests" This reverts commitc0808c8009. * update * fix test * state management * add test * delete dead code * update cursor * update cursor cartesian * delete unused state class * fix * missing test * update cursor substreams * missing test * fix typing * fix typing * delete unused field * delete unused method * update datetime stream slice * cleanup * assert * request options * request option cartesian * assert when passing by path * request options for substreams * always return a map * pass stream_state * refactor and almost done fixing tests * fix tests * rename to inject_into * only accept enum * delete conditional paginator * only return body data * missing test * update docstrings * update docstrings * update comment * rename * tests * class_name -> type * improve interface * fix some of the tests * fix more of the tests * fix tests * reset * reset * Revert "reset" This reverts commiteb9a918a09. * remove extra argument * docstring * update * delete unused file * reset * reset * rename * fix timewindow * create InterpolatedString * helper method * assert on request option * better asserts * format * docstrings * docstrings * remove optional from type hint * Update airbyte-cdk/python/airbyte_cdk/sources/declarative/stream_slicers/cartesian_product_stream_slicer.py Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * inherit from request options provider * inherit from request options provider * remove optional from type hint * remove extra parameter * none check Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Tobias Macey <tmacey@boundlessnotions.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Bas Beelen <bjgbeelen@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Guy Feldman <gfeldman@86labs.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Yiyang Li <yiyangli2010@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Connector Development Kit (Python)
The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:
- HTTP APIs (REST APIs, GraphQL, etc..)
- Singer Taps
- Generic Python sources (anything not covered by the above)
The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.
This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.
Getting Started
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
cd airbyte-integrations/connector-templates/generator
./generate.sh
then follow the interactive prompt. Next, find all TODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.
Concepts & Documentation
See the concepts docs for a tour through what the API offers.
Example Connectors
HTTP Connectors:
Singer connectors:
Simple Python connectors using the barebones Source abstraction:
Contributing
First time setup
We assume python points to python >=3.9.
Setup a virtual env:
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]" # [dev] installs development-only dependencies
Iteration
- Iterate on the code locally
- Run tests via
pytest -s unit_tests - Perform static type checks using
mypy airbyte_cdk.MyPyconfiguration is in.mypy.ini. - The
type_check_and_test.shscript bundles both type checking and testing in one convenient command. Feel free to use it!
Testing
All tests are located in the unit_tests directory. Run pytest --cov=airbyte_cdk unit_tests/ to run them. This also presents a test coverage report.
Publishing a new version to PyPi
- Bump the package version in
setup.py - Open a PR
- An Airbyte member must comment
/publish-cdk dry-run=trueto publish the package to test.pypi.org or/publish-cdk dry-run=falseto publish it to the real index of pypi.org.
Coming Soon
- Full OAuth 2.0 support (including refresh token issuing flow via UI or CLI)
- Airbyte Java HTTP CDK
- CDK for Async HTTP endpoints (request-poll-wait style endpoints)
- CDK for other protocols
- Don't see a feature you need? Create an issue and let us know how we can help!