mirror of synced 2026-01-11 03:04:14 -05:00

Files

Davin Chia 2188201c7c Upgrade to Pip 21.1. (#3070 )

Upgrade to Pip 21.1 in preparation to upgrade DBT 19.

The latest Pip has

- a stricter resolver
- access to more up to date Python wheels (e.g. #2267 is solved since as any version after Pip 21 has access to ready-compiled Python wheels, see https://issues.apache.org/jira/browse/ARROW-11835).
- in theory faster pip installs (https://github.com/pypa/pip/issues/9187#issuecomment-826037738 makes me think the current Pip version is stable enough be consumed)

Also relax the constraint for `requests` and `backoff` since both were actually specifying incompatible versions that the old Pip resolver was silently swallowing.

Test this locally first by running git clean -dfx to remove all the existing venv files. The two different Pip versions do not work well.

2021-04-28 06:40:31 +08:00

base_python

Add template generation for Santa aka CDK. (#3034 )

2021-04-25 18:02:33 +08:00

docs

Proof read CDK Tutorial docs. (#3085 )

2021-04-27 09:36:15 -07:00

unit_tests

Add template generation for Santa aka CDK. (#3034 )

2021-04-25 18:02:33 +08:00

.dockerignore

Integrations Reorganization: Bases (#673 )

2020-10-21 11:23:49 -07:00

build.gradle

improve instructions in templates & publish base docker images to dockerhub (#2441 )

2021-03-15 02:16:36 -07:00

Dockerfile

Santa: Implement OAuth (#3000 )

2021-04-23 10:12:13 +08:00

main_dev.py

Add template generation for Santa aka CDK. (#3034 )

2021-04-25 18:02:33 +08:00

README.md

Add README.md for base-python (#3080 )

2021-04-27 00:30:04 -07:00

requirements.txt

split airbyte protocol generated structs out of base_python (#721 )

2020-10-28 10:03:15 -07:00

setup.py

Upgrade to Pip 21.1. (#3070 )

2021-04-28 06:40:31 +08:00

README.md

Airbyte Python Connector Development Framework (CDK)

The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors. The CDK currently offers helpers specific for creating Airbyte source connectors for:

HTTP APIs (REST APIs, GraphQL, etc..)
Singer Taps
Generic Python sources (anything not covered by the above)

The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.

This document is a general introduction to the CDK. Readers should have basic familiarity with the Airbyte Specification before proceeding.

Getting started

Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run

cd airbyte-integrations/connector-templates/generator
npm run generate

then follow the interactive prompt. Next, find all TODOs in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.

Additionally, you can follow this tutorial for a complete walkthrough of creating an HTTP connector using the Airbyte CDK.

Concepts & Documentation

See the overview docs for a tour through what the API offers.

Airbyte Specification

Find the reference docs for the Airbyte Specification (the interface for how sources and destinations interact) here.

Example Connectors

HTTP Connectors:

Singer connectors:

Simple Python connectors using the barebones Source abstraction: