airbyte/docs/connector-development

Connector Development

Airbyte supports two types of connectors: Sources and Destinations. A connector takes the form of a Docker image which follows the Airbyte specification.

To build a new connector in Java or Python, we provide templates so you don't need to start everything from scratch.

Note: you are not required to maintain the connectors you create. The goal is that the Airbyte core team and the community help maintain the connector.

Airbyte provides some Connector Development Kits (CDKs) to help you build connectors.

If you need help from our team for connector development, we offer premium support to our open-source users, talk to our team to get access to it.

Connector builder UI

The connector builder UI is based on the low-code development framework below and allows to develop and use connectors without leaving the Airbyte UI (no local development environment required).

Low-code Connector-Development Framework

You can use the low-code framework to build source connectors for REST APIs by modifying boilerplate YAML files.

Python Connector-Development Kit (CDK)

You can build a connector very quickly in Python with the Airbyte CDK, which generates 75% of the code required for you.

Community maintained CDKs

The Airbyte community also maintains some CDKs:

• The Typescript CDK is actively maintained by Faros.ai for use in their product.
• The Airbyte Dotnet CDK comes with C# templates which can be used to generate 75% of the code required for you

The Airbyte specification

Before building a new connector, review Airbyte's data protocol specification.

Adding a new connector

Requirements

To add a new connector you need to:

1. Implement & Package your connector in an Airbyte Protocol compliant Docker image
2. Add integration tests for your connector. At a minimum, all connectors must pass Airbyte's standard test suite, but you can also add your own tests.
3. Document how to build & test your connector
4. Publish the Docker image containing the connector

Each requirement has a subsection below.

1. Implement & package the connector

If you are building a connector in any of the following languages/frameworks, then you're in luck! We provide autogenerated templates to get you started quickly:

Sources

• Python Source Connector
• Singer-based Python Source Connector. Singer.io is an open source framework with a large community and many available connectors (known as taps & targets). To build an Airbyte connector from a Singer tap, wrap the tap in a thin Python package to make it Airbyte Protocol-compatible. See the Github Connector for an example of an Airbyte Connector implemented on top of a Singer tap.
• Generic Connector: This template provides a basic starting point for any language.

Destinations

• Java Destination Connector
• Python Destination Connector

Creating a connector from a template

Run the interactive generator:

cd airbyte-integrations/connector-templates/generator
./generate.sh

and choose the relevant template by using the arrow keys. This will generate a new connector in the airbyte-integrations/connectors/<your-connector> directory.

Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial:

• Speedrun: Building a HTTP source with the CDK
• Building a HTTP source with the CDK
• Building a Python source
• Building a Python destination
• Building a Java destination

As you implement your connector, make sure to review the Best Practices for Connector Development guide. Following best practices is not a requirement for merging your contribution to Airbyte, but it certainly doesn't hurt ;)

2. Integration tests

At a minimum, your connector must implement the acceptance tests described in Testing Connectors

Note: Acceptance tests are not yet available for Python destination connectors. Coming soon!

3. Document building & testing your connector

If you're writing in Python or Java, skip this section -- it is provided automatically.

If you're writing in another language, please document the commands needed to:

1. Build your connector docker image (usually this is just docker build . but let us know if there are necessary flags, gotchas, etc..)
2. Run any unit or integration tests in a Docker image.

Your integration and unit tests must be runnable entirely within a Docker image. This is important to guarantee consistent build environments.

When you submit a PR to Airbyte with your connector, the reviewer will use the commands you provide to integrate your connector into Airbyte's build system as follows:

1. :airbyte-integrations:connectors:source-<name>:build should run unit tests and build the integration's Docker image
2. :airbyte-integrations:connectors:source-<name>:integrationTest should run integration tests including Airbyte's Standard test suite.

4. Publish the connector

Typically this will be handled as part of code review by an Airbyter. There is a section below on what steps are needed for publishing a connector and will mostly be used by Airbyte employees publishing the connector.

Updating an existing connector

The steps for updating an existing connector are the same as for building a new connector minus the need to use the autogenerator to create a new connector. Therefore the steps are:

1. Iterate on the connector to make the needed changes
2. Run tests
3. Add any needed docs updates
4. Create a PR to get the connector published

Adding normalization to a connector

In order to enable normalization for a destination connector, you'll need to set some fields on the destination definitions entry for the connector. This is done in the metadata.yaml file found at the root of each connector.

Here's an example of normalization fields being set to enable normalization for the Postgres destination:

data:
    # ... other fields
    normalizationConfig:
        normalizationRepository: airbyte/normalization
        normalizationTag: 0.2.25
        normalizationIntegrationType: postgres

For more information about what these fields mean, see the NormalizationDestinationDefinitionConfig schema.

The presence of these fields will enable normalization for the connector, and determine which docker image will run.

Publishing a connector

Once you've finished iterating on the changes to a connector as specified in its README.md, follow these instructions to ship the new version of the connector with Airbyte out of the box.

1. Bump the version in the Dockerfile of the connector (LABEL io.airbyte.version=X.X.X).
2. Bump the docker image version in the metadata.yaml of the connector.
3. Submit a PR containing the changes you made.
4. One of Airbyte maintainers will review the change in the new version and make sure the tests are passing.
5. You our an Airbyte maintainer can merge the PR once it is approved and all the required CI checks are passing you.
6. Once the PR is merged the new connector version will be published to DockerHub and the connector should now be available for everyone who uses it. Thank you!

Updating Connector Metadata

When a new (or updated version) of a connector is ready, our automations will check your branch for a few things:

• Does the connector have an icon?
• Does the connector have documentation and is it in the proper format?
• Does the connector have a changelog entry for this version?
• The metadata.yaml file is valid.

If any of the above are failing, you won't be able to merge your PR or publish your connector.

Connector icons should be square SVGs and be located in this directory.

Connector documentation and changelogs are markdown files living either here for sources, or here for destinations.

Using credentials in CI

In order to run integration tests in CI, you'll often need to inject credentials into CI. There are a few steps for doing this:

1. Place the credentials into Google Secret Manager(GSM): Airbyte uses a project 'Google Secret Manager' service as the source of truth for all CI secrets. Place the credentials exactly as they should be used by the connector into a GSM secret here i.e.: it should basically be a copy paste of the config.json passed into a connector via the --config flag. We use the following naming pattern: SECRET_<capital source OR destination name>_CREDS e.g: SECRET_SOURCE-S3_CREDS or SECRET_DESTINATION-SNOWFLAKE_CREDS.
2. Add the GSM secret's labels:
   • connector (required) -- unique connector's name or set of connectors' names with '_' as delimiter i.e.: connector=source-s3, connector=destination-snowflake
   • filename (optional) -- custom target secret file. Unfortunately Google doesn't use '.' into labels' values and so Airbyte CI scripts will add '.json' to the end automatically. By default secrets will be saved to ./secrets/config.json i.e: filename=config_auth => secrets/config_auth.json
3. Save a necessary JSON value Example.
4. That should be it.

Access CI secrets on GSM

Access to GSM storage is limited to Airbyte employees. To give an employee permissions to the project:

1. Go to the permissions' page
2. Add a new principal to dataline-integration-testing:

• input their login email
• select the role Development_CI_Secrets

3. Save