jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-05 12:05:28 -05:00
Code Issues Packages Projects Releases Wiki Activity

GitBook: [master] 55 pages and 36 assets modified

Browse Source
This commit is contained in:
John Lafleur
2021-04-05 00:43:26 +00:00
committed by gitbook-bot
parent 0609011e05
commit 55256d293d
85 changed files with 231 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/tutorials/building-a-python-source.md
View File

@@ -66,13 +66,13 @@ Select the `python` template and then input the name of your connector. For this
### Step 2: Build the newly generated source
Build the source by running:
Build the source by running:
```
```text
cd airbyte-integrations/connectors/source-<name>
python -m venv .venv # Create a virtual environment in the .venv directory
source .venv/bin/activate # enable the venv
pip install -r requirements.txt
pip install -r requirements.txt
```
This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
@@ -90,8 +90,7 @@ The generator creates a file `source_<source_name>/source.py`. This will be wher
Python dependencies for your source should be declared in `airbyte-integrations/connectors/source-<source-name>/setup.py` in the `install_requires` field. You will notice that a couple of Airbyte dependencies are already declared there. Do not remove these; they give your source access to the helper interface that is provided by the generator.
You may notice that there is a `requirements.txt` in your source's directory as well. Do not touch this. It is autogenerated and used to provide Airbyte
dependencies. All your dependencies should be declared in `setup.py`.
You may notice that there is a `requirements.txt` in your source's directory as well. Do not touch this. It is autogenerated and used to provide Airbyte dependencies. All your dependencies should be declared in `setup.py`.
#### Development Environment
@@ -123,8 +122,7 @@ The nice thing about this approach is that you can iterate completely within in
**Run the source using docker**
If you want to run your source exactly as it will be run by Airbyte \(i.e. within a docker container\), you can use the following commands from the
connector module directory (`airbyte-integrations/connectors/source-example-python`):
If you want to run your source exactly as it will be run by Airbyte \(i.e. within a docker container\), you can use the following commands from the connector module directory \(`airbyte-integrations/connectors/source-example-python`\):
```text
# First build the container
@@ -161,7 +159,7 @@ The generated code that Airbyte provides, handles implementing the `spec` method
As described in the template code, this method takes in a json object called config that has the values described in the `spec.json` filled in. In other words if the `spec.json` said that the source requires a `username` and `password` the config object might be `{ "username": "airbyte", "password": "password123" }`. It returns a json object that reports, given the credentials in the config, whether we were able to connect to the source. For example, with the given credentials could the source connect to the database server.
While developing, we recommend storing this object in `secrets/config.json`. The `secrets` directory is gitignored by default.
While developing, we recommend storing this object in `secrets/config.json`. The `secrets` directory is gitignored by default.
### Step 6: Implement `discover`
@@ -177,8 +175,7 @@ As described in the template code, this method takes in the same config object a
The Standard Tests are a set of tests that run against all sources. These tests are run in the Airbyte CI to prevent regressions. They also can help you sanity check that your source works as expected. The following [article](../contributing-to-airbyte/building-new-connector/testing-connectors.md) explains Standard Tests and how to run them.
You can run the tests using `./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest`. Make sure to run this command from
the Airbyte repository root.
You can run the tests using `./gradlew :airbyte-integrations:connectors:source-<source-name>:integrationTest`. Make sure to run this command from the Airbyte repository root.
{% hint style="info" %}
In some rare cases we make exceptions and allow a source to not need to pass all the standard tests. If for some reason you think your source cannot reasonably pass one of the tests cases, reach out to us on github or slack, and we can determine whether there's a change we can make so that the test will pass or if we should skip that test for your source.
@@ -186,7 +183,7 @@ In some rare cases we make exceptions and allow a source to not need to pass all
### Step 9: Write unit tests and/or integration tests
The Standard Tests are meant to cover the basic functionality of a source. Think of it as the bare minimum required for us to add a source to Airbyte. In case you need to test additional functionality of your source, write unit or integration tests.
The Standard Tests are meant to cover the basic functionality of a source. Think of it as the bare minimum required for us to add a source to Airbyte. In case you need to test additional functionality of your source, write unit or integration tests.
#### Unit Tests
@@ -196,9 +193,9 @@ You can run the tests using `python -m pytest -s unit_tests`
#### Integration Tests
Place any integration tests in the `integration_tests` directory such that they can be [discovered by pytest](https://docs.pytest.org/en/reorganize-docs/new-docs/user/naming_conventions.html).
Place any integration tests in the `integration_tests` directory such that they can be [discovered by pytest](https://docs.pytest.org/en/reorganize-docs/new-docs/user/naming_conventions.html).
Run integration tests using `python -m pytest -s integration_tests`.
Run integration tests using `python -m pytest -s integration_tests`.
#### Step 10: Update the `README.md`
@@ -206,7 +203,7 @@ The template fills in most of the information for the readme for you. Unless the
#### Step 11: Add the connector to the API/UI
Open the following file: `airbyte-config/init/src/main/resources/seed/source_definitions.yaml`. You'll find a list of all the connectors that Airbyte displays in the UI. Pattern match to add your own connector. Make sure to generate a new _unique_ UUIDv4 for the `sourceDefinitionId` field. You can get one [here](https://www.uuidgenerator.net/).
Open the following file: `airbyte-config/init/src/main/resources/seed/source_definitions.yaml`. You'll find a list of all the connectors that Airbyte displays in the UI. Pattern match to add your own connector. Make sure to generate a new _unique_ UUIDv4 for the `sourceDefinitionId` field. You can get one [here](https://www.uuidgenerator.net/).
Note that for simple and quick testing use cases, you can also do this step [using the UI](../integrations/custom-connectors.md#adding-your-connectors-in-the-ui).