jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2026-01-10 09:04:48 -05:00
Code Issues Packages Projects Releases Wiki Activity

GitBook: [master] 161 pages and 75 assets modified

Browse Source
This commit is contained in:
Marcos Marx
2021-05-16 22:51:02 +00:00
committed by gitbook-bot
parent 18c1d6eb1b
commit a18b0e80d0
77 changed files with 1322 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
docs/contributing-to-airbyte/python/tutorials/cdk-tutorial-python-http/2-install-dependencies.md Normal file
View File

@@ -0,0 +1,78 @@
# Step 2: Install Dependencies
Now that you've generated the module, let's navigate to its directory and install dependencies:
```text
cd ../../connectors/source-<name>
python -m venv .venv # Create a virtual environment in the .venv directory
source .venv/bin/activate # enable the venv
pip install -r requirements.txt
```
This step sets up the initial python environment. **All** subsequent `python` or `pip` commands assume you have activated your virtual environment.
Let's verify everything is working as intended. Run:
```text
python main.py spec
```
You should see some output:
```text
{"type": "SPEC", "spec": {"documentationUrl": "https://docsurl.com", "connectionSpecification": {"$schema": "http://json-schema.org/draft-07/schema#", "title": "Python Http Tutorial Spec", "type": "object", "required": ["TODO"], "additionalProperties": false, "properties": {"TODO: This schema defines the configuration required for the source. This usually involves metadata such as database and/or authentication information.": {"type": "string", "description": "describe me"}}}}}
```
We just ran Airbyte Protocol's `spec` command! We'll talk more about this later, but this is a simple sanity check to make sure everything is wired up correctly.
Note that the `main.py` file is a simple script that makes it easy to run your connector. Its invocation format is `python main.py <command> [args]`. See the module's generated `README.md` for the commands it supports.
## Notes on iteration cycle
### Dependencies
Python dependencies for your source should be declared in `airbyte-integrations/connectors/source-<source-name>/setup.py` in the `install_requires` field. You will notice that a couple of Airbyte dependencies are already declared there. Do not remove these; they give your source access to the helper interfaces provided by the generator.
You may notice that there is a `requirements.txt` in your source's directory as well. Don't edit this. It is autogenerated and used to provide Airbyte dependencies. All your dependencies should be declared in `setup.py`.
### Development Environment
The commands we ran above created a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html) for your source. If you want your IDE to auto complete and resolve dependencies properly, point it at the virtual env `airbyte-integrations/connectors/source-<source-name>/.venv`. Also anytime you change the dependencies in the `setup.py` make sure to re-run `pip install -r requirements.txt`.
### Iterating on your implementation
There are two ways we recommend iterating on a source. Consider using whichever one matches your style.
**Run the source using python**
You'll notice in your source's directory that there is a python file called `main.py`. This file exists as convenience for development. You run it to test that your source works:
```text
# from airbyte-integrations/connectors/source-<name>
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json
```
The nice thing about this approach is that you can iterate completely within python. The downside is that you are not quite running your source as it will actually be run by Airbyte. Specifically, you're not running it from within the docker container that will house it.
**Run the source using docker**
If you want to run your source exactly as it will be run by Airbyte \(i.e. within a docker container\), you can use the following commands from the connector module directory \(`airbyte-integrations/connectors/source-python-http-example`\):
```text
# First build the container
docker build . -t airbyte/source-<name>:dev
# Then use the following commands to run it
docker run --rm airbyte/source-python-http-example:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-python-http-example:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-python-http-example:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-python-http-example:dev read --config /secrets/config.json --catalog /sample_files/configured_catalog.json
```
Note: Each time you make a change to your implementation you need to re-build the connector image via `docker build . -t airbyte/source-<name>:dev`. This ensures the new python code is added into the docker container.
The nice thing about this approach is that you are running your source exactly as it will be run by Airbyte. The tradeoff is iteration is slightly slower, as the connector is re-built between each change.