jprdonnelly/airbyte
1
0
Fork 0
mirror of synced 2025-12-25 02:09:19 -05:00
Code Issues Packages Projects Releases Wiki Activity

[#3078] [CDK] Add support for enabling debug from command line and some basic general debug logs (#14521)

Browse Source 
* allow for command line debug option and basic debug statements + declarative

* feedback from pr comments

* fix some tests w/ req/res mixed up and fixing logging tests

* formatting

* pr feedback: cleaning up traces in logger.py and update docs with debug configuration

* remove unneeded trace logger test

* remove extra print statement
This commit is contained in:
Brian Lai
2022-07-13 18:01:07 -04:00
committed by GitHub
parent 1671def1a6
commit 7bff12aea5
11 changed files with 143 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docs/connector-development/cdk-python/README.md
View File

@@ -86,6 +86,18 @@ pip install -e ".[tests]" # [tests] installs test-only dependencies
* Perform static type checks using `mypy airbyte_cdk`. `MyPy` configuration is in `.mypy.ini`.
* The `type_check_and_test.sh` script bundles both type checking and testing in one convenient command. Feel free to use it!
#### Debugging
While developing your connector, you can print detailed debug information during a sync by specifying the `--debug` flag. This allows you to get a better picture of what is happening during each step of your sync.
```text
python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
```
In addition to preset CDK debug statements, you can also add your own statements to emit debug information specific to your connector:
```python
self.logger.debug("your debug message here", extra={"debug_field": self.value})
```
#### Testing
All tests are located in the `unit_tests` directory. Run `pytest --cov=airbyte_cdk unit_tests/` to run them. This also presents a test coverage report.

18
docs/connector-development/tutorials/building-a-python-source.md
View File

@@ -147,6 +147,24 @@ Note: Each time you make a change to your implementation you need to re-build th
The nice thing about this approach is that you are running your source exactly as it will be run by Airbyte. The tradeoff is that iteration is slightly slower, because you need to re-build the connector between each change.
**Detailed Debug Messages**
During development of your connector, you can enable the printing of detailed debug information during a sync by specifying the `--debug` flag. This will allow you to get a better picture of what is happening during each step of your sync.
```text
python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json --debug
```
In addition to the preset CDK debug statements, you can also emit custom debug information from your connector by introducing your own debug statements:
```python
self.logger.debug(
"your debug message here",
extra={
"debug_field": self.value,
"custom_field": your_object.field
}
)
```
**TDD using standard tests**
Airbyte provides a standard test suite that is run against every source. The objective of these tests is to provide some "free" tests that can sanity check that the basic functionality of the source works. One approach to developing your connector is to simply run the tests between each change and use the feedback from them to guide your development.