mirror of synced 2026-01-02 21:02:43 -05:00

Files

Alexandre Girard f3799280f2 connector builder: Emit message at start of slice (#25180 )

* Move condition for yielding the slice message to an overwritable method

* Automated Commit - Formatting Changes

* yield the slice log messages

* same for incremental

* refactor

* Revert "refactor"

This reverts commit c594365bd8.

* move flag from factory to source

* set the flag

* remove debug print

* halfmock

* clean up

* Add a test for a single page

* Add another test

* Pass the flag

* rename

---------

Co-authored-by: girarda <girarda@users.noreply.github.com>

2023-04-14 10:23:59 -07:00

__init__.py

Connector Builder: Make connector_builder part of the CDK package (#24280 )

2023-03-21 13:31:16 -04:00

connector_builder_handler.py

connector builder: Emit message at start of slice (#25180 )

2023-04-14 10:23:59 -07:00

main.py

Connector builder: set pages and slices limits (#25121 )

2023-04-12 14:46:43 -07:00

message_grouper.py

Add missing type hints (#24345 )

2023-03-22 07:15:54 -07:00

models.py

Connector builder: Port "send stacktrace when error on read" to CDK connector builder module (#24173 )

2023-03-21 17:22:08 -07:00

README.md

Decouple flags for debug messages from connector builder log messages (#24881 )

2023-04-06 12:16:05 -04:00

README.md

Connector Builder Backend

This is the backend for requests from the Connector Builder.

Local development

Locally running the Connector Builder backend

python main.py read --config path/to/config --catalog path/to/catalog

Note:

Requires the keys __injected_declarative_manifest and __command in its config, where __injected_declarative_manifest is a JSON manifest and __command is one of the commands handled by the ConnectorBuilderHandler (stream_read, list_streams, or resolve_manifest), i.e.

{
  "config": <normal config>,
  "__injected_declarative_manifest": {...},
  "__command": <"resolve_manifest" | "list_streams" | "test_read">
}

*See ConnectionSpecification for details on the "config" key if needed.

When the __command is list_streams or resolve_manifest, the argument to catalog should be an empty string.

Locally running the docker image

Build

First, make sure you build the latest Docker image:

./gradlew airbyte-cdk:python:airbyteDocker

The docker image name and tag, respectively, are the values of the io.airbyte.name and io.airbyte.version LABELs in the Dockerfile.

Run

Then run any of the connector commands as follows:

docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-declarative-manifest:dev read --config /secrets/config.json