mirror of synced 2026-01-04 18:04:31 -05:00

Files

LiRen Tu 973f0b1165 Make connector adaptable based on deployment mode (#13522 )

* Add deployment mode to env shared with jobs

* Add adaptive runners

* Migrate postgres source to use adaptive runner

* Add an array of specs in docker image spec definition

* Add copyright

* Parse docker image spec with specs list

* Update spec yaml files

* Pass in DEPLOYMENT_MODE to docker compose file

* Revert "Parse docker image spec with specs list"

This reverts commit 8fe41dd3b7.

* Revert changes in docker image spec

* Read cloud specific spec files based on deployment mode

* Revert "Update spec yaml files"

This reverts commit 059f326432.

* Publish cloud spec file if necessary

* Fix upload script

* Move test files

* Update docker compose file

* Format code

* Add comment about spec filename

* Add unit tests

* Remove redundant jdbc acceptance test

When running `PostgresStrictEncryptJdbcSourceAcceptanceTest`, the `discover` method tests always fail because there are unexpected columns in the catalog:
- `wakeup_at`
- `last_visited_at`
- `last_comment_at`

These columns only exist in `PostgresJdbcSourceAcceptanceTest`. And this failure cannot be reproduced locally.

The hypothesis is that when the JDBC unit tests are run on CI, they are run in parallel, and the same testcontainer is used for both tests. That's why the strict encrypt test can discover columns from the oridinary unit test.

Given that the JDBC strict encrypt test is basically redundant, it is removed.

2022-06-15 08:23:54 -07:00

src

Make connector adaptable based on deployment mode (#13522 )

2022-06-15 08:23:54 -07:00

build.gradle

Start publishing proper artifacts. (#13484 )

2022-06-06 17:15:25 +08:00

README.md

REDO: Generate seed connector specs on build (#7613 )

2021-11-04 17:06:16 -07:00

README.md

Generating Seed Connector Specs

The catalog of seeded connector definitions is stored and manually updated in the airbyte-config/init/src/main/resources/seed/*_definitions.yaml files. These manually-maintained connector definitions intentionally do not contain the connector specs, in an effort to keep these files human-readable and easily-editable, and because specs can be automatically fetched.

This automatic fetching of connector specs is the goal of the SeedConnectorSpecGenerator. This class reads the connector definitions in the airbyte-config/init/src/main/resources/seed/*_definitions.yaml files, fetches the corresponding specs from the GCS bucket cache, and writes the specs to the airbyte-config/init/src/main/resources/seed/*_specs.yaml files. See the SeedConnectorSpecGenerator class for more details.

Therefore, whenever a connector definition is updated in the airbyte-config/init/src/main/resources/seed/*_definitions.yaml files, the SeedConnectorSpecGenerator should be re-ran to generate the updated connector specs files. To do so, run ./gradlew :airbyte-config:init:processResources, or just build the platform project, and commit the changes to your PR. If you do not do this, the build in the CI will fail because there will be a diff in the generated files as you have not checked in the changes that were applied by the generator.