09fce77107
* add specs module with logic to fetch specs on build * format + build and add gradle dependency for new script * check seed file for existing specs + refactor * add tests + a bit more refactoring * run gw format * update yaml config persistence to merge specs into definitions * add comment * add dep * add tests for GcsBucketSpecFetcher * get rid of static block + format * DRY up parse call * add GCS details to comment * formatting + fix test * update comment * do not format seed specs files * change signature of run to allow cloud to reuse this script * run gw format * revert commits that change signature of run * fix comment typo Co-authored-by: Davin Chia <davinchia@gmail.com> * rename enum to be distinct from the enum in cloud * add missing dependencies between modules * add readme for seed connector spec generator * reword * reference readme in comment * ignore 'spec' field in newFields logic * rearrange dependencies so that CONNECTORS_BASE build does not depend on SeedConnectorSpecGenerator * run format * add some more helpful info to the GCS fetch failure message * add more info * get rid of unnecessary static block * Fix publishing docs (#7589) * Fix publishing docs * Reorder steps and add a comment about rebuilding the platform * Update README.md Co-authored-by: Lake Mossman <lake@airbyte.io> * add dependency and rebuild * update PR template with seed connector generation steps * revert formatting changes to PR template * Update build.gradle * Remove unnecessary dep Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
1.4 KiB
Generating Seed Connector Specs
The catalog of seeded connector definitions is stored and manually updated in the airbyte-config/init/src/main/resources/seed/*_definitions.yaml
files. These manually-maintained connector definitions intentionally do not contain the connector specs, in an effort to keep these files
human-readable and easily-editable, and because specs can be automatically fetched.
This automatic fetching of connector specs is the goal of the SeedConnectorSpecGenerator. This class reads the connector definitions in
the airbyte-config/init/src/main/resources/seed/*_definitions.yaml files, fetches the corresponding specs from the GCS bucket cache, and writes the
specs to the airbyte-config/init/src/main/resources/seed/*_specs.yaml files. See the
SeedConnectorSpecGenerator class for more details.
Therefore, whenever a connector definition is updated in the airbyte-config/init/src/main/resources/seed/*_definitions.yaml files, the
SeedConnectorSpecGenerator should be re-ran to generate the updated connector specs files. To do so,
run ./gradlew :airbyte-config:init:processResources, or just build the platform project, and commit the changes to your PR. If you do not do this,
the build in the CI will fail because there will be a diff in the generated files as you have not checked in the changes that were applied by the
generator.