## What
Currently the connector image publishing flow use project name/connector
directory to construct the repo name in dockerhub and ignoring the repo
defined in metadata.yml file. It creates inconsistency between
publishing the pulling image flow. This PR is to fix the issue
## How
<!--
* Describe how code changes achieve the solution.
-->
## Review guide
<!--
1. `x.py`
2. `y.py`
-->
## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them.
-->
## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌
## What
Adds automation for syncing evergreen documentation to versioned docs,
and applies it to update version-2.0 docs. Follow-up to #68098 which
reorganized breaking changes documentation.
Requested by @aaronsteers in
https://app.devin.ai/sessions/644caae698024ce99ebb5a34609b785c.
## How
**New Poe Task** (`poe-tasks/repo-root-tasks.toml`):
- Created `poe docs-update-latest-evergreen` task
- Reads latest version from `platform_versions.json` (currently 2.0)
- For each specified directory (`connector-development/`,
`contributing-to-airbyte/`):
- Removes old versioned directory content with `rm -rf`
- Copies fresh content from main docs
- This ensures versioned docs are fully replaced (no stale files remain)
**Version-2.0 Docs Updates**:
- Applied task to sync latest connector development and contribution
guides
- This brought in changes from #68098 including the new
`connector-breaking-changes.md`
- Removed obsolete files that no longer exist in main docs:
- `contributing-to-airbyte/change-cdk-connector.md` (replaced by
connector-breaking-changes.md)
- `contributing-to-airbyte/submit-new-connector.md` (moved to
connector-development/)
## Review Guide
**🔴 CRITICAL - Destructive Operation:**
1. `poe-tasks/repo-root-tasks.toml` (lines 39-57)
- **Review the `rm -rf "$DEST"` approach**: This completely removes
directory contents before copying
- **Verify directory scope is appropriate**: Only
`connector-development/` and `contributing-to-airbyte/` are affected
- **Question: Are these directories truly 100% evergreen?** No
version-specific content should exist in these paths
**🟡 IMPORTANT - File Organization:**
2. Deleted files:
- `contributing-to-airbyte/change-cdk-connector.md` - Verify no incoming
links are broken
- `contributing-to-airbyte/submit-new-connector.md` - Now in
`connector-development/`
3. Link updates:
- `connector-development/README.md` - Updated submit-new-connector link
- `contributing-to-airbyte/README.md` - Updated connector guide
references
- `contributing-to-airbyte/developing-locally.md` - Updated
submit-new-connector link
- Verify all relative paths are correct from new locations
**🟢 MINOR - Documentation Content:**
4. `connector-development/connector-breaking-changes.md` - New
comprehensive guide (from #68098)
5. `connector-development/local-connector-development.md` - Added
breaking changes reference
## User Impact
**Positive:**
- Version-2.0 docs now include updated breaking changes guidance from
#68098
- Automated task reduces manual effort for future doc syncs
- Contributors viewing versioned docs get accurate connector development
information
**Potential Issues:**
- Task must be run manually after main docs updates (no automated
trigger)
- If any version-specific content exists in synced directories, it will
be incorrectly overwritten
- The `rm -rf` approach is aggressive - any mistake in directory paths
could be destructive
## Can this PR be safely reverted and rolled back?
- [x] YES 💚
Pure documentation changes. Reverting removes the Poe task and reverts
versioned docs to previous state. Main (latest) docs are unaffected.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The validate-connector-metadata.sh script was failing for strict-encrypt connectors because it was looking for documentation files like "postgres-strict-encrypt.md" when these connectors actually share documentation with their base connector (e.g., "postgres.md").
🤖 Generated with Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
We previously removed the whitelist here since all JVM connectors were migrated.
However this script doesn't work for non-JVM connectors for now. Fix this to only build images for JVM connectors.
Now that we have fully moved to gradle for JVM we can remove Python and airbyte-ci dependencies from the connectors build system to simplify Java connector builds and eliminate unnecessary Python toolchain requirements.
Any Java CDK changes should also trigger tests on JVM connectors whose CDK versions are set to local.
This helps CDK developers understand when they are stepping on connector developer toes when both the CDK and Connectors are being worked on eagerly.
- Modify the get_modified_connectors script to accept a local-cdk flag telling it to 'detect' connectors on local CDK. This is correct behaviour since the Connector has in fact changed, as the CDK has changed, even though the connector doesn't have direct code changes.
- Modify the regular connector ci check workflow to notice cdk changes and test connectors on local CDK.
I experimented with a gradle approach here and concluded it was more suited for local development than CI, since all the tests would be run within a single job which means 1) longer test execution 2) more confusing test output.
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
In response to #61570.
It looks like we publish images on merge to master even if version numbers are not bumped. I'm going to revert this for now and investigate.
What
A bug currently exists in the publishing flow for JVM connectors - we always build and publish connectors on changes.
This is fine if the connector's version actually changed. Some destinations use requireVersionIncrementsInPullRequests to false to allow for rapid iteration without publishing versions.
Because of this, we want to add logic here to only publish images if the image does not exists.
How
Curl dockerhub to see if the image exists. If it does, do not publish.
I also added a --publish flag and defaulted the script to dry-run for better testing. This was previously requested, and I though I'd do so since I was here.
After this PR all JVM destinations will be on the new non-dagger flow.
Note on the failed builds:
destination-csv - docker only, which is why the tests are failing. I am going to deprecate this if I cannot convince Junie to fix this for me.
destination-local-json - same as destination csv.
destination-postgres-strict-encrypt - passing for me locally, looks like a certificate error
destination-postgres - passing for me locall
destination-oracle - these are failing on container errors - the test containers they are trying to download aren't able to start up
destination-oracle-strict-encrypt - same as base oracle
destination-starburst-galaxy - actual test failures
destination-yellowbrick - actual test failures
destination-redis - actual test failures
The failed tests mirror master so we aren't adding any new failures here.
