Any Java CDK changes should also trigger tests on JVM connectors whose CDK versions are set to local.
This helps CDK developers understand when they are stepping on connector developer toes when both the CDK and Connectors are being worked on eagerly.
- Modify the get_modified_connectors script to accept a local-cdk flag telling it to 'detect' connectors on local CDK. This is correct behaviour since the Connector has in fact changed, as the CDK has changed, even though the connector doesn't have direct code changes.
- Modify the regular connector ci check workflow to notice cdk changes and test connectors on local CDK.
I experimented with a gradle approach here and concluded it was more suited for local development than CI, since all the tests would be run within a single job which means 1) longer test execution 2) more confusing test output.
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
Modify the publishing workflows for JVM to publish images outside of dagger, while relies on airbyte-ci to continue updating registry/metadata.
This relies on the fact that airbyte-ci first checks for a published image, and if the image is published, skips image publishing and only updates metadata.
One edge case here: airbyte-ci will always try to build a pre-release image, regardless of whether docker images already exists. The changes here will build and publish the JVM image before dagger attempts to do so, so dagger attempts will rely on the cache and get the same result. Eventually we want to remove this behaviour from airbyte-ci. I tried to do so, and it's more complicated, so we'll do this later.
- update the get modified file script to get the connectors modified since the last commit. This support the flow in master.
- create a new script that given a list of connectors replicates airbyte-ci's current behaviour
- if main-release is set (default), build and publish images according to the dockerImageTag in metadata.yaml.
- if pre-release is set, builds and publishes images by taking the dockerImageTag in metadata.yaml and adding -dev.<git-first-10-digits>.
- this supports both the auto-publish flow in master, and the pre-release flows using the publish action.
- update the publish connector GHA workflow to be compatible with all of this.
NOTE: we only support publishing destination-dev-null to start to be safe. This is fine since airbyte-ci will continue to build images if it detects the image does not exist.
Co-authored-by: Aaron ("AJ") Steers <aj@airbyte.io>
