Use Cloudrepo.io to host temporary built jars. (#4949)

Continuation of my saga to publish a public jar while we wait for the upstream fix to be merged in.

Turns out using the GCS bucket as a public maven repository still requires GCS login. This is not feasible since it requires all our community contributors to have and sign into gcloud. This also required injecting this everywhere in the build and was extremely ugly.

We could publish to the central maven repository, but it takes 2 days for us to get an OSS account. We'd also have to dedup on artifact id.
I explored GCP's Artifact Registry, but I didn't manage to set it up after an hour. It is also only possible to upload via Maven/Gradle and there were a lot of steps to jump through to authenticate.
I went with CloudRepo as they were the best in term of set up, UX and performance. I was able to set something up in 10 mins. I was able to upload my Jars manually via the UI. CoudRepo also supports public read-only repository, which is exactly what we need (publish is still gated on authentication). Ran into a slight hiccup and support got back to me in 30 mins. Frankly think we should consider using them for our private cloud. They are also fairly cheap. Anyone who wants to upload jars will have to use credentials that I've added to shared-engineering folder. Just navigate to the CloudRepo url.

.github/workflows/gradle.yml

@@ -89,14 +89,11 @@ jobs:
       - name: Generate Template scaffold
         run: ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates --scan
 
-      # TODO(Issue-4915): Remove this once fix has been merged into upstream library. Rely on caching and only load credentials once.
       - name: Format
-        run: ./tools/bin/setup_gcs_creds.sh && GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" SUB_BUILD=CONNECTORS_BASE ./gradlew format --scan --info --stacktrace
-        env:
-          GOOGLE_CLOUD_STORAGE_CREDS: ${{ secrets.GOOGLE_CLOUD_STORAGE_TEST_CREDS }}
+        run: SUB_BUILD=CONNECTORS_BASE ./gradlew format --scan --info --stacktrace
 
       - name: Build
-        run: GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" SUB_BUILD=CONNECTORS_BASE ./gradlew build --scan
+        run: SUB_BUILD=CONNECTORS_BASE ./gradlew build --scan
 
       - name: Ensure no file change
         run: git status --porcelain && test -z "$(git status --porcelain)"
@@ -223,17 +220,14 @@ jobs:
       - name: Install Pyenv
         run: python3 -m pip install virtualenv==16.7.9 --user
 
-      # TODO(Issue-4915): Remove this once fix has been merged into upstream library. Rely on caching and only load credentials once.
       - name: Format
-        run: ./tools/bin/setup_gcs_creds.sh && GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" SUB_BUILD=PLATFORM ./gradlew format --scan --info --stacktrace
-        env:
-          GOOGLE_CLOUD_STORAGE_CREDS: ${{ secrets.GOOGLE_CLOUD_STORAGE_TEST_CREDS }}
+        run: SUB_BUILD=PLATFORM ./gradlew format --scan --info --stacktrace
 
       - name: Ensure no file change
         run: git status --porcelain && test -z "$(git status --porcelain)"
 
       - name: Build
-        run: GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" && SUB_BUILD=PLATFORM ./gradlew build --scan
+        run: SUB_BUILD=PLATFORM ./gradlew build --scan
 
       - name: Ensure no file change
         run: git status --porcelain && test -z "$(git status --porcelain)"
@@ -348,11 +342,8 @@ jobs:
       - name: Install Cypress Test Dependencies
         run: sudo apt-get update && sudo apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb
 
-      # TODO(Issue-4915): Remove this once fix has been merged into upstream library. Rely on caching and only load credentials once.
       - name: Build Platform Docker Images
-        run: ./tools/bin/setup_gcs_creds.sh && GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" SUB_BUILD=PLATFORM ./gradlew --no-daemon composebuild --scan
-        env:
-          GOOGLE_CLOUD_STORAGE_CREDS: ${{ secrets.GOOGLE_CLOUD_STORAGE_TEST_CREDS }}
+        run: SUB_BUILD=PLATFORM ./gradlew --no-daemon composebuild --scan
 
       - name: Run End-to-End Frontend Tests
         run: ./tools/bin/e2e_test.sh
@@ -446,11 +437,8 @@ jobs:
           HOME: /home/runner
           CHANGE_MINIKUBE_NONE_USER: true
 
-      # TODO(Issue-4915): Remove this once fix has been merged into upstream library. Rely on caching and only load credentials once.
       - name: Build Platform Docker Images
-        run: ./tools/bin/setup_gcs_creds.sh && GOOGLE_APPLICATION_CREDENTIALS="/tmp/gcs.json" SUB_BUILD=PLATFORM ./gradlew composeBuild --scan
-        env:
-          GOOGLE_CLOUD_STORAGE_CREDS: ${{ secrets.GOOGLE_CLOUD_STORAGE_TEST_CREDS }}
+        run: SUB_BUILD=PLATFORM ./gradlew composeBuild --scan
 
       - name: Run Logging Tests
         run: ./tools/bin/cloud_storage_logging_test.sh

build.gradle

@@ -152,6 +152,10 @@ subprojects {
 
     repositories {
         mavenCentral()
+        maven {
+            // TODO(Issue-4915): Remove this when upstream is merged in.
+            url 'https://airbyte.mycloudrepo.io/public/repositories/airbyte-public-jars/'
+        }
     }
 
     pmd {
@@ -213,7 +217,7 @@ subprojects {
         implementation 'org.slf4j:log4j-over-slf4j:1.7.30' // log4j1.2 bridge
 
         // Dependencies for logging to cloud storage, as well as the various clients used to do so.
-        implementation 'com.therealvan:appender-log4j2:3.1.0'
+        implementation 'com.therealvan:appender-log4j2:3.1.2-SNAPSHOT'
         implementation 'com.amazonaws:aws-java-sdk-s3:1.12.6'
         implementation 'com.google.cloud:google-cloud-storage:1.115.0'
 

tools/bin/setup_gcs_creds.sh

@@ -1,6 +0,0 @@
-#!/usr/bin/env bash
-
-# TODO(Issue-4915): Remove this hook as part of #4915.
-
-echo "$GOOGLE_CLOUD_STORAGE_CREDS" > "/tmp/gcs.json"
-