* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
* Update platform containers to use non-root users
* Update kube template for the webapp container to use port 8080
After having updated the webbapp nginx image to expose port 8080 instead of 80
* missing 80 -> 8080 changes
Co-authored-by: alafanechere <augustin.lafanechere@gmail.com>
* Rename GcsStorageBucket to GcsLogBucket.
* Update all references to GCP_STORAGE_BUCKET to GCS_LOG_BUCKET.
* Undo this for configuration files for older Airbyte versions.
* Clean up Job env vars. (#8462)
* Rename MAX_SYNC_JOB_ATTEMPTS to SYNC_JOB_MAX_ATTEMPTS.
* Rename MAX_SYNC_TIMEOUT_DAYS to SYNC_JOB_MAX_TIMEOUT_DAYS.
* Rename WORKER_POD_TOLERATIONS to JOB_POD_TOLERATIONS.
* Rename WORKER_POD_NODE_SELECTORS to JOB_POD_NODE_SELECTORS.
* Rename JOB_IMAGE_PULL_POLICY to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_POLICY.
* Rename JOBS_IMAGE_PULL_SECRET to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_SECRET.
* Rename JOB_SOCAT_IMAGE to JOB_POD_SOCAT_IMAGE.
* Rename JOB_BUSYBOX_IMAGE to JOB_POD_BUSYBOX_IMAGE.
* Rename JOB_CURL_IMAGE to JOB_POD_CURL_IMAGE.
* Rename KUBE_NAMESPACE to JOB_POD_KUBE_NAMESPACE.
* Rename RESOURCE_CPU_REQUEST to JOB_POD_MAIN_CONTAINER_CPU_REQUEST.
* Rename RESOURCE_CPU_LIMIT to JOB_POD_MAIN_CONTAINER_CPU_LIMIT.
* Rename RESOURCE_MEMORY_REQUEST to JOB_POD_MAIN_CONTAINER_MEMORY_REQUEST.
* Rename RESOURCE_MEMORY_LIMIT to JOB_POD_MAIN_CONTAINER_MEMORY_LIMIT.
* Remove worker suffix from created pods to reduce confusion with actual worker pods.
* Use sync instead of worker to name job pods.
* improve link checker
* use ref instead of base_ref
* remove base, always compare to master for modified
* add failing to test
* don't do quiet for testing
* switch error to 404 not 403
* yes to both
* turn off verbose mode
* fix
* actually check things
* fix outstanding link problems
* revet change to run for everything
* use new format
* ignore gitbook failures
* switch back to only running on master
* docker-compose split of scheduler and worker
* fix heartbeat location bug + add support for kubernetes
* use two workers in integration tests
* capture logs in AirbyteTestContainer
* add waiting
* rename to make it easier to review
* rename module
* fix remaining conflicts
* allow configuring max workers of each type and document usage
* fix build
* remove comment
* add worker resource requiremetns
* try to fix for connector build
* fix regression in biuld
* add env comments for SUBMITTER_NUM_THREADS
* Update airbyte-workers/src/main/java/io/airbyte/workers/WorkerApp.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* Update airbyte-workers/src/main/java/io/airbyte/workers/temporal/TemporalPool.java
Co-authored-by: Davin Chia <davinchia@gmail.com>
* merge temporalpool into workerapp
* output docker system info
* move check to before
* remove unnecessary parts of the patch
* could this be the problem? i thought i added this
* show disk usage
* add print statements
* add pruning
* fix prune option
* use force
Co-authored-by: Davin Chia <davinchia@gmail.com>
See bluedenim/log4j-s3-search#101 for a summary.
Because this is upstream, I built the jar locally and uploaded it to a temp gcs bucket serving as a maven repository.
I've added this location and the new version in the build.gradle.
I want to merge this in now since there are several users waiting on this. Created #4915 to follow up with the OSS PR and clean up work.
Implement logging to and reading from Minio. Use the same S3 client for this.
Configure Airbyte Kube Prod and Staging to use Minio by default, so Airbyte Kube is a standalone deployment.
Also update documentation to reflect this.
* Making single port 8000 default and update NetworkErrorBoundary.tsx response code 502
* add .env files
* Updating instructions in docs/deploying-airbyte/*
* fix mounting
* svc accounts and other fixes
* remove redundant sync closure call
* switch back to using dev tags and format
* do not overwrite in initContainers
* rename to scheduler-service-account
* add documentation
* fix lack of cp -n on busybox
* use kubectl run, still needs affinity
* add affinity
* downgrade kubectl on scheduler to match docker-desktop latest
* fix overrides
* THE MISISNG PIECE
* add attach permissions
* fmt
* update kube documentation
* put docs on the docs site instead
