* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
## What
Finale of https://github.com/airbytehq/airbyte/pull/13122.
We've renamed all directories in previous PRs. Here we remove the fat jar configuration and add publishing to all subprojects.
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
* Remove fat jar configuration.
* Add publishing to all subprojects.
* Migrate OSS to temporal scheduler
* add comment about migration being performed in server
* add comments about removing migration logic
* formatting and add tests for migration logic
* rm duplicated test
* remove more duplicated build task
* remove retry
* disable acceptance tests that call temporal directly when on kube
* set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent
* set default value of container orchestrator enabled to true
* Revert "set default value of container orchestrator enabled to true"
This reverts commit 21b36703a9.
* Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent"
This reverts commit 6dd2ec04a2.
* Revert "Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent""
This reverts commit 2f40f9da50.
* Revert "Revert "set default value of container orchestrator enabled to true""
This reverts commit 26068d5b31.
* fix sync workflow test
* remove defunct cancellation tests due to internal temporal error
* format - remove unused imports
* revert changes that set container orchestrator enabled to true everywhere
* remove NEW_SCHEDULER feature flag from .env files, and set CONTAINER_ORCHESTRATOR_ENABLED flag to true for kube .env files
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
This is adding a metadata entry in order to make the use of the DB secret store to be the default one.
It will avoid having secret outside of the secret table.
Introduce a migration to a secret manager
If a secret manager is specify, it will go though all the config, save the secret in the configured secret store. If the secret is already in a store, it will not migrate the secret to the secret store.
This restore the Json traversal library. A bug was introduce in the Json path library, the PR fix it.
In a json schema we can define an enum without specifying a "type" attribute. It wasn't handle in the previous implemantation. We now return a right type in the getType method and process it the same way than the an integer/boolean/string type.
* Revert "Revert "Build OSS branch for deploying to Cloud env (#11474)""
This reverts commit 55e3c1e051.
* add action to get dev branch tag to OSS project instead of doing it in cloud
* remove dev branch version action, going to do this in cloud afterall
* add VERSION buildArg to Dockerfiles, default to current airbyte version but overwritable
* use VERSION env var consistently as Dockerfile buildArg, jar version, and tag
pass version and image_tag into docker build task function
* add github action for building and pushing an OSS branch for Cloud to consume
* allow AirbyteVersion to validate versions containing 'oss-branch' prefix
* change oss-branch prefix to dev for branch-based versions
* better action name
* add docker-compose-cloud.build.yaml to define minimum set of cloud images that are pushed by oss branch action
* update local dev docs to describe optional usage of VERSION env var
* make branch_version_tag input optional, if not provided, generates dev-<commit_hash>
* fix typo
* fix missed merge conflict
* update docker docs
* update integrationRunner isDev check
The OSS deployment allows to do an export and import of the workspace configuration. If the users is not using a secret manager in their deployment, this will return API key and password in plain text. This is a serious security issue, especially since it has been presented by @timroes that some instance are exposed on the public internet. In order to avoid returning password or other values that should be in a secret we need to sanitize the export. This is what this PR is doing.
* Add Disable Failing Connections feature
* Rename and cleanup
* list jobs based off connection id
* Move variables to env config and update unit tests
* Fix env flag name
* Fix missing name changes
* Add comments to unit test
* Address PR comments
* Support multiple config types
* Update unit tests
* Remove the attemptId notion in the connectionManagerWorkflow (#10780)
This is removing the attemptId from the create attempt activity to replace it with the attemptNumber. This will be modified in the workflow in a later commit.
* Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)
This reverts commit 99338c852a.
* Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)
This reverts commit 892dc7ec66.
* Revert "Revert "Revert "Remove the attemptId notion in the connectionManagerWorkflow (#10780)" (#11057)" (#11073)" (#11081)
This reverts commit e27bb74050.
* Add Disable Failing Connections feature
* Rename and cleanup
* Fix rebase
* only disable if first job is older than max days
* Return boolean for activity
* Return boolean for activity
* Add unit tests for ConnectionManagerWorkflow
* Utilize object output for activity and ignore non success or failed runs
* Utilize object output for activity and ignore non success or failed runs
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
Add a metric to monitor the monitoring app's rate of publishing metrics. Though this isn't perfect, it gives us some insight into whether metric publishing is okay or running into issues.
Refactor for readability.
Create a separate enum class. This enum is added to when a dev needs to add another metric to the reporter.
This helps us isolate the emission logic + scheduling configuration from the actual threads pushing the logic.
* auto-upgrade connectors there are in use with patch version only
* update check version docstring
* remove try/catch from hasNewPatchVersion
* refactor write std defs function
* run format
* add unit test and change exception
* update airbyte version function name to be more clear
* correct unit test in migration tests
* run format
* Pass worker metadata to connector
* Fix compilation
* Pass in job id and image from worker
* Remove application version
* Add default job environment variables
* Add back removed comment
* Rename env map to job metadata
* Fix env configs
* Read connector from application
* Use empty string
* Remove println
* Fix unit test
* Fix compilation error
* Introduce constants for worker env
* Add worker env to ENV_VARS_TO_TRANSFER
* Pass into getWorkerMetadata map to all constructions
* Format code
* Format octavia cli
* Fix test compilation
* Fix typos
* test: new failures metadata for segment tracking
* new failures metadata for segment tracking
failure_reasons: array of all failures (as json objects) for a job
- for general analytics on failures
main_failure_reason: main failure reason (as json object) for this job
- for operational usage (for Intercom)
- currently this is just the first failure reason chronologically
- we'll probably to change this when we have more data on how to
determine failure reasons more intelligently
- added an attempt_id to failures so we can group failures by attempt
- removed stacktrace from failures since it's not clear how we'd use
these in an analytics use case (and because segment has a 32kb size
limit for events)
* remove attempt_id
attempt info is already in failure metadata
* explicitly sort failures array chronologically
* replace "unknown" enums with null
note: ImmutableMaps don't allow nulls
* move sorting to the correct place
* upgrade log4j2 to 2.17.0
* Use colors instead of background.
* Put this back.
* Fix syntax.
* go back to original log colors
* fix comments + remove system property -> env var fallback mechanism
* upgrade all the way to 2.17.1
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: lmossman <lake@airbyte.io>
* add misc todos
* save work so far
* configure async pod processing
* remove comment
* fmt
* working except logging propagation?
* add comment
* add logging and misc configuration fixes
* add output propagation
* fix state reading
* logging is working (but background highlighting is not)
* fix log highlighting
* use sys instead of ctx
* comment
* clean up and test state management
* clean up orchestrator app construction
* unify launcher workers and handle resuming
* respond to comments
* misc
* disable
* fix comment
* respond to comments
* Add a job notification
The new scheduler was missing a notification step after the job is done.
This is needed in order to report the number of record of a sync.
* Acceptance test with the new scheduler
Add a new github action task to run the acceptances test with the new scheduler
* Retry if the failure
* PR comments
- Add comments to the interface methods in Configs.java.
- Add new document on configuring airbyte. Transfer the non internal-only variables to this document.
This getting rid of scheduling with the scheduler app and the job submitter. It is replaced by a temporal workflow which will be responsible to schedule the syncs on time.
* Rename GcsStorageBucket to GcsLogBucket.
* Update all references to GCP_STORAGE_BUCKET to GCS_LOG_BUCKET.
* Undo this for configuration files for older Airbyte versions.
* Clean up Job env vars. (#8462)
* Rename MAX_SYNC_JOB_ATTEMPTS to SYNC_JOB_MAX_ATTEMPTS.
* Rename MAX_SYNC_TIMEOUT_DAYS to SYNC_JOB_MAX_TIMEOUT_DAYS.
* Rename WORKER_POD_TOLERATIONS to JOB_POD_TOLERATIONS.
* Rename WORKER_POD_NODE_SELECTORS to JOB_POD_NODE_SELECTORS.
* Rename JOB_IMAGE_PULL_POLICY to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_POLICY.
* Rename JOBS_IMAGE_PULL_SECRET to JOB_POD_MAIN_CONTAINER_IMAGE_PULL_SECRET.
* Rename JOB_SOCAT_IMAGE to JOB_POD_SOCAT_IMAGE.
* Rename JOB_BUSYBOX_IMAGE to JOB_POD_BUSYBOX_IMAGE.
* Rename JOB_CURL_IMAGE to JOB_POD_CURL_IMAGE.
* Rename KUBE_NAMESPACE to JOB_POD_KUBE_NAMESPACE.
* Rename RESOURCE_CPU_REQUEST to JOB_POD_MAIN_CONTAINER_CPU_REQUEST.
* Rename RESOURCE_CPU_LIMIT to JOB_POD_MAIN_CONTAINER_CPU_LIMIT.
* Rename RESOURCE_MEMORY_REQUEST to JOB_POD_MAIN_CONTAINER_MEMORY_REQUEST.
* Rename RESOURCE_MEMORY_LIMIT to JOB_POD_MAIN_CONTAINER_MEMORY_LIMIT.
* Remove worker suffix from created pods to reduce confusion with actual worker pods.
* Use sync instead of worker to name job pods.
* Source-MySql: transform binary data base64 format, add integration tests
* Source-MySql: fix code style
* Source-MySql: bump versions
* Source-MySql: bump versions in source_specs.yaml
* Source-MySql: added test for stream with binary data for DestinationAbstractTest
* Source-MySql: added format
