* switch to temporal cloud client for now
* format
* use client cert/key env secret instead of path to secret
* add TODO comments
* format
* add logging to debug timeout issue
* add more logging
* change workflow task timeout
* PR feedback: consolidate as much as possible, add missing javadoc
* fix acceptance test, needs to specify localhost
* add internal-use only comments
* format
* refactor to clean up TemporalClient and prepare it for future dependency injection framework
* remove extraneous log statements
* PR feedback
* fix test
* return isInitialized true in test
* sweep all scheduler application code and new-scheduler conditional logic
* remove airbyte-scheduler from deployments and docs
* format
* remove 'v2' from github actions
* add back scheduler in delete deployment command
* remove scheduler parameters from helm chart values
* add back job cleaner + test and add comment
* remove now-unused env vars from code and docs
* format
* remove feature flags from web backend connection handler as it is no longer needed
* remove feature flags from config api as it is now longer needed
* remove feature flags input from config api test
* format + shorter url
* remove scheduler parameters from helm chart readme
## What
Finale of https://github.com/airbytehq/airbyte/pull/13122.
We've renamed all directories in previous PRs. Here we remove the fat jar configuration and add publishing to all subprojects.
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
* Remove fat jar configuration.
* Add publishing to all subprojects.
## What
Part 3 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13478 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
## How
Rename airbyte-scheduler:models to airbyte-scheduler:scheduler-models.
Rename airbyte-scheduler:persistence to airbyte-scheduler:scheduler-persistence.
## What
Part 2 of https://github.com/airbytehq/airbyte/pull/13122.
Follow up to #13476 .
Explanation for what is happening:
Identically named subprojects have the following issues:
* publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
* deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
* the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See https://github.com/gradle/gradle/issues/847 for more info.
* given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Rename airbyte-config:models to airbyte-config:config-models.
* Rename airbyte-config:persistence to airbyte-config:config-persistence.
Part 1 of #13122.
Rename airbyte-db:lib to airbyte-db:db-lib.
Rename airbyte-metrics:lib to airbyte-metrics:metrics-lib
Rename airbyte-protocol:models to airbyte-protocol:protocol-models.
Explanation for what is happening:
Identically named subprojects have the following issues:
- publishing as is leads to classpath confusion when the jars with the same names are placed in the Java distribution. This leads to NoClassDefFound errors on runtime.
- deconflicting the jar names without changing directory names leads to dependency errors as the OSS jar pom files are generated using project dependencies (suggesting a dependency a sibling subproject in the same repo) that use subprojects group and name as a reference. This means the generated jars look for Jars that do not exists (as their names have been changed) and cannot compile.
- the workaround to changing a subproject's name involves resetting the subproject's name in the settings.gradle and depending on the new name in each build.gradle. This increases configuration burden and decreases the ease of reading, since one will have to check the settings.gradle to know what the right subproject name is. See Projects with same name lead to unintended conflict resolution gradle/gradle#847 for more info.
- given that Gradle itself doesn't have support for identically named subprojects (see the linked issue), the simplest solution is to not allow duplicated directories. I've only renamed conflicting directories here to keep things simple. I will create a follow up issues to enforce non-identical subproject names in our builds.
* Display addtional failure information when sync is expanded
* rename
* Update airbyte-webapp/src/components/JobItem/components/FailureDetails.tsx
Co-authored-by: Tim Roes <tim@airbyte.io>
* fix bad merge
* jump to timestamp
* cleat timestamp when logs colapsed
* speed up search
* rename
* Do what @pedroslopez says
* Rebase from master after big API update
* `floor` matchtimes for greater range matching
* Update airbyte-webapp/src/components/JobItem/components/ErrorDetails.tsx
Co-authored-by: Tim Roes <tim@airbyte.io>
* Update airbyte-webapp/src/components/JobItem/components/ErrorDetails.tsx
Co-authored-by: Tim Roes <tim@airbyte.io>
* Update airbyte-webapp/src/components/JobItem/components/Logs.tsx
Co-authored-by: Tim Roes <tim@airbyte.io>
* Update airbyte-webapp/src/components/JobItem/JobItem.tsx
Co-authored-by: Tim Roes <tim@airbyte.io>
* replace regexp with `dayJs`
* Extract dayjs into globals module
* mach time in full-second resolution
* revert `dayJs` and use `Date.parse`
* Just show failure timestamp rather than scroll
Co-authored-by: Tim Roes <tim@airbyte.io>
* Migrate OSS to temporal scheduler
* add comment about migration being performed in server
* add comments about removing migration logic
* formatting and add tests for migration logic
* rm duplicated test
* remove more duplicated build task
* remove retry
* disable acceptance tests that call temporal directly when on kube
* set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent
* set default value of container orchestrator enabled to true
* Revert "set default value of container orchestrator enabled to true"
This reverts commit 21b36703a9.
* Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent"
This reverts commit 6dd2ec04a2.
* Revert "Revert "set NEW_SCHEDULER and CONTAINER_ORCHESTRATOR_ENABLED env vars to true to be consistent""
This reverts commit 2f40f9da50.
* Revert "Revert "set default value of container orchestrator enabled to true""
This reverts commit 26068d5b31.
* fix sync workflow test
* remove defunct cancellation tests due to internal temporal error
* format - remove unused imports
* revert changes that set container orchestrator enabled to true everywhere
* remove NEW_SCHEDULER feature flag from .env files, and set CONTAINER_ORCHESTRATOR_ENABLED flag to true for kube .env files
Co-authored-by: Benoit Moriceau <benoit@airbyte.io>
* Better database connection handling for connectors
* Log connection error
* Properly close connection
* Remove unused method
* Close data source
* Use utility to close data source
* Use utility to close data source
* PR feedback
* Add Databricks driver
* Use driver class enum
* Use correct config
* Ensure config created before use
* Fix failing integration test
* Create DSLContext before use
* Address integration test failures
* Ensure DSLContext is closed
* Fix compile error
* Use correct datasource
* Use correct connection properties
* Close DSLContext
* Close DSLContext
* Fix integration test failures
* Properly close datasource
* Fix compilation issues
* Use existing database object
* Wrap close in try/finally
* Update test
* Wrap close in try/finally
* Ensure DSLContext is created
* Revert change to test
* Use correct data source
* Remove unused import
* More cleanup
* Add missing annotation
* Only initialize data source once
* Remove unused import
* Force testcontainers version
* Fix testcontainer issue
* Fix failing test
* Properly close all data sources
* Clear data sources after closing
* Fix compile error
* Fix compilation error
* Add missing method
* Repair temporal state when performing manual actions
* refactor temporal client and fix tests
* add unreachable workflow exception
* format
* test repeated deletion
* add acceptance tests for automatic workflow repair
* rename and DRY up manual operation methods in SchedulerHandler
* refactor temporal client to batch signal and start requests together in repair case
* add comment
* remove main method
* fix job id fetching
* only overwrite workflowState if reset flags are true on input
* fix test
* fix cancel endpoint
* Clean job state before creating new jobs in connection manager workflow (#12589)
* first working iteration of cleaning job state on first workflow run
* second iteration, with tests
* undo local testing changes
* move method
* add comment explaining placement of clean job state logic
* change connection_workflow failure origin value to platform
* remove cast from new query
* create static var for non terminal job statuses
* change failure origin value to airbyte_platform
* tweak external message wording
* remove unused variable
* reword external message
* fix merge conflict
* remove log lines
* move cleaning job state to beginning of workflow
* do not clean job state if there is already a job id for this workflow, and add test
* see if sleeping fixes test on CI
* add repeated test annotation to protect from flakiness
* fail jobs before creating new ones to protect from quarantined state
* update external message for cleaning job state error
currently on m1 macs, switching between building for arm vs amd64
architectures is a bit cumbersome because some of the base docker images
have not been parameterized yet, so you will run into build errors unless
you untag those base images every time you switch between architectures.
This PR should allow you switch freely between the two without needing
that manual step.
This PR also adds a single env var BUILD_ARCH that can be used to
switch between building for arm vs amd64.
With this PR we can build and push images for individual platform components
which is much faster than trying to redeploy everything when iterating on
changes that are limited to only a few components.
Ideally we'd have a github action that allowed us to deploy individual platform
components, but until that exists this seems like a reasonable solution for faster
iteration.
