Concurrency limit is based on a counter that increment and decrement the limit each time a flow is started and terminated.
This count should always be accurate.
But if some unexpected event occurs (bug or user manually do something wrong), the count may not be accurate anymore.
To avoid any potential issue, when we decrement the counter, we chech that concurrency count is bellow the limit before unqueing an execution.
Fixes#12031
Closes #13301
If a test mis-behave, for ex starting an execution but not terminating it, as the default termination grace period is 5mn it can take very long time to wait for post-test terminaison.
Switching to a termination grace period of 5s may help.
I also detect that the ExecutionControllerRunner test when launching the test suite, would not properly kill the `sleep-long` flow so waiting for it to complete, or the termination grace period. When a test that use this flow is launched separatly it works properly. As a safety net I reduce the sleep from 5mn to 30s.
When computing the next tasks to run, all task runs are created in the CREATED state.
Then when computed tasks to send to the worker, CREATED task runs are listed and converted into worker task.
The issue is that on the next execution message, if tasks sent to the worker are still in CREATED (for ex because the Worker didn't start them yet), they would still be evaluted as to send to the worker.
Setting them to a new SUBMITTED state would prevent them to be taken into account again until they are really terminated.
This should avoid the deduplicateWorkerTask state but this is kept for now with a warning and would be removed later if it proves to work in all cases.
A counter allow to lock by flow which solves the race when two executions are created at the same time and the executoion_runnings table is empty.
Evaluating concurrency limit on the main executionQueue method also avoid an unexpected behavior where the CREATED execution is processed twice as its status didn't change immediatly when QUEUED.
Closes https://github.com/kestra-io/kestra-ee/issues/4877
By evaluating multiple condition in a separate queue, we serialize their evaluation which avoir races when we compute the outputs for flow triggers.
This is because evaluation is a multi step process: first you get the existing condtion, then you evaluate, then you store the result. As this is not guarded by a lock you must not do it concurrently.
The race can still occurs if muiltiple executors run but this is less probable. A re-implementation would be needed probably in 2.0 for that.
Fixes https://github.com/kestra-io/kestra-ee/issues/4602
* feat: implement Flows as a DataSource for dashboards
* chore: review changes
* fix: method signature changes from another commit apply in new flow fetchData method
* feat(tests): rework runner utils to not use the queue during testing
* feat(tests): rework runner utils to not use the queue during testing
* test: rework RetryCaseTest to not rely on executionQueue
* fix(tests): don't catch the Queue exception
* fix(tests): don't catch the Queue exception
* fix compile
* fix(test): concurrency error and made runner test parallel ready
* fix(tests): remove test instance
* feat(tests): use Test Runner Utils
* fix(tests): flaky tests
* fix(test): flaky tests
* feat(tests): rework runner utils to not use the queue during testing
* feat(tests): rework runner utils to not use the queue during testing
* test: rework RetryCaseTest to not rely on executionQueue
* fix(tests): don't catch the Queue exception
* fix(tests): don't catch the Queue exception
* fix compile
* fix(test): concurrency error and made runner test parallel ready
* fix(tests): remove test instance
* feat(tests): use Test Runner Utils
* fix(tests): flaky tests
* fix(test): flaky tests
* fix(tests): flaky set test
* fix(tests): remove RunnerUtils
* fix(tests): fix flaky
* feat(test): rework runner tests to remove the queue usage
* feat(test): fix a flaky and remove parallelism from mysql test suit
* fix(tests): flaky tests
* clean(tests): unwanted test
* add debug exec when fail
* feat(tests): add thread to mysql thread pool
* fix(test): flaky and disable a test
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: Roman Acevedo <roman.acevedo62@gmail.com>
* test: make AbstractExecutionRepositoryTest parallelizable
* feat(tests): play jdbc h2 tests in parallel
* fix(tests): failing unit tests
* tests: add await until timeout on some tests
* fix(tests): failing unit tests
* fix(tests): failing unit tests
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: Nicolas K. <nk_mikmak@hotmail.com>
This table contains executions for flows that have a concurrency that are currently running.
It has been added in 0.24 but in that release there was a bug that may prevent some records to being correctly removed from this table.
To fix that, we truncate it once.
This occurs in Postgres using the `\u0000` unicode sequence. Postgres refuse to store any JSONB with this sequence as it has no textual representation.
We now properly detect that and fail the task.
Fixes#10326
This is due to the fact that we now process the execution queue concurrently so there is a race when counting currently running executions. This can be seen easily using a ForEachItem as it could create tens or hundreds of executions almost instantly leading to almost all those executions started as they would all see 0 executions running...
Using a dedicated execution running queue, as done in EE, would serialize the messages and fix the issue.
However, if using multiple executor instances and concurrency limit = 1, there is a theoretical race as no locks will be done if no execution is running. A max surge of executions could be as high as the number of executor but this race is less probable to happen in real world scenario.
Fixes#10167
- When creating an execution, you can pass a breakpoint of the form `taskId.value` and an execution kind.
- An execution with a breakpoint will be suspended in the `BREAKPOINT` state when arriving at the point where the breakpoint task should be executed
- You can resume an execution from a breakpoint, this would resume the execution and remove the existing breakpoint. At this time a new breakpoint can be passed.
- You can pass a breakpoint when replaying an execution.
Part-of: https://github.com/kestra-io/kestra-ee/issues/1547
