Concurrency limit is based on a counter that increment and decrement the limit each time a flow is started and terminated.
This count should always be accurate.
But if some unexpected event occurs (bug or user manually do something wrong), the count may not be accurate anymore.
To avoid any potential issue, when we decrement the counter, we chech that concurrency count is bellow the limit before unqueing an execution.
Fixes#12031
Closes #13301
If a test mis-behave, for ex starting an execution but not terminating it, as the default termination grace period is 5mn it can take very long time to wait for post-test terminaison.
Switching to a termination grace period of 5s may help.
I also detect that the ExecutionControllerRunner test when launching the test suite, would not properly kill the `sleep-long` flow so waiting for it to complete, or the termination grace period. When a test that use this flow is launched separatly it works properly. As a safety net I reduce the sleep from 5mn to 30s.
When computing the next tasks to run, all task runs are created in the CREATED state.
Then when computed tasks to send to the worker, CREATED task runs are listed and converted into worker task.
The issue is that on the next execution message, if tasks sent to the worker are still in CREATED (for ex because the Worker didn't start them yet), they would still be evaluted as to send to the worker.
Setting them to a new SUBMITTED state would prevent them to be taken into account again until they are really terminated.
This should avoid the deduplicateWorkerTask state but this is kept for now with a warning and would be removed later if it proves to work in all cases.
A counter allow to lock by flow which solves the race when two executions are created at the same time and the executoion_runnings table is empty.
Evaluating concurrency limit on the main executionQueue method also avoid an unexpected behavior where the CREATED execution is processed twice as its status didn't change immediatly when QUEUED.
Closes https://github.com/kestra-io/kestra-ee/issues/4877
By evaluating multiple condition in a separate queue, we serialize their evaluation which avoir races when we compute the outputs for flow triggers.
This is because evaluation is a multi step process: first you get the existing condtion, then you evaluate, then you store the result. As this is not guarded by a lock you must not do it concurrently.
The race can still occurs if muiltiple executors run but this is less probable. A re-implementation would be needed probably in 2.0 for that.
Fixes https://github.com/kestra-io/kestra-ee/issues/4602
* feat: implement Flows as a DataSource for dashboards
* chore: review changes
* fix: method signature changes from another commit apply in new flow fetchData method
* feat(tests): rework runner utils to not use the queue during testing
* feat(tests): rework runner utils to not use the queue during testing
* test: rework RetryCaseTest to not rely on executionQueue
* fix(tests): don't catch the Queue exception
* fix(tests): don't catch the Queue exception
* fix compile
* fix(test): concurrency error and made runner test parallel ready
* fix(tests): remove test instance
* feat(tests): use Test Runner Utils
* fix(tests): flaky tests
* fix(test): flaky tests
* feat(tests): rework runner utils to not use the queue during testing
* feat(tests): rework runner utils to not use the queue during testing
* test: rework RetryCaseTest to not rely on executionQueue
* fix(tests): don't catch the Queue exception
* fix(tests): don't catch the Queue exception
* fix compile
* fix(test): concurrency error and made runner test parallel ready
* fix(tests): remove test instance
* feat(tests): use Test Runner Utils
* fix(tests): flaky tests
* fix(test): flaky tests
* fix(tests): flaky set test
* fix(tests): remove RunnerUtils
* fix(tests): fix flaky
* feat(test): rework runner tests to remove the queue usage
* feat(test): fix a flaky and remove parallelism from mysql test suit
* fix(tests): flaky tests
* clean(tests): unwanted test
* add debug exec when fail
* feat(tests): add thread to mysql thread pool
* fix(test): flaky and disable a test
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: Roman Acevedo <roman.acevedo62@gmail.com>
* test: make AbstractExecutionRepositoryTest parallelizable
* feat(tests): play jdbc h2 tests in parallel
* fix(tests): failing unit tests
* tests: add await until timeout on some tests
* fix(tests): failing unit tests
* fix(tests): failing unit tests
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: Nicolas K. <nk_mikmak@hotmail.com>
This table contains executions for flows that have a concurrency that are currently running.
It has been added in 0.24 but in that release there was a bug that may prevent some records to being correctly removed from this table.
To fix that, we truncate it once.
This occurs in Postgres using the `\u0000` unicode sequence. Postgres refuse to store any JSONB with this sequence as it has no textual representation.
We now properly detect that and fail the task.
Fixes#10326
This is due to the fact that we now process the execution queue concurrently so there is a race when counting currently running executions. This can be seen easily using a ForEachItem as it could create tens or hundreds of executions almost instantly leading to almost all those executions started as they would all see 0 executions running...
Using a dedicated execution running queue, as done in EE, would serialize the messages and fix the issue.
However, if using multiple executor instances and concurrency limit = 1, there is a theoretical race as no locks will be done if no execution is running. A max surge of executions could be as high as the number of executor but this race is less probable to happen in real world scenario.
Fixes#10167
- When creating an execution, you can pass a breakpoint of the form `taskId.value` and an execution kind.
- An execution with a breakpoint will be suspended in the `BREAKPOINT` state when arriving at the point where the breakpoint task should be executed
- You can resume an execution from a breakpoint, this would resume the execution and remove the existing breakpoint. At this time a new breakpoint can be passed.
- You can pass a breakpoint when replaying an execution.
Part-of: https://github.com/kestra-io/kestra-ee/issues/1547
We created miltiple worker job queue because the bean was in the prototype scope.
This was needed only for tests as they are closing it.
Switching to singleton and rebuilding the context of the test that needs it fixes the issue.
We previously starts 2 queues: one for emit, and a specific one for receive wich handle WorkerJobRunning under the cover.
We can replace by using a single queue that will transparently handle WorkerJobRunning. As queues starts thread pools their cost is not negligeable.
Fixes#9007
* feat:
- Implement width property
- Replace custom dashboard
- Started to integrate the KPI chart
* feat(ui): introduce dashboard chart layout system
* feat(ui): introduce dashboard chart kpi card
* chore(ui): amend layout widths for sm screen size
* chore(ui): prevent editing of default dashboard
* chore(ui): centering the KPI text inside the box
* chore(ui): dashboard edit preview to respect set layouts
* chore(ui): initial work on setting the default flow & namespace dashboards
* fix(ui): make sure there is no naming clashes
* feat: KPI chart backend implementation
* feat: validation annotations
* chore(ui): make chart legend align to right side
* chore(ui): properly show chart labels
* chore(ui): improve state and ID components inside custom tabels
* chore(ui): add proper link to execution in tables
* feat: implemented Triggers as Datasource for custom dashboards
close kestra-io/kestra-ee#3740
* feat: modified the Markdown chart so now it accept different sources
* feat: rename KPI property to numerator & where
close#3739
* chore(ui): improve markdown component
* chore(ui): markdown charts
* chore(ui): markdown charts
* chore(ui): markdown charts remove padding
* chore(ui): markdown charts
* feat: fixes + define custom dashboard equivalent to current default dashboard with some modification
* fix: round double value
* chore(ui): improve flows and ns charts
* chore(ui): make sure that table shows execution links only if namespace and flowId exist
* chore(ui): make sure markdown is properly shown on dashboard edititng
* fix: correctly do preview instead of load on homepage
* fix: correctly preview markdown chart and add description in default flow dashboard
* fix: apply review changes
* fix: modify test following classes modifications on charts
* tests: restore package-lock
* remove chromatic tools and a warning
---------
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
Co-authored-by: Bart Ledoux <bledoux@kestra.io>
* fix(filters): change label filtering to 'and' instead of 'or'
link to #8489
* test(execution): add test to validate and behavior
* test(execution): fix test
* feat(core): #3427 add OSS tenant migration scripts
* clean(core): fixes after review
* clean(core): make only one command for oss and EE migration
* fix(core): user synchronisation command and clean PR
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Purge service instance in EMPTY state after a certain duration, 30 days by default, to avoid never ending groth on the service_instances table.
Fixes#8514
Add new FlowId, FlowInterface and GenericFlow classes to support
deserialization of flow with un-typed plugins (i.e., tasks, triggers)
in order to inject defaults prior to strongly-typed deserialization.
