Compare commits

...

203 Commits

Author SHA1 Message Date
Roman Acevedo
93d53b9d57 try out to run in parallel JdbcRunnerRetryTest, lower exec run duration 2025-09-17 12:24:53 +02:00
Roman Acevedo
11e5e14e4e fix Timeline flamegraph 2025-09-17 11:44:04 +02:00
Roman Acevedo
72ce317c3d Update workflow-backend-test.yml 2025-09-17 11:44:04 +02:00
Roman Acevedo
4da44013c1 add Timeline flamegraph to temp debug tests on this branch 2025-09-17 11:44:04 +02:00
nKwiatkowski
c9995c6f42 fix(tests): failing unit tests 2025-09-17 10:31:39 +02:00
nKwiatkowski
a409299dd8 feat(tests): play jdbc h2 tests in parallel 2025-09-16 19:23:07 +02:00
Roman Acevedo
34cf67b0a4 test: make AbstractExecutionRepositoryTest parallelizable 2025-09-16 19:23:07 +02:00
Ludovic DEHON
d092556bc2 chore(build): use remote actions 2025-09-16 18:09:54 +02:00
Roman Acevedo
308106d532 ci: make generated test report retrocompatible with older releases (#11308)
* ci: make generated test report retrocompatible with older realeases

* ci: fix cli
2025-09-16 15:21:56 +02:00
Piyush Bhaskar
8fe8f96278 refactor(core): use el-splitter instead of custom sliders (#11309)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-16 18:35:57 +05:30
Miloš Paunović
a5cad6d87c chore(core): improve coloring scheme for dependencies graph (#11306) 2025-09-16 14:26:15 +02:00
Loïc Mathieu
199d67fbe2 chore(system): share the application.yaml config file between OSS and EE 2025-09-16 10:53:53 +02:00
Loïc Mathieu
558a2e3f01 fix(flows): properly coompute flow dependencies with preconditions
When both upstream flows and where are set, it should be a AND between the two as dependencies must match the upstream flows.

Fixes #11164
2025-09-16 10:43:55 +02:00
HARSH THAKARE
e1d2c30e54 fix(core): add validation to prevent empty label values in Labels task (#11273)
part of #11227

---------

Co-authored-by: harshinfomaticae <harsh.thakare@infomaticae.co.in>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-16 10:26:46 +02:00
Loïc Mathieu
700c6de411 fix(system): allow flattening a map with duplicated keys 2025-09-16 10:24:43 +02:00
Florian Hussonnois
2b838a5012 fix(executions): add missing CrudEvent on purge execution
Related-to: kestra-io/kestra-ee#5061
2025-09-16 09:34:19 +02:00
Loïc Mathieu
617daa79db fix(executions): truncate the execution_running table as in 0.24 there was an issue in the purge
This table contains executions for flows that have a concurrency that are currently running.
It has been added in 0.24 but in that release there was a bug that may prevent some records to being correctly removed from this table.
To fix that, we truncate it once.
2025-09-15 17:29:28 +02:00
Roman Acevedo
1791127acb test: unflaky FileChangedEventListener and PluginDefaultServiceTest, debug log on JdbcServiceLivenessCoordinatorTest
* test: parallelize AbstractRunnerTest

* test: add TestsUtils.randomTenant(..) function

* test: i think i found a bug

* revert debug

* test: add comment on potential bug, make test pass

* test: fix test metadata

* test: unflaky PluginDefaultServiceTest by separating class

* test: add log on JdbcServiceLivenessCoordinatorTest to debug

* test: cleanup debug log

* fix
2025-09-15 17:07:37 +02:00
brian-mulier-p
7feb571fb3 fix(test): add tenant-in-path storage test (#11292)
part of kestra-io/storage-s3#166
2025-09-15 16:49:02 +02:00
brian-mulier-p
a315bd0e1c fix(security): enhance basic auth security (#11285)
closes kestra-io/kestra-ee#5111
2025-09-15 16:27:14 +02:00
Roman Acevedo
e2ac1e7e98 ci: prevent commenting PR test report when cancelled 2025-09-15 16:01:07 +02:00
Miloš Paunović
c6f40eff52 fix(core): adjust positioning of default tour elements (#11286)
The problem occurred when `No Code` was selected as the `Default Editor Type` in `Settings`. This `PR` resolves the issue.

Closes https://github.com/kestra-io/kestra/issues/9556.
2025-09-15 14:55:00 +02:00
Miloš Paunović
ccd42f7a1a chore(core): remove superfluous button attribute in settings page (#11283) 2025-09-15 12:27:19 +02:00
Florian Hussonnois
ef08c8ac30 fix(plugins): remove regex validation on version property
Changes:
* Fixes stable method in Version class
* Remove regex validation on 'version' property

Related-to: kestra-io/kestra-ee#5090
2025-09-15 11:54:10 +02:00
github-actions[bot]
7b527c85a9 chore(core): localize to languages other than english (#11280)
Extended localization support by adding translations for multiple languages using English as the base. This enhances accessibility and usability for non-English-speaking users while keeping English as the source reference.

Co-authored-by: GitHub Action <actions@github.com>
2025-09-15 11:09:17 +02:00
Hamza
d121867066 chore(flows): trigger editor autocompletion when backspace is pressed (#10797)
Closes https://github.com/kestra-io/kestra/issues/10776.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-15 11:07:20 +02:00
Roman Acevedo
a084a9f6f0 ci: fix Summary report test path 2025-09-15 10:50:25 +02:00
Karthik D
f6fff11081 chore(core): add reset to defaults option to settings page (#11226)
Closes https://github.com/kestra-io/kestra/issues/10640.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-15 10:45:11 +02:00
Roman Acevedo
3d5015938f ci: add total header to generateTestReportSummary 2025-09-15 10:32:22 +02:00
Florian Hussonnois
951c93cedb fix(core): fix CrudEvent model for DELETE operation
Refactor XxxRepository class to use new factory methods
from the CrudEvent class

Related-to: kestra-io/kestra-ee#4727
2025-09-15 10:06:52 +02:00
Antoine Gauthier
9c06b37989 chore(core): resolve button text overflow on system overview page (#11271)
Closes https://github.com/kestra-io/kestra/issues/11245.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-15 09:57:10 +02:00
Anna Geller
a916a03fdd fix(stats): update edition comparison with latest features and improved descriptions (#11272) 2025-09-14 12:35:26 +02:00
Roman Acevedo
4e728da331 test: disable one last test 2025-09-12 20:24:08 +02:00
Roman Acevedo
166a3932c9 test: do not parallelize yet AbstractRunnerTest 2025-09-12 20:24:08 +02:00
Roman Acevedo
0a21971bbf ci: only comment PR with test report in PR 2025-09-12 20:24:08 +02:00
Roman Acevedo
8c4d7c0f9e test: disable failing tests, they will be fixed soon
- will be treated in https://github.com/kestra-io/kestra/issues/11269
2025-09-12 20:24:08 +02:00
Nicolas K.
b709913071 test: run core tests in parallel (#11265)
- advance on #11264

* feat(ci-cd): play tests in parallel and synchronize plugin registry init

* fix(tests): change memory to h2 because the configuration have changed

* feat(tests): use tenant id to run runner tests in parallel

* run AbstractRunnerTest test methods in parallel

* feat(tests): use tenant id to run runner tests in parallel

* feat(tests): remove unwanted generated files

---------

Co-authored-by: Roman Acevedo <roman.acevedo62@gmail.com>
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-09-12 19:29:38 +02:00
Roman Acevedo
5be401d23c ci: add a kestra-devtools cli, and comment PR with failed tests
this is a POC, I think it can already be useful. Next step will be to move kestra-devtools to a separate repo and publish it to npm
2025-09-12 18:48:12 +02:00
Roman Acevedo
bb9f4be8c2 Revert "chore(sanitycheck): refactor PurgeCurrentExecutionFiles (#11115)"
This reverts commit fc690bf7cd.
Python task cannot be used here, it is not available. This commit was
wrongly merged with a red CI
2025-09-12 17:49:02 +02:00
François Delbrayelle
01e8e46b77 Revert "feat(retry): use the retry policy on HttpClient (#10922)" (#11263)
This reverts commit a236688be6.
2025-09-12 17:46:28 +02:00
Miloš Paunović
d00f4b0768 chore(core): ensure editor suggestion widget renders above other elements (#11258)
Closes https://github.com/kestra-io/kestra/issues/10702.
Closes https://github.com/kestra-io/kestra/issues/11033.
2025-09-12 14:48:56 +02:00
Barthélémy Ledoux
279f59c874 fix(core): only display close all tabs when there is more than one tab (#11257) 2025-09-12 14:20:54 +02:00
Barthélémy Ledoux
d897509726 fix(flows): clear tasks list when last task is deleted (#11255) 2025-09-12 14:20:42 +02:00
Pradumna Saraf
0d592342af chore(sanitycheck): add for OutputValues (#11105) 2025-09-12 16:53:13 +05:30
Pradumna Saraf
fc690bf7cd chore(sanitycheck): refactor PurgeCurrentExecutionFiles (#11115) 2025-09-12 16:52:37 +05:30
Antoine Gauthier
0a1b919863 chore(logs): display copy button only on row hover (#11254)
Closes https://github.com/kestra-io/kestra/issues/11220.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-12 12:00:08 +02:00
Piyush Bhaskar
2f4e981a29 fix(core): add gradient at footer to avoid hard cut (#11252) 2025-09-12 14:35:47 +05:30
brian-mulier-p
5e7739432e fix(core): add ability to remap sort keys (#11233)
part of kestra-io/kestra-ee#5075
2025-09-12 09:43:39 +02:00
Miloš Paunović
8aba863b8c feat(core): introduce close all panels functionality (#11225)
Closes https://github.com/kestra-io/kestra/issues/10785.
2025-09-12 09:01:24 +02:00
dependabot[bot]
7eaa43c50f build(deps): bump axios (#11243)
Bumps the npm_and_yarn group with 1 update in the /ui directory: [axios](https://github.com/axios/axios).


Updates `axios` from 1.11.0 to 1.12.0
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.11.0...v1.12.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.12.0
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-12 08:36:02 +02:00
Piyush Bhaskar
267ff78bfe fix(admin): change the header and add description on hover (#11241)
Co-authored-by: GitHub Action <actions@github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-12 12:00:41 +05:30
François Delbrayelle
7272cfe01f feat(ai_copilot): gray italic placeholder + rename AiAgent to AiCopilot (#11235) 2025-09-11 20:24:04 +02:00
brian.mulier
91e2fdb2cc fix(ai): increase maxOutputToken default 2025-09-11 18:11:52 +02:00
François Delbrayelle
a236688be6 feat(retry): use the retry policy on HttpClient (#10922) 2025-09-11 15:00:25 +02:00
Antoine Gauthier
81763d40ae fix(docs): center main container in DocsLayout (#11222)
Co-authored-by: Piyush Bhaskar <impiyush0012@gmail.com>
2025-09-11 16:18:12 +05:30
Miloš Paunović
677efb6739 fix(namespaces): open details page at top (#11221)
Closes https://github.com/kestra-io/kestra/issues/10536.
2025-09-11 10:52:47 +02:00
Nicolas K.
b35924fef1 fix(tests): add server type mock in the kestra context (#11176)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-09-11 09:45:51 +02:00
Jaem Dessources
9dd93294b6 fix(core): align copy logs button to each row’s right edge (#11216)
Closes https://github.com/kestra-io/kestra/issues/10898.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-11 08:55:01 +02:00
Piyush Bhaskar
fac6dfe9a0 fix(core): update router usage in loadAutocomplete. (#11219) 2025-09-11 12:13:05 +05:30
Bisesh
3bf9764505 fix(core): make sidebar tab color consistent when unfocused (#11217)
Closes https://github.com/kestra-io/kestra/issues/11156.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-11 08:33:57 +02:00
Piyush Bhaskar
c35cea5d19 fix(core): override the ns module. (#11218) 2025-09-11 11:53:00 +05:30
Barthélémy Ledoux
4d8e9479f1 refactor: finally get rid of vuex (#11211) 2025-09-10 22:44:21 +02:00
Florian Hussonnois
3f24e8e838 fix(core): make CRC32 for plugin JARs lazy
Make CRC32 calculation for lazy plugin JAR files
to avoid excessive startup time and performance impact.

Avoid byte buffer reallocation while computing CRC32.
2025-09-10 17:42:02 +02:00
Miloš Paunović
7175fcb666 fix(executions): refactor link creation to ensure the id is rendered as a clickable link (#11209)
Related to https://github.com/kestra-io/kestra/issues/10906.
2025-09-10 15:01:29 +02:00
Barthélémy Ledoux
2ddfa13b1b refactor: make-axios-composable (#11177) 2025-09-10 14:54:00 +02:00
Barthélémy Ledoux
ba2a5dfec8 chore: revert monaco update (#11207) 2025-09-10 13:34:33 +02:00
Loïc Mathieu
f84441dac7 fix(ci): disable publishing docker image on fork
I should have not trusted an AI for this but copy/paste what I know work: the Quarkus CI!
2025-09-10 12:17:25 +02:00
Barthélémy Ledoux
433b788e4a chore: a bunch of performance fixes detected by oxlint (eslint-unicorn) (#10050) 2025-09-10 11:35:07 +02:00
dependabot[bot]
65c5fd6331 build(deps): bump org.projectlombok:lombok from 1.18.38 to 1.18.40
Bumps [org.projectlombok:lombok](https://github.com/projectlombok/lombok) from 1.18.38 to 1.18.40.
- [Changelog](https://github.com/projectlombok/lombok/blob/master/doc/changelog.markdown)
- [Commits](https://github.com/projectlombok/lombok/compare/v1.18.38...v1.18.40)

---
updated-dependencies:
- dependency-name: org.projectlombok:lombok
  dependency-version: 1.18.40
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 11:12:24 +02:00
dependabot[bot]
421ab40276 build(deps): bump io.micrometer:micrometer-core from 1.15.3 to 1.15.4
Bumps [io.micrometer:micrometer-core](https://github.com/micrometer-metrics/micrometer) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/micrometer-metrics/micrometer/releases)
- [Commits](https://github.com/micrometer-metrics/micrometer/compare/v1.15.3...v1.15.4)

---
updated-dependencies:
- dependency-name: io.micrometer:micrometer-core
  dependency-version: 1.15.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 11:11:48 +02:00
dependabot[bot]
efb2779693 build(deps): bump flyingSaucerVersion from 9.13.3 to 10.0.0
Bumps `flyingSaucerVersion` from 9.13.3 to 10.0.0.

Updates `org.xhtmlrenderer:flying-saucer-core` from 9.13.3 to 10.0.0
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.13.3...v10.0.0)

Updates `org.xhtmlrenderer:flying-saucer-pdf` from 9.13.3 to 10.0.0
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.13.3...v10.0.0)

---
updated-dependencies:
- dependency-name: org.xhtmlrenderer:flying-saucer-core
  dependency-version: 10.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
- dependency-name: org.xhtmlrenderer:flying-saucer-pdf
  dependency-version: 10.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 11:10:59 +02:00
dependabot[bot]
74d371c0ca build(deps): bump com.azure:azure-sdk-bom from 1.2.37 to 1.2.38
Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.37 to 1.2.38.
- [Release notes](https://github.com/azure/azure-sdk-for-java/releases)
- [Commits](https://github.com/azure/azure-sdk-for-java/compare/azure-sdk-bom_1.2.37...azure-sdk-bom_1.2.38)

---
updated-dependencies:
- dependency-name: com.azure:azure-sdk-bom
  dependency-version: 1.2.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 11:10:10 +02:00
Loïc Mathieu
90a7869020 fixsystem): always load netty from the app classloader
As Netty is used in core and a lot of plugins, and we already load project reactor from the app classloader that depends in Netty.

Fixes https://github.com/kestra-io/kestra-ee/issues/5038
2025-09-10 10:50:22 +02:00
dependabot[bot]
d9ccb50b0f build(deps): bump actions/github-script from 7 to 8
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:47:40 +02:00
dependabot[bot]
aea0b87ef8 build(deps): bump aquasecurity/trivy-action from 0.33.0 to 0.33.1
Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.33.0 to 0.33.1.
- [Release notes](https://github.com/aquasecurity/trivy-action/releases)
- [Commits](https://github.com/aquasecurity/trivy-action/compare/0.33.0...0.33.1)

---
updated-dependencies:
- dependency-name: aquasecurity/trivy-action
  dependency-version: 0.33.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:47:17 +02:00
Loïc Mathieu
9a144fc3fe fix(system): we don't need to advance the parser anymore to the first token 2025-09-10 10:46:44 +02:00
Loïc Mathieu
ddd9cebc63 chore(deps): upgrade to Jackson 2.20.0
Jackson annotation now uses a version scheme without micro version so it has been updated to 2.20.

Closes #11069
2025-09-10 10:46:44 +02:00
dependabot[bot]
1bebbb9b73 build(deps): bump com.gorylenko.gradle-git-properties
Bumps com.gorylenko.gradle-git-properties from 2.5.2 to 2.5.3.

---
updated-dependencies:
- dependency-name: com.gorylenko.gradle-git-properties
  dependency-version: 2.5.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:46:26 +02:00
dependabot[bot]
8de4dc867e build(deps): bump actions/setup-python from 5 to 6
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:46:08 +02:00
dependabot[bot]
fc49694e76 build(deps): bump actions/setup-node from 4 to 5
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 5.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:45:47 +02:00
dependabot[bot]
152300abae build(deps): bump io.micronaut.openapi:micronaut-openapi-bom
Bumps [io.micronaut.openapi:micronaut-openapi-bom](https://github.com/micronaut-projects/micronaut-openapi) from 6.17.3 to 6.18.0.
- [Release notes](https://github.com/micronaut-projects/micronaut-openapi/releases)
- [Commits](https://github.com/micronaut-projects/micronaut-openapi/compare/v6.17.3...v6.18.0)

---
updated-dependencies:
- dependency-name: io.micronaut.openapi:micronaut-openapi-bom
  dependency-version: 6.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:45:13 +02:00
dependabot[bot]
1ff5dda4e1 build(deps): bump software.amazon.awssdk:bom from 2.33.2 to 2.33.5
Bumps software.amazon.awssdk:bom from 2.33.2 to 2.33.5.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-version: 2.33.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 10:44:50 +02:00
Miloš Paunović
84f9b8876d chore(deps): regular dependency update (#11200)
Performing a weekly round of dependency updates in the NPM ecosystem to keep everything up to date.
2025-09-10 10:18:33 +02:00
brian-mulier-p
575955567f fix(flows): avoid failing flow dependencies with dynamic defaults (#11166)
closes #11117
2025-09-10 09:59:51 +02:00
brian-mulier-p
d6d2580b45 fix(namespaces): avoid adding 'company.team' as default ns (#11174)
closes #11168
2025-09-09 17:13:48 +02:00
Miloš Paunović
070e54b902 chore(flows): display correct flow dependency count (#11169)
Closes https://github.com/kestra-io/kestra/issues/11127.
2025-09-09 13:56:17 +02:00
Roman Acevedo
829ca4380f fix(flows): topology would not load when having many flows and cyclic relations
- this will probably fix https://github.com/kestra-io/kestra-ee/issues/4980

the issue was recursiveFlowTopology was returning a lot of duplicates, it was aggravated when having many Flows and multiple Flow triggers
2025-09-09 13:06:20 +02:00
Karthik D
381c7a75ad chore(core): use simple search input on blueprints listing (#11034)
Closes https://github.com/kestra-io/kestra/issues/11002.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-09 12:54:58 +02:00
louispy
1688c489a9 chore(flows): improve visibility of horizontal scroll bar on listing (#11163)
Closes https://github.com/kestra-io/kestra/issues/11158.

Co-authored-by: louispy <louisleslie98@gmail.com>
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-09-09 12:40:28 +02:00
AKSHAT GUPTA
93ccbf5f9b chore(core): separate data loading from graph node rendering on dependency view (#11155)
Relates to https://github.com/kestra-io/kestra/issues/11125.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-09 12:25:58 +02:00
Barthélémy Ledoux
ac1cb235e5 refactor: avoid importing all of lodash when we only need groupBy (#10870) 2025-09-09 11:34:13 +02:00
dependabot[bot]
9d3d3642e8 build(deps): bump kafkaVersion from 4.0.0 to 4.1.0
Bumps `kafkaVersion` from 4.0.0 to 4.1.0.

Updates `org.apache.kafka:kafka-clients` from 4.0.0 to 4.1.0

Updates `org.apache.kafka:kafka-streams` from 4.0.0 to 4.1.0

Updates `org.apache.kafka:kafka-streams-test-utils` from 4.0.0 to 4.1.0

---
updated-dependencies:
- dependency-name: org.apache.kafka:kafka-clients
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.kafka:kafka-streams
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.kafka:kafka-streams-test-utils
  dependency-version: 4.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-09 09:56:38 +02:00
Suguresh
3d306a885e feat(core): add extra date format options (#10237)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-09 09:31:49 +02:00
Antoine Gauthier
ef193c5774 feat(core): add a new date format option with milliseconds (#11108)
Closes https://github.com/kestra-io/kestra/issues/11028.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-09 09:20:37 +02:00
AmbarMishra973
d0f46169f4 feat(executions): make the id field a link that can be opened in a new tab (#10963)
Closes https://github.com/kestra-io/kestra/issues/10906.

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-09-09 09:13:49 +02:00
François Delbrayelle
3005ab527c fix(outputs): open external file was not working (#11154) 2025-09-08 17:45:19 +02:00
Barthélémy Ledoux
688e2af12b chore: update eslint config for vue files (#9891) 2025-09-08 16:42:33 +02:00
Nicolas K.
4c0a05f484 fix(test): flaky Scheduler trigger change test (#11153)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-09-08 16:33:23 +02:00
zaib shamsi
108f8fc2c7 feat(executions): nicer exception message for the HttpFunction
### What I did

- Improved the exception message in HttpFunction.java to make debugging easier.

### Why

- The original message was too generic. This change makes it clearer where the issue occurs.
2025-09-08 15:04:12 +02:00
Barthélémy Ledoux
8b81a37559 refactor: make folder structure of no-code use "no-code" (#11122) 2025-09-08 14:15:04 +02:00
Barthélémy Ledoux
9222f97d63 fix(core): multipanel split creates super big panels (#11123) 2025-09-08 14:14:40 +02:00
brian.mulier
43e3591417 chore(ci): fail-safe update-plugin-kestra-version.sh 2025-09-08 12:02:28 +02:00
brian.mulier
438dc9ecf6 chore(ci): create branch if not exist on update-plugin-kestra-version.sh 2025-09-08 11:45:15 +02:00
brian-mulier-p
7292837c58 chore(ci): add LTS tagging (#11131) 2025-09-08 11:13:16 +02:00
brian.mulier
7fa93d7764 chore(version): update to version 'v1.1.0-SNAPSHOT'. 2025-09-08 10:08:34 +02:00
brian.mulier
a3c9b35b25 fix(ci): no more RC semver check on plugins 2025-09-08 09:56:11 +02:00
brian.mulier
2c03101422 fix(ci): no more RC semver check on plugins 2025-09-08 09:54:42 +02:00
brian.mulier
7ee2cca3ae fix(ci): no more RC semver check 2025-09-08 09:54:42 +02:00
brian.mulier
ddb48a4384 fix(ci): no more RC semver check 2025-09-08 09:51:31 +02:00
AJ Emerich
a62c5ab637 fix(docs): correct core properties and other docs (#11026)
* fix(docs): correct core properties and other docs

Correct grammar, punctuation, and consistency

* Apply suggestions from code review

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>

* Apply suggestions from code review

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>

* Apply suggestions from code review

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>

---------

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>
2025-09-05 17:44:04 +02:00
AJ Emerich
1b934d31f1 fix(docs): fix trigger docs#11018 2025-09-05 17:40:02 +02:00
Miloš Paunović
f887f53c25 fix(core): allow triggering the task from playground again after a 422 error (#11112)
Closes https://github.com/kestra-io/kestra/issues/11109.
2025-09-05 15:43:43 +02:00
Roman Acevedo
098fa5a4ef fix(topology): prevent flowTopology failing on a Flow for bugged relation
- fixes https://github.com/kestra-io/kestra/issues/11096
2025-09-05 15:16:00 +02:00
Barthélémy Ledoux
c833ab6bc1 fix(flows): avoid updating code if no-code has not changed (#11094) 2025-09-05 14:14:08 +02:00
Krie
dfb7c61455 docs: clarify java version on devcontainer 2025-09-05 13:56:13 +02:00
Florian Hussonnois
ed509d4461 fix(plugins): fix registration of a same plugins and doc reload
Compute a quick CRC32 of each plugin based on the Central Directory
of the JarFile to ensure change detection - allowing re-upload
of a same plugin version (EE)

Fix: kestra-io/kestra-ee#4925
Fix: kestra-io/kestra-ee#4882
2025-09-05 12:55:19 +02:00
dependabot[bot]
8b50f191d8 build(deps): bump mermaid in /ui in the npm_and_yarn group (#10833)
Bumps the npm_and_yarn group in /ui with 1 update: [mermaid](https://github.com/mermaid-js/mermaid).


Updates `mermaid` from 11.8.1 to 11.10.0
- [Release notes](https://github.com/mermaid-js/mermaid/releases)
- [Commits](https://github.com/mermaid-js/mermaid/compare/mermaid@11.8.1...mermaid@11.10.0)

---
updated-dependencies:
- dependency-name: mermaid
  dependency-version: 11.10.0
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-05 12:36:38 +02:00
Piyush Bhaskar
7c5b531117 fix(core): export flow from settings (#11111) 2025-09-05 14:03:18 +05:30
Miloš Paunović
810579bce9 chore(core): amend dependency graph coloring scheme (#11107) 2025-09-05 09:59:46 +02:00
Piyush Bhaskar
a0e7c50b28 fix(core): show inherited kv pairs button only on kv tab (#11104) 2025-09-05 11:40:50 +05:30
François Delbrayelle
bfbc3f70a4 build(main): add skip-test as an option for workflows (#11103) 2025-09-04 19:33:06 +02:00
Ludovic DEHON
9b5c4b0052 feat(build): allow skip test on main ci 2025-09-04 18:55:03 +02:00
AJ Emerich
b7063d986b docs(flow-trigger): add example with conditions (#11099)
Closes https://github.com/kestra-io/kestra/issues/10197
2025-09-04 18:33:37 +02:00
brian.mulier
46ec0ae701 fix(ai): move maxOutputToken default from 50000 to 4000 2025-09-04 18:28:02 +02:00
brian.mulier
ba0615ba01 fix(ai): add some properties to Posthog events 2025-09-04 18:28:02 +02:00
Florian Hussonnois
0a26098a91 fix(core): add method to DefaultPluginRegistry to override of plugin registration 2025-09-04 17:55:56 +02:00
Barthélémy Ledoux
0e2863e6fd feat(flows): update DEFAULT_ACTIVE_TABS to follow settings (#11095) 2025-09-04 16:55:55 +02:00
brian-mulier-p
4b6559203c fix(dashboards): blueprints working for dashboards (#11082)
part of #11076
2025-09-04 15:39:23 +02:00
Malaydewangan09
b6993d71f2 feat(plugins): add plugins 2025-09-04 18:59:28 +05:30
Barthélémy Ledoux
ea6daf381d fix(core): Multipanel layout fixes (#11092) 2025-09-04 15:23:57 +02:00
dependabot[bot]
0649c1309b build(deps): bump software.amazon.awssdk:bom from 2.32.31 to 2.33.1
Bumps software.amazon.awssdk:bom from 2.32.31 to 2.33.1.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-version: 2.33.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 15:01:05 +02:00
François Delbrayelle
6e21d650f9 feat(plugin): add new priority attr on Plugin annotation (#10975) 2025-09-04 14:21:38 +02:00
Piyush Bhaskar
88acc91323 chore(version): update @kestra-io/ui-libs to version 0.0.244 (#11088) 2025-09-04 17:44:45 +05:30
Miloš Paunović
a822f3b372 fix(flow)*: properly handle tab closing by clicking the cross icon in the corner of the panel (#11086)
Closes https://github.com/kestra-io/kestra/issues/10981.
2025-09-04 14:11:34 +02:00
github-actions[bot]
4a3c6ee9e9 chore(core): remove empty lines at the end of translation files (#11089) 2025-09-04 14:07:38 +02:00
Loïc Mathieu
0ed4e5853d fix(deps): add worker to the platform 2025-09-04 13:33:42 +02:00
Loïc Mathieu
71cdd02230 fix(executions): add logs in case of concurrency limit failure
Fixes #11004
2025-09-04 13:03:00 +02:00
brian.mulier
fedddcde00 fix(ai): move back to Gemini as default 2025-09-04 12:49:18 +02:00
Ludovic DEHON
03f256cb9b chore(deps): follow platform on com.microsoft.playwright:playwright 2025-09-04 11:37:01 +02:00
dependabot[bot]
f9317ba8ea build(deps): bump google-github-actions/setup-gcloud from 2 to 3
Bumps [google-github-actions/setup-gcloud](https://github.com/google-github-actions/setup-gcloud) from 2 to 3.
- [Release notes](https://github.com/google-github-actions/setup-gcloud/releases)
- [Changelog](https://github.com/google-github-actions/setup-gcloud/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/setup-gcloud/compare/v2...v3)

---
updated-dependencies:
- dependency-name: google-github-actions/setup-gcloud
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 11:35:56 +02:00
dependabot[bot]
eefca3d7a4 build(deps): bump google-github-actions/auth from 2 to 3
Bumps [google-github-actions/auth](https://github.com/google-github-actions/auth) from 2 to 3.
- [Release notes](https://github.com/google-github-actions/auth/releases)
- [Changelog](https://github.com/google-github-actions/auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/google-github-actions/auth/compare/v2...v3)

---
updated-dependencies:
- dependency-name: google-github-actions/auth
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 11:35:37 +02:00
dependabot[bot]
96b9e3c74b build(deps): bump mailchecker from 6.0.17 to 6.0.18 in /ui
Bumps [mailchecker](https://github.com/FGRibreau/mailchecker) from 6.0.17 to 6.0.18.
- [Changelog](https://github.com/FGRibreau/mailchecker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/FGRibreau/mailchecker/compare/v6.0.17...v6.0.18)

---
updated-dependencies:
- dependency-name: mailchecker
  dependency-version: 6.0.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 11:34:34 +02:00
dependabot[bot]
0a78778e5c build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.38.9 to 0.38.11.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.38.9...v0.38.11)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-version: 0.38.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 11:27:48 +02:00
dependabot[bot]
5342948bfb build(deps): bump aquasecurity/trivy-action from 0.32.0 to 0.33.0
Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.32.0 to 0.33.0.
- [Release notes](https://github.com/aquasecurity/trivy-action/releases)
- [Commits](https://github.com/aquasecurity/trivy-action/compare/0.32.0...0.33.0)

---
updated-dependencies:
- dependency-name: aquasecurity/trivy-action
  dependency-version: 0.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-04 11:26:05 +02:00
brian.mulier
f9beb0f4af fix(ai): add model-name to configuration example when Copilot is not configured 2025-09-04 11:25:21 +02:00
Ludovic DEHON
70c1621025 chore(deps): update most of js deps 2025-09-04 11:24:37 +02:00
Loïc Mathieu
a9098e7dc9 fix(tests): reduce test duration 2025-09-04 11:04:13 +02:00
Loïc Mathieu
249839833c chore(system): move the standalone runner to the cli module 2025-09-04 11:04:13 +02:00
Loïc Mathieu
97ec24fc6a chore(system): merge Indexer with the IndexerInterface 2025-09-04 11:04:13 +02:00
Loïc Mathieu
be5e24217b chore(system): extract the scheduler to its own module 2025-09-04 11:04:13 +02:00
Loïc Mathieu
a5724bcb18 chore(system): extract the executor to its own module 2025-09-04 11:04:13 +02:00
Loïc Mathieu
f3057d2d57 chore(system): extract the worker to its own module 2025-09-04 11:04:13 +02:00
Roman Acevedo
e8a953fc6b test: disable flaky test AbstractRunnerTest.flowTriggerWithConcurrencyLimit
- sometimes fails on Kafka tests: https://github.com/kestra-io/kestra-ee/actions/runs/17382883576/attempts/1#summary-49344101977
- it will be fixed during cooldown here https://github.com/kestra-io/kestra/issues/10758
2025-09-04 10:42:17 +02:00
Barthélémy Ledoux
267f4fcc86 fix(flow): cleanup executions' graph and flow on execution reset (#11075)
* fix: typescript issue in LowCodeEditor

* fix(flow): cleanup executions' graph and flow on execution reset

* add tsexpect error
2025-09-04 10:17:48 +02:00
github-actions[bot]
af1e2e3059 chore(core): localize to languages other than english (#11074)
Extended localization support by adding translations for multiple languages using English as the base. This enhances accessibility and usability for non-English-speaking users while keeping English as the source reference.

Co-authored-by: GitHub Action <actions@github.com>
2025-09-04 09:51:32 +02:00
Anna Geller
638d58697b fix: too long sidebar names (#11073) 2025-09-04 09:44:36 +02:00
brian-mulier-p
0ae9374bf5 fix(ai): optional flowYaml in generateFlow endpoint (#10974)
closes kestra-io/kestra-ee#4286
2025-09-03 14:50:59 +02:00
Miloš Paunović
6a0344a09e chore(core): improve dependency graph user experience (#10994)
Closes https://github.com/kestra-io/kestra/issues/10925.
Closes https://github.com/kestra-io/kestra-ee/issues/4865.
2025-09-03 14:41:40 +02:00
Miloš Paunović
e46b4a75d1 fix(flows): pass proper namespace id on flow creation (#11031) 2025-09-03 13:57:04 +02:00
Ludovic DEHON
8fe1bae739 chore(deps): remove unrequired deps from MakeFile 2025-09-03 12:54:57 +02:00
Ludovic DEHON
9ef59fdd23 chore(deps): upgrade some platform deps 2025-09-03 12:54:09 +02:00
Roman Acevedo
79ab4415ad fix(storage): avoid NPE in PurgeKV when no metadata or expiration date
- fixes https://github.com/kestra-io/kestra/issues/11019
2025-09-03 12:52:20 +02:00
brian.mulier
dd3829cc48 fix(ai): enhance FlowYamlBuilder prompt (#11027) 2025-09-03 12:08:54 +02:00
brian.mulier
fa187904f9 fix(ai): remove @Requires on API key as it's already done transitively (#11027) 2025-09-03 12:08:54 +02:00
brian.mulier
c659599b1f chore(deps): add langchain4j version to platform (#11027) 2025-09-03 12:08:54 +02:00
Florian Hussonnois
85db1eafa7 fix(plugins): add plugin registry hash to invalidate cache
Fixes: kestra-io/kestra-ee#4882
2025-09-03 11:51:39 +02:00
Florian Hussonnois
9863f0807f fix(flows): fix dynamic default inputs (#11014)
Fixes: #11014
2025-09-03 10:25:33 +02:00
brian-mulier-p
b3b0d630cf feat(ai): langchain4j implementation of AI Copilot (#10995)
closes kestra-io/kestra-ee#4710
2025-09-03 10:04:07 +02:00
Florian Hussonnois
97665449a8 fix(system): rename service EMPTY state to INACTIVE (kestra-io/kestra-ee#4838)
Related-to: kestra-io/kestra-ee#4838
2025-09-03 09:11:36 +02:00
AJ Emerich
2e1ed792e9 fix(docs): fix grammar and punctuation in core storage (#11020)
* fix(docs): fix grammar and punctuation in core storage

* Apply suggestions from code review

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>

---------

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>
2025-09-02 23:49:01 +02:00
Ludovic DEHON
59e3ae5922 chore(deps): use a common http5 client 2025-09-02 22:49:53 +02:00
Anna Geller
d46b54746a fix(docs): improve grammar and examples (#11017)
* fix(docs): improve grammar and examples

* fix: comma and whitespace issues in CI
2025-09-02 20:04:44 +02:00
Ludovic DEHON
4fbaed744b chore(deps): bump micronaut to 4.9.3 2025-09-02 19:44:19 +02:00
AJ Emerich
b0638437d5 fix(kv-purge): clean up the docs (#11016) 2025-09-02 19:21:01 +02:00
Ludovic DEHON
e3f9d0f8ff chore(deps): bump org.sonarqube from 6.2.0.5505 to 6.3.1.5724 2025-09-02 18:23:30 +02:00
Barthélémy Ledoux
ae80738f33 fix(docs): remove url hash change when opening a docs (#11000) 2025-09-02 15:44:55 +02:00
Barthélémy Ledoux
2b4f208569 refactor(namespaces): less code in namespace files editor (#9912) 2025-09-02 14:41:24 +02:00
Barthélémy Ledoux
8acbc8ba03 fix(nocode): KeyValue Pairs have a bug (#10998) 2025-09-02 14:19:55 +02:00
github-actions[bot]
d92cc099c7 chore(core): localize to languages other than english (#11012)
Extended localization support by adding translations for multiple languages using English as the base. This enhances accessibility and usability for non-English-speaking users while keeping English as the source reference.

Co-authored-by: GitHub Action <actions@github.com>
2025-09-02 14:07:51 +02:00
Piyush Bhaskar
3d806022bc feat(executions): allow changing the value of input before replaying an execution (#11010) 2025-09-02 16:39:28 +05:30
Roman Acevedo
b72bafe344 Revert "chore(deps): bump com.gradleup.shadow to 9.1.0"
This reverts commit 55c89244b4.
2025-09-02 12:22:05 +02:00
Barthélémy Ledoux
5e2063aa57 fix(no-code): explore allOf clauses for $deprecated flags (#10999) 2025-09-02 11:35:46 +02:00
Miloš Paunović
3838f8c87f fix(executions): clear errors on selected value change in outputs tab (#11007)
Closes https://github.com/kestra-io/kestra/issues/10979.
2025-09-02 11:16:59 +02:00
Ludovic DEHON
55c89244b4 chore(deps): bump com.gradleup.shadow to 9.1.0 2025-09-02 10:02:03 +02:00
Roman Acevedo
59c9ae57b7 ci: fix setversion-tag.yml not triggering a main.yml job on a pushed tag
the missing token: ${{ secrets.GH_PERSONAL_TOKEN }} is the only difference between this CI and EE CI, so it is probably the right fix
2025-09-01 16:25:04 +02:00
Roman Acevedo
1b8a2cd19a tests: add default false value to ui-anonymous-usage-report.enabled
to avoid having to configure it in tests
2025-09-01 16:02:17 +02:00
Miloš Paunović
16992626d2 fix(core): allow removal of block items from no code editor (#10992)
Closes https://github.com/kestra-io/kestra-ee/issues/4862.

Co-authored-by: Bart Ledoux <bledoux@kestra.io>
2025-09-01 10:50:06 +02:00
Ludovic DEHON
2c94922736 feat(core): add thread http client, deadlock and virtual thread metrics 2025-08-31 23:57:34 +02:00
Ludovic DEHON
b45c0b13be refactor(ui): posthog as composable and option for ui telemetry
relate to kestra-io/kestra-ee#4831
2025-08-29 19:36:58 +02:00
Nicolas K.
aed055dcb1 fix(storage): delete the metadata with the key when we delete an KV s… (#10990)
* fix(storage): delete the metadata with the key when we delete an KV store entry

* Update core/src/main/java/io/kestra/core/storages/kv/InternalKVStore.java

Co-authored-by: brian-mulier-p <bmmulier@hotmail.fr>

* fix(storage): fix typo

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: brian-mulier-p <bmmulier@hotmail.fr>
2025-08-29 17:28:39 +02:00
Nicolas K.
cfe107705b feat(storage): add purge kv task (#10964)
* feat(storage): add purge kv task

* feat(storage): add purge kv task

* feat(storage): add sanity check

* feat(storage): clean code

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-08-29 16:24:41 +02:00
brian-mulier-p
58da5fe7d8 fix(flows): add version autocompletion (#10973)
closes kestra-io/kestra-ee#4702
2025-08-29 15:34:37 +02:00
brian.mulier
869f7c718c fix(secrets): base64-encoded secrets value obfuscation
closes kestra-io/kestra-ee#4639
2025-08-29 15:33:52 +02:00
brian.mulier
985ed2ac89 fix(secrets): empty secrets ignored in secret obfuscation
closes #10526
2025-08-29 15:33:52 +02:00
Loïc Mathieu
bf0361778d fix(executions): clear errors/finally/afterExecution branches when changing the state of a taskrun
As changing the state of a taskrun will restart the flow, if we didn't clear those branches, the flow would not resart properly.

Fixes https://github.com/kestra-io/kestra-ee/issues/3211
2025-08-29 15:13:06 +02:00
Miloš Paunović
04a6adf012 chore(executions): make dependency graph table links navigate to execution pages (#10988)
Closes https://github.com/kestra-io/kestra-ee/issues/4866.
2025-08-29 14:27:08 +02:00
Piyush Bhaskar
0ffb71c25d fix(ui): do not allow white space in password (#10987) 2025-08-29 16:47:45 +05:30
Miloš Paunović
63659eca79 chore(executions): optimize dependency graph state updates (#10985)
Closes https://github.com/kestra-io/kestra/issues/10795.
2025-08-29 12:58:03 +02:00
Loïc Mathieu
357d4e0d69 feat(executions): allow flow trigger on concurrency limit
Closes https://github.com/kestra-io/kestra-ee/issues/3270

This allow listening to the QUEUED state on the flow trigger.
This also fixes an issue that when concurrency limit is setup, you would not listen to the RUNNING state.
2025-08-29 11:07:43 +02:00
Piyush Bhaskar
cf301a1192 chore(core): fix border color of validation btn and add loading state in Action button (#10982) 2025-08-29 14:08:23 +05:30
Ludovic DEHON
bc08fc7d07 fix(core): disable useless health check 2025-08-28 20:47:02 +02:00
Ludovic DEHON
85ac124740 feat(core): add netty metrics on micrometer 2025-08-28 20:46:28 +02:00
Ludovic DEHON
8021257bf4 fix(core): align open source & ee configuration 2025-08-28 20:45:50 +02:00
717 changed files with 20522 additions and 9368 deletions

View File

@@ -23,7 +23,15 @@ In the meantime, you can move onto the next step...
---
### Requirements
- Java 21 (LTS versions).
> ⚠️ Java 24 and above are not supported yet and will fail with `invalid source release: 21`.
- Gradle (comes with wrapper `./gradlew`)
- Docker (optional, for running Kestra in containers)
### Development:
- (Optional) By default, your dev server will target `localhost:8080`. If your backend is running elsewhere, you can create `.env.development.local` under `ui` folder with this content:
```
VITE_APP_API_URL={myApiUrl}

View File

@@ -26,7 +26,7 @@ jobs:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: "3.x"
@@ -39,7 +39,7 @@ jobs:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Set up Node
uses: actions/setup-node@v4
uses: actions/setup-node@v5
with:
node-version: "20.x"

View File

@@ -37,7 +37,7 @@ jobs:
path: kestra
# Setup build
- uses: kestra-io/actions/.github/actions/setup-build@main
- uses: kestra-io/actions/composite/setup-build@main
name: Setup - Build
id: build
with:

View File

@@ -4,7 +4,7 @@ on:
workflow_dispatch:
inputs:
releaseVersion:
description: 'The release version (e.g., 0.21.0-rc1)'
description: 'The release version (e.g., 0.21.0)'
required: true
type: string
nextVersion:
@@ -25,21 +25,13 @@ jobs:
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v5
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
- uses: kestra-io/actions/composite/setup-build@main
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
caches-enabled: true
# Get Plugins List
- name: Get Plugins List
@@ -60,7 +52,7 @@ jobs:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./dev-tools/release-plugins.sh;
./dev-tools/release-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--next-version=${{github.event.inputs.nextVersion}} \
@@ -73,10 +65,10 @@ jobs:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./dev-tools/release-plugins.sh;
./dev-tools/release-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--next-version=${{github.event.inputs.nextVersion}} \
--dry-run \
--yes \
${{ steps.plugins-list.outputs.repositories }}
${{ steps.plugins-list.outputs.repositories }}

View File

@@ -4,7 +4,7 @@ on:
workflow_dispatch:
inputs:
releaseVersion:
description: 'The release version (e.g., 0.21.0-rc1)'
description: 'The release version (e.g., 0.21.0)'
required: true
type: string
nextVersion:
@@ -23,8 +23,8 @@ jobs:
# Checks
- name: Check Inputs
run: |
if ! [[ "$RELEASE_VERSION" =~ ^[0-9]+(\.[0-9]+)\.0-rc[01](-SNAPSHOT)?$ ]]; then
echo "Invalid release version. Must match regex: ^[0-9]+(\.[0-9]+)\.0-rc[01](-SNAPSHOT)?$"
if ! [[ "$RELEASE_VERSION" =~ ^[0-9]+(\.[0-9]+)\.0$ ]]; then
echo "Invalid release version. Must match regex: ^[0-9]+(\.[0-9]+)\.0$"
exit 1
fi
@@ -38,15 +38,8 @@ jobs:
fetch-depth: 0
path: kestra
# Checkout GitHub Actions
- uses: actions/checkout@v5
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
- uses: kestra-io/actions/composite/setup-build@main
id: build
with:
java-enabled: true
@@ -78,7 +71,6 @@ jobs:
git checkout develop;
if [[ "$RELEASE_VERSION" == *"-SNAPSHOT" ]]; then
# -SNAPSHOT qualifier maybe used to test release-candidates
./gradlew release -Prelease.useAutomaticVersion=true \
-Prelease.releaseVersion="${RELEASE_VERSION}" \
-Prelease.newVersion="${NEXT_VERSION}" \
@@ -89,4 +81,4 @@ jobs:
-Prelease.releaseVersion="${RELEASE_VERSION}" \
-Prelease.newVersion="${NEXT_VERSION}" \
-Prelease.pushReleaseVersionBranch="${PUSH_RELEASE_BRANCH}"
fi
fi

View File

@@ -0,0 +1,32 @@
name: kestra-devtools test
on:
pull_request:
branches:
- develop
paths:
- 'dev-tools/kestra-devtools/**'
env:
# to save corepack from itself
COREPACK_INTEGRITY_KEYS: 0
jobs:
test:
name: kestra-devtools tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Npm - install
working-directory: 'dev-tools/kestra-devtools'
run: npm ci
- name: Run tests
working-directory: 'dev-tools/kestra-devtools'
run: npm run test
- name: Npm - Run build
working-directory: 'dev-tools/kestra-devtools'
run: npm run build

View File

@@ -3,6 +3,14 @@ name: Main Workflow
on:
workflow_dispatch:
inputs:
skip-test:
description: 'Skip test'
type: choice
required: true
default: 'false'
options:
- "true"
- "false"
plugin-version:
description: "plugins version"
required: false
@@ -24,13 +32,14 @@ jobs:
tests:
name: Execute tests
uses: ./.github/workflows/workflow-test.yml
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
with:
report-status: false
release:
name: Release
needs: [tests]
if: "!startsWith(github.ref, 'refs/heads/releases')"
if: "!failure() && !cancelled() && !startsWith(github.ref, 'refs/heads/releases')"
uses: ./.github/workflows/workflow-release.yml
with:
plugin-version: ${{ inputs.plugin-version != '' && inputs.plugin-version || (github.ref == 'refs/heads/develop' && 'LATEST-SNAPSHOT' || 'LATEST') }}
@@ -44,13 +53,12 @@ jobs:
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE }}
GH_PERSONAL_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
SLACK_RELEASES_WEBHOOK_URL: ${{ secrets.SLACK_RELEASES_WEBHOOK_URL }}
end:
runs-on: ubuntu-latest
needs:
- release
if: always()
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
steps:
- name: Trigger EE Workflow
uses: peter-evans/repository-dispatch@v3
@@ -60,14 +68,9 @@ jobs:
repository: kestra-io/kestra-ee
event-type: "oss-updated"
# Slack
- name: Slack - Notification
uses: Gamesight/slack-workflow-status@master
if: ${{ always() && env.SLACK_WEBHOOK_URL != 0 }}
if: ${{ failure() && env.SLACK_WEBHOOK_URL != 0 && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop') }}
uses: kestra-io/actions/composite/slack-status@main
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
slack_webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
name: GitHub Actions
icon_emoji: ":github-actions:"
channel: "C02DQ1A7JLR" # _int_git channel
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}

View File

@@ -60,19 +60,3 @@ jobs:
name: E2E - Tests
uses: ./.github/workflows/e2e.yml
end:
name: End
runs-on: ubuntu-latest
if: always()
needs: [frontend, backend]
steps:
# Slack
- name: Slack notification
uses: Gamesight/slack-workflow-status@master
if: ${{ always() && env.SLACK_WEBHOOK_URL != 0 }}
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
slack_webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
name: GitHub Actions
icon_emoji: ":github-actions:"
channel: "C02DQ1A7JLR"

View File

@@ -34,11 +34,14 @@ jobs:
fi
# Checkout
- uses: actions/checkout@v5
- name: Checkout
uses: actions/checkout@v5
with:
fetch-depth: 0
token: ${{ secrets.GH_PERSONAL_TOKEN }}
- name: Configure Git
# Configure
- name: Git - Configure
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
@@ -54,4 +57,4 @@ jobs:
git commit -m"chore(version): update to version '$RELEASE_VERSION'"
git push
git tag -a "v$RELEASE_VERSION" -m"v$RELEASE_VERSION"
git push --tags
git push --tags

View File

@@ -21,13 +21,6 @@ jobs:
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v5
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
@@ -70,15 +63,8 @@ jobs:
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v5
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
- uses: kestra-io/actions/composite/setup-build@main
id: build
with:
java-enabled: false
@@ -87,7 +73,7 @@ jobs:
# Run Trivy image scan for Docker vulnerabilities, see https://github.com/aquasecurity/trivy-action
- name: Docker Vulnerabilities Check
uses: aquasecurity/trivy-action@0.32.0
uses: aquasecurity/trivy-action@0.33.1
with:
image-ref: kestra/kestra:develop
format: 'template'
@@ -115,24 +101,16 @@ jobs:
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v5
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
- uses: kestra-io/actions/composite/setup-build@main
id: build
with:
java-enabled: false
node-enabled: false
caches-enabled: true
# Run Trivy image scan for Docker vulnerabilities, see https://github.com/aquasecurity/trivy-action
- name: Docker Vulnerabilities Check
uses: aquasecurity/trivy-action@0.32.0
uses: aquasecurity/trivy-action@0.33.1
with:
image-ref: kestra/kestra:latest
format: table

View File

@@ -20,6 +20,7 @@ permissions:
contents: write
checks: write
actions: read
pull-requests: write
jobs:
test:
@@ -35,7 +36,7 @@ jobs:
fetch-depth: 0
# Setup build
- uses: kestra-io/actions/.github/actions/setup-build@main
- uses: kestra-io/actions/composite/setup-build@main
name: Setup - Build
id: build
with:
@@ -59,6 +60,30 @@ jobs:
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp-service-account.json
./gradlew check javadoc --parallel
- name: comment PR with test report
if: ${{ !cancelled() && github.event_name == 'pull_request' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_AUTH_TOKEN }}
run: |
export KESTRA_PWD=$(pwd) && sh -c 'cd dev-tools/kestra-devtools && npm ci && npm run build && node dist/kestra-devtools-cli.cjs generateTestReportSummary --only-errors --ci $KESTRA_PWD' > report.md
cat report.md
# Gradle check
- name: 'generate Timeline flamegraph'
if: always()
env:
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
shell: bash
run: |
echo $GOOGLE_SERVICE_ACCOUNT | base64 -d > ~/.gcp-service-account.json
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp-service-account.json
./gradlew mergeTestTimeline
- name: 'Upload Timeline flamegraph'
uses: actions/upload-artifact@v4
if: always()
with:
name: all-test-timelines.json
path: build/reports/test-timelines-report/all-test-timelines.json
retention-days: 5
# report test
- name: Test - Publish Test Results
uses: dorny/test-reporter@v2
@@ -86,13 +111,13 @@ jobs:
id: auth
if: always() && env.GOOGLE_SERVICE_ACCOUNT != ''
continue-on-error: true
uses: "google-github-actions/auth@v2"
uses: "google-github-actions/auth@v3"
with:
credentials_json: "${{ secrets.GOOGLE_SERVICE_ACCOUNT }}"
- name: GCP - Setup Cloud SDK
if: env.GOOGLE_SERVICE_ACCOUNT != ''
uses: "google-github-actions/setup-gcloud@v2"
uses: "google-github-actions/setup-gcloud@v3"
# Allure check
- uses: rlespinasse/github-slug-action@v5

View File

@@ -26,7 +26,7 @@ jobs:
run: npm ci
# Setup build
- uses: kestra-io/actions/.github/actions/setup-build@main
- uses: kestra-io/actions/composite/setup-build@main
name: Setup - Build
id: build
with:

View File

@@ -25,15 +25,6 @@ jobs:
fetch-depth: 0
submodules: true
# Checkout GitHub Actions
- name: Checkout - Actions
uses: actions/checkout@v5
with:
repository: kestra-io/actions
sparse-checkout-cone-mode: true
path: actions
sparse-checkout: |
.github/actions
# Download Exec
# Must be done after checkout actions
@@ -59,7 +50,7 @@ jobs:
# GitHub Release
- name: Create GitHub release
uses: ./actions/.github/actions/github-release
uses: kestra-io/actions/composite/github-release@main
if: ${{ startsWith(github.ref, 'refs/tags/v') }}
env:
MAKE_LATEST: ${{ steps.is_latest.outputs.latest }}
@@ -82,7 +73,7 @@ jobs:
- name: Merge Release Notes
if: ${{ startsWith(github.ref, 'refs/tags/v') }}
uses: ./actions/.github/actions/github-release-note-merge
uses: kestra-io/actions/composite/github-release-note-merge@main
env:
GITHUB_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
RELEASE_TAG: ${{ github.ref_name }}

View File

@@ -11,6 +11,14 @@ on:
options:
- "true"
- "false"
retag-lts:
description: 'Retag LTS Docker images'
required: true
type: choice
default: "false"
options:
- "true"
- "false"
release-tag:
description: 'Kestra Release Tag (by default, deduced with the ref)'
required: false
@@ -179,6 +187,11 @@ jobs:
run: |
regctl image copy ${{ format('kestra/kestra:{0}{1}', steps.vars.outputs.tag, matrix.image.name) }} ${{ format('kestra/kestra:latest{0}', matrix.image.name) }}
- name: Retag to LTS
if: startsWith(github.ref, 'refs/tags/v') && inputs.retag-lts == 'true'
run: |
regctl image copy ${{ format('kestra/kestra:{0}{1}', steps.vars.outputs.tag, matrix.image.name) }} ${{ format('kestra/kestra:latest-lts{0}', matrix.image.name) }}
end:
runs-on: ubuntu-latest
needs:
@@ -187,14 +200,9 @@ jobs:
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
steps:
# Slack
- name: Slack notification
uses: Gamesight/slack-workflow-status@master
if: ${{ always() && env.SLACK_WEBHOOK_URL != 0 }}
if: ${{ failure() && env.SLACK_WEBHOOK_URL != 0 }}
uses: kestra-io/actions/composite/slack-status@main
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
slack_webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
name: GitHub Actions
icon_emoji: ':github-actions:'
channel: 'C02DQ1A7JLR' # _int_git channel
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}

View File

@@ -29,7 +29,7 @@ jobs:
# Setup build
- name: Setup - Build
uses: kestra-io/actions/.github/actions/setup-build@main
uses: kestra-io/actions/composite/setup-build@main
id: build
with:
java-enabled: true

View File

@@ -7,7 +7,7 @@ on:
jobs:
publish:
name: Pull Request - Delete Docker
if: github.repository == github.event.pull_request.head.repo.full_name # prevent running on forks
if: github.repository == 'kestra-io/kestra' # prevent running on forks
runs-on: ubuntu-latest
steps:
- uses: dataaxiom/ghcr-cleanup-action@v1

View File

@@ -8,12 +8,12 @@ on:
jobs:
build-artifacts:
name: Build Artifacts
if: github.repository == github.event.pull_request.head.repo.full_name # prevent running on forks
if: github.repository == 'kestra-io/kestra' # prevent running on forks
uses: ./.github/workflows/workflow-build-artifacts.yml
publish:
name: Publish Docker
if: github.repository == github.event.pull_request.head.repo.full_name # prevent running on forks
if: github.repository == 'kestra-io/kestra' # prevent running on forks
runs-on: ubuntu-latest
needs: build-artifacts
env:
@@ -62,7 +62,7 @@ jobs:
# Add comment on pull request
- name: Add comment to PR
uses: actions/github-script@v7
uses: actions/github-script@v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |

View File

@@ -84,14 +84,12 @@ jobs:
name: Notify - Slack
runs-on: ubuntu-latest
needs: [ frontend, backend ]
if: github.event_name == 'schedule'
steps:
- name: Notify failed CI
id: send-ci-failed
if: |
always() && (needs.frontend.result != 'success' ||
needs.backend.result != 'success')
uses: kestra-io/actions/.github/actions/send-ci-failed@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
always() &&
(needs.frontend.result != 'success' || needs.backend.result != 'success') &&
(github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop')
uses: kestra-io/actions/composite/slack-status@main
with:
webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}

View File

@@ -36,6 +36,7 @@
#plugin-gemini:io.kestra.plugin:plugin-gemini:LATEST
#plugin-git:io.kestra.plugin:plugin-git:LATEST
#plugin-github:io.kestra.plugin:plugin-github:LATEST
#plugin-gitlab:io.kestra.plugin:plugin-gitlab:LATEST
#plugin-googleworkspace:io.kestra.plugin:plugin-googleworkspace:LATEST
#plugin-graalvm:io.kestra.plugin:plugin-graalvm:LATEST
#plugin-graphql:io.kestra.plugin:plugin-graphql:LATEST
@@ -108,16 +109,17 @@
#plugin-serdes:io.kestra.plugin:plugin-serdes:LATEST
#plugin-servicenow:io.kestra.plugin:plugin-servicenow:LATEST
#plugin-sifflet:io.kestra.plugin:plugin-sifflet:LATEST
#plugin-singer:io.kestra.plugin:plugin-singer:LATEST
#plugin-soda:io.kestra.plugin:plugin-soda:LATEST
#plugin-solace:io.kestra.plugin:plugin-solace:LATEST
#plugin-spark:io.kestra.plugin:plugin-spark:LATEST
#plugin-sqlmesh:io.kestra.plugin:plugin-sqlmesh:LATEST
#plugin-supabase:io.kestra.plugin:plugin-supabase:LATEST
#plugin-surrealdb:io.kestra.plugin:plugin-surrealdb:LATEST
#plugin-terraform:io.kestra.plugin:plugin-terraform:LATEST
#plugin-transform:io.kestra.plugin:plugin-transform-grok:LATEST
#plugin-transform:io.kestra.plugin:plugin-transform-json:LATEST
#plugin-tika:io.kestra.plugin:plugin-tika:LATEST
#plugin-trivy:io.kestra.plugin:plugin-trivy:LATEST
#plugin-weaviate:io.kestra.plugin:plugin-weaviate:LATEST
#plugin-zendesk:io.kestra.plugin:plugin-zendesk:LATEST
#plugin-typesense:io.kestra.plugin:plugin-typesense:LATEST

View File

@@ -89,7 +89,7 @@ build-docker: build-exec
--compress \
--rm \
-f ./Dockerfile \
--build-arg="APT_PACKAGES=python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip jattach" \
--build-arg="APT_PACKAGES=python3 python-is-python3 python3-pip curl jattach" \
--build-arg="PYTHON_LIBRARIES=kestra" \
-t ${DOCKER_IMAGE}:${VERSION} ${DOCKER_PATH} || exit 1 ;

View File

@@ -21,7 +21,7 @@ plugins {
// test
id "com.adarshr.test-logger" version "4.0.0"
id "org.sonarqube" version "6.2.0.5505"
id "org.sonarqube" version "6.3.1.5724"
id 'jacoco-report-aggregation'
// helper
@@ -32,7 +32,7 @@ plugins {
// release
id 'net.researchgate.release' version '3.1.0'
id "com.gorylenko.gradle-git-properties" version "2.5.2"
id "com.gorylenko.gradle-git-properties" version "2.5.3"
id 'signing'
id "com.vanniktech.maven.publish" version "0.34.0"
@@ -168,8 +168,9 @@ allprojects {
/**********************************************************************************************************************\
* Test
**********************************************************************************************************************/
subprojects {
if (it.name != 'platform' && it.name != 'jmh-benchmarks') {
subprojects {subProj ->
if (subProj.name != 'platform' && subProj.name != 'jmh-benchmarks') {
apply plugin: "com.adarshr.test-logger"
java {
@@ -207,6 +208,13 @@ subprojects {
test {
useJUnitPlatform()
reports {
junitXml.required = true
junitXml.outputPerTestCase = true
junitXml.mergeReruns = true
junitXml.includeSystemErrLog = true;
junitXml.outputLocation = layout.buildDirectory.dir("test-results/test")
}
// set Xmx for test workers
maxHeapSize = '4g'
@@ -222,6 +230,52 @@ subprojects {
environment 'SECRET_PASSWORD', "cGFzc3dvcmQ="
environment 'ENV_TEST1', "true"
environment 'ENV_TEST2', "Pass by env"
// === Test Timeline Trace (Chrome trace format) ===
// Produces per-JVM ndjson under build/test-timelines/*.jsonl and a merged array via :mergeTestTimeline
// Each event has: start time (ts, µs since epoch), end via dur, and absolute duration (dur, µs)
doFirst {
file("${buildDir}/test-results/test-timelines").mkdirs()
}
def jvmName = java.lang.management.ManagementFactory.runtimeMXBean.name
def pid = jvmName.tokenize('@')[0]
def traceDir = file("${buildDir}/test-results/test-timelines")
def traceFile = new File(traceDir, "${project.name}-${name}-${pid}.jsonl")
def starts = new java.util.concurrent.ConcurrentHashMap<Object, Long>()
beforeTest { org.gradle.api.tasks.testing.TestDescriptor d ->
// epoch millis to allow cross-JVM merge
starts.put(d, System.currentTimeMillis())
}
afterTest { org.gradle.api.tasks.testing.TestDescriptor d, org.gradle.api.tasks.testing.TestResult r ->
def st = starts.remove(d)
if (st != null) {
def en = System.currentTimeMillis()
long tsMicros = st * 1000L // start time (µs since epoch)
long durMicros = (en - st) * 1000L // duration (µs)
def ev = [
name: (d.className ? d.className + '.' + d.name : d.name),
cat : 'test',
ph : 'X', // Complete event with duration
ts : tsMicros,
dur : durMicros,
pid : project.name, // group by project/module
tid : "${name}-worker-${pid}",
args: [result: r.resultType.toString()]
]
synchronized (traceFile.absolutePath.intern()) {
traceFile << (groovy.json.JsonOutput.toJson(ev) + System.lineSeparator())
}
}
}
if (subProj.name == 'core' || subProj.name == 'jdbc-h2') {
// JUnit 5 parallel settings
systemProperty 'junit.jupiter.execution.parallel.enabled', 'true'
systemProperty 'junit.jupiter.execution.parallel.mode.default', 'concurrent'
systemProperty 'junit.jupiter.execution.parallel.mode.classes.default', 'same_thread'
systemProperty 'junit.jupiter.execution.parallel.config.strategy', 'dynamic'
}
}
testlogger {
@@ -236,7 +290,53 @@ subprojects {
}
}
}
// Root-level aggregator: merge timelines from ALL modules into one Chrome trace
if (project == rootProject) {
tasks.register('mergeTestTimeline') {
group = 'verification'
description = 'Merge per-worker test timeline ndjson from all modules into a single Chrome Trace JSON array.'
doLast {
def collectedFiles = [] as List<File>
// Collect *.jsonl files from every subproject
rootProject.subprojects.each { p ->
def dir = p.file("${p.buildDir}/test-results/test-timelines")
if (dir.exists()) {
collectedFiles.addAll(p.fileTree(dir: dir, include: '*.jsonl').files)
}
}
if (collectedFiles.isEmpty()) {
logger.lifecycle("No timeline files found in any subproject. Run tests first (e.g., './gradlew test --parallel').")
return
}
collectedFiles = collectedFiles.sort { it.name }
def outDir = rootProject.file("${rootProject.buildDir}/reports/test-timelines-report")
outDir.mkdirs()
def out = new File(outDir, "all-test-timelines.json")
out.withWriter('UTF-8') { w ->
w << '['
boolean first = true
collectedFiles.each { f ->
f.eachLine { line ->
def trimmed = line?.trim()
if (trimmed) {
if (!first) w << ','
w << trimmed
first = false
}
}
}
w << ']'
}
logger.lifecycle("Merged ${collectedFiles.size()} files into ${out} — open it in chrome://tracing or Perfetto UI.")
}
}
}
/**********************************************************************************************************************\
* End-to-End Tests
**********************************************************************************************************************/

View File

@@ -33,8 +33,13 @@ dependencies {
implementation project(":storage-local")
// Kestra server components
implementation project(":executor")
implementation project(":scheduler")
implementation project(":webserver")
implementation project(":worker")
//test
testImplementation project(':tests')
testImplementation "org.wiremock:wiremock-jetty12"
}

View File

@@ -89,11 +89,24 @@ public class App implements Callable<Integer> {
*/
protected static ApplicationContext applicationContext(Class<?> mainClass,
String[] args) {
return applicationContext(mainClass, new String [] { Environment.CLI }, args);
}
/**
* Create an {@link ApplicationContext} with additional properties based on configuration files (--config) and
* forced Properties from current command.
*
* @param args args passed to java app
* @return the application context created
*/
protected static ApplicationContext applicationContext(Class<?> mainClass,
String[] environments,
String[] args) {
ApplicationContextBuilder builder = ApplicationContext
.builder()
.mainClass(mainClass)
.environments(Environment.CLI);
.environments(environments);
CommandLine cmd = new CommandLine(mainClass, CommandLine.defaultFactory());
continueOnParsingErrors(cmd);

View File

@@ -1,15 +1,14 @@
package io.kestra.core.runners;
package io.kestra.cli;
import io.kestra.core.schedulers.AbstractScheduler;
import io.kestra.core.runners.*;
import io.kestra.core.server.Service;
import io.kestra.core.utils.Await;
import io.kestra.core.utils.ExecutorsUtils;
import io.kestra.worker.DefaultWorker;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.annotation.Requires;
import io.micronaut.context.annotation.Value;
import jakarta.annotation.PreDestroy;
import jakarta.inject.Inject;
import jakarta.inject.Singleton;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
@@ -24,9 +23,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
@SuppressWarnings("try")
@Slf4j
@Singleton
@Requires(missingBeans = RunnerInterface.class)
public class StandAloneRunner implements RunnerInterface, AutoCloseable {
public class StandAloneRunner implements Runnable, AutoCloseable {
@Setter protected int workerThread = Math.max(3, Runtime.getRuntime().availableProcessors());
@Setter protected boolean schedulerEnabled = true;
@Setter protected boolean workerEnabled = true;
@@ -45,7 +42,7 @@ public class StandAloneRunner implements RunnerInterface, AutoCloseable {
private final AtomicBoolean running = new AtomicBoolean(false);
private volatile ExecutorService poolExecutor;
private ExecutorService poolExecutor;
@Override
public void run() {
@@ -57,20 +54,20 @@ public class StandAloneRunner implements RunnerInterface, AutoCloseable {
if (workerEnabled) {
// FIXME: For backward-compatibility with Kestra 0.15.x and earliest we still used UUID for Worker ID instead of IdUtils
String workerID = UUID.randomUUID().toString();
Worker worker = applicationContext.createBean(Worker.class, workerID, workerThread, null);
Worker worker = applicationContext.createBean(DefaultWorker.class, workerID, workerThread, null);
applicationContext.registerSingleton(worker); //
poolExecutor.execute(worker);
servers.add(worker);
}
if (schedulerEnabled) {
AbstractScheduler scheduler = applicationContext.getBean(AbstractScheduler.class);
Scheduler scheduler = applicationContext.getBean(Scheduler.class);
poolExecutor.execute(scheduler);
servers.add(scheduler);
}
if (indexerEnabled) {
IndexerInterface indexer = applicationContext.getBean(IndexerInterface.class);
Indexer indexer = applicationContext.getBean(Indexer.class);
poolExecutor.execute(indexer);
servers.add(indexer);
}

View File

@@ -8,7 +8,7 @@ import io.kestra.core.repositories.FlowRepositoryInterface;
import io.kestra.core.repositories.LocalFlowRepositoryLoader;
import io.kestra.core.runners.FlowInputOutput;
import io.kestra.core.runners.RunnerUtils;
import io.kestra.core.runners.StandAloneRunner;
import io.kestra.cli.StandAloneRunner;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;
import jakarta.validation.ConstraintViolationException;
@@ -72,7 +72,6 @@ public class FlowTestCommand extends AbstractApiCommand {
public Integer call() throws Exception {
super.call();
StandAloneRunner runner = applicationContext.getBean(StandAloneRunner.class);
LocalFlowRepositoryLoader repositoryLoader = applicationContext.getBean(LocalFlowRepositoryLoader.class);
FlowRepositoryInterface flowRepository = applicationContext.getBean(FlowRepositoryInterface.class);
FlowInputOutput flowInputOutput = applicationContext.getBean(FlowInputOutput.class);
@@ -89,7 +88,7 @@ public class FlowTestCommand extends AbstractApiCommand {
inputs.put(this.inputs.get(i), this.inputs.get(i+1));
}
try {
try (StandAloneRunner runner = applicationContext.createBean(StandAloneRunner.class);){
runner.run();
repositoryLoader.load(tenantService.getTenantId(tenantId), file.toFile());
@@ -103,8 +102,6 @@ public class FlowTestCommand extends AbstractApiCommand {
(flow, execution) -> flowInputOutput.readExecutionInputs(flow, execution, inputs),
Duration.ofHours(1)
);
runner.close();
} catch (ConstraintViolationException e) {
throw new CommandLine.ParameterException(this.spec.commandLine(), e.getMessage());
} catch (IOException | TimeoutException e) {

View File

@@ -3,7 +3,7 @@ package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.ExecutorInterface;
import io.kestra.core.services.SkipExecutionService;
import io.kestra.executor.SkipExecutionService;
import io.kestra.core.services.StartExecutorService;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;

View File

@@ -2,7 +2,7 @@ package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.IndexerInterface;
import io.kestra.core.runners.Indexer;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;
@@ -29,7 +29,7 @@ public class IndexerCommand extends AbstractServerCommand {
public Integer call() throws Exception {
super.call();
IndexerInterface indexer = applicationContext.getBean(IndexerInterface.class);
Indexer indexer = applicationContext.getBean(Indexer.class);
indexer.run();
Await.until(() -> !this.applicationContext.isRunning());

View File

@@ -2,7 +2,7 @@ package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.models.ServerType;
import io.kestra.core.schedulers.AbstractScheduler;
import io.kestra.scheduler.AbstractScheduler;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;

View File

@@ -6,8 +6,8 @@ import io.kestra.cli.services.TenantIdSelectorService;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.repositories.LocalFlowRepositoryLoader;
import io.kestra.core.runners.StandAloneRunner;
import io.kestra.core.services.SkipExecutionService;
import io.kestra.cli.StandAloneRunner;
import io.kestra.executor.SkipExecutionService;
import io.kestra.core.services.StartExecutorService;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
@@ -109,26 +109,27 @@ public class StandAloneCommand extends AbstractServerCommand {
}
}
StandAloneRunner standAloneRunner = applicationContext.getBean(StandAloneRunner.class);
try (StandAloneRunner standAloneRunner = applicationContext.getBean(StandAloneRunner.class)) {
if (this.workerThread == 0) {
standAloneRunner.setWorkerEnabled(false);
} else {
standAloneRunner.setWorkerThread(this.workerThread);
if (this.workerThread == 0) {
standAloneRunner.setWorkerEnabled(false);
} else {
standAloneRunner.setWorkerThread(this.workerThread);
}
if (this.indexerDisabled) {
standAloneRunner.setIndexerEnabled(false);
}
standAloneRunner.run();
if (fileWatcher != null) {
fileWatcher.startListeningFromConfig();
}
Await.until(() -> !this.applicationContext.isRunning());
}
if (this.indexerDisabled) {
standAloneRunner.setIndexerEnabled(false);
}
standAloneRunner.run();
if (fileWatcher != null) {
fileWatcher.startListeningFromConfig();
}
Await.until(() -> !this.applicationContext.isRunning());
return 0;
}
}

View File

@@ -2,7 +2,7 @@ package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.IndexerInterface;
import io.kestra.core.runners.Indexer;
import io.kestra.core.utils.Await;
import io.kestra.core.utils.ExecutorsUtils;
import io.micronaut.context.ApplicationContext;
@@ -54,7 +54,7 @@ public class WebServerCommand extends AbstractServerCommand {
if (!indexerDisabled) {
log.info("Starting an embedded indexer, this can be disabled by using `--no-indexer`.");
poolExecutor = executorsUtils.cachedThreadPool("webserver-indexer");
poolExecutor.execute(applicationContext.getBean(IndexerInterface.class));
poolExecutor.execute(applicationContext.getBean(Indexer.class));
shutdownHook(false, () -> poolExecutor.shutdown());
}

View File

@@ -262,6 +262,8 @@ public class FileChangedEventListener {
}
private String getTenantIdFromPath(Path path) {
// FIXME there is probably a bug here when a tenant has '_' in its name,
// a valid tenant name is defined with following regex: "^[a-z0-9][a-z0-9_-]*"
return path.getFileName().toString().split("_")[0];
}
}

View File

@@ -18,6 +18,10 @@ micronaut:
root:
paths: classpath:root
mapping: /**
codec:
json:
additional-types:
- application/scim+json
server:
max-request-size: 10GB
multipart:
@@ -78,8 +82,19 @@ micronaut:
type: scheduled
core-pool-size: 1
# Disable OpenTelemetry metrics by default, users that need it must enable it and configure the collector URL.
metrics:
binders:
retry:
enabled: true
netty:
queues:
enabled: true
bytebuf-allocators:
enabled: true
channels:
enabled: true
# Disable OpenTelemetry metrics by default, users that need it must enable it and configure the collector URL.
export:
otlp:
enabled: false
@@ -92,6 +107,8 @@ jackson:
serialization-inclusion: non_null
deserialization:
FAIL_ON_UNKNOWN_PROPERTIES: false
mapper:
ACCEPT_CASE_INSENSITIVE_ENUMS: true
endpoints:
all:
@@ -100,6 +117,10 @@ endpoints:
sensitive: false
health:
details-visible: ANONYMOUS
disk-space:
enabled: false
discovery-client:
enabled: false
loggers:
write-sensitive: false
env:
@@ -133,12 +154,44 @@ kestra:
tutorial-flows:
# Automatically loads all tutorial flows at startup.
enabled: true
retries:
attempts: 5
multiplier: 2.0
delay: 1s
maxDelay: ""
server:
basic-auth:
# These URLs will not be authenticated, by default we open some of the Micronaut default endpoints but not all for security reasons
open-urls:
- "/ping"
- "/api/v1/executions/webhook/"
preview:
initial-rows: 100
max-rows: 5000
# The expected time for this server to complete all its tasks before initiating a graceful shutdown.
terminationGracePeriod: 5m
workerTaskRestartStrategy: AFTER_TERMINATION_GRACE_PERIOD
# Configuration for Liveness and Heartbeat mechanism between servers.
liveness:
enabled: true
# The expected time between liveness probe.
interval: 10s
# The timeout used to detect service failures.
timeout: 1m
# The time to wait before executing a liveness probe.
initialDelay: 1m
# The expected time between service heartbeats.
heartbeatInterval: 3s
service:
purge:
initial-delay: 1h
fixed-delay: 1d
retention: 30d
jdbc:
queues:
min-poll-interval: 25ms
@@ -150,7 +203,7 @@ kestra:
fixed-delay: 1h
retention: 7d
types:
- type : io.kestra.core.models.executions.LogEntry
- type: io.kestra.core.models.executions.LogEntry
retention: 1h
- type: io.kestra.core.models.executions.MetricEntry
retention: 1h
@@ -182,37 +235,12 @@ kestra:
traces:
root: DISABLED
server:
basic-auth:
# These URLs will not be authenticated, by default we open some of the Micronaut default endpoints but not all for security reasons
open-urls:
- "/ping"
- "/api/v1/executions/webhook/"
preview:
initial-rows: 100
max-rows: 5000
# The expected time for this server to complete all its tasks before initiating a graceful shutdown.
terminationGracePeriod: 5m
workerTaskRestartStrategy: AFTER_TERMINATION_GRACE_PERIOD
# Configuration for Liveness and Heartbeat mechanism between servers.
liveness:
enabled: true
# The expected time between liveness probe.
interval: 10s
# The timeout used to detect service failures.
timeout: 1m
# The time to wait before executing a liveness probe.
initialDelay: 1m
# The expected time between service heartbeats.
heartbeatInterval: 3s
service:
purge:
initial-delay: 1h
fixed-delay: 1d
retention: 30d
ui-anonymous-usage-report:
enabled: true
anonymous-usage-report:
enabled: true
uri: https://api.kestra.io/v1/server-events/
uri: https://api.kestra.io/v1/reports/server-events
initial-delay: 5m
fixed-delay: 1h

View File

@@ -4,11 +4,11 @@ import io.kestra.core.models.flows.Flow;
import io.kestra.core.models.flows.GenericFlow;
import io.kestra.core.repositories.FlowRepositoryInterface;
import io.kestra.core.utils.Await;
import io.kestra.core.utils.TestsUtils;
import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
import jakarta.inject.Inject;
import org.apache.commons.io.FileUtils;
import org.junit.jupiter.api.*;
import org.junitpioneer.jupiter.RetryingTest;
import java.io.IOException;
import java.nio.file.Files;
@@ -19,7 +19,6 @@ import java.util.concurrent.Executors;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import static io.kestra.core.tenant.TenantService.MAIN_TENANT;
import static io.kestra.core.utils.Rethrow.throwRunnable;
import static org.assertj.core.api.Assertions.assertThat;
@@ -57,10 +56,11 @@ class FileChangedEventListenerTest {
}
}
@RetryingTest(5) // Flaky on CI but always pass locally
@Test
void test() throws IOException, TimeoutException {
var tenant = TestsUtils.randomTenant(FileChangedEventListenerTest.class.getSimpleName(), "test");
// remove the flow if it already exists
flowRepository.findByIdWithSource(MAIN_TENANT, "io.kestra.tests.watch", "myflow").ifPresent(flow -> flowRepository.delete(flow));
flowRepository.findByIdWithSource(tenant, "io.kestra.tests.watch", "myflow").ifPresent(flow -> flowRepository.delete(flow));
// create a basic flow
String flow = """
@@ -73,14 +73,14 @@ class FileChangedEventListenerTest {
message: Hello World! 🚀
""";
GenericFlow genericFlow = GenericFlow.fromYaml(MAIN_TENANT, flow);
GenericFlow genericFlow = GenericFlow.fromYaml(tenant, flow);
Files.write(Path.of(FILE_WATCH + "/" + genericFlow.uidWithoutRevision() + ".yaml"), flow.getBytes());
Await.until(
() -> flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "myflow").isPresent(),
() -> flowRepository.findById(tenant, "io.kestra.tests.watch", "myflow").isPresent(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
Flow myflow = flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "myflow").orElseThrow();
Flow myflow = flowRepository.findById(tenant, "io.kestra.tests.watch", "myflow").orElseThrow();
assertThat(myflow.getTasks()).hasSize(1);
assertThat(myflow.getTasks().getFirst().getId()).isEqualTo("hello");
assertThat(myflow.getTasks().getFirst().getType()).isEqualTo("io.kestra.plugin.core.log.Log");
@@ -88,16 +88,17 @@ class FileChangedEventListenerTest {
// delete the flow
Files.delete(Path.of(FILE_WATCH + "/" + genericFlow.uidWithoutRevision() + ".yaml"));
Await.until(
() -> flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "myflow").isEmpty(),
() -> flowRepository.findById(tenant, "io.kestra.tests.watch", "myflow").isEmpty(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
}
@RetryingTest(5) // Flaky on CI but always pass locally
@Test
void testWithPluginDefault() throws IOException, TimeoutException {
var tenant = TestsUtils.randomTenant(FileChangedEventListenerTest.class.getName(), "testWithPluginDefault");
// remove the flow if it already exists
flowRepository.findByIdWithSource(MAIN_TENANT, "io.kestra.tests.watch", "pluginDefault").ifPresent(flow -> flowRepository.delete(flow));
flowRepository.findByIdWithSource(tenant, "io.kestra.tests.watch", "pluginDefault").ifPresent(flow -> flowRepository.delete(flow));
// create a flow with plugin default
String pluginDefault = """
@@ -113,14 +114,14 @@ class FileChangedEventListenerTest {
values:
message: Hello World!
""";
GenericFlow genericFlow = GenericFlow.fromYaml(MAIN_TENANT, pluginDefault);
GenericFlow genericFlow = GenericFlow.fromYaml(tenant, pluginDefault);
Files.write(Path.of(FILE_WATCH + "/" + genericFlow.uidWithoutRevision() + ".yaml"), pluginDefault.getBytes());
Await.until(
() -> flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "pluginDefault").isPresent(),
() -> flowRepository.findById(tenant, "io.kestra.tests.watch", "pluginDefault").isPresent(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
Flow pluginDefaultFlow = flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "pluginDefault").orElseThrow();
Flow pluginDefaultFlow = flowRepository.findById(tenant, "io.kestra.tests.watch", "pluginDefault").orElseThrow();
assertThat(pluginDefaultFlow.getTasks()).hasSize(1);
assertThat(pluginDefaultFlow.getTasks().getFirst().getId()).isEqualTo("helloWithDefault");
assertThat(pluginDefaultFlow.getTasks().getFirst().getType()).isEqualTo("io.kestra.plugin.core.log.Log");
@@ -128,7 +129,7 @@ class FileChangedEventListenerTest {
// delete both files
Files.delete(Path.of(FILE_WATCH + "/" + genericFlow.uidWithoutRevision() + ".yaml"));
Await.until(
() -> flowRepository.findById(MAIN_TENANT, "io.kestra.tests.watch", "pluginDefault").isEmpty(),
() -> flowRepository.findById(tenant, "io.kestra.tests.watch", "pluginDefault").isEmpty(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);

View File

@@ -42,10 +42,10 @@ dependencies {
// plugins
implementation 'org.apache.maven.resolver:maven-resolver-impl'
implementation 'org.apache.maven.resolver:maven-resolver-supplier'
implementation 'org.apache.maven.resolver:maven-resolver-supplier-mvn3'
implementation 'org.apache.maven.resolver:maven-resolver-connector-basic'
implementation 'org.apache.maven.resolver:maven-resolver-transport-file'
implementation 'org.apache.maven.resolver:maven-resolver-transport-http'
implementation 'org.apache.maven.resolver:maven-resolver-transport-apache'
// scheduler
implementation group: 'com.cronutils', name: 'cron-utils'
@@ -63,6 +63,10 @@ dependencies {
exclude group: 'com.fasterxml.jackson.core'
}
// micrometer
implementation "io.micronaut.micrometer:micronaut-micrometer-observation"
implementation 'io.micrometer:micrometer-java21'
// test
testAnnotationProcessor project(':processor')
testImplementation project(':tests')
@@ -70,6 +74,9 @@ dependencies {
testImplementation project(':repository-memory')
testImplementation project(':runner-memory')
testImplementation project(':storage-local')
testImplementation project(':worker')
testImplementation project(':scheduler')
testImplementation project(':executor')
testImplementation "io.micronaut:micronaut-http-client"
testImplementation "io.micronaut:micronaut-http-server-netty"

View File

@@ -3,30 +3,88 @@ package io.kestra.core.events;
import io.micronaut.core.annotation.Nullable;
import io.micronaut.http.HttpRequest;
import io.micronaut.http.context.ServerRequestContext;
import lombok.AllArgsConstructor;
import lombok.Getter;
@AllArgsConstructor
import java.util.Objects;
@Getter
public class CrudEvent<T> {
T model;
private final T model;
@Nullable
T previousModel;
CrudEventType type;
HttpRequest<?> request;
private final T previousModel;
private final CrudEventType type;
private final HttpRequest<?> request;
/**
* Static helper method for creating a new {@link CrudEventType#UPDATE} CrudEvent.
*
* @param model the new created model.
* @param <T> type of the model.
* @return the new {@link CrudEvent}.
*/
public static <T> CrudEvent<T> create(T model) {
Objects.requireNonNull(model, "Can't create CREATE event with a null model");
return new CrudEvent<>(model, null, CrudEventType.CREATE);
}
/**
* Static helper method for creating a new {@link CrudEventType#DELETE} CrudEvent.
*
* @param model the deleted model.
* @param <T> type of the model.
* @return the new {@link CrudEvent}.
*/
public static <T> CrudEvent<T> delete(T model) {
Objects.requireNonNull(model, "Can't create DELETE event with a null model");
return new CrudEvent<>(null, model, CrudEventType.DELETE);
}
/**
* Static helper method for creating a new CrudEvent.
*
* @param before the model before the update.
* @param after the model after the update.
* @param <T> type of the model.
* @return the new {@link CrudEvent}.
*/
public static <T> CrudEvent<T> of(T before, T after) {
if (before == null && after == null) {
throw new IllegalArgumentException("Both before and after cannot be null");
}
if (before == null) {
return create(after);
}
if (after == null) {
return delete(before);
}
return new CrudEvent<>(after, before, CrudEventType.UPDATE);
}
/**
* @deprecated use the static factory methods.
*/
@Deprecated
public CrudEvent(T model, CrudEventType type) {
this.model = model;
this.type = type;
this.previousModel = null;
this.request = ServerRequestContext.currentRequest().orElse(null);
this(
CrudEventType.DELETE.equals(type) ? null : model,
CrudEventType.DELETE.equals(type) ? model : null,
type,
ServerRequestContext.currentRequest().orElse(null)
);
}
public CrudEvent(T model, T previousModel, CrudEventType type) {
this(model, previousModel, type, ServerRequestContext.currentRequest().orElse(null));
}
public CrudEvent(T model, T previousModel, CrudEventType type, HttpRequest<?> request) {
this.model = model;
this.previousModel = previousModel;
this.type = type;
this.request = ServerRequestContext.currentRequest().orElse(null);
this.request = request;
}
}

View File

@@ -6,16 +6,24 @@ import io.kestra.core.http.HttpRequest;
import io.kestra.core.http.HttpResponse;
import io.kestra.core.http.client.apache.*;
import io.kestra.core.http.client.configurations.HttpConfiguration;
import io.kestra.core.runners.DefaultRunContext;
import io.kestra.core.runners.RunContext;
import io.kestra.core.serializers.JacksonMapper;
import io.micrometer.common.KeyValues;
import io.micrometer.core.instrument.binder.httpcomponents.hc5.ApacheHttpClientContext;
import io.micrometer.core.instrument.binder.httpcomponents.hc5.DefaultApacheHttpClientObservationConvention;
import io.micrometer.core.instrument.binder.httpcomponents.hc5.ObservationExecChainHandler;
import io.micrometer.observation.ObservationRegistry;
import io.micronaut.http.MediaType;
import jakarta.annotation.Nullable;
import lombok.Builder;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.hc.client5.http.ContextBuilder;
import org.apache.hc.client5.http.auth.*;
import org.apache.hc.client5.http.config.ConnectionConfig;
import org.apache.hc.client5.http.impl.ChainElement;
import org.apache.hc.client5.http.impl.DefaultAuthenticationStrategy;
import org.apache.hc.client5.http.impl.auth.BasicCredentialsProvider;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
@@ -30,7 +38,6 @@ import org.apache.hc.core5.http.io.HttpClientResponseHandler;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.ssl.SSLContexts;
import org.apache.hc.core5.util.Timeout;
import org.codehaus.plexus.util.StringUtils;
import java.io.Closeable;
import java.io.IOException;
@@ -50,11 +57,16 @@ public class HttpClient implements Closeable {
private transient CloseableHttpClient client;
private final RunContext runContext;
private final HttpConfiguration configuration;
private ObservationRegistry observationRegistry;
@Builder
public HttpClient(RunContext runContext, @Nullable HttpConfiguration configuration) throws IllegalVariableEvaluationException {
this.runContext = runContext;
this.configuration = configuration == null ? HttpConfiguration.builder().build() : configuration;
if (runContext instanceof DefaultRunContext defaultRunContext) {
this.observationRegistry = defaultRunContext.getApplicationContext().findBean(ObservationRegistry.class).orElse(null);
}
this.client = this.createClient();
}
@@ -67,6 +79,13 @@ public class HttpClient implements Closeable {
.disableDefaultUserAgent()
.setUserAgent("Kestra");
if (observationRegistry != null) {
// micrometer, must be placed before the retry strategy (see https://docs.micrometer.io/micrometer/reference/reference/httpcomponents.html#_retry_strategy_considerations)
builder.addExecInterceptorAfter(ChainElement.RETRY.name(), "micrometer",
new ObservationExecChainHandler(observationRegistry, new CustomApacheHttpClientObservationConvention())
);
}
// logger
if (this.configuration.getLogs() != null && this.configuration.getLogs().length > 0) {
if (ArrayUtils.contains(this.configuration.getLogs(), HttpConfiguration.LoggingType.REQUEST_HEADERS) ||
@@ -297,4 +316,14 @@ public class HttpClient implements Closeable {
this.client.close();
}
}
public static class CustomApacheHttpClientObservationConvention extends DefaultApacheHttpClientObservationConvention {
@Override
public KeyValues getLowCardinalityKeyValues(ApacheHttpClientContext context) {
return KeyValues.concat(
super.getLowCardinalityKeyValues(context),
KeyValues.of("type", "core-client")
);
}
}
}

View File

@@ -0,0 +1,34 @@
package io.kestra.core.metrics;
import io.micrometer.core.instrument.binder.jvm.JvmThreadDeadlockMetrics;
import io.micrometer.java21.instrument.binder.jdk.VirtualThreadMetrics;
import io.micronaut.configuration.metrics.annotation.RequiresMetrics;
import io.micronaut.context.annotation.Bean;
import io.micronaut.context.annotation.Factory;
import io.micronaut.context.annotation.Primary;
import io.micronaut.context.annotation.Requires;
import jakarta.inject.Singleton;
import static io.micronaut.configuration.metrics.micrometer.MeterRegistryFactory.MICRONAUT_METRICS_BINDERS;
import static io.micronaut.core.util.StringUtils.FALSE;
@Factory
@RequiresMetrics
public class MeterRegistryBinderFactory {
@Bean
@Primary
@Singleton
@Requires(property = MICRONAUT_METRICS_BINDERS + ".jvm.enabled", notEquals = FALSE)
public VirtualThreadMetrics virtualThreadMetrics() {
return new VirtualThreadMetrics();
}
@Bean
@Primary
@Singleton
@Requires(property = MICRONAUT_METRICS_BINDERS + ".jvm.enabled", notEquals = FALSE)
public JvmThreadDeadlockMetrics threadDeadlockMetricsMetrics() {
return new JvmThreadDeadlockMetrics();
}
}

View File

@@ -6,7 +6,6 @@ import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.triggers.AbstractTrigger;
import io.kestra.core.models.triggers.TriggerContext;
import io.kestra.core.runners.*;
import io.kestra.core.schedulers.SchedulerExecutionWithTrigger;
import io.micrometer.core.instrument.*;
import io.micrometer.core.instrument.binder.MeterBinder;
import io.micrometer.core.instrument.search.Search;
@@ -395,19 +394,6 @@ public class MetricRegistry {
return triggerContext.getTenantId() == null ? baseTags : ArrayUtils.addAll(baseTags, TAG_TENANT_ID, triggerContext.getTenantId());
}
/**
* Return tags for current {@link SchedulerExecutionWithTrigger}.
*
* @param schedulerExecutionWithTrigger the current SchedulerExecutionWithTrigger
* @return tags to apply to metrics
*/
public String[] tags(SchedulerExecutionWithTrigger schedulerExecutionWithTrigger, String... tags) {
return ArrayUtils.addAll(
this.tags(schedulerExecutionWithTrigger.getExecution()),
tags
);
}
/**
* Return tags for current {@link ExecutionKilled}
*

View File

@@ -1,16 +1,33 @@
package io.kestra.core.models;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.Valid;
import jakarta.validation.constraints.Pattern;
import java.util.List;
import java.util.Map;
/**
* Interface that can be implemented by classes supporting plugin versioning.
*
* @see Plugin
*/
public interface PluginVersioning {
String TITLE = "Plugin Version";
String DESCRIPTION = """
Defines the version of the plugin to use.
@Pattern(regexp="\\d+\\.\\d+\\.\\d+(-[a-zA-Z0-9-]+)?|([a-zA-Z0-9]+)")
@Schema(title = "The version of the plugin to use.")
The version must follow the Semantic Versioning (SemVer) specification:
- A single-digit MAJOR version (e.g., `1`).
- A MAJOR.MINOR version (e.g., `1.1`).
- A MAJOR.MINOR.PATCH version, optionally with any qualifier
(e.g., `1.1.2`, `1.1.0-SNAPSHOT`).
""";
@Schema(
title = TITLE,
description = DESCRIPTION
)
String getVersion();
}

View File

@@ -3,7 +3,6 @@ package io.kestra.core.models.triggers.multipleflows;
import io.kestra.core.exceptions.InternalException;
import io.kestra.core.models.conditions.Condition;
import io.kestra.core.models.conditions.ConditionContext;
import io.kestra.core.models.executions.Execution;
import io.kestra.core.models.triggers.TimeWindow;
import io.kestra.core.utils.Rethrow;
import org.slf4j.Logger;
@@ -24,7 +23,7 @@ public interface MultipleCondition extends Rethrow.PredicateChecked<ConditionCon
/**
* This conditions will only validate previously calculated value on
* {@link io.kestra.core.services.FlowTriggerService#computeExecutionsFromFlowTriggers(Execution, List, Optional)}} and {@link MultipleConditionStorageInterface#save(List)} by the executor.
* io.kestra.executor.FlowTriggerService#computeExecutionsFromFlowTriggers(Execution, List, Optional) and {@link MultipleConditionStorageInterface#save(List)} by the executor.
* The real validation is done here.
*/
@Override

View File

@@ -43,7 +43,7 @@ public class DefaultPluginRegistry implements PluginRegistry {
static final DefaultPluginRegistry INSTANCE = new DefaultPluginRegistry();
}
private final Map<PluginIdentifier, PluginClassAndMetadata<? extends Plugin>> pluginClassByIdentifier = new ConcurrentHashMap<>();
protected final Map<PluginIdentifier, PluginClassAndMetadata<? extends Plugin>> pluginClassByIdentifier = new ConcurrentHashMap<>();
private final Map<PluginBundleIdentifier, RegisteredPlugin> plugins = new ConcurrentHashMap<>();
private final PluginScanner scanner = new PluginScanner(DefaultPluginRegistry.class.getClassLoader());
private final AtomicBoolean initialized = new AtomicBoolean(false);
@@ -56,7 +56,7 @@ public class DefaultPluginRegistry implements PluginRegistry {
*
* @return the {@link DefaultPluginRegistry}.
*/
public static DefaultPluginRegistry getOrCreate() {
public synchronized static DefaultPluginRegistry getOrCreate() {
DefaultPluginRegistry instance = LazyHolder.INSTANCE;
if (!instance.isInitialized()) {
instance.init();
@@ -74,7 +74,7 @@ public class DefaultPluginRegistry implements PluginRegistry {
/**
* Initializes the registry by loading all core plugins.
*/
protected void init() {
protected synchronized void init() {
if (initialized.compareAndSet(false, true)) {
register(scanner.scan());
}
@@ -103,11 +103,13 @@ public class DefaultPluginRegistry implements PluginRegistry {
*/
@Override
public void registerIfAbsent(final Path pluginPath) {
long start = System.currentTimeMillis();
if (isPluginPathValid(pluginPath) && !isPluginPathScanned(pluginPath)) {
List<RegisteredPlugin> scanned = scanner.scan(pluginPath);
scanned.forEach(this::register);
scannedPluginPaths.add(pluginPath);
}
log.debug("Registered if absent plugins from path {} in {} ms", pluginPath, System.currentTimeMillis() - start);
}
private boolean isPluginPathScanned(final Path pluginPath) {
@@ -119,10 +121,12 @@ public class DefaultPluginRegistry implements PluginRegistry {
*/
@Override
public void register(final Path pluginPath) {
long start = System.currentTimeMillis();
if (isPluginPathValid(pluginPath)) {
List<RegisteredPlugin> scanned = scanner.scan(pluginPath);
scanned.forEach(this::register);
}
log.debug("Registered plugins from path {} in {} ms", pluginPath, System.currentTimeMillis() - start);
}
/**
@@ -191,21 +195,28 @@ public class DefaultPluginRegistry implements PluginRegistry {
*/
public void register(final RegisteredPlugin plugin) {
final PluginBundleIdentifier identifier = PluginBundleIdentifier.of(plugin);
// Skip registration if plugin-bundle already exists in the registry.
if (containsPluginBundle(identifier)) {
return;
// Skip registration if the same plugin already exists in the registry.
final RegisteredPlugin existing = plugins.get(identifier);
if (existing != null && existing.crc32() == plugin.crc32()) {
return; // same plugin already registered
}
lock.lock();
try {
if (existing != null) {
unregister(List.of(existing));
}
plugins.put(PluginBundleIdentifier.of(plugin), plugin);
pluginClassByIdentifier.putAll(getPluginClassesByIdentifier(plugin));
registerAll(getPluginClassesByIdentifier(plugin));
} finally {
lock.unlock();
}
}
protected void registerAll(Map<PluginIdentifier, PluginClassAndMetadata<? extends Plugin>> plugins) {
pluginClassByIdentifier.putAll(plugins);
}
@SuppressWarnings("unchecked")
protected Map<PluginIdentifier, PluginClassAndMetadata<? extends Plugin>> getPluginClassesByIdentifier(final RegisteredPlugin plugin) {
Map<PluginIdentifier, PluginClassAndMetadata<? extends Plugin>> classes = new HashMap<>();

View File

@@ -6,6 +6,12 @@ import lombok.Getter;
import lombok.ToString;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.Enumeration;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.zip.CRC32;
@AllArgsConstructor
@Getter
@@ -14,4 +20,59 @@ import java.net.URL;
public class ExternalPlugin {
private final URL location;
private final URL[] resources;
private volatile Long crc32; // lazy-val
public ExternalPlugin(URL location, URL[] resources) {
this.location = location;
this.resources = resources;
}
public Long getCrc32() {
if (this.crc32 == null) {
synchronized (this) {
if (this.crc32 == null) {
this.crc32 = computeJarCrc32(location);
}
}
}
return crc32;
}
/**
* Compute a CRC32 of the JAR File without reading the whole file
*
* @param location of the JAR File.
* @return the CRC32 of {@code -1} if the checksum can't be computed.
*/
private static long computeJarCrc32(final URL location) {
CRC32 crc = new CRC32();
try (JarFile jar = new JarFile(location.toURI().getPath(), false)) {
Enumeration<JarEntry> entries = jar.entries();
byte[] buffer = new byte[Long.BYTES]; // reusable buffer to avoid re-allocation
while (entries.hasMoreElements()) {
JarEntry entry = entries.nextElement();
crc.update(entry.getName().getBytes(StandardCharsets.UTF_8));
updateCrc32WithLong(crc, buffer, entry.getSize());
updateCrc32WithLong(crc, buffer, entry.getCrc());
}
return crc.getValue();
} catch (Exception e) {
return -1;
}
}
private static void updateCrc32WithLong(CRC32 crc32, byte[] reusable, long val) {
// fast long -> byte conversion
reusable[0] = (byte) (val >>> 56);
reusable[1] = (byte) (val >>> 48);
reusable[2] = (byte) (val >>> 40);
reusable[3] = (byte) (val >>> 32);
reusable[4] = (byte) (val >>> 24);
reusable[5] = (byte) (val >>> 16);
reusable[6] = (byte) (val >>> 8);
reusable[7] = (byte) val;
crc32.update(reusable);;
}
}

View File

@@ -46,6 +46,7 @@ public class PluginClassLoader extends URLClassLoader {
+ "|dev.failsafe"
+ "|reactor"
+ "|io.opentelemetry"
+ "|io.netty"
+ ")\\..*$");
private final ClassLoader parent;

View File

@@ -2,10 +2,14 @@ package io.kestra.core.plugins;
import io.kestra.core.models.Plugin;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.file.Path;
import java.util.List;
import java.util.Optional;
import java.util.function.Predicate;
import java.util.zip.CRC32;
import java.util.zip.Checksum;
/**
* Registry for managing all Kestra's {@link Plugin}.
@@ -123,4 +127,24 @@ public interface PluginRegistry {
* @return {@code true} if supported. Otherwise {@code false}.
*/
boolean isVersioningSupported();
/**
* Computes a CRC32 hash value representing the current content of the plugin registry.
*
* @return a {@code long} containing the CRC32 checksum value, serving as a compact
* representation of the registry's content
*/
default long hash() {
Checksum crc32 = new CRC32();
for (RegisteredPlugin plugin : plugins()) {
Optional.ofNullable(plugin.getExternalPlugin())
.map(ExternalPlugin::getCrc32)
.ifPresent(checksum -> {
byte[] bytes = ByteBuffer.allocate(Long.BYTES).putLong(checksum).array();
crc32.update(bytes, 0, bytes.length);
});
}
return crc32.getValue();
}
}

View File

@@ -5,11 +5,15 @@ import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.InvalidPathException;
import java.nio.file.Path;
import java.util.*;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.zip.CRC32;
@Slf4j
public class PluginResolver {
@@ -119,4 +123,5 @@ public class PluginResolver {
return urls;
}
}

View File

@@ -308,6 +308,10 @@ public class RegisteredPlugin {
}
return null;
}
public long crc32() {
return Optional.ofNullable(externalPlugin).map(ExternalPlugin::getCrc32).orElse(-1L);
}
@Override
public String toString() {

View File

@@ -83,7 +83,9 @@ public class LocalFlowRepositoryLoader {
}
public void load(String tenantId, File basePath) throws IOException {
Map<String, FlowInterface> flowByUidInRepository = flowRepository.findAllForAllTenants().stream()
Map<String, FlowInterface> flowByUidInRepository = flowRepository.findAllForAllTenants()
.stream()
.filter(flow -> tenantId.equals(flow.getTenantId()))
.collect(Collectors.toMap(FlowId::uidWithoutRevision, Function.identity()));
try (Stream<Path> pathStream = Files.walk(basePath.toPath())) {

View File

@@ -11,6 +11,10 @@ import lombok.Getter;
import java.util.ArrayList;
import java.util.List;
// TODO for 2.0: this class is used as a queue consumer (which should have been the ExecutorInterface instead),
// a queue message (only in Kafka) and an execution context.
// At some point, we should rename it to ExecutorContext and move it to the executor module,
// then rename the ExecutorInterface to just Executor (to be used as a queue consumer)
@Getter
@AllArgsConstructor
public class Executor {

View File

@@ -1,7 +1,7 @@
package io.kestra.core.runners;
import java.io.Closeable;
import io.kestra.core.server.Service;
public interface ExecutorInterface extends Closeable, Runnable {
public interface ExecutorInterface extends Service, Runnable {
}

View File

@@ -382,6 +382,15 @@ public class FlowInputOutput {
.stream()
.collect(HashMap::new, (m, v) -> m.put(v.getKey(), v.getValue().value()), HashMap::putAll)
);
// Hack: Pre-inject all inputs that have a default value with 'null' to prevent
// RunContextFactory from attempting to render them when absent, which could
// otherwise cause an exception if a Pebble expression is involved.
List<Input<?>> inputs = Optional.ofNullable(flow).map(FlowInterface::getInputs).orElse(List.of());
for (Input<?> input : inputs) {
if (input.getDefaults() != null && !flattenInputs.containsKey(input.getId())) {
flattenInputs.put(input.getId(), null);
}
}
return runContextFactory.of(flow, execution, vars -> vars.withInputs(flattenInputs));
}

View File

@@ -1,4 +1,7 @@
package io.kestra.core.runners;
// NOTE: this class is not used anymore but must be kept as it is used in as queue consumer both in JDBC and Kafka
public class Indexer {
import io.kestra.core.server.Service;
public interface Indexer extends Service, Runnable {
}

View File

@@ -20,9 +20,11 @@ import io.kestra.core.queues.QueueInterface;
import jakarta.annotation.Nullable;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.*;
import java.util.function.Supplier;
@@ -142,8 +144,9 @@ public class RunContextLogger implements Supplier<org.slf4j.Logger> {
}
public void usedSecret(String secret) {
if (secret != null) {
if (secret != null && !secret.isEmpty()) {
this.useSecrets.add(secret);
this.useSecrets.add(Base64.getEncoder().encodeToString(secret.getBytes(StandardCharsets.UTF_8)));
}
}

View File

@@ -10,6 +10,7 @@ import io.kestra.core.models.flows.FlowInterface;
import io.kestra.core.models.flows.Input;
import io.kestra.core.models.flows.State;
import io.kestra.core.models.flows.input.SecretInput;
import io.kestra.core.models.property.Property;
import io.kestra.core.models.property.PropertyContext;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.triggers.AbstractTrigger;
@@ -282,15 +283,15 @@ public final class RunVariables {
if (flow != null && flow.getInputs() != null) {
// we add default inputs value from the flow if not already set, this will be useful for triggers
flow.getInputs().stream()
.filter(input -> input.getDefaults() != null && !inputs.containsKey(input.getId()))
.forEach(input -> {
try {
inputs.put(input.getId(), FlowInputOutput.resolveDefaultValue(input, propertyContext));
} catch (IllegalVariableEvaluationException e) {
throw new RuntimeException("Unable to inject default value for input '" + input.getId() + "'", e);
}
});
flow.getInputs().stream()
.filter(input -> input.getDefaults() != null && !inputs.containsKey(input.getId()))
.forEach(input -> {
try {
inputs.put(input.getId(), FlowInputOutput.resolveDefaultValue(input, propertyContext));
} catch (IllegalVariableEvaluationException e) {
// Silent catch, if an input depends on another input, or a variable that is populated at runtime / input filling time, we can't resolve it here.
}
});
}
if (!inputs.isEmpty()) {

View File

@@ -1,5 +0,0 @@
package io.kestra.core.runners;
public interface RunnerInterface {
void run();
}

View File

@@ -1,4 +1,4 @@
package io.kestra.core.schedulers;
package io.kestra.core.runners;
import java.util.function.Consumer;

View File

@@ -2,6 +2,5 @@ package io.kestra.core.runners;
import io.kestra.core.server.Service;
public interface IndexerInterface extends Service, Runnable {
public interface Scheduler extends Service, Runnable {
}

View File

@@ -1,4 +1,4 @@
package io.kestra.core.schedulers;
package io.kestra.core.runners;
import io.kestra.core.models.conditions.ConditionContext;
import io.kestra.core.models.flows.Flow;

File diff suppressed because it is too large Load Diff

View File

@@ -5,6 +5,8 @@ import io.kestra.core.http.HttpRequest;
import io.kestra.core.http.HttpResponse;
import io.kestra.core.http.client.HttpClient;
import io.kestra.core.http.client.HttpClientException;
import io.kestra.core.http.client.HttpClientRequestException;
import io.kestra.core.http.client.HttpClientResponseException;
import io.kestra.core.http.client.configurations.HttpConfiguration;
import io.kestra.core.runners.RunContext;
import io.kestra.core.runners.RunContextFactory;
@@ -101,8 +103,15 @@ public class HttpFunction<T> implements Function {
try (HttpClient httpClient = new HttpClient(runContext, httpConfiguration)) {
HttpResponse<Object> response = httpClient.request(httpRequest, Object.class);
return response.getBody();
} catch (HttpClientException | IllegalVariableEvaluationException | IOException e) {
throw new PebbleException(e, "Unable to execute HTTP request", lineNumber, self.getName());
} catch (HttpClientResponseException e) {
if (e.getResponse() != null) {
String msg = "Failed to execute HTTP Request, server respond with status " + e.getResponse().getStatus().getCode() + " : " + e.getResponse().getStatus().getReason();
throw new PebbleException(e, msg , lineNumber, self.getName());
} else {
throw new PebbleException( e, "Failed to execute HTTP request ", lineNumber, self.getName());
}
} catch(HttpClientException | IllegalVariableEvaluationException | IOException e ) {
throw new PebbleException( e, "Failed to execute HTTP request ", lineNumber, self.getName());
}
}

View File

@@ -1,9 +0,0 @@
package io.kestra.core.schedulers;
import jakarta.inject.Singleton;
@SuppressWarnings("try")
@Singleton
public interface Scheduler extends Runnable, AutoCloseable {
}

View File

@@ -180,23 +180,13 @@ public final class FileSerde {
}
private static <T> MappingIterator<T> createMappingIterator(ObjectMapper objectMapper, Reader reader, TypeReference<T> type) throws IOException {
// See https://github.com/FasterXML/jackson-dataformats-binary/issues/493
// There is a limitation with the MappingIterator that cannot differentiate between an array of things (of whatever shape)
// and a sequence/stream of things (of Array shape).
// To work around that, we need to create a JsonParser and advance to the first token.
try (var parser = objectMapper.createParser(reader)) {
parser.nextToken();
return objectMapper.readerFor(type).readValues(parser);
}
}
private static <T> MappingIterator<T> createMappingIterator(ObjectMapper objectMapper, Reader reader, Class<T> type) throws IOException {
// See https://github.com/FasterXML/jackson-dataformats-binary/issues/493
// There is a limitation with the MappingIterator that cannot differentiate between an array of things (of whatever shape)
// and a sequence/stream of things (of Array shape).
// To work around that, we need to create a JsonParser and advance to the first token.
try (var parser = objectMapper.createParser(reader)) {
parser.nextToken();
return objectMapper.readerFor(type).readValues(parser);
}
}

View File

@@ -172,22 +172,19 @@ public final class JacksonMapper {
return Pair.of(patchPrevToNew, patchNewToPrev);
}
public static String applyPatches(Object object, List<JsonNode> patches) throws JsonProcessingException {
public static JsonNode applyPatchesOnJsonNode(JsonNode jsonObject, List<JsonNode> patches) {
for (JsonNode patch : patches) {
try {
// Required for ES
if (patch.findValue("value") == null) {
((ObjectNode) patch.get(0)).set("value", (JsonNode) null);
((ObjectNode) patch.get(0)).set("value", null);
}
JsonNode current = MAPPER.valueToTree(object);
object = JsonPatch.fromJson(patch).apply(current);
jsonObject = JsonPatch.fromJson(patch).apply(jsonObject);
} catch (IOException | JsonPatchException e) {
throw new RuntimeException(e);
}
}
return MAPPER.writeValueAsString(object);
return jsonObject;
}
}

View File

@@ -171,7 +171,7 @@ public abstract class AbstractServiceLivenessCoordinator extends AbstractService
protected void handleAllServiceInNotRunningState() {
// Soft delete all services which are NOT_RUNNING anymore.
store.findAllInstancesInStates(Set.of(Service.ServiceState.NOT_RUNNING))
.forEach(instance -> safelyUpdate(instance, Service.ServiceState.EMPTY, null));
.forEach(instance -> safelyUpdate(instance, Service.ServiceState.INACTIVE, null));
}
protected void handleAllServicesForTerminatedStates(final Instant now) {

View File

@@ -1,9 +1,11 @@
package io.kestra.core.server;
import com.fasterxml.jackson.annotation.JsonCreator;
import io.kestra.core.utils.Enums;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
/**
@@ -106,12 +108,12 @@ public interface Service extends AutoCloseable {
* |
* v
* +------+-------+
* | Empty (8) |
* | Inactive (8) |
* +------+-------+
* </pre>
*/
enum ServiceState {
CREATED(1, 2, 3, 4, 9), // 0
CREATED(1, 2, 3, 4, 9), // 0
RUNNING(2, 3, 4, 9), // 1
ERROR(4), // 2
DISCONNECTED(4, 7), // 3
@@ -119,14 +121,24 @@ public interface Service extends AutoCloseable {
TERMINATED_GRACEFULLY(7), // 5
TERMINATED_FORCED(7), // 6
NOT_RUNNING(8), // 7
EMPTY(), // 8 FINAL STATE
MAINTENANCE(1, 2, 3, 4); // 9
INACTIVE(), // 8 FINAL STATE
MAINTENANCE(1, 2, 3, 4); // 9
private final Set<Integer> validTransitions = new HashSet<>();
ServiceState(final Integer... validTransitions) {
this.validTransitions.addAll(Arrays.asList(validTransitions));
}
@JsonCreator
public static ServiceState fromString(final String value) {
try {
// EMPTY state was renamed to INACTIVE in Kestra 1.0
return Enums.getForNameIgnoreCase(value, ServiceState.class, Map.of("EMPTY", INACTIVE));
} catch (IllegalArgumentException e) {
return INACTIVE;
}
}
public boolean isValidTransition(final ServiceState newState) {
return validTransitions.contains(newState.ordinal()) || equals(newState);
@@ -145,7 +157,7 @@ public interface Service extends AutoCloseable {
return equals(TERMINATED_GRACEFULLY)
|| equals(TERMINATED_FORCED)
|| equals(NOT_RUNNING)
|| equals(EMPTY);
|| equals(INACTIVE);
}
public static Set<ServiceState> allRunningStates() {

View File

@@ -392,7 +392,7 @@ public class ServiceLivenessManager extends AbstractServiceLivenessTask {
// More especially, this handles the case where a WORKER is configured with a short gracefulTerminationPeriod
// and the JVM was unresponsive for more than this period.
// In this context, the worker's tasks have already been resubmitted by the executor; the worker must therefore stop immediately.
if (state.equals(Service.ServiceState.NOT_RUNNING) || state.equals(Service.ServiceState.EMPTY)) {
if (state.equals(Service.ServiceState.NOT_RUNNING) || state.equals(Service.ServiceState.INACTIVE)) {
service.skipGracefulTermination(true);
}
KestraContext.getContext().shutdown();

View File

@@ -130,7 +130,7 @@ public class ConditionService {
return this.conditionContext(runContext, flow, execution, null);
}
boolean valid(FlowInterface flow, List<Condition> list, ConditionContext conditionContext) {
public boolean valid(FlowInterface flow, List<Condition> list, ConditionContext conditionContext) {
return list
.stream()
.allMatch(condition -> {

View File

@@ -317,6 +317,32 @@ public class ExecutionService {
return revision != null ? newExecution.withFlowRevision(revision) : newExecution;
}
public Execution changeTaskRunState(final Execution execution, Flow flow, String taskRunId, State.Type newState) throws Exception {
Execution newExecution = markAs(execution, flow, taskRunId, newState);
// if the execution was terminated, it could have executed errors/finally/afterExecutions, we must remove them as the execution will be restarted
if (execution.getState().isTerminated()) {
List<TaskRun> newTaskRuns = newExecution.getTaskRunList();
// We need to remove global error tasks and flowable error tasks if any
flow
.allErrorsWithChildren()
.forEach(task -> newTaskRuns.removeIf(taskRun -> taskRun.getTaskId().equals(task.getId())));
// We need to remove global finally tasks and flowable error tasks if any
flow
.allFinallyWithChildren()
.forEach(task -> newTaskRuns.removeIf(taskRun -> taskRun.getTaskId().equals(task.getId())));
// We need to remove afterExecution tasks
ListUtils.emptyOnNull(flow.getAfterExecution())
.forEach(task -> newTaskRuns.removeIf(taskRun -> taskRun.getTaskId().equals(task.getId())));
return newExecution.withTaskRunList(newTaskRuns);
} else {
return newExecution;
}
}
public Execution markAs(final Execution execution, FlowInterface flow, String taskRunId, State.Type newState) throws Exception {
return this.markAs(execution, flow, taskRunId, newState, null, null);
}

View File

@@ -3,12 +3,7 @@ package io.kestra.core.services;
import com.fasterxml.jackson.core.JsonProcessingException;
import io.kestra.core.exceptions.FlowProcessingException;
import io.kestra.core.models.executions.Execution;
import io.kestra.core.models.flows.Flow;
import io.kestra.core.models.flows.FlowId;
import io.kestra.core.models.flows.FlowInterface;
import io.kestra.core.models.flows.FlowWithException;
import io.kestra.core.models.flows.FlowWithSource;
import io.kestra.core.models.flows.GenericFlow;
import io.kestra.core.models.flows.*;
import io.kestra.core.models.tasks.RunnableTask;
import io.kestra.core.models.topologies.FlowTopology;
import io.kestra.core.models.triggers.AbstractTrigger;
@@ -30,16 +25,7 @@ import org.apache.commons.lang3.builder.EqualsBuilder;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Objects;
import java.util.Optional;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -408,7 +394,7 @@ public class FlowService {
return latestFlows.values().stream().filter(flow -> !flow.isDeleted());
}
protected boolean removeUnwanted(Flow f, Execution execution) {
public boolean removeUnwanted(Flow f, Execution execution) {
// we don't allow recursive
return !f.uidWithoutRevision().equals(FlowId.uidWithoutRevision(execution));
}
@@ -551,23 +537,24 @@ public class FlowService {
return expandAll ? recursiveFlowTopology(new ArrayList<>(), tenant, namespace, id, destinationOnly) : flowTopologyRepository.get().findByFlow(tenant, namespace, id, destinationOnly).stream();
}
private Stream<FlowTopology> recursiveFlowTopology(List<FlowId> flowIds, String tenantId, String namespace, String id, boolean destinationOnly) {
private Stream<FlowTopology> recursiveFlowTopology(List<String> visitedTopologies, String tenantId, String namespace, String id, boolean destinationOnly) {
if (flowTopologyRepository.isEmpty()) {
throw noRepositoryException();
}
List<FlowTopology> flowTopologies = flowTopologyRepository.get().findByFlow(tenantId, namespace, id, destinationOnly);
FlowId flowId = FlowId.of(tenantId, namespace, id, null);
if (flowIds.contains(flowId)) {
return flowTopologies.stream();
}
flowIds.add(flowId);
var flowTopologies = flowTopologyRepository.get().findByFlow(tenantId, namespace, id, destinationOnly);
return flowTopologies.stream()
.flatMap(topology -> Stream.of(topology.getDestination(), topology.getSource()))
// recursively fetch child nodes
.flatMap(node -> recursiveFlowTopology(flowIds, node.getTenantId(), node.getNamespace(), node.getId(), destinationOnly));
// ignore already visited topologies
.filter(x -> !visitedTopologies.contains(x.uid()))
.flatMap(topology -> {
visitedTopologies.add(topology.uid());
Stream<FlowTopology> subTopologies = Stream
.of(topology.getDestination(), topology.getSource())
// recursively visit children and parents nodes
.flatMap(relationNode -> recursiveFlowTopology(visitedTopologies, relationNode.getTenantId(), relationNode.getNamespace(), relationNode.getId(), destinationOnly));
return Stream.concat(Stream.of(topology), subTopologies);
});
}
private IllegalStateException noRepositoryException() {

View File

@@ -10,6 +10,7 @@ import jakarta.annotation.Nullable;
import java.io.ByteArrayInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.time.Instant;
import java.util.Collections;
@@ -112,7 +113,14 @@ public class InternalKVStore implements KVStore {
@Override
public boolean delete(String key) throws IOException {
KVStore.validateKey(key);
return this.storage.delete(this.tenant, this.namespace, this.storageUri(key));
URI uri = this.storageUri(key);
boolean deleted = this.storage.delete(this.tenant, this.namespace, uri);
URI metadataURI = URI.create(uri.getPath() + ".metadata");
if (this.storage.exists(this.tenant, this.namespace, metadataURI)){
this.storage.delete(this.tenant, this.namespace, metadataURI);
}
return deleted;
}
/**
@@ -120,18 +128,32 @@ public class InternalKVStore implements KVStore {
*/
@Override
public List<KVEntry> list() throws IOException {
List<FileAttributes> list;
try {
list = this.storage.list(this.tenant, this.namespace, this.storageUri(null));
} catch (FileNotFoundException e) {
return Collections.emptyList();
}
List<FileAttributes> list = listAllFromStorage();
return list.stream()
.map(throwFunction(KVEntry::from))
.filter(kvEntry -> Optional.ofNullable(kvEntry.expirationDate()).map(expirationDate -> Instant.now().isBefore(expirationDate)).orElse(true))
.toList();
}
/**
* {@inheritDoc}
*/
@Override
public List<KVEntry> listAll() throws IOException {
List<FileAttributes> list = listAllFromStorage();
return list.stream()
.map(throwFunction(KVEntry::from))
.toList();
}
private List<FileAttributes> listAllFromStorage() throws IOException {
try {
return this.storage.list(this.tenant, this.namespace, this.storageUri(null));
} catch (FileNotFoundException e) {
return Collections.emptyList();
}
}
/**
* {@inheritDoc}
*/

View File

@@ -1,13 +1,14 @@
package io.kestra.core.storages.kv;
import io.kestra.core.storages.FileAttributes;
import jakarta.annotation.Nullable;
import java.io.IOException;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.Optional;
public record KVEntry(String key, String description, Instant creationDate, Instant updateDate, Instant expirationDate) {
public record KVEntry(String key, @Nullable String description, Instant creationDate, Instant updateDate, @Nullable Instant expirationDate) {
public static KVEntry from(FileAttributes fileAttributes) throws IOException {
Optional<KVMetadata> kvMetadata = Optional.ofNullable(fileAttributes.getMetadata()).map(KVMetadata::new);
return new KVEntry(

View File

@@ -1,9 +1,7 @@
package io.kestra.core.storages.kv;
import io.kestra.core.exceptions.ResourceExpiredException;
import io.kestra.core.runners.RunContext;
import io.kestra.core.storages.StorageContext;
import jakarta.annotation.Nullable;
import java.io.IOException;
import java.net.URI;
@@ -80,6 +78,14 @@ public interface KVStore {
*/
List<KVEntry> list() throws IOException;
/**
* Lists all the K/V store entries, expired or not.
*
* @return The list of all {@link KVEntry}.
* @throws IOException if an error occurred while executing the operation on the K/V store.
*/
List<KVEntry> listAll() throws IOException;
/**
* Finds the K/V store entry for the given key.
*

View File

@@ -18,6 +18,7 @@ import io.kestra.core.repositories.FlowRepositoryInterface;
import io.kestra.core.repositories.FlowTopologyRepositoryInterface;
import io.kestra.core.services.ConditionService;
import io.kestra.core.utils.ListUtils;
import io.kestra.core.utils.MapUtils;
import io.kestra.plugin.core.condition.*;
import io.micronaut.core.annotation.Nullable;
import jakarta.inject.Inject;
@@ -175,9 +176,6 @@ public class FlowTopologyService {
protected boolean isTriggerChild(Flow parent, Flow child) {
List<AbstractTrigger> triggers = ListUtils.emptyOnNull(child.getTriggers());
// simulated execution: we add a "simulated" label so conditions can know that the evaluation is for a simulated execution
Execution execution = Execution.newExecution(parent, (f, e) -> null, List.of(SIMULATED_EXECUTION), Optional.empty());
// keep only flow trigger
List<io.kestra.plugin.core.trigger.Flow> flowTriggers = triggers
.stream()
@@ -189,13 +187,16 @@ public class FlowTopologyService {
return false;
}
// simulated execution: we add a "simulated" label so conditions can know that the evaluation is for a simulated execution
Execution execution = Execution.newExecution(parent, (f, e) -> null, List.of(SIMULATED_EXECUTION), Optional.empty());
boolean conditionMatch = flowTriggers
.stream()
.flatMap(flow -> ListUtils.emptyOnNull(flow.getConditions()).stream())
.allMatch(condition -> validateCondition(condition, parent, execution));
boolean preconditionMatch = flowTriggers.stream()
.anyMatch(flow -> flow.getPreconditions() == null || validateMultipleConditions(flow.getPreconditions().getConditions(), parent, execution));
.anyMatch(flow -> flow.getPreconditions() == null || validatePreconditions(flow.getPreconditions(), parent, execution));
return conditionMatch && preconditionMatch;
}
@@ -209,7 +210,13 @@ public class FlowTopologyService {
return validateMultipleConditions(multipleCondition.getConditions(), child, execution);
}
return this.conditionService.isValid(condition, child, execution);
try {
return this.conditionService.isValid(condition, child, execution);
} catch (Exception e) {
// extra safety net, it means there is a bug
log.error("unable to validate condition in FlowTopologyService, flow: {}, condition: {}", child.uid(), condition, e);
return false;
}
}
private boolean validateMultipleConditions(Map<String, Condition> multipleConditions, FlowInterface child, Execution execution) {
@@ -233,11 +240,24 @@ public class FlowTopologyService {
}
private boolean isMandatoryMultipleCondition(Condition condition) {
return Stream
.of(
Expression.class
)
.anyMatch(aClass -> condition.getClass().isAssignableFrom(aClass));
return condition.getClass().isAssignableFrom(Expression.class);
}
private boolean validatePreconditions(io.kestra.plugin.core.trigger.Flow.Preconditions preconditions, FlowInterface child, Execution execution) {
boolean upstreamFlowMatched = MapUtils.emptyOnNull(preconditions.getUpstreamFlowsConditions())
.values()
.stream()
.filter(c -> !isFilterCondition(c))
.anyMatch(c -> validateCondition(c, child, execution));
boolean whereMatched = MapUtils.emptyOnNull(preconditions.getWhereConditions())
.values()
.stream()
.filter(c -> !isFilterCondition(c))
.allMatch(c -> validateCondition(c, child, execution));
// to be a dependency, if upstream flow is set it must be either inside it so it's a AND between upstream flow and where
return upstreamFlowMatched && whereMatched;
}
private boolean isFilterCondition(Condition condition) {

View File

@@ -0,0 +1,15 @@
package io.kestra.core.utils;
import jakarta.inject.Singleton;
@Singleton
public class EditionProvider {
public Edition get() {
return Edition.OSS;
}
public enum Edition {
OSS,
EE
}
}

View File

@@ -206,22 +206,17 @@ public class MapUtils {
/**
* Utility method that flatten a nested map.
* <p>
* NOTE: for simplicity, this method didn't allow to flatten maps with conflicting keys that would end up in different flatten keys,
* this could be related later if needed by flattening {k1: k2: {k3: v1}, k1: {k4: v2}} to {k1.k2.k3: v1, k1.k4: v2} is prohibited for now.
*
* @param nestedMap the nested map.
* @return the flattened map.
*
* @throws IllegalArgumentException if any entry contains a map of more than one element.
*/
public static Map<String, Object> nestedToFlattenMap(@NotNull Map<String, Object> nestedMap) {
Map<String, Object> result = new TreeMap<>();
for (Map.Entry<String, Object> entry : nestedMap.entrySet()) {
if (entry.getValue() instanceof Map<?, ?> map) {
Map.Entry<String, Object> flatten = flattenEntry(entry.getKey(), (Map<String, Object>) map);
result.put(flatten.getKey(), flatten.getValue());
Map<String, Object> flatten = flattenEntry(entry.getKey(), (Map<String, Object>) map);
result.putAll(flatten);
} else {
result.put(entry.getKey(), entry.getValue());
}
@@ -229,18 +224,19 @@ public class MapUtils {
return result;
}
private static Map.Entry<String, Object> flattenEntry(String key, Map<String, Object> value) {
if (value.size() > 1) {
throw new IllegalArgumentException("You cannot flatten a map with an entry that is a map of more than one element, conflicting key: " + key);
private static Map<String, Object> flattenEntry(String key, Map<String, Object> value) {
Map<String, Object> result = new TreeMap<>();
for (Map.Entry<String, Object> entry : value.entrySet()) {
String newKey = key + "." + entry.getKey();
Object newValue = entry.getValue();
if (newValue instanceof Map<?, ?> map) {
result.putAll(flattenEntry(newKey, (Map<String, Object>) map));
} else {
result.put(newKey, newValue);
}
}
Map.Entry<String, Object> entry = value.entrySet().iterator().next();
String newKey = key + "." + entry.getKey();
Object newValue = entry.getValue();
if (newValue instanceof Map<?, ?> map) {
return flattenEntry(newKey, (Map<String, Object>) map);
} else {
return Map.entry(newKey, newValue);
}
return result;
}
}

View File

@@ -32,17 +32,23 @@ public class Version implements Comparable<Version> {
* @param version the version.
* @return a new {@link Version} instance.
*/
public static Version of(String version) {
public static Version of(final Object version) {
if (version.startsWith("v")) {
version = version.substring(1);
if (Objects.isNull(version)) {
throw new IllegalArgumentException("Invalid version, cannot parse null version");
}
String strVersion = version.toString();
if (strVersion.startsWith("v")) {
strVersion = strVersion.substring(1);
}
int qualifier = version.indexOf("-");
int qualifier = strVersion.indexOf("-");
final String[] versions = qualifier > 0 ?
version.substring(0, qualifier).split("\\.") :
version.split("\\.");
strVersion.substring(0, qualifier).split("\\.") :
strVersion.split("\\.");
try {
final int majorVersion = Integer.parseInt(versions[0]);
final int minorVersion = versions.length > 1 ? Integer.parseInt(versions[1]) : 0;
@@ -52,28 +58,54 @@ public class Version implements Comparable<Version> {
majorVersion,
minorVersion,
incrementalVersion,
qualifier > 0 ? version.substring(qualifier + 1) : null,
version
qualifier > 0 ? strVersion.substring(qualifier + 1) : null,
strVersion
);
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Invalid version, cannot parse '" + version + "'");
}
}
/**
* Static helper method for returning the most recent stable version for a current {@link Version}.
* Returns the most recent stable version compatible with the given {@link Version}.
*
* @param from the current version.
* @param versions the list of version.
* <p>Resolution strategy:</p>
* <ol>
* <li>If {@code from} is present in {@code versions}, return it directly.</li>
* <li>Otherwise, return the latest version with the same major and minor.</li>
* <li>If none found and {@code from.majorVersion() > 0}, return the latest with the same major.</li>
* <li>Return {@code null} if no compatible version is found.</li>
* </ol>
*
* @return the last stable version.
* @param from the current version
* @param versions available versions
* @return the most recent compatible stable version, or {@code null} if none
*/
public static Version getStable(final Version from, final Collection<Version> versions) {
List<Version> compatibleVersions = versions.stream()
if (versions.contains(from)) {
return from;
}
// Prefer same major+minor stable versions
List<Version> sameMinorStable = versions.stream()
.filter(v -> v.majorVersion() == from.majorVersion() && v.minorVersion() == from.minorVersion())
.toList();
if (compatibleVersions.isEmpty()) return null;
return Version.getLatest(compatibleVersions);
if (!sameMinorStable.isEmpty()) {
return Version.getLatest(sameMinorStable);
}
if (from.majorVersion() > 0) {
// Fallback: any stable version with the same major
List<Version> sameMajorStable = versions.stream()
.filter(v -> v.majorVersion() == from.majorVersion())
.toList();
if (!sameMajorStable.isEmpty()) {
return Version.getLatest(sameMajorStable);
}
}
return null;
}
/**

View File

@@ -29,7 +29,7 @@ import java.util.Map;
@Getter
@NoArgsConstructor
@Schema(
title = "Condition that check labels of an execution."
title = "Condition that checks labels of an execution."
)
@Plugin(
examples = {

View File

@@ -35,8 +35,8 @@ import static io.kestra.core.utils.MapUtils.mergeWithNullableValues;
examples = {
@Example(
title = """
The upstream `flow_a` must explicitly define its outputs
to be used in the `ExecutionOutputs` condition.
The upstream `flow_a` must explicitly define its outputs
to be used in the `ExecutionOutputs` condition.
```yaml
id: flow_a
@@ -58,7 +58,7 @@ import static io.kestra.core.utils.MapUtils.mergeWithNullableValues;
value: "{{ outputs.hello.value }}"
```
The `flow_condition_executionoutputs` will run whenever `flow_a` finishes successfully
The `flow_condition_executionoutputs` will run whenever `flow_a` finishes successfully
and returns an output matching the value 'hello':
""",
full = true,
@@ -105,7 +105,7 @@ public class ExecutionOutputs extends Condition implements ScheduleCondition {
conditionContext.getVariables(),
Map.of(TRIGGER_VAR, Map.of(OUTPUTS_VAR, conditionContext.getExecution().getOutputs()))
);
return conditionContext.getRunContext().render(expression).skipCache().as(Boolean.class, variables).orElseThrow();
}

View File

@@ -35,7 +35,7 @@ import jakarta.validation.constraints.NotNull;
code = """
id: myflow
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log

View File

@@ -47,7 +47,7 @@ public class FlowNamespaceCondition extends Condition {
@Builder.Default
@Schema(
title = "If we must look at the flow namespace by prefix (checked using startWith). The prefix is case sensitive."
title = "If we must look at the flow namespace by prefix (checked using startsWith). The prefix is case sensitive."
)
@PluginProperty
private final Boolean prefix = false;

View File

@@ -26,7 +26,7 @@ import java.util.*;
@Getter
@NoArgsConstructor
@Schema(
title = "Run a flow if the list of preconditions are met in a time window.",
title = "Run a flow if the list of preconditions is met in a time window.",
description = """
**This task is deprecated**, use the `preconditions` property of the `io.kestra.plugin.core.trigger.Flow` trigger instead.
Will trigger an executions when all the flows defined by the preconditions are successfully executed in a specific period of time.

View File

@@ -27,7 +27,7 @@ import static io.kestra.core.utils.Rethrow.throwPredicate;
@Getter
@NoArgsConstructor
@Schema(
title = "Condition to exclude others conditions."
title = "Condition to exclude other conditions."
)
@Plugin(
examples = {
@@ -62,7 +62,7 @@ public class Not extends Condition implements ScheduleCondition {
@NotEmpty
@Schema(
title = "The list of conditions to exclude.",
description = "If any conditions is true, it will prevent the event's execution."
description = "If any condition is true, it will prevent the event's execution."
)
@PluginProperty
private List<Condition> conditions;

View File

@@ -31,7 +31,7 @@ import static io.kestra.core.utils.Rethrow.throwPredicate;
@Plugin(
examples = {
@Example(
title = "Trigger condition to execute the flow when any of the condition is satisfied.",
title = "Trigger condition to execute the flow when any of the conditions are satisfied.",
full = true,
code = """
id: schedule_condition_or

View File

@@ -5,14 +5,12 @@ import de.focus_shift.jollyday.core.ManagerParameters;
import io.kestra.core.exceptions.InternalException;
import io.kestra.core.models.annotations.Example;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.conditions.Condition;
import io.kestra.core.models.conditions.ConditionContext;
import io.kestra.core.models.conditions.ScheduleCondition;
import io.kestra.core.models.property.Property;
import io.kestra.core.utils.DateUtils;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.NotNull;
import lombok.*;
import lombok.experimental.SuperBuilder;
@@ -31,7 +29,7 @@ import java.time.LocalDate;
examples = {
@Example(
full = true,
title = "Trigger condition to excute the flow only on public holidays.",
title = "Trigger condition to execute the flow only on public holidays.",
code = """
id: schedule_condition_public-holiday
namespace: company.team
@@ -52,7 +50,7 @@ import java.time.LocalDate;
),
@Example(
full = true,
title = "Trigger condition to excute the flow only on work days in France.",
title = "Trigger condition to execute the flow only on work days in France.",
code = """
id: schedule-condition-work-days
namespace: company.team

View File

@@ -25,7 +25,7 @@ import lombok.experimental.SuperBuilder;
@Plugin(
examples = {
@Example(
title = "Display a bar chart with with Executions per Namespace.",
title = "Display a bar chart with Executions per Namespace.",
full = true,
code = { """
charts:

View File

@@ -42,7 +42,6 @@ import lombok.experimental.SuperBuilder;
- Use this insight to identify trends and optimize performance.
"""
}
)
}

View File

@@ -25,7 +25,7 @@ import lombok.experimental.SuperBuilder;
@Plugin(
examples = {
@Example(
title = "Display a pie chart with with Executions per State.",
title = "Display a pie chart with Executions per State.",
full = true,
code = { """
charts:

View File

@@ -25,7 +25,7 @@ import lombok.experimental.SuperBuilder;
@Plugin(
examples = {
@Example(
title = "Display a table with a Log count for each level by Namespace.",
title = "Display a table with Log counts for each level by Namespace.",
full = true,
code = { """
charts:

View File

@@ -29,7 +29,7 @@ import lombok.experimental.SuperBuilder;
@Example(
title = "Display a chart with Executions over the last week.",
full = true,
code = { """
code = { """
charts:
- id: executions_timeseries
type: io.kestra.plugin.core.dashboard.chart.TimeSeries

View File

@@ -31,7 +31,7 @@ import lombok.experimental.SuperBuilder;
title = "Display a chart with a Executions per Namespace broken out by State.",
full = true,
code = { """
charts:
charts:
- id: executions_per_namespace_bars
type: io.kestra.plugin.core.dashboard.chart.Bar
chartOptions:

View File

@@ -50,7 +50,7 @@ import java.util.Optional;
Bearer {{ inputs.token }}
{%- elseif inputs.username is not empty and inputs.password is not empty -%}
Basic {{ (inputs.username + ':' + inputs.password) | base64encode }}
{%- endif -%}
{%- endif -%}
"""
)
},

View File

@@ -73,7 +73,7 @@ public class Assert extends Task implements RunnableTask<VoidOutput> {
@PluginProperty(dynamic = true)
private List<String> conditions;
@Schema(title = "Optional error message.")
@Schema(title = "Optional error message")
private Property<String> errorMessage;
@Override

View File

@@ -37,7 +37,7 @@ import static io.kestra.core.utils.Rethrow.throwPredicate;
@Plugin(
examples = {
@Example(
title = "Send a slack notification if there is no execution for a flow for the last 24 hours.",
title = "Send a slack notification if there is no execution for a flow in the last 24 hours.",
full = true,
code = """
id: executions_count
@@ -78,32 +78,32 @@ import static io.kestra.core.utils.Rethrow.throwPredicate;
)
public class Count extends Task implements RunnableTask<Count.Output> {
@Schema(
title = "A list of flows to be filtered.",
title = "A list of flows to be filtered",
description = "If not provided, namespaces must be set."
)
@PluginProperty
protected List<Flow> flows;
@Schema(
title = "A list of states to be filtered."
title = "A list of states to be filtered"
)
protected Property<List<State.Type>> states;
@NotNull
@Schema(
title = "The start date."
title = "The start date"
)
protected Property<String> startDate;
@Schema(
title = "The end date."
title = "The end date"
)
protected Property<String> endDate;
@NotNull
@Schema(
title = "The expression to look at against each flow.",
description = "The expression is such that expression must return `true` in order to keep the current line.\n" +
title = "The expression to check against each flow",
description = "The expression is such that the expression must return `true` in order to keep the current line.\n" +
"Some examples: \n" +
"- ```yaml {{ eq count 0 }} ```: no execution found\n" +
"- ```yaml {{ gte count 5 }} ```: more than 5 executions\n"
@@ -121,7 +121,7 @@ public class Count extends Task implements RunnableTask<Count.Output> {
.getBean(ExecutionRepositoryInterface.class);
if (flows == null && namespaces == null) {
throw new IllegalArgumentException("You must provide a list of flows or namespaces");
throw new IllegalArgumentException("You must provide a list of flows or namespaces.");
}
var flowInfo = runContext.flowInfo();

View File

@@ -119,7 +119,7 @@ public class Fail extends Task implements RunnableTask<VoidOutput> {
)
private Property<String> condition;
@Schema(title = "Optional error message.")
@Schema(title = "Optional error message")
@Builder.Default
private Property<String> errorMessage = Property.ofValue("Task failure");

View File

@@ -36,7 +36,7 @@ import static io.kestra.core.utils.Rethrow.throwFunction;
@Getter
@NoArgsConstructor
@Schema(
title = "Allow to add or overwrite labels for the current execution at runtime.",
title = "Add or overwrite labels for the current execution at runtime.",
description = "Trying to pass a system label (a label starting with `system.`) will fail the task."
)
@Plugin(
@@ -76,7 +76,7 @@ public class Labels extends Task implements ExecutionUpdatableTask {
private static final ObjectMapper MAPPER = JacksonMapper.ofJson();
@Schema(
title = "Labels to add to the current execution.",
title = "Labels to add to the current execution",
description = "The value should result in a list of labels or a labelKey:labelValue map",
oneOf = {
String.class,
@@ -127,9 +127,24 @@ public class Labels extends Task implements ExecutionUpdatableTask {
}
// check for system labels: none can be passed at runtime
Optional<Map.Entry<String, String>> first = labelsAsMap.entrySet().stream().filter(entry -> entry.getKey().startsWith(SYSTEM_PREFIX)).findFirst();
if (first.isPresent()) {
throw new IllegalArgumentException("System labels can only be set by Kestra itself, offending label: " + first.get().getKey() + "=" + first.get().getValue());
Optional<Map.Entry<String, String>> systemLabel = labelsAsMap.entrySet().stream()
.filter(entry -> entry.getKey().startsWith(SYSTEM_PREFIX))
.findFirst();
if (systemLabel.isPresent()) {
throw new IllegalArgumentException(
"System labels can only be set by Kestra itself, offending label: " +
systemLabel.get().getKey() + "=" + systemLabel.get().getValue()
);
}
// check for empty label values
Optional<Map.Entry<String, String>> emptyValue = labelsAsMap.entrySet().stream()
.filter(entry -> entry.getValue().isEmpty())
.findFirst();
if (emptyValue.isPresent()) {
throw new IllegalArgumentException(
"Label values cannot be empty, offending label: " + emptyValue.get().getKey()
);
}
Map<String, String> newLabels = ListUtils.emptyOnNull(execution.getLabels()).stream()

View File

@@ -45,59 +45,59 @@ import java.util.List;
)
public class PurgeExecutions extends Task implements RunnableTask<PurgeExecutions.Output> {
@Schema(
title = "Namespace whose flows need to be purged, or namespace of the flow that needs to be purged.",
title = "Namespace whose flows need to be purged, or namespace of the flow that needs to be purged",
description = "If `flowId` isn't provided, this is a namespace prefix, else the namespace of the flow."
)
private Property<String> namespace;
@Schema(
title = "The flow ID to be purged.",
title = "The flow ID to be purged",
description = "You need to provide the `namespace` properties if you want to purge a flow."
)
private Property<String> flowId;
@Schema(
title = "The minimum date to be purged.",
title = "The date after which data should be purged",
description = "All data of flows executed after this date will be purged."
)
private Property<String> startDate;
@Schema(
title = "The maximum date to be purged.",
title = "The date before which data should be purged.",
description = "All data of flows executed before this date will be purged."
)
@NotNull
private Property<String> endDate;
@Schema(
title = "The state of the executions to be purged.",
title = "The state of the executions to be purged",
description = "If not set, executions for any states will be purged."
)
private Property<List<State.Type>> states;
@Schema(
title = "Whether to purge executions."
title = "Flag specifying whether to purge executions"
)
@Builder.Default
private Property<Boolean> purgeExecution = Property.ofValue(true);
@Schema(
title = "Whether to purge execution's logs.",
title = "Flag specifying whether to purge execution logs",
description = """
This will only purge logs from executions not from triggers, and it will do it execution by execution.
The `io.kestra.plugin.core.log.PurgeLogs` task is a better fit to purge logs as it will purge logs in bulk, and will also purge logs not tied to an execution like trigger logs."""
This will only purge logs from executions, not from triggers, and it will do it execution by execution.
The `io.kestra.plugin.core.log.PurgeLogs` task is a better fit to purge, as it will purge logs in bulk and will also purge logs not tied to an execution like trigger logs."""
)
@Builder.Default
private Property<Boolean> purgeLog = Property.ofValue(true);
@Schema(
title = "Whether to purge execution's metrics."
title = "Flag specifying whether to purge execution's metrics."
)
@Builder.Default
private Property<Boolean> purgeMetric = Property.ofValue(true);
@Schema(
title = "Whether to purge execution's files from the Kestra's internal storage."
title = "Flag specifying whether to purge execution's files from the Kestra's internal storage"
)
@Builder.Default
private Property<Boolean> purgeStorage = Property.ofValue(true);
@@ -141,22 +141,22 @@ public class PurgeExecutions extends Task implements RunnableTask<PurgeExecution
@Getter
public static class Output implements io.kestra.core.models.tasks.Output {
@Schema(
title = "The count of deleted executions."
title = "The count of deleted executions"
)
private int executionsCount;
@Schema(
title = "The count of deleted logs."
title = "The count of deleted logs"
)
private int logsCount;
@Schema(
title = "The count of deleted storage files."
title = "The count of deleted storage files"
)
private int storagesCount;
@Schema(
title = "The count of deleted metrics."
title = "The count of deleted metrics"
)
private int metricsCount;
}

View File

@@ -64,14 +64,14 @@ public class Resume extends Task implements RunnableTask<VoidOutput> {
description = """
If you explicitly define an `executionId`, Kestra will use that specific ID.
If another `namespace` and `flowId` properties are set, Kestra will look for a paused execution for that corresponding flow.
If `executionId` is not set and `namespace` and `flowId` properties are set, Kestra will look for a paused execution for that corresponding flow.
If `executionId` is not set, the task will use the ID of the current execution."""
)
private Property<String> executionId;
@Schema(
title = "Inputs to be passed to the execution when it's resumed."
title = "Inputs to be passed to the execution when it's resumed"
)
private Property<Map<String, Object>> inputs;
@@ -93,7 +93,7 @@ public class Resume extends Task implements RunnableTask<VoidOutput> {
Execution execution = executionRepository.findById(executionInfo.tenantId(), executionInfo.id())
.orElseThrow(() -> new IllegalArgumentException("No execution found for execution id " + executionInfo.id()));
FlowInterface flow = flowExecutor.findByExecution(execution).orElseThrow(() -> new IllegalArgumentException("Flow not found for execution id " + executionInfo.id()));
FlowInterface flow = flowExecutor.findByExecution(execution).orElseThrow(() -> new IllegalArgumentException("Flow not found for execution ID " + executionInfo.id()));
Map<String, Object> renderedInputs = runContext.render(this.inputs).asMap(String.class, Object.class);
renderedInputs = !renderedInputs.isEmpty() ? renderedInputs : null;

View File

@@ -22,7 +22,7 @@ import java.util.Map;
@Getter
@NoArgsConstructor
@Schema(
title = "Allow to set execution variables. These variables will then be available via the `{{ vars.name }}` expression."
title = "Allow to set execution variables. These variables are available via the `{{ vars.name }}` expression."
)
@Plugin(
examples = {
@@ -53,7 +53,7 @@ public class SetVariables extends Task implements ExecutionUpdatableTask {
@NotNull
private Property<Map<String, Object>> variables;
@Schema(title = "Whether to overwrite existing variables")
@Schema(title = "Flag specifying whether to overwrite existing variables")
@NotNull
@Builder.Default
private Property<Boolean> overwrite = Property.ofValue(true);

View File

@@ -21,7 +21,7 @@ import java.util.Map;
@Getter
@NoArgsConstructor
@Schema(
title = "Allow to unset execution variables."
title = "Unset execution variables."
)
@Plugin(
examples = {
@@ -57,7 +57,7 @@ public class UnsetVariables extends Task implements ExecutionUpdatableTask {
@NotNull
private Property<List<String>> variables;
@Schema(title = "Whether to ignore missing variables")
@Schema(title = "Flag specifying whether to ignore missing variables")
@NotNull
@Builder.Default
private Property<Boolean> ignoreMissing = Property.ofValue(false);

View File

@@ -91,7 +91,7 @@ public class Dag extends Task implements FlowableTask<VoidOutput> {
@NotNull
@Builder.Default
@Schema(
title = "Number of concurrent parallel tasks that can be running at any point in time.",
title = "Number of concurrent parallel tasks that can be running at any point in time",
description = "If the value is `0`, no concurrency limit exists for the tasks in a DAG and all tasks that can run in parallel will start at the same time."
)
private final Property<Integer> concurrent = Property.ofValue(0);
@@ -134,7 +134,7 @@ public class Dag extends Task implements FlowableTask<VoidOutput> {
private void controlTask() throws IllegalVariableEvaluationException {
List<String> dagCheckNotExistTasks = this.dagCheckNotExistTask(this.tasks);
if (!dagCheckNotExistTasks.isEmpty()) {
throw new IllegalVariableEvaluationException("Some task doesn't exists on task '" + this.id + "': " + String.join(", ", dagCheckNotExistTasks));
throw new IllegalVariableEvaluationException("Some task doesn't exist on task '" + this.id + "': " + String.join(", ", dagCheckNotExistTasks));
}
ArrayList<String> cyclicDependenciesTasks = this.dagCheckCyclicDependencies(this.tasks);
@@ -238,14 +238,14 @@ public class Dag extends Task implements FlowableTask<VoidOutput> {
public static class DagTask {
@NotNull
@Schema(
title = "The task within the DAG."
title = "The task within the DAG"
)
@PluginProperty
private Task task;
@PluginProperty
@Schema(
title = "The list of task IDs that should have been successfully executed before starting this task."
title = "The list of task IDs that should have been successfully executed before starting this task"
)
private List<String> dependsOn;
}

View File

@@ -126,15 +126,15 @@ public class EachParallel extends Parallel implements FlowableTask<VoidOutput> {
@NotNull
@Builder.Default
@Schema(
title = "Number of concurrent parallel tasks that can be running at any point in time.",
description = "If the value is `0`, no limit exist and all the tasks will start at the same time."
title = "Number of concurrent parallel tasks that can be running at any point in time",
description = "If the value is `0`, no limit exists and all the tasks will start at the same time."
)
private final Property<Integer> concurrent = Property.ofValue(0);
@NotNull
@PluginProperty(dynamic = true)
@Schema(
title = "The list of values for this task.",
title = "The list of values for this task",
description = "The value can be passed as a string, a list of strings, or a list of objects.",
oneOf = {String.class, Object[].class}
)

Some files were not shown because too many files have changed in this diff Show More