Compare commits

...

681 Commits

Author SHA1 Message Date
Florian Hussonnois
8617eb0c7b chore: upgrade to version 'v0.22.0-rc1-SNAPSHOT' 2025-03-25 17:32:59 +01:00
YannC
2a002e9531 chore(ci): lower build-artifacts workflow so github release can use it 2025-03-25 17:32:59 +01:00
Barthélémy Ledoux
bc8bd7b00b feat: cleaner multipanel tab move (#8029)
* feat: better/simpler display for tab movement

* Revert "remove tests to make sure we merge"

This reverts commit a979232f6c.

* fix some tests

* better tests

* fix: make it work even after first move

* fix the blinking

* add a few tests

* one fix for split panel one for size

* fix test ?

* fix: scroll all the way

* make sure tab 4 is visible

* better borders

* fix dem darn tests
2025-03-25 15:46:05 +01:00
YannC
40f6334b0b fix(): datetime filtering issues (#8027)
close #7959
2025-03-25 14:48:27 +01:00
Nicolas K.
71151f5ac2 feat(core): #5467 add inheritance for KV in pebble and Get task (#8031)
* feat(core): #5467 add inheritance for KV in pebble and Get task

* fix(core): #5467 error when namespace don't contain dot

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-25 14:46:17 +01:00
brian.mulier
75e7635505 fix(ui): make sure global secret view iterates over all secrets 2025-03-25 13:43:05 +01:00
Florian Hussonnois
6c9dc8fba4 fix(core): fix NPE when closing standalone runner 2025-03-25 12:26:50 +01:00
brian.mulier
28d1f005a4 fix(ui): fail-safe secrets API calls on global secrets view 2025-03-25 12:09:07 +01:00
AJ Emerich
7fd7a6fcbc feat(docs): add example for best practice with multiline json http post request (#8023) 2025-03-25 11:42:27 +01:00
yuri
57707faf37 fix(ui): amend operator value of labels inside the filter (#8028) 2025-03-25 11:20:04 +01:00
brian.mulier
c64c2c7104 fix(ui): fail-safe secrets API calls 2025-03-25 11:08:22 +01:00
YannC
8f9b2fc0db fix(): use correct property for blueprint type
close #3227
2025-03-25 10:22:49 +01:00
Satvik Kushwaha
aab9e55794 chore(ui): limit flow run dialog maximum height (#7938)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-03-25 10:02:50 +01:00
Miloš Paunović
f9d48d6f74 fix(ui): allow multi label filtering (#8022) 2025-03-25 09:52:54 +01:00
Florian Hussonnois
267d848d3f fix(core): ensure defaults can be injected in flows (#3206)
Flow revisions create from older Kestra versions may not be linked to their original source.
In such cases, fall back to the generated source approach to enable plugin default injection.

close: #3206
2025-03-25 09:17:55 +01:00
brian.mulier
edbf14c1b8 fix(ui): handle properly layout of global Secrets when there is a secret manager 2025-03-25 02:45:21 +01:00
brian.mulier
d9ac267161 feat(ui): introduce global Secrets page
closes kestra-io/kestra-ee#1257
2025-03-25 01:39:18 +01:00
brian.mulier
74455ad993 fix(ui): prevent function parameters autocompletion from deleting parenthesis 2025-03-25 01:39:18 +01:00
YannC
e2dd02173f fix(): use correctly context-path when needed (#8018) 2025-03-24 20:30:55 +01:00
YannC
c7b3a42f34 fix(): make absolute date work with metrics (#8015)
close #7951
2025-03-24 18:46:45 +01:00
Rudhra Bharathy G
fe882bbee6 fix(): moved the logic to the backend (#5917)
close #5740
Co-authored-by: YannC <ycoornaert@kestra.io>
2025-03-24 17:59:54 +01:00
YannC
1efb21c087 fix(): make KestraFilter works with legacy default dashboard + avoid showing custom dashboard in namespace/flow pages (#8012)
close #7959
2025-03-24 16:33:48 +01:00
github-actions[bot]
beb6182104 chore(translations): localize to languages other than English (#8013)
Co-authored-by: GitHub Action <actions@github.com>
2025-03-24 16:24:15 +01:00
Miloš Paunović
7f0d3521db chore(ui): add anchors to plugin documentation (#8014) 2025-03-24 16:11:47 +01:00
benjoEK1337
7e4453632c fix(core): Handling for trailing slash in the KESTRA_URL configuration (#6373)
* fix(core): Handling for trailing slash in the KESTRA_URL configuration

* Update webserver/src/main/java/io/kestra/webserver/controllers/api/ExecutionController.java

---------

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
Co-authored-by: Loïc Mathieu <loikeseke@gmail.com>
2025-03-24 16:10:43 +01:00
Loïc Mathieu
4a3d6b30d2 feat(core,jdbc): reset the trigger inside the JdbcExecutor
Fixes #7237
2025-03-24 15:54:53 +01:00
Florian Hussonnois
fe7c14c048 chore: refactor StorageInterfaceFactory to be a bean
changes:
* extract class for ServiceType

related-to: kestra-io/kestra-ee#3021
2025-03-24 15:45:24 +01:00
Barthélémy Ledoux
7b007aafc0 feat: parse docs is 2 steps to avoid user waiting with nothing (#7149) 2025-03-24 15:25:46 +01:00
Barthélémy Ledoux
f6aa2729b2 feat: multi panel editor without the refactor (#7971)
* refactor: move logic out of EditorView.vue and into store flow.js

* fix create mode

* feat: add multi panel flow editor

* missing flow id

* fix helpers tests ?

* restore helpers test

* fix one failing test

* fix: remove non-working stories

* remove size when one does split

* fix duplicated simulation

* fix tests

* try and fix the test

* remove tests to make sure we merge
2025-03-24 15:13:26 +01:00
Nicolas K.
0ca25445fd fix(platform): move slf4j api to enforce platform to fix it's version in test (#8007)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-24 14:25:16 +01:00
Piyush Bhaskar
4900963cb9 fix(ui): amend task source tabs colors (#8010) 2025-03-24 14:01:48 +01:00
Piyush Bhaskar
21d264f444 chore(ui): highlight first item in global search bar autocomplete (#8009) 2025-03-24 13:43:27 +01:00
Miloš Paunović
b6a3d0cd54 chore(ui): removing dynamic section of collapsible plugin properties (#8008) 2025-03-24 13:11:29 +01:00
Bart Ledoux
3244b1c293 fix: remove unwanted change in Curl 2025-03-24 11:52:49 +01:00
Florian Hussonnois
0987d0b349 feat(core): add new subkey arg to secret pebble function
Add new `subkey` named argument to the pebble function `secret`
to be able to select a specific field inside a secret containing
a JSON value

close: kestra-io/kestra-ee#3200
2025-03-24 11:43:14 +01:00
dependabot[bot]
5a2ac895ec build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.36.2 to 0.36.3.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.36.2...v0.36.3)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 11:14:10 +01:00
Loïc Mathieu
7421693445 fix(core): race while initializing trigger + possible duplicate update 2025-03-24 11:10:17 +01:00
Loïc Mathieu
19894dbcd6 fix(core): race in the FlowListener
It sometimes throw ConcurrentModificationException while sending flow updates to the consumers.
2025-03-24 11:10:17 +01:00
dependabot[bot]
62e37f3b15 build(deps): bump software.amazon.awssdk:bom from 2.31.1 to 2.31.6
Bumps software.amazon.awssdk:bom from 2.31.1 to 2.31.6.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 10:48:44 +01:00
dependabot[bot]
37af61f41d build(deps): bump com.github.oshi:oshi-core from 6.7.1 to 6.8.0
Bumps [com.github.oshi:oshi-core](https://github.com/oshi/oshi) from 6.7.1 to 6.8.0.
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oshi/oshi/compare/oshi-parent-6.7.1...oshi-parent-6.8.0)

---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 10:29:32 +01:00
dependabot[bot]
491f07296a build(deps): bump com.google.cloud:libraries-bom from 26.56.0 to 26.57.0
Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.56.0 to 26.57.0.
- [Release notes](https://github.com/googleapis/java-cloud-bom/releases)
- [Changelog](https://github.com/googleapis/java-cloud-bom/blob/main/release-please-config.json)
- [Commits](https://github.com/googleapis/java-cloud-bom/compare/v26.56.0...v26.57.0)

---
updated-dependencies:
- dependency-name: com.google.cloud:libraries-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 10:29:14 +01:00
dependabot[bot]
6d59630a67 build(deps): bump com.microsoft.playwright:playwright
Bumps [com.microsoft.playwright:playwright](https://github.com/microsoft/playwright-java) from 1.50.0 to 1.51.0.
- [Release notes](https://github.com/microsoft/playwright-java/releases)
- [Commits](https://github.com/microsoft/playwright-java/compare/v1.50.0...v1.51.0)

---
updated-dependencies:
- dependency-name: com.microsoft.playwright:playwright
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 10:28:36 +01:00
YannC
a098650559 fix(): restarting trigger does not break the list anymore (#8004)
close #7712
2025-03-24 10:21:19 +01:00
Barthélémy Ledoux
505049a69e refactor: move flow editor logic into flow store (#7968)
* refactor: move logic out of EditorView.vue and into store flow.js

* fix create mode

* missing flow id

* fix helpers tests ?

* restore helpers test

* remove unfinished test files

* fix plugin docs

* remove issues on lowcode
2025-03-24 09:48:17 +01:00
yuri1969
a1abd28a3d fix(build): Gradle space-assignment deprecation 2025-03-24 09:26:44 +01:00
yuri1969
49a29c4bf1 fix(core): amend server start announcements
Servers announced their start even when their async startup process
was still in progress. This lead to confusing situations - a server
announced a successfull start but in fact it failed just few
moments later.

Note, the log message needs to be ported to EE server impls.
2025-03-24 09:25:00 +01:00
Miloš Paunović
49af8f336a fix(ui): include parameters into request for plugin schema fetching (#8002) 2025-03-24 09:12:52 +01:00
Nicolas K.
8642771a2b fix(core): working dir interface contract by putting back putFile(path, content) (#7980)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-21 16:58:51 +01:00
Nicolas K.
e44487c8d8 feat(core): #7932 add file exist comportment to NamespaceFiles (#7979)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-21 15:17:12 +01:00
Ludovic DEHON
f29aab0719 feat(core): speed up namespace file download and add a log & metrics
close #7871
close #7870
2025-03-21 13:44:16 +01:00
Loïc Mathieu
73c7a2d3d3 feat(core): allow null or empty proxy address to bypass proxy conf
Fixes #7961
2025-03-21 12:36:11 +01:00
brian.mulier
773a6e9093 fix(ui): change plugin doc properly upon switching plugin type
closes #7876
2025-03-21 11:51:44 +01:00
Ludovic DEHON
df1bbcfb7f fix(cicd): display ui unit test status 2025-03-21 09:53:49 +01:00
brian.mulier
f3419084fa fix(ui): filter tests were not up-to-date 2025-03-21 09:49:07 +01:00
brian.mulier
ab7788aeaf feat(ui): introduce global KV Store view
part of kestra-io/kestra-ee#1257
closes kestra-io/kestra-ee#1743
2025-03-21 09:49:07 +01:00
YannC
99e06632e0 fix(): prevent using LABELS in JDBC
link #6655
2025-03-20 21:54:41 +01:00
YannC
7b3d5a593e fix(): remove old '$eq' to avoid error in dashboard
link #7959
2025-03-20 21:09:09 +01:00
Bart Ledoux
cca7ed0bff fix: update ui-libs 2025-03-20 18:33:56 +01:00
Loïc Mathieu
481138e433 fix(core): add metric Publish task doc & icon
fixes #7154
2025-03-20 16:24:38 +01:00
Barthélémy Ledoux
267ffb69dc fix: only run posthog in prod mode (#7952) 2025-03-20 14:40:47 +01:00
Loïc Mathieu
43f1374aa3 chore: add GraalVM plugin 2025-03-20 13:10:39 +01:00
Loïc Mathieu
fe944ccc5b chore: add jattach into our Docker image 2025-03-20 11:57:55 +01:00
YannC
cec8702fc6 feat(): allow to specify a parent namespace in the flow updates command, and the possibility to allow child namespace in the flow bulk create command (#7956) 2025-03-20 11:24:23 +01:00
YannC
f898a679c4 fix(): Adapt the usage of the new query filter for custom dashboard (#7935)
close #7091
2025-03-19 17:13:56 +01:00
Nicolas K.
9a9d59f433 fix(core-ee): NPE when execution labels are null (#7950)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-19 17:09:17 +01:00
Loïc Mathieu
0579e23a20 fix(core): triggers don't have an execution ID already 2025-03-19 15:39:08 +01:00
YannC
343367d8c5 feat(): save last dashboard used in local storage (#7948)
close kestra-io/kestra-ee#2615
2025-03-19 14:45:45 +01:00
Miloš Paunović
d47e8083ac fix(ui): prevent filter text prefix to show up when searching locally (#7947) 2025-03-19 13:51:24 +01:00
Miloš Paunović
7c4d6b02a7 chore(ui): introduce a refresh functionality to execution logs page (#7946) 2025-03-19 12:07:34 +01:00
Miloš Paunović
95dd1b8a55 chore(deps): regular dependency update (#7944) 2025-03-19 10:41:29 +01:00
Loïc Mathieu
0ec2d88429 Revert "build(deps): bump protobufVersion from 3.25.5 to 4.30.1"
This reverts commit 6946c92683.
2025-03-19 10:27:44 +01:00
dependabot[bot]
c3b504a9ed build(deps): bump pdfjs-dist from 4.10.38 to 5.0.375 in /ui (#7942)
Bumps [pdfjs-dist](https://github.com/mozilla/pdf.js) from 4.10.38 to 5.0.375.
- [Release notes](https://github.com/mozilla/pdf.js/releases)
- [Commits](https://github.com/mozilla/pdf.js/compare/v4.10.38...v5.0.375)

---
updated-dependencies:
- dependency-name: pdfjs-dist
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-19 10:25:58 +01:00
Loïc Mathieu
8cba4dab62 fix(core): possible NPE if no manifest 2025-03-19 10:03:12 +01:00
Miloš Paunović
1b22d1e3d1 fix(ui): amend dependabot errors with parsing of package-lock.json file (#7941) 2025-03-19 09:33:20 +01:00
Loïc Mathieu
6194f244c8 feat(core): require existing namespace
Part-of: https://github.com/kestra-io/kestra-ee/issues/3071
2025-03-18 15:56:36 +01:00
Loïc Mathieu
11a1664865 feat(*): allow disabling flow logs and use a specific logger for executions, tasks and triggers
Fixes #7753
2025-03-18 15:53:30 +01:00
Miloš Paunović
db394f6371 chore(ui): remove the flow topology control button from image export (#7937) 2025-03-18 15:32:21 +01:00
Loïc Mathieu
a7433c6f62 feat(core): add correlationId to the Flow trigger
Fixes #7920
2025-03-18 14:35:59 +01:00
Loïc Mathieu
745c64c4b0 feat(model, core): document tasks props that are internal storage URI
Fixes https://github.com/kestra-io/kestra/issues/681
2025-03-18 14:29:57 +01:00
YannC
5d15e2081c fix(*): Improve log timestamp precision + allow to override timestamp… (#7847)
* fix(*): Improve log timestamp precision + allow to override timestamp in log consumer

close #5915
close #7243
close #5662

* chore(): review changes
2025-03-18 14:25:24 +01:00
Miloš Paunović
7d098eaa4e feat(ui): add beta badge global component (#7934) 2025-03-18 11:56:43 +01:00
Florian Hussonnois
0ee9abb372 fix: OpenTelemetry should be optional
OpenTelementry must be optional when OTEL is globally disabled
in micronaut.
2025-03-18 10:34:23 +01:00
github-actions[bot]
3f542ae737 chore(translations): localize to languages other than English (#7929) 2025-03-18 10:20:49 +01:00
rajatsingh23
811cc7722a feat(ui): add the option to copy single/all logs to clipboard (#7755)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-03-18 09:49:54 +01:00
Florian Hussonnois
f4f8ecb247 feat!: make kv pebble function raise error by default (#7855)
BREAKING CHANGE: use 'true' as default value for kv function arg errorOnMissing.

close: #7855
2025-03-18 09:24:16 +01:00
Florian Hussonnois
3566c4d365 fix(cli): return exit 0 in CLI plugins cmd
The CLI plugins install command must return exit code 0
even when no plugin are installed
2025-03-18 09:23:58 +01:00
YannC
a7421987b7 chore(): change docker image used for AbstractTaskRunnerTest 2025-03-18 09:19:17 +01:00
ben8t
c640b8208b [fix] fix sleep assert comparison (#7924) 2025-03-18 09:16:20 +01:00
ben8t
db71b56dcd Update documentation Switch core task (#6792) 2025-03-18 09:13:39 +01:00
Florian Hussonnois
8076fcc990 fix: disable micronaut otel by default 2025-03-18 08:47:19 +01:00
Miloš Paunović
bd520c2150 fix(ui): amend displaying large amount of logs (#7923)
* chore(ui): amend plugin documentation console errors

* fix(ui): amend displaying large amount of logs
2025-03-18 08:35:55 +01:00
Florian Hussonnois
2c731a0192 fix(core): avoid ClassCastException when parsing flow inputs (#7882)
Use toString() instead of casting objects directly to String
to avoid undesirable ClasCastException we expect a string type

close: #7882
2025-03-17 14:48:15 +01:00
github-actions[bot]
3b291b75cf chore(translations): localize to languages other than English (#7918) 2025-03-17 14:23:48 +01:00
Miloš Paunović
1563d1d2da feat(ui): implement a default tab setting for flows (#7917) 2025-03-17 13:44:09 +01:00
dependabot[bot]
5c73953c81 build(deps): bump org.aspectj:aspectjweaver from 1.9.22.1 to 1.9.23
Bumps [org.aspectj:aspectjweaver](https://github.com/eclipse/org.aspectj) from 1.9.22.1 to 1.9.23.
- [Release notes](https://github.com/eclipse/org.aspectj/releases)
- [Commits](https://github.com/eclipse/org.aspectj/commits)

---
updated-dependencies:
- dependency-name: org.aspectj:aspectjweaver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 13:16:50 +01:00
dependabot[bot]
f89aa8d278 build(deps): bump org.slf4j:slf4j-api from 2.0.16 to 2.0.17
Bumps org.slf4j:slf4j-api from 2.0.16 to 2.0.17.

---
updated-dependencies:
- dependency-name: org.slf4j:slf4j-api
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 13:16:41 +01:00
github-actions[bot]
7efa3975fc chore(translations): localize to languages other than English (#7915) 2025-03-17 12:54:11 +01:00
Miloš Paunović
a3362b0c85 chore(ui): properly sanitize execution errors markdown content before rendering (#7914) 2025-03-17 12:52:02 +01:00
dependabot[bot]
ab9ba91e5a build(deps): bump org.jooq:jooq from 3.19.18 to 3.20.2
Bumps org.jooq:jooq from 3.19.18 to 3.20.2.

---
updated-dependencies:
- dependency-name: org.jooq:jooq
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 12:04:50 +01:00
Loïc Mathieu
355e24c9da chore(core): merge outputs only when necessary 2025-03-17 11:36:59 +01:00
github-actions[bot]
78dcca64ae chore(translations): localize to languages other than English (#7909) 2025-03-17 11:28:11 +01:00
Anna Geller
57c749d812 Revert "Adjusted the CI to skip frontend and backend tests for commits/PRs co…" (#7910)
This reverts commit 71a59e7b85.
2025-03-17 11:18:15 +01:00
Ash
2d7c233a5e chore(ui): improve charts horizontal scrollbar (#7508) 2025-03-17 10:48:16 +01:00
Piyush Bhaskar
cfac2c6fef chore(ui): change display on gantt page for created & queued executions (#7706) 2025-03-17 10:31:40 +01:00
Piyush Bhaskar
06352e46dc chore(ui): use the correct shadow on charts (#7752) 2025-03-17 10:22:55 +01:00
Piyush Bhaskar
08312aef97 chore(ui): show loading skeletons on dashboard (#7417) 2025-03-17 10:19:50 +01:00
github-actions[bot]
e05554e12d chore(translations): localize to languages other than English (#7908) 2025-03-17 10:10:57 +01:00
Miloš Paunović
cbb6ca77f7 feat(ui): add copy button to kv store listing rows (#7907) 2025-03-17 09:59:11 +01:00
dependabot[bot]
3dcd3c9785 build(deps): bump nl.basjes.gitignore:gitignore-reader
Bumps [nl.basjes.gitignore:gitignore-reader](https://github.com/nielsbasjes/codeowners) from 1.11.1 to 1.11.3.
- [Release notes](https://github.com/nielsbasjes/codeowners/releases)
- [Changelog](https://github.com/nielsbasjes/codeowners/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nielsbasjes/codeowners/compare/v1.11.1...v1.11.3)

---
updated-dependencies:
- dependency-name: nl.basjes.gitignore:gitignore-reader
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:52:28 +01:00
dependabot[bot]
7f6e15ec4a build(deps): bump aquasecurity/trivy-action from 0.29.0 to 0.30.0
Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.29.0 to 0.30.0.
- [Release notes](https://github.com/aquasecurity/trivy-action/releases)
- [Commits](https://github.com/aquasecurity/trivy-action/compare/0.29.0...0.30.0)

---
updated-dependencies:
- dependency-name: aquasecurity/trivy-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:51:53 +01:00
dependabot[bot]
ab061e9a12 build(deps): bump dorny/test-reporter from 1 to 2
Bumps [dorny/test-reporter](https://github.com/dorny/test-reporter) from 1 to 2.
- [Release notes](https://github.com/dorny/test-reporter/releases)
- [Changelog](https://github.com/dorny/test-reporter/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dorny/test-reporter/compare/v1...v2)

---
updated-dependencies:
- dependency-name: dorny/test-reporter
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:51:41 +01:00
dependabot[bot]
8eb91b75e9 build(deps): bump com.github.oshi:oshi-core from 6.7.0 to 6.7.1
Bumps [com.github.oshi:oshi-core](https://github.com/oshi/oshi) from 6.7.0 to 6.7.1.
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oshi/oshi/compare/oshi-parent-6.7.0...oshi-parent-6.7.1)

---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:51:30 +01:00
dependabot[bot]
0134d5e5c2 build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.36.1 to 0.36.2.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.36.1...v0.36.2)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:51:18 +01:00
dependabot[bot]
6946c92683 build(deps): bump protobufVersion from 3.25.5 to 4.30.1
Bumps `protobufVersion` from 3.25.5 to 4.30.1.

Updates `com.google.protobuf:protobuf-java` from 3.25.5 to 4.30.1
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.25.5...v4.30.1)

Updates `com.google.protobuf:protobuf-java-util` from 3.25.5 to 4.30.1

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
  update-type: version-update:semver-major
- dependency-name: com.google.protobuf:protobuf-java-util
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:50:36 +01:00
dependabot[bot]
149dcac5f9 build(deps): bump software.amazon.awssdk:bom from 2.30.36 to 2.31.1
Bumps software.amazon.awssdk:bom from 2.30.36 to 2.31.1.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-17 09:50:27 +01:00
Miloš Paunović
0af5aea287 chore(ui): create flow button on welcome page to start product tour (#7906) 2025-03-17 08:40:57 +01:00
Barthélémy Ledoux
9413d4df06 fix(ui): use container queries for dashboard (#7889) 2025-03-17 08:24:39 +01:00
Miloš Paunović
87ee267f10 chore(ui): complete refactor of yaml utils (#7888) 2025-03-17 08:14:48 +01:00
Will Russell
02336ed397 fix(docs): typos 2025-03-14 19:04:38 +00:00
Will Russell
379199e186 fix(docs): fail examples 2025-03-14 18:47:57 +00:00
Will Russell
ebec8c2121 fix(docs): schedule example 2025-03-14 18:41:36 +00:00
Malay Dewangan
564c36ceab fix: remove labels from an execution (#7256)
* fix: remove labels from an execution

* feat(test): add test for removing labels from execution

* fix(test): update test for removing labels from execution

* feat: handle system labels and add unit test for system labels

---------

Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-03-14 15:19:29 +01:00
Miloš Paunović
3d248d0d74 refactor(ui): remove the obsolete console statement (#7887) 2025-03-14 15:00:35 +01:00
Loïc Mathieu
70e6d47c1a feat(webserver): add Kestra URL to the config endpoint 2025-03-14 14:57:52 +01:00
Loïc Mathieu
1ba54cd08e feat(core): add Kestra env name and URL to the expression context 2025-03-14 14:57:52 +01:00
Florian Hussonnois
eb6b40a04b feat(cli): add new --all option to plugin install cmd (#7375)
This commit adds the option all to the plugin install command
for installing all available public plugins.

Changes:
* add new service class PluginCatalogService

close: #7375
2025-03-14 14:44:30 +01:00
Anna Geller
d314c52924 docs: kv pebble (#7886) 2025-03-14 14:31:52 +01:00
Trinaya Kantevari
71a59e7b85 Adjusted the CI to skip frontend and backend tests for commits/PRs containing the 'docs-only' text. #6729 (#7756)
* Update pull-request.yml to skip tests for PRs with 'docs-only'.

Updated the if condition in the frontend and backend jobs to skip tests if 'docs-only' is present in the Pull Request title, body, or labels.

* Update workflow-test.yml to skip tests for commits with 'docs-only'.

Updated the if condition in the frontend and backend jobs to skip tests if 'docs-only' is present in the commit message.

* Apply suggestions from code review

---------

Co-authored-by: Anna Geller <anna.m.geller@gmail.com>
2025-03-14 14:29:26 +01:00
Will Russell
ff504afd8f fix(docs): add parallel example 2025-03-14 12:37:00 +00:00
YannC
64e5b80049 fix(ui): display back core property if not in a task
close #7879
2025-03-14 11:44:22 +01:00
Loïc Mathieu
c9baaf8565 fix(core): Avoid calling Worker post construct method twice 2025-03-14 09:58:30 +01:00
Miloš Paunović
02ad2af305 chore(ui): re-organize visuals for empty state component (#7875) 2025-03-14 08:46:57 +01:00
Florian Hussonnois
bed11f1541 chore(core): add PluginVersioning interface and version for TaskRunner
Changes:
- add new interface PluginVersioning
- update interfaces TaskInterface, TriggerInterface, TaskRunner

close: kestra-io/kestra-ee#3122
2025-03-14 08:26:52 +01:00
Miloš Paunović
6d14773ea7 chore(ui): prevent node state history to be shown if there is no date present (#7867) 2025-03-13 16:26:25 +01:00
Loïc Mathieu
74577cb059 chore(deps): upgrade @kestra-io/ui-libs to 0.0.157 2025-03-13 15:50:45 +01:00
Miloš Paunović
e63b019857 chore(ui): re-organize visuals for empty state component (#7866) 2025-03-13 15:18:29 +01:00
github-actions[bot]
8684bd9481 chore(translations): localize to languages other than English (#7863) 2025-03-13 14:36:31 +01:00
Miloš Paunović
586e089ada feat(ui): add afterExecution block to no code editor (#7848) 2025-03-13 13:53:52 +01:00
Loïc Mathieu
acebfef0d1 fix(deps): move OTLP metrics lib to CLI to avoid warning in tests 2025-03-13 13:05:20 +01:00
Loïc Mathieu
f7019af9d5 feat(core): add afterExecution to the topology 2025-03-13 13:02:34 +01:00
Loïc Mathieu
1419680006 fix(core): missing afterExecution in FlowForExecution 2025-03-13 13:02:34 +01:00
Shruti Mantri
64290f5d17 feat: add finally to the flow assets in basic.md (#7857) 2025-03-13 10:27:43 +01:00
Loïc Mathieu
e2da2dfeb0 fix(core): allowedNamespace is already called in the KvStoreService 2025-03-12 16:37:16 +01:00
Will Russell
977fe222a4 fix(docs): 2025 get started video 2025-03-12 13:11:18 +00:00
ben8t
0a24a5d13f Remove pebble function references that don't exist (#7698)
* [fix] remove pebble function references that don't exist

* [fix] pebble documentation

* [fix] remove non-existing functions
2025-03-12 12:23:22 +01:00
Loïc Mathieu
ecfe925ece feat(core): redact KESTRA_JAVA_OPTS from the env available to executions 2025-03-12 11:31:53 +01:00
dependabot[bot]
76f6c93e48 build(deps): bump @babel/runtime-corejs3 from 7.26.9 to 7.26.10 in /ui (#7840)
Bumps [@babel/runtime-corejs3](https://github.com/babel/babel/tree/HEAD/packages/babel-runtime-corejs3) from 7.26.9 to 7.26.10.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.26.10/packages/babel-runtime-corejs3)

---
updated-dependencies:
- dependency-name: "@babel/runtime-corejs3"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-12 08:43:56 +01:00
dependabot[bot]
cfb76f4279 build(deps): bump @babel/helpers from 7.26.9 to 7.26.10 in /ui (#7841)
Bumps [@babel/helpers](https://github.com/babel/babel/tree/HEAD/packages/babel-helpers) from 7.26.9 to 7.26.10.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.26.10/packages/babel-helpers)

---
updated-dependencies:
- dependency-name: "@babel/helpers"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-12 08:43:18 +01:00
Miloš Paunović
50bbf03d53 chore(deps): regular dependency update (#7839) 2025-03-12 08:29:38 +01:00
Loïc Mathieu
2c9f09a82e feat(build): configure heap size to max 50% of available memory (#7800)
* feat(build): configure heap size to max 50% of available memory

* Apply suggestions from code review

Co-authored-by: brian-mulier-p <bmmulier@hotmail.fr>

---------

Co-authored-by: brian-mulier-p <bmmulier@hotmail.fr>
2025-03-11 16:38:51 +01:00
github-actions[bot]
c9579e5dcd chore(translations): localize to languages other than English (#7803)
Co-authored-by: GitHub Action <actions@github.com>
2025-03-11 16:34:42 +01:00
Miloš Paunović
1b13559a61 chore(ui): add the ability to change word wrap and to copy the preview output content (#7801) 2025-03-11 15:12:51 +01:00
Loïc Mathieu
c2e62d653d feat(core,jdbc): switch back to a cached thread pool for async JDBC queue
But with less threads as 2xnbCore was too much.
2025-03-11 14:25:32 +01:00
Bart Ledoux
14ff4438fe chore: update ui-libs 2025-03-11 14:06:59 +01:00
Loïc Mathieu
39b8fc1039 feat(core): afterExecution tasks
Fixes https://github.com/kestra-io/kestra-ee/issues/2792
2025-03-11 13:35:08 +01:00
Florian Hussonnois
1cb323b7aa fix(core): fix properly map MavenPluginRepositoryConfig 2025-03-11 13:17:23 +01:00
brian.mulier
4a12827684 fix(ui): remove errors from dashboard validation if it's fixed
closes #7748
2025-03-11 12:26:50 +01:00
brian.mulier
82a346b2ce fix(ui): make switch view buttons from dashboard editor the same as flow editor ones 2025-03-11 12:23:18 +01:00
dependabot[bot]
dea392a941 build(deps): bump io.micrometer:micrometer-core from 1.14.3 to 1.14.5
Bumps [io.micrometer:micrometer-core](https://github.com/micrometer-metrics/micrometer) from 1.14.3 to 1.14.5.
- [Release notes](https://github.com/micrometer-metrics/micrometer/releases)
- [Commits](https://github.com/micrometer-metrics/micrometer/compare/v1.14.3...v1.14.5)

---
updated-dependencies:
- dependency-name: io.micrometer:micrometer-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-11 11:22:43 +01:00
Loïc Mathieu
d0af5767d5 feat(core): add execution state in Pebbe context
Fixes #7155
2025-03-11 11:21:50 +01:00
brian.mulier
a064c7a956 feat(ui): introduce kv() and secret() pebble autocompletions
closes kestra-io/kestra-ee#77
2025-03-11 00:10:29 +01:00
brian.mulier
9b5b2b981f fix(webserver): add endpoint for inherited secrets 2025-03-10 23:42:33 +01:00
YannC
fac2ae813c fix(jdbc): return correct total when paginating custom dashboard chart (#7790)
close #7164
2025-03-10 18:05:44 +01:00
Nicolas K.
233a871c58 feat(core-ee): add json format to file log exporter (#7789)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-10 17:47:51 +01:00
Nicolas K.
05f24f3d65 fix(core): #7740 http configuration bearer token may change to basic because of allowFailed (#7788)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-10 16:33:15 +01:00
dependabot[bot]
66e5a7ca31 build(deps): bump software.amazon.awssdk:bom from 2.30.31 to 2.30.36
Bumps software.amazon.awssdk:bom from 2.30.31 to 2.30.36.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:48:03 +01:00
dependabot[bot]
fefaa7cdbb build(deps): bump org.testcontainers:testcontainers
Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.20.5 to 1.20.6.
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testcontainers/testcontainers-java/compare/1.20.5...1.20.6)

---
updated-dependencies:
- dependency-name: org.testcontainers:testcontainers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:47:46 +01:00
YannC
fd6b9dc065 fix(tests): increase seconds diff between dates (#7785) 2025-03-10 15:08:20 +01:00
dependabot[bot]
5935308e43 build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.34.0 to 0.36.1.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.34.0...v0.36.1)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:02:29 +01:00
dependabot[bot]
83f06f3374 build(deps): bump org.jsoup:jsoup from 1.18.3 to 1.19.1
Bumps [org.jsoup:jsoup](https://github.com/jhy/jsoup) from 1.18.3 to 1.19.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES.md)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.18.3...jsoup-1.19.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:02:19 +01:00
dependabot[bot]
8d7bc6fdd4 build(deps): bump com.github.docker-java:docker-java from 3.4.1 to 3.4.2
Bumps [com.github.docker-java:docker-java](https://github.com/docker-java/docker-java) from 3.4.1 to 3.4.2.
- [Release notes](https://github.com/docker-java/docker-java/releases)
- [Changelog](https://github.com/docker-java/docker-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/docker-java/docker-java/compare/3.4.1...3.4.2)

---
updated-dependencies:
- dependency-name: com.github.docker-java:docker-java
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:01:44 +01:00
dependabot[bot]
03a44c1039 build(deps): bump org.opensearch.client:opensearch-java
Bumps [org.opensearch.client:opensearch-java](https://github.com/opensearch-project/opensearch-java) from 2.21.0 to 2.22.0.
- [Release notes](https://github.com/opensearch-project/opensearch-java/releases)
- [Changelog](https://github.com/opensearch-project/opensearch-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/opensearch-project/opensearch-java/compare/v2.21.0...v2.22.0)

---
updated-dependencies:
- dependency-name: org.opensearch.client:opensearch-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:00:59 +01:00
dependabot[bot]
5af0858445 build(deps): bump org.testcontainers:junit-jupiter from 1.20.5 to 1.20.6
Bumps [org.testcontainers:junit-jupiter](https://github.com/testcontainers/testcontainers-java) from 1.20.5 to 1.20.6.
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testcontainers/testcontainers-java/compare/1.20.5...1.20.6)

---
updated-dependencies:
- dependency-name: org.testcontainers:junit-jupiter
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:00:31 +01:00
dependabot[bot]
0a304ff1d3 build(deps): bump com.gorylenko.gradle-git-properties
Bumps com.gorylenko.gradle-git-properties from 2.4.2 to 2.5.0.

---
updated-dependencies:
- dependency-name: com.gorylenko.gradle-git-properties
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 15:00:13 +01:00
dependabot[bot]
92418841fc build(deps): bump com.azure:azure-sdk-bom from 1.2.31 to 1.2.32
Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.31 to 1.2.32.
- [Release notes](https://github.com/azure/azure-sdk-for-java/releases)
- [Commits](https://github.com/azure/azure-sdk-for-java/compare/azure-sdk-bom_1.2.31...azure-sdk-bom_1.2.32)

---
updated-dependencies:
- dependency-name: com.azure:azure-sdk-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 14:59:56 +01:00
Florian Hussonnois
01036c829d fix(core): wait for service-manager-task thread to be stopped 2025-03-10 14:22:38 +01:00
Loïc Mathieu
5ffeee532c fix(jdbc): flaky tests JdbcServiceLivenessCoordinatorTest 2025-03-10 12:38:56 +01:00
Loïc Mathieu
c606760522 chore(core): disable OTLP metrics on tests 2025-03-10 12:38:56 +01:00
Loïc Mathieu
0870d8ebd8 fix(core): flaky test ExitTest.shouldExitAndKillTheExecution() 2025-03-10 12:38:56 +01:00
Florian Hussonnois
d2d0726f73 fix(cli): fix double shutdown warn messages
Changes:
- Avoid registering two shutdown-hooks that will both log the
shutdown message - leading to some confusion
- Some code cleanup/refactoring
2025-03-10 12:31:06 +01:00
Nicolas K.
868a232527 fix(core): failing schedule test (#7783)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-10 12:03:11 +01:00
Loïc Mathieu
3ef11044a2 feat(core): custom log filter
Since Logback 1.5.13, there is no more default implementation for the EvaluatorFilter so we need to supply our own.
See https://logback.qos.ch/manual/filters.html#evaluatorFilter
2025-03-10 10:56:13 +01:00
Miloš Paunović
2d1582f761 chore(ui): respect line numbers prop as part of editor options (#7781) 2025-03-10 10:34:15 +01:00
github-actions[bot]
94d36fdc01 chore(translations): localize to languages other than English (#7780) 2025-03-10 10:13:49 +01:00
dependabot[bot]
0388909828 build(deps): bump robinraju/release-downloader from 1.11 to 1.12
Bumps [robinraju/release-downloader](https://github.com/robinraju/release-downloader) from 1.11 to 1.12.
- [Release notes](https://github.com/robinraju/release-downloader/releases)
- [Commits](https://github.com/robinraju/release-downloader/compare/v1.11...v1.12)

---
updated-dependencies:
- dependency-name: robinraju/release-downloader
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 09:55:11 +01:00
Miloš Paunović
ff4f43c39b chore(ui): re-order namespace tabs (#7778) 2025-03-10 09:38:51 +01:00
Miloš Paunović
140d058beb chore(ui): remove single empty space between label key and value so it can be copied (#7774) 2025-03-10 08:08:22 +01:00
Loïc Mathieu
4276a0afd9 Revert "fix(core): ThresholdFilter is now stricly lower"
This reverts commit bd82f5e3b6.
2025-03-07 18:12:33 +01:00
Loïc Mathieu
1692cab533 feat(core): add outputs to the Flow trigger
Fixes #6848
2025-03-07 18:10:00 +01:00
Nicolas K.
3013e9dfd5 feat(core): #5467 add namespaces in the namespaceFiles parameter (#7749)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-07 17:18:45 +01:00
brian.mulier
440a942900 fix(ci): generate_translations.py is now deleting keys that are no longer in en translation to avoid ghost translations 2025-03-07 16:49:45 +01:00
Loïc Mathieu
7696d41d5f fix(jdbc): resubmit worker job to the good worker group
Fixes #7730
2025-03-07 16:44:28 +01:00
github-actions[bot]
45abaa146e chore(translations): localize to languages other than English (#7746)
Co-authored-by: GitHub Action <actions@github.com>
2025-03-07 14:57:33 +01:00
Loïc Mathieu
e9f2711cd3 feat(core): handle worker group fallback inside the scheduler 2025-03-07 14:48:02 +01:00
brian.mulier
166262209a fix(tests): reject promise with 404 instead of empty resolve if non-mocked store call in flowAutoCompletionProvider.spec.ts 2025-03-07 14:01:37 +01:00
Miloš Paunović
99bad6abb0 chore(ui): use uniformed pagination component for custom dashboard tables (#7744) 2025-03-07 13:39:40 +01:00
Barthélémy Ledoux
acc10ed638 fix(flow editor): enhance behavior when switching file tabs (#7722)
* fix(flow editor): when opening a new file and coming back to flow, validation + lowcode are broken

* fix: sync flowYaml only onEdit

* fix: files can now be saved again

* fix: make editor saving button ready for better ux

* allow for a bit of time for storybook to launch the inputs story

* add storybook for flow editor

* setup storybook
2025-03-07 11:29:54 +01:00
Loïc Mathieu
88341bb5c9 feat(webserver): mask the secret() function result from eval outputs
Fixes https://github.com/kestra-io/kestra-ee/issues/3073
2025-03-07 11:26:28 +01:00
brian-mulier-p
e8411b6b11 refacto(ui): move autocompletion to its own service (#7710)
Co-authored-by: Bart Ledoux <bledoux@kestra.io>
2025-03-07 10:46:44 +01:00
github-actions[bot]
6af105a8bf chore(translations): localize to languages other than English (#7739)
Co-authored-by: GitHub Action <actions@github.com>
2025-03-07 10:14:51 +01:00
Miloš Paunović
9a2e09cc8c chore(ui): uniforming empty state for components (#7737) 2025-03-07 09:11:16 +01:00
Nicolas K.
9b1a9d64bc feat(core): #7721 add namespace to pebble file functions (#7729)
* feat(core): #7721 add namespace to pebble file functions

* clean(core): #7721 extract tenantId only once

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-06 18:05:19 +01:00
Bart Ledoux
4678616520 fix: repair collapsed menu submenus 2025-03-06 17:12:24 +01:00
Bart Ledoux
07e4598fa4 fix(demo): on pages stop showing the docs without a button
closes #7726
2025-03-06 16:54:52 +01:00
YannC
01fe48b47a fix(ui): improved fetch of type for pluginDoc and avoid removing doc if map has "type" as property but without doc (like ENUM value) (#7727)
close #6101
close #7708
2025-03-06 16:47:04 +01:00
YannC
f7e61a46df feat(core): add new crudEventType value 2025-03-06 15:49:45 +01:00
Loïc Mathieu
dfe5552a1e feat(core): allow reading file from any namespaces
Fixes https://github.com/kestra-io/kestra-ee/issues/1934
2025-03-06 15:36:24 +01:00
YannC
593558dd22 fix(runner-memory): delete MemorySchedulerTriggerState back due to cherry-pick 2025-03-06 15:28:29 +01:00
Miloš Paunović
b22764290e chore(ui): properly sanitize markdown content before rendering (#7724) 2025-03-06 15:13:10 +01:00
YannC
78bfbf0d5e fix(): align to EE 2025-03-06 14:13:06 +01:00
ben8t
f505f29360 Add core plugin sanitychecks (#7592)
* [feature] add sanitychecks

* [feature] add dag sanitychecks

* [fix] use assert

* [fix] use assert

* [feature] add for each sanity
2025-03-06 13:43:54 +01:00
ben8t
1726347dbf change email oss auth default (#7717) 2025-03-06 10:47:24 +01:00
Miloš Paunović
bbf232ba52 fix(ui): prevent context docs open on editor custom blueprints click (#7716) 2025-03-06 09:53:54 +01:00
yuri1969
8f9fc5fe4b fix(core): stop Docker runner gracefully 2025-03-06 09:34:11 +01:00
Ludovic DEHON
9a56b763f4 chore(core): make registry unregister usable with immutable list 2025-03-06 08:48:36 +01:00
Miloš Paunović
40d37d9e42 chore(ui): auto expand first element in execution overview cascaders (#7715) 2025-03-06 08:48:22 +01:00
Florian Hussonnois
e200bbdb6b fix(cli): fix regression on CLI plugin doc cmd 2025-03-05 21:24:09 +01:00
brian.mulier
ef65623b13 fix(core): avoid duplicates in plugins subgroups + properly retrieve subgroup title 2025-03-05 18:36:45 +01:00
Nicolas K.
d8b2e92e8d feat(core-ee): #2838 add audit log shipper (#7701)
* feat(core-ee): #2838 add audit log shipper

* fix: scheduler flacky tests

* feat(core-ee): add pretty string audit log and clean PR

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-05 16:30:11 +01:00
Loïc Mathieu
2c63112a59 feat(core, jdbc): dynamic worker group key
Fixes https://github.com/kestra-io/kestra-ee/issues/3064
2025-03-05 15:37:01 +01:00
dependabot[bot]
20b87f1c9c build(deps): bump jacksonVersion from 2.18.2 to 2.18.3
Bumps `jacksonVersion` from 2.18.2 to 2.18.3.

Updates `com.fasterxml.jackson:jackson-bom` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-bom/compare/jackson-bom-2.18.2...jackson-bom-2.18.3)

Updates `com.fasterxml.jackson.core:jackson-core` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.18.2...jackson-core-2.18.3)

Updates `com.fasterxml.jackson.core:jackson-databind` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson/commits)

Updates `com.fasterxml.jackson.core:jackson-annotations` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson/commits)

Updates `com.fasterxml.jackson.module:jackson-module-parameter-names` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-modules-java8/compare/jackson-modules-java8-2.18.2...jackson-modules-java8-2.18.3)

Updates `com.fasterxml.jackson.dataformat:jackson-dataformat-yaml` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-dataformats-text/compare/jackson-dataformats-text-2.18.2...jackson-dataformats-text-2.18.3)

Updates `com.fasterxml.jackson.dataformat:jackson-dataformat-smile` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-dataformats-binary/compare/jackson-dataformats-binary-2.18.2...jackson-dataformats-binary-2.18.3)

Updates `com.fasterxml.jackson.dataformat:jackson-dataformat-cbor` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-dataformats-binary/compare/jackson-dataformats-binary-2.18.2...jackson-dataformats-binary-2.18.3)

Updates `com.fasterxml.jackson.dataformat:jackson-dataformat-ion` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-dataformat-ion/commits)

Updates `com.fasterxml.jackson.dataformat:jackson-dataformat-xml` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-dataformat-xml/compare/jackson-dataformat-xml-2.18.2...jackson-dataformat-xml-2.18.3)

Updates `com.fasterxml.jackson.datatype:jackson-datatype-guava` from 2.18.2 to 2.18.3
- [Commits](https://github.com/FasterXML/jackson-datatypes-collections/compare/jackson-datatypes-collections-2.18.2...jackson-datatypes-collections-2.18.3)

Updates `com.fasterxml.jackson.datatype:jackson-datatype-jsr310` from 2.18.2 to 2.18.3

Updates `com.fasterxml.jackson.datatype:jackson-datatype-jdk8` from 2.18.2 to 2.18.3

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson:jackson-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.core:jackson-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.core:jackson-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.module:jackson-module-parameter-names
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-yaml
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-smile
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-cbor
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-ion
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.dataformat:jackson-dataformat-xml
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-guava
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jsr310
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jdk8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-05 15:36:23 +01:00
dependabot[bot]
ca4e6a4b33 build(deps): bump org.opensearch.client:opensearch-java
Bumps [org.opensearch.client:opensearch-java](https://github.com/opensearch-project/opensearch-java) from 2.20.0 to 2.21.0.
- [Release notes](https://github.com/opensearch-project/opensearch-java/releases)
- [Changelog](https://github.com/opensearch-project/opensearch-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/opensearch-project/opensearch-java/compare/v2.20.0...v2.21.0)

---
updated-dependencies:
- dependency-name: org.opensearch.client:opensearch-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-05 13:11:00 +01:00
dependabot[bot]
82df58d26f build(deps): bump com.google.cloud:libraries-bom from 26.54.0 to 26.56.0
Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.54.0 to 26.56.0.
- [Release notes](https://github.com/googleapis/java-cloud-bom/releases)
- [Changelog](https://github.com/googleapis/java-cloud-bom/blob/main/release-please-config.json)
- [Commits](https://github.com/googleapis/java-cloud-bom/compare/v26.54.0...v26.56.0)

---
updated-dependencies:
- dependency-name: com.google.cloud:libraries-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-05 12:28:30 +01:00
dependabot[bot]
b6f91128a1 build(deps): bump opensearchRestVersion from 2.18.0 to 2.19.1
Bumps `opensearchRestVersion` from 2.18.0 to 2.19.1.

Updates `org.opensearch.client:opensearch-rest-client` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/opensearch-project/OpenSearch/releases)
- [Changelog](https://github.com/opensearch-project/OpenSearch/blob/main/CHANGELOG-3.0.md)
- [Commits](https://github.com/opensearch-project/OpenSearch/compare/2.18.0...2.19.1)

Updates `org.opensearch.client:opensearch-rest-high-level-client` from 2.18.0 to 2.19.1
- [Release notes](https://github.com/opensearch-project/OpenSearch/releases)
- [Changelog](https://github.com/opensearch-project/OpenSearch/blob/main/CHANGELOG-3.0.md)
- [Commits](https://github.com/opensearch-project/OpenSearch/compare/2.18.0...2.19.1)

---
updated-dependencies:
- dependency-name: org.opensearch.client:opensearch-rest-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.opensearch.client:opensearch-rest-high-level-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-05 12:15:10 +01:00
Miloš Paunović
a7abc17c0b chore(deps): regular dependency update (#7705) 2025-03-05 12:01:18 +01:00
Miloš Paunović
9607d0152e chore(ui): properly sanitize markdown content before rendering (#7697) 2025-03-05 08:17:02 +01:00
AJ Emerich
fc3c1a4c54 docs: fix typo in storage reverse task title (#7667) 2025-03-04 18:31:56 +01:00
Miloš Paunović
4cee4fca61 chore(ui): make sure chart stacks are following the same order every time (#7664) 2025-03-04 15:22:14 +01:00
Miloš Paunović
475573d13e feat(ui): add markdown formatting to flow run dialog (#7663) 2025-03-04 14:34:15 +01:00
Miloš Paunović
cfac9f339f chore(ui): properly sanitize markdown content before rendering (#7662) 2025-03-04 14:02:07 +01:00
Loïc Mathieu
cd97705d87 feat(core, jdbc): DashboardRepository.findAll() 2025-03-04 11:03:31 +01:00
Nicolas K.
56406c6b5c feat(#7636): add default options for HttpClient (#7650)
* feat(#7636): add default options for HttpClient

* fix(webserver): flacky execution controller tests

* fix(webserver): flacky execution controller tests

* fix(webserver): flacky execution controller tests

* fix(webserver): rework executor controller kill test

* fix(webserver): rework executor controller kill test

* fix(webserver): rework executor controller kill test

* fix(webserver): rework executor controller kill test

* fix(webserver): rework executor controller kill test

* fix(webserver): rework executor controller kill test

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-03-04 10:02:10 +01:00
YannC
38921a265a chore(deps): upgrade ui-libs version to v0.0.154 2025-03-03 22:11:53 +01:00
Shruti Mantri
be61869642 feat: add example for expression condition (#7660) 2025-03-03 18:22:49 +01:00
Shruti Mantri
1f09afe564 feat: add examples for flow conditions (#7659) 2025-03-03 17:52:56 +01:00
Shruti Mantri
35a0520dba feat: add examples for conditions - 3 (#7657)
Co-authored-by: AJ Emerich <aemerich@kestra.io>
2025-03-03 17:52:20 +01:00
YannC
cf6fad0896 fix(): avoid crash on injectDefault 2025-03-03 17:42:21 +01:00
Miloš Paunović
af0c1134e1 chore(ui): improve the topology tooltip label for adding task button (#7656) 2025-03-03 16:14:40 +01:00
Loïc Mathieu
12180457ea fix(core): MultipleCondition documentation 2025-03-03 15:14:55 +01:00
Loïc Mathieu
1eb9adf30a fix(core): validation error when timeWindow.type is null 2025-03-03 15:14:55 +01:00
dependabot[bot]
e7f2ec2aee build(deps): bump software.amazon.awssdk:bom from 2.30.16 to 2.30.31
Bumps software.amazon.awssdk:bom from 2.30.16 to 2.30.31.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 15:06:43 +01:00
dependabot[bot]
94421f141e build(deps): bump org.owasp.dependencycheck from 12.0.2 to 12.1.0
Bumps org.owasp.dependencycheck from 12.0.2 to 12.1.0.

---
updated-dependencies:
- dependency-name: org.owasp.dependencycheck
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 14:36:41 +01:00
dependabot[bot]
65b071adfd chore(deps): bump org.testcontainers:testcontainers
Bumps [org.testcontainers:testcontainers](https://github.com/testcontainers/testcontainers-java) from 1.20.4 to 1.20.5.
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testcontainers/testcontainers-java/compare/1.20.4...1.20.5)

---
updated-dependencies:
- dependency-name: org.testcontainers:testcontainers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 14:31:35 +01:00
Ludovic DEHON
d2bf56fecf fix(cicd): missing acls for test reports 2025-03-03 14:17:45 +01:00
dependabot[bot]
998a5cca32 chore(deps): bump flyingSaucerVersion from 9.11.3 to 9.11.4
Bumps `flyingSaucerVersion` from 9.11.3 to 9.11.4.

Updates `org.xhtmlrenderer:flying-saucer-core` from 9.11.3 to 9.11.4
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.11.3...v9.11.4)

Updates `org.xhtmlrenderer:flying-saucer-pdf` from 9.11.3 to 9.11.4
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.11.3...v9.11.4)

---
updated-dependencies:
- dependency-name: org.xhtmlrenderer:flying-saucer-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.xhtmlrenderer:flying-saucer-pdf
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 14:15:30 +01:00
dependabot[bot]
4b48ad597d build(deps): bump nl.basjes.gitignore:gitignore-reader
Bumps [nl.basjes.gitignore:gitignore-reader](https://github.com/nielsbasjes/codeowners) from 1.10.0 to 1.11.1.
- [Release notes](https://github.com/nielsbasjes/codeowners/releases)
- [Changelog](https://github.com/nielsbasjes/codeowners/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nielsbasjes/codeowners/compare/v1.10.0...v1.11.1)

---
updated-dependencies:
- dependency-name: nl.basjes.gitignore:gitignore-reader
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 14:14:41 +01:00
dependabot[bot]
66fdb58f4b build(deps): bump com.github.oshi:oshi-core from 6.6.6 to 6.7.0
Bumps [com.github.oshi:oshi-core](https://github.com/oshi/oshi) from 6.6.6 to 6.7.0.
- [Release notes](https://github.com/oshi/oshi/releases)
- [Changelog](https://github.com/oshi/oshi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/oshi/oshi/compare/oshi-parent-6.6.6...oshi-parent-6.7.0)

---
updated-dependencies:
- dependency-name: com.github.oshi:oshi-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-03 14:14:27 +01:00
Loïc Mathieu
8c708e2d53 feat(core): improve merging outputs by merging them at the task level 2025-03-03 13:57:48 +01:00
Loïc Mathieu
677585213a feat(core): use HashMap.newHashMap(int) 2025-03-03 13:57:48 +01:00
Loïc Mathieu
ff3f90465d feat(core): replace new ArrayList by Collections.emptyList() 2025-03-03 13:57:48 +01:00
YannC
8f5189df49 fix(): recoverMissedSchedules behavior on long running executions (#7617)
* fix(): handle missed scheduled for long running execution

close #7549

* tests(core): add test for backfill last/none on long running execution

* fix(): review pr

* fix(): shorter test

* fix(): fix tests and add missing getStopAfter

* tests(): fix scheduler tests
2025-03-03 13:14:16 +01:00
Miloš Paunović
9446eefd94 fix(ui): make sure adding labels on flow run dialog is possible (#7652) 2025-03-03 11:29:35 +01:00
YannC
334410ab58 fix(): make subflow not found a warning instead of an error (#7649) 2025-03-03 10:38:28 +01:00
nKwiatkowski
7ccb97a963 Revert "feat(#7636): add default options for HttpClient"
This reverts commit 5b42d0adba.
2025-03-03 10:09:08 +01:00
nKwiatkowski
3957177455 Merge remote-tracking branch 'origin/develop' into develop 2025-03-03 10:05:13 +01:00
nKwiatkowski
5b42d0adba feat(#7636): add default options for HttpClient 2025-03-03 10:05:01 +01:00
Loïc Mathieu
8bd3c2fef8 fix(ci): fix vulnerability check workflow 2025-03-03 09:41:13 +01:00
Miloš Paunović
d8fc4a9ce2 fix(ui): allow sidebar theme toggle to update the editor theme (#7648) 2025-03-03 09:15:24 +01:00
Florian Hussonnois
bfd82e0b5c chore(core): eval value property once for flowable task Switch 2025-02-28 18:38:31 +01:00
Loïc Mathieu
fa07cbd3b9 feat(core): improve performance of ExecutorService.handleChildWorkerTaskResult
Searching for a retry in all parents is a costly operation, doing it only wgen we are retrying or failing avoid it most of the time.
2025-02-28 17:40:29 +01:00
github-actions[bot]
2c77a43935 chore(translations): localize to languages other than English (#7635)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-28 16:09:24 +01:00
Piyush Bhaskar
60a84a49f8 chore(ui): improve empty state of the namespace files editor (#7495) 2025-02-28 15:54:39 +01:00
Piyush Bhaskar
d8295ef4ae feat(ui): allow reordering tabs in the editor (#7531) 2025-02-28 15:46:30 +01:00
Miloš Paunović
b02325a2e4 chore(ui): remove crud details from execution overview (#7634) 2025-02-28 15:39:43 +01:00
Florian Hussonnois
a098847c65 feat(core): enhance plugin management
Changes:
* add new interface PluginManager
* add new CLI for un-installing plugins
* add new option --locally to CLI plugin install
* refactor service for downloading plugins
* refactor PluginController
* move Version util class from EE to OSS
* migrate aether lib to maven-resolver (#915)

part-of: #915
2025-02-28 15:04:40 +01:00
brian.mulier
5f21eb5790 fix(ui): use watch with ref instead of accessing the value 2025-02-28 14:16:05 +01:00
MilosPaunovic
036a7cf4f7 fix(ui): improve check for text label of filters section 2025-02-28 14:00:32 +01:00
MilosPaunovic
446a034d6b fix(ui): additional check for text label of filters section 2025-02-28 12:48:38 +01:00
github-actions[bot]
3fff36881a chore(translations): localize to languages other than English (#7633)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-28 12:23:36 +01:00
Anna Geller
174ac280d2 Update README.md 2025-02-28 11:12:41 +01:00
Miloš Paunović
c610ccdaee chore(ui): improve label for text search in filters section (#7631)
* chore(ui): improve label for text search in filters section

* chore(ci): prevent running translation spec on ci:cd workflow
2025-02-28 11:07:19 +01:00
Hashim Khalifa
d8ed5b8b88 feat:string filter (#7428)
* feat:string filter

* chore: include string filter in docs
2025-02-28 10:38:32 +01:00
brian.mulier
17e54134ce fix(core): no longer lowercasing PluginClassIdentifier to have proper validation upon Plugin deserialization
closes #7387
2025-02-28 10:01:57 +01:00
Anna Geller
cacbd069f9 Update README.md 2025-02-28 01:07:18 +01:00
Anna Geller
de5a0f4623 Update README.md 2025-02-28 01:05:24 +01:00
Piyush Bhaskar
0f6c8d3521 fix(ui): executions naviation based on start Date. (#7626) 2025-02-27 21:21:21 +01:00
Anna Geller
c8207b8706 fix: confusing trace log 2025-02-27 20:33:26 +01:00
brian.mulier
39cd6189ac chore(deps): bump ui-libs to 0.0.151 2025-02-27 18:47:43 +01:00
brian.mulier
9b4f3148fc fix(ui): replace alert blocks upon markdown rendering to display them properly
closes #7393
2025-02-27 16:46:23 +01:00
Shruti Mantri
ca6a7a9e16 Add full examples for dayweek, dayweekinmonth and publicholidays conditions (#7615)
* Add full examples for dayweek, dayweekinmonth and publicholidays conditions

* Update core/src/main/java/io/kestra/plugin/core/condition/DayWeekInMonth.java

---------

Co-authored-by: AJ Emerich <aemerich@kestra.io>
2025-02-27 16:39:41 +01:00
Bart Ledoux
3dc8e98ed3 fix: make sure nocode edits the right task from the topology
closes #7619
2025-02-27 15:58:17 +01:00
github-actions[bot]
ea79be9de8 chore(translations): localize to languages other than English (#7618)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-27 15:17:13 +01:00
Anna Geller
3804bdc7f7 fix: improve translation prompt 2025-02-27 15:14:02 +01:00
Anna Geller
d04764814a fix: turn CI flag to a dropdown 2025-02-27 14:41:31 +01:00
Anna Geller
020d674d8c fix(translations): allow retranslating modified keys when needed 2025-02-27 14:39:17 +01:00
Miloš Paunović
ec7458dce5 feat(ui): make filter dropdown be positioned below the input caret (#7614) 2025-02-27 14:24:53 +01:00
Anna Geller
ae75ea06d2 fix: improve some wording 2025-02-27 14:19:30 +01:00
github-actions[bot]
53cd056871 chore(translations): localize to languages other than English (#7613)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-27 14:14:52 +01:00
Piyush Bhaskar
9bab2d2cfa feat(ui): introduce the execution timeline section on overview tab (#7498)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-27 14:09:48 +01:00
Loïc Mathieu
4d87d95eac chore(deps): upgrade Micronaut core to 4.7.15 2025-02-27 14:09:22 +01:00
Anna Geller
81635ddc84 fix: clean up translations script 2025-02-27 13:11:46 +01:00
github-actions[bot]
781d1338e9 chore(translations): localize to languages other than English (#7610)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-27 12:47:51 +01:00
Anna Geller
37ba85a130 Revert "chore(translations): localize to languages other than English (#7605)" (#7609)
This reverts commit 04b8a0a14c.
2025-02-27 12:41:42 +01:00
Anna Geller
7c7ac5a70d fix: only re-translate if the key is not already in the target dict (#7608)
* fix: only re-translate if the key is not already in the target dict or is empty

* Update generate_translations.py

* Update generate_translations.py
2025-02-27 12:40:46 +01:00
Shruti Mantri
f315485fc6 feat: add full examples for datetimebetween condition (#7598) 2025-02-27 11:30:59 +00:00
github-actions[bot]
04b8a0a14c chore(translations): localize to languages other than English (#7605)
Co-authored-by: GitHub Action <actions@github.com>
2025-02-27 11:53:09 +01:00
Anna Geller
8dc702ed16 fix(translations): fix translation key detection by comparing against last modifying commit (#7604) 2025-02-27 11:47:24 +01:00
Anna Geller
6dee52da16 fix: ci for translations (#7602)
* fix: ci for translations

* clean uo
2025-02-27 10:39:36 +01:00
Anna Geller
31dce9aadd Update generate-translations.yml 2025-02-27 10:23:30 +01:00
Anna Geller
cf1d98f56d chore(translations): standalone action for translations (#7597) 2025-02-27 10:22:01 +01:00
Harshit Dhaduk
3bbe65653b feat(ui): add ability to have persistent filter options (#7276)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-27 09:31:45 +01:00
Piyush Bhaskar
5108dd5e25 feat(ui): improve inspecting details of multiple executions (#7516) 2025-02-27 08:33:02 +01:00
Tanvir Ahmed
1ea331b8ab chore: introduce Devcontainer setup (#7507)
Co-authored-by: brian.mulier <bmmulier@hotmail.fr>
2025-02-26 19:26:55 +01:00
aeSouid
e9ceda4666 Fix issue text filter after label 2025-02-26 19:18:16 +01:00
brian.mulier
a996347de2 fix(ui): LabelInput.vue was causing UI freeze 2025-02-26 16:38:31 +01:00
Nicolas K.
d8e3a9ab8b Feat/npe on runcontext cleanup (#7585)
* fix(#7548): NPE on runContext cleanup if logger is null

* clean(#7548): unused import

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-26 14:06:41 +01:00
Miloš Paunović
e097bdbe53 chore(deps): regular dependency update (#7588) 2025-02-26 13:41:58 +01:00
Miloš Paunović
0090caad3f fix(ui): make sure adding labels on flow run dialog is possible (#7587) 2025-02-26 12:43:56 +01:00
Loïc Mathieu
7fbe433221 fix(core): properly render list properties 2025-02-26 12:03:55 +01:00
Miloš Paunović
b489848ab9 chore(ui): remove background and check mark from selected filters dropdown list (#7583) 2025-02-26 11:23:05 +01:00
Loïc Mathieu
fc7ef1ca38 fix(core): log can have no executionId
Fixes https://github.com/kestra-io/kestra-ee/issues/2980
2025-02-26 11:03:43 +01:00
Miloš Paunović
0dfe54baa8 fix(ui): amend inconsistencies inside the product tour (#7582) 2025-02-26 11:02:40 +01:00
Miloš Paunović
45c5592609 fix(ui): make flow deletion work as expected (#7579) 2025-02-26 09:04:18 +01:00
Miloš Paunović
3134d2abe2 chore(ui): respect default execution tab settings field when opening single execution (#7576) 2025-02-26 08:35:07 +01:00
aeSouid
2bb5ac61c2 Labels multiselect: (#7544)
combine the labels filter by operation to handle multiselect when building the query
2025-02-25 16:50:38 +01:00
Malaydewangan09
be04c168fd refactor: update http client and fix tests 2025-02-25 16:31:27 +01:00
Malaydewangan09
04c4916ac6 chore: add wiremock and update tests 2025-02-25 16:31:27 +01:00
Malaydewangan09
8d2af87db3 fix: refactor PluginSearchCommand 2025-02-25 16:31:27 +01:00
Malaydewangan09
e7950279bb feat: add plugin search command 2025-02-25 16:31:27 +01:00
Đỗ Trọng Hải
44936e9cdf fix(ui): improve coloring for task object tooltips in no code editor (#7515) 2025-02-25 15:38:57 +01:00
yuri
27755a3791 fix(ui): preselect filter comparator option (#7518) 2025-02-25 15:33:21 +01:00
brian.mulier
ff4f7abb0f fix(ui): styling enhancements for plugin doc
closes kestra-io/ui-libs#196
2025-02-25 15:21:24 +01:00
brian.mulier
48b117b351 fix(ui): plugins TOC is now handling every type of plugins 2025-02-25 15:21:24 +01:00
yuri
daca5675d0 fix(ui): amend label filter encoding after values change (#7536) 2025-02-25 14:05:12 +01:00
Miloš Paunović
b3278bf208 feat(ui): introduce topology export to image files (#7541) 2025-02-25 13:42:13 +01:00
Loïc Mathieu
8a26fdd83e chore(core): move run context property validation to the run context
This would avoid loading the Validator from the application context each time we render a property
2025-02-25 11:09:46 +01:00
Loïc Mathieu
38b8190be9 fix(core): Subflow using the old task name never ends
Fixes #7506
2025-02-25 11:06:31 +01:00
Miloš Paunović
52cb63c436 chore(ui): prevent system labels to be shown in set labels dialog (#7539)
* chore(ui): prevent system labels to be shown in set labels dialog

* chore(ui): improve the removal of labels
2025-02-25 10:42:07 +01:00
dependabot[bot]
f3cff1b8cd build(deps): bump io.micronaut.platform:micronaut-platform
Bumps [io.micronaut.platform:micronaut-platform](https://github.com/micronaut-projects/micronaut-platform) from 4.7.5 to 4.7.6.
- [Release notes](https://github.com/micronaut-projects/micronaut-platform/releases)
- [Commits](https://github.com/micronaut-projects/micronaut-platform/compare/v4.7.5...v4.7.6)

---
updated-dependencies:
- dependency-name: io.micronaut.platform:micronaut-platform
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-25 10:14:04 +01:00
Loïc Mathieu
a2c89e508c fix(core): handle space in HTTP request URI
Fixes #6936
2025-02-25 09:03:58 +01:00
aeSouid
c8aae742a2 fix(ui): amend filtering of logs (#7535) 2025-02-25 08:33:11 +01:00
aeSouid
eab7b746e5 Fix the search by labels input as single value 2025-02-24 19:31:14 +01:00
nKwiatkowski
1d5981665f fix(core): #172 add reactor into classloader blacklist 2025-02-24 18:41:15 +01:00
Ludovic DEHON
4500c976d6 fix(core): content type encoding should not be mandatory 2025-02-24 18:33:25 +01:00
Loïc Mathieu
cf10269f25 fix(jdbc): be resilient to DataException
We usually fail fast, but when a DataException is thrown it means the JDBC driver throws an exception with error code 22: data exception.
As the exception is from the data not the database or the network, there is no point of failfast, we throw a QueueException that may or may not be handled gracefully by the call site.
2025-02-24 16:56:10 +01:00
MilosPaunovic
cf635058cd chore(ui): use system namespace label from configs 2025-02-24 14:45:28 +01:00
aeSouid
bd6937a9f0 Fix the multi select status for the IAM/invitaiton page (#7528) 2025-02-24 13:36:34 +01:00
Miloš Paunović
59cc6f32d0 refactor(ui): remove obsolete chartjs-chart-treemap library (#7529) 2025-02-24 13:33:48 +01:00
Miloš Paunović
b8423ef70e chore(ui): purge empty labels on execution (#7527) 2025-02-24 12:24:50 +01:00
Miloš Paunović
0ec1b3a983 chore(ui): prevent sending random strings as child filter values (#7526)
* chore(ui): properly handle child filter

* chore(ui): prevent sending random strings as child filter values
2025-02-24 11:19:16 +01:00
Anna Geller
8cebd39f0d docs: debug expression (#7514)
* docs: debug expression

* fix: translation script

* fix: translations commit depth

* fix: comment
2025-02-24 10:22:21 +01:00
Ludovic DEHON
9fce6cfe65 fix(core): missing content type on http client 2025-02-21 23:18:27 +01:00
brian.mulier
c8c0c4e63f fix(core): camel to snake-case for app-blocks in RegisteredPlugin 2025-02-21 19:44:06 +01:00
Loïc Mathieu
25723b1acb fix(core): typo in PluginScanner 2025-02-21 19:40:16 +01:00
rajatsingh23
065d299422 chore(ui): restore automatic scroll to bottom on logs (#7365) 2025-02-21 15:08:13 +01:00
brian.mulier
9ec4d9282b fix(core): rename dashboards subgroups 2025-02-21 14:48:03 +01:00
brian.mulier
16284e5b95 fix(core): move package-info.java to proper dashboard packages 2025-02-21 14:35:36 +01:00
Ludovic DEHON
a115eb5373 fix(core): try to log message for unhandled realtime trigger exception 2025-02-21 12:50:21 +01:00
brian.mulier
ac2643c108 fix(core): add package-info.java to script + handle subgroups properly 2025-02-21 12:23:24 +01:00
Adam Hirshson
4710baec7d chore(ui): improve the scope of translations (#7505)
Co-authored-by: Yailan Bordas <bordasyailan@gmail.com>
Co-authored-by: Yailan Bordas <109761932+YAI-cs@users.noreply.github.com>
Co-authored-by: Christian Arauz <carau022@fiu.edu>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-21 09:20:52 +01:00
Miloš Paunović
1efe531228 chore(ui): add translation key/value pairs (#7509) 2025-02-21 09:19:09 +01:00
brian.mulier
df92491e55 fix(core): add package-info.java to dashboard package 2025-02-20 19:05:58 +01:00
aeSouid
e7c65e020a chore(ui): improve scope type filters (#7504) 2025-02-20 14:14:08 +01:00
Miloš Paunović
90212ee7dc fix(ui): properly filter out log levels (#7503) 2025-02-20 12:18:58 +01:00
Miloš Paunović
ac1984b349 fix(ui): make flow metrics behave as expected (#7502) 2025-02-20 11:59:27 +01:00
YannC
c5767fd313 fix(core): require condition in Flow trigger (#7494)
close #971
2025-02-20 10:19:06 +01:00
Ludovic DEHON
8ee79264ed fix(tasks): remove useless format metrics on return (#7486) 2025-02-19 22:56:29 +01:00
Mathieu Gabelle
c6d21776ec fix: render before command with options in CommandsWrapper (#7496) 2025-02-19 17:02:31 +01:00
Aabhas Sao
45a9ea0cc5 chore(ui): simplify query search with spaces inside (#7404)
Signed-off-by: Aabhas Sao <aabhassao0@gmail.com>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-19 16:26:47 +01:00
YannC
1f60a86739 fix(core): if subflow is disabled, raise an error (#7490) 2025-02-19 14:17:28 +01:00
Piyush Bhaskar
d0e8b691e3 chore(ui): amend margins of EE locked pages (#7446) 2025-02-19 14:11:45 +01:00
Piyush Bhaskar
0687a97eb8 feat(ui): add keyboard shortcuts for changing editor font size (#7450)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-19 13:54:20 +01:00
Miloš Paunović
dcf89b5db8 chore(translations): add missing key/value pairs (#7492) 2025-02-19 13:53:46 +01:00
Piyush Bhaskar
b1e29fbe31 chore(ui): improve styling of editor file tree (#7420) 2025-02-19 13:40:38 +01:00
Miloš Paunović
4b2113fb9b chore(ui): remove obsolete props (#7487) 2025-02-19 13:37:19 +01:00
Piyush Bhaskar
dc1728e420 fix(ui): fix the light theme contrast for editor. (#7438)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-19 13:31:47 +01:00
Miloš Paunović
8c83290d7a chore(deps): regular dependency update (#7485) 2025-02-19 12:54:49 +01:00
Miloš Paunović
212b3a82a9 fix(ui): improve handling of main filter labels on page load (#7456) 2025-02-19 12:08:53 +01:00
Miloš Paunović
1612c65ea3 fix(ui): properly cast value to boolean (#7455) 2025-02-19 11:29:07 +01:00
Miloš Paunović
6257b1508c chore(ui): disable click on search button if filtering is automatic (#7454) 2025-02-19 11:08:57 +01:00
yuri
f1670b8ac8 fix(webserver): allow special chars in label key (#7419) 2025-02-19 09:06:48 +01:00
brian.mulier
ce15ca1cac fix(ci): QEMU issue 2025-02-18 23:36:53 +01:00
Ludovic DEHON
a53395ab3f fix(core): http proxy was not passed to configuration
close #7449
2025-02-18 22:46:15 +01:00
Mathieu Gabelle
e0eaf675b2 refactor: return only command when no interpreter and no beforeCommands (#7452) 2025-02-18 18:59:04 +01:00
Ludovic DEHON
666f8a7ad9 fix(cicd): update concurrency key 2025-02-18 18:21:25 +01:00
brian.mulier
cebe8f3545 fix(tests): increase timeout on JdbcServiceLivenessCoordinatorTest.taskResubmitSkipExecution 2025-02-18 15:48:36 +01:00
brian.mulier
16e3830c9c fix(core): render delete property at the beginning in Docker task runner 2025-02-18 15:48:36 +01:00
brian.mulier
15b85ac952 fix(core): remove props with default from required in json schema to avoid validation errors
closes #7406
2025-02-18 15:48:36 +01:00
Miloš Paunović
db016a085f chore(ui): add arrows to namespace listing (#7448) 2025-02-18 13:54:07 +01:00
YannC
d5f6da10dd fix(): invert condition 2025-02-18 12:38:54 +01:00
brian.mulier
3723275817 fix(core): move back to the old worker thread pool because it was restricting it to 1 thread 2025-02-18 12:24:21 +01:00
YannC
ef29883138 fix(core): provide tenantId when looking for subflow (#7442) 2025-02-18 11:38:25 +01:00
Mathieu Gabelle
5ced9b6c4f refactor: remove rendering from Docker (#7439)
rendering is done in the CommandsWrapper so no need to render again in the task runner
2025-02-18 11:15:14 +01:00
Miloš Paunović
11f3629f38 chore(ui): add missing translations (#7444) 2025-02-18 11:11:50 +01:00
Miloš Paunović
d8c5a1e14c chore(ui): amend namespace flow creation label (#7443) 2025-02-18 11:09:55 +01:00
Miloš Paunović
cee2734817 fix(ui): improve modifying inputs from no code editor (#7440) 2025-02-18 10:36:15 +01:00
brian.mulier
13cb0fb96d fix(tests): logs are asynchronously inserted so we wait for them to be fully in 2025-02-18 10:19:58 +01:00
YannC
a020e3f3ae ci(test): force test if ref is a tag 2025-02-18 09:46:28 +01:00
brian.mulier
12fef24279 chore(deps): bump ui-libs to 0.0.141 2025-02-18 01:38:22 +01:00
rajatsingh23
4a55485cd6 fix(ui): better duration consistency on Gantt chart 2025-02-18 00:38:54 +01:00
brian.mulier
aed5734e17 chore(deps): bump ui-libs to 0.0.140 2025-02-18 00:25:39 +01:00
brian.mulier
d55ce16f57 fix(tests): wider maxDuration for retry-failed-flow-duration.yml 2025-02-18 00:05:14 +01:00
aeSouid
4abf3e9e6b Filter feat: (#7432)
Fix handle namespace column in filtering on scope
2025-02-17 14:43:03 +01:00
Bart Ledoux
fbd8934346 fix: restore red dot when there is news 2025-02-17 14:17:13 +01:00
Bart Ledoux
df6d339310 refactor: avoid en.json warning when building 2025-02-17 14:16:47 +01:00
YannC
a89ef71580 ci(publish-docker): attempts with command on qemu docker image 2025-02-17 11:15:14 +01:00
YannC
89c2d3951c ci(publish-docker): attempts with ubuntu 0.20 (#7431) 2025-02-17 10:48:10 +01:00
Mathieu Gabelle
e061a3617c refactor: introduce render in commands wrapper for property string (#7430) 2025-02-17 10:44:54 +01:00
Mathieu Gabelle
410cf0e389 fix: enable rendering of commands properties inside CommandsWrapper (#7381)
* fix: move commands to Property

migrate to Property in TaskCommands and CommandsWrapper
implement beforeCommand and interpreter
2025-02-17 09:49:49 +01:00
YannC
f783ab72b5 fix(test): attempt at making the test not flakky (#7400) 2025-02-17 09:20:26 +01:00
Anna Geller
72e78c8c31 Update ExecutionOutputs.java (#7394) 2025-02-17 09:20:10 +01:00
YannC
a716094a9d ci(): now run the file changes detection also in the workflow test 2025-02-17 09:15:26 +01:00
YannC
c75def0eac ci():allow release to be run 2025-02-17 09:10:08 +01:00
Piyush Bhaskar
c27d5ce480 chore(ui): improved Logs empty page (#7415) 2025-02-16 22:44:00 +01:00
aeSouid
49960d67ab chore(ui): improve filter parameters decoding for absolute date (#7409) 2025-02-14 14:51:13 +01:00
Loïc Mathieu
d9d2f86971 fix(core): ForEachItem inside an If task
Fixes #6801
2025-02-14 14:46:09 +01:00
Laibrez
95f5862201 chore(ui): improve execution outputs section (#7377)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-14 14:45:20 +01:00
AJ Emerich
e6419cf7a2 fix(ui): correct english translations (#7401)
Closes https://github.com/kestra-io/kestra-ee/issues/2913
2025-02-14 14:19:38 +01:00
Miloš Paunović
c1faeab068 chore(ui): make import flows button be a secondary one, styling-wise (#7405) 2025-02-14 12:59:44 +01:00
Aabhas Sao
dd3ff3e771 chore(ui): vertically center all elements in table rows (#7372)
Signed-off-by: Aabhas Sao <aabhassao0@gmail.com>
2025-02-14 11:52:37 +01:00
Miloš Paunović
e635338b47 chore(ui): improve colors of filter dropdown selector (#7403) 2025-02-14 11:47:10 +01:00
Loïc Mathieu
ddfed2e65c fix(core): taskrun list can be null 2025-02-14 11:45:33 +01:00
aeSouid
52fe6afae0 Filter feat: (#7389)
namespace mulitple choice in and not in
2025-02-14 11:37:57 +01:00
Miloš Paunović
a9ff2af159 fix(ui): properly handle the operation labels in filter component (#7399) 2025-02-14 10:50:16 +01:00
Miloš Paunović
bf19a8f5ad chore(ui): improve the labels behavior (#7397) 2025-02-14 10:30:46 +01:00
Loïc Mathieu
41712b8d80 feat(jdbc): consume multiple times the execution and worker task result queues
So that the executor use multiple threads to process messages.
2025-02-14 09:19:51 +01:00
Loïc Mathieu
f8a6e3fb04 feat(core, jdbc): use an elastic thread pool
To avoid using too many threads when it's not needed.
2025-02-14 09:19:39 +01:00
Ludovic DEHON
ad651cdc53 fix(ui): fix slack button on error toast 2025-02-13 19:27:20 +01:00
Miloš Paunović
f57a463983 chore(ui): improve breadcrumbs on namespace view (#7386) 2025-02-13 15:46:04 +01:00
Miloš Paunović
b3e66d8844 chore(ui): include autocompletion shortcut in the preview list (#7384) 2025-02-13 15:03:42 +01:00
Miloš Paunović
dc64d81dd0 chore(ui): add link to filtered triggers page from backfill dialog (#7380) 2025-02-13 12:17:53 +01:00
YannC
be30034143 ci(): moved build artifacts directly in docker workflow 2025-02-13 11:20:16 +01:00
YannC
f8d8e0998c ci(): fix outputs 2025-02-13 10:47:58 +01:00
Bart Ledoux
b8d0ae3ecf fix: changing language should work with providers too 2025-02-13 10:32:20 +01:00
Loïc Mathieu
def8fa3ff1 fix(cli): disable by default OTEL metrics 2025-02-13 10:21:18 +01:00
YannC
cfc0c9f9f2 ci(docker): revert fixed qemu version 2025-02-13 10:06:14 +01:00
YannC
49fe36250a ci(docker): fixed version of qemu 2025-02-13 10:00:32 +01:00
YannC
0c19f4c518 ci(): map outputs in build artifacts 2025-02-13 09:40:55 +01:00
YannC
1e6d553ce6 ci(): correct output in workflow build artifacts 2025-02-13 09:08:00 +01:00
YannC
1ebbdbbae6 fix(scheduler): delete trigger when flow is not found (#7366)
close #7312
2025-02-13 08:52:19 +01:00
Ludovic DEHON
5b29a0d070 fix(cicd): add mariadb plugins on docker image 2025-02-12 23:48:50 +01:00
Ludovic DEHON
239fb6a685 fix(core): handle http request with no content type 2025-02-12 19:48:50 +01:00
YannC
9924ba5ae2 fix(ui): refresh dashboard list (#7370)
close #2791
2025-02-12 17:44:24 +01:00
Loïc Mathieu
13ac335c93 feat(jdbc): clean more eagerly some queues based on configuration 2025-02-12 16:35:46 +01:00
Loïc Mathieu
9d717caf95 feat(jdbc): queues.updated should be set when the record is updated 2025-02-12 16:35:46 +01:00
Loïc Mathieu
1d65fd96b4 fix(jdbc): delete the executor state at the correct stage
Without this change, the executor state is deleted when the execution is terminated.
Then the last execution message is emitted, which goes again into the queue wich recreates the executor state.
2025-02-12 16:11:32 +01:00
Loïc Mathieu
b45a44bd35 fix(core): render list
Fixes #7253
2025-02-12 16:09:16 +01:00
YannC
bd9ba882fb test(webserver): flaky ExecutionControllerTest for kill (#7368) 2025-02-12 15:59:39 +01:00
Miloš Paunović
4b682931e0 chore(ui): add validation messages for custom dashboard crud actions (#7367) 2025-02-12 14:38:10 +01:00
YannC
d657f48278 ci: cleanup 2025-02-12 14:28:49 +01:00
Eduardo Goncalvez
714938bec6 chore(ui): add shadows to main charts (#7314)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-12 11:38:59 +01:00
Barthélémy Ledoux
73a9eb485f fix: make menu hierarchy get closer to the original designs (#7102)
* fix: make menu hierarchy get closer to the original designs

* smaller spacing in lower hierachy

* chore: add sidebar story
2025-02-12 10:52:14 +01:00
Miloš Paunović
4d9f2a130f fix(ui): make sure bulk selection is taking into account only selected items (#7362) 2025-02-12 10:47:41 +01:00
Barthélémy Ledoux
a040ec44b9 fix: trim bookmarks better (#7359) 2025-02-12 10:19:22 +01:00
Barthélémy Ledoux
b1c27b1869 fix: add proper ellipsis to sidemenu (#7361) 2025-02-12 10:19:06 +01:00
Loïc Mathieu
ed14493632 feat(core): add system.correlationId label to triggered executions 2025-02-12 10:12:56 +01:00
Miloš Paunović
3803f4ea4c chore(deps): regular dependency update (#7358) 2025-02-12 09:59:20 +01:00
Miloš Paunović
4992cf34fa chore(translations): amend theme key (#7356) 2025-02-12 09:23:03 +01:00
YannC
420afd7758 ci():fix frontend being skipped (#7355) 2025-02-12 09:21:59 +01:00
Shamar
c3852a2933 chore(ui): improve the doughnut chart legend (#7321)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-12 08:59:43 +01:00
YannC
41417adf8b ci(): fixed vulnerabilities-check.yml wrong character 2025-02-12 08:57:16 +01:00
Pravesh-Sudha
cd89cd8fb6 chore(ui): introduce filters bar to flow triggers page (#7292)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-12 08:45:47 +01:00
YannC
9b5e16a02d ci(): call reusable workflow correctly 2025-02-12 08:42:09 +01:00
Ludovic DEHON
f4b78755ad docs(core): better documentation on docker tasks 2025-02-11 22:12:43 +01:00
Ludovic DEHON
7bf42cb1cb fix(cicd): npm install in the wrong folder for vulnerabilities checks 2025-02-11 22:12:38 +01:00
Loïc Mathieu
7dbf86d54e fix(core): possible NPE when an execution has no labels 2025-02-11 18:01:27 +01:00
Loïc Mathieu
e7f551edc2 feat(jdbc): purge worker job queue early 2025-02-11 16:30:40 +01:00
Loïc Mathieu
31d2212419 feat(jdbc): puerge worker task result queue early 2025-02-11 16:30:40 +01:00
Loïc Mathieu
dd8ae5e642 feat(jdbc): purge execution queue early 2025-02-11 16:30:40 +01:00
Nicolas K.
0aadbab9c4 fix(core): add request.yaml file back (#7308)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-11 16:15:25 +01:00
YannC
49759da3b4 ci(): Remove if from test workflow 2025-02-11 15:48:20 +01:00
YannC
489e01cf38 ci(): Use correct workflow 2025-02-11 15:46:39 +01:00
YannC
1821ec27c9 ci(): Correct outputs 2025-02-11 15:44:01 +01:00
YannC
a6918b1f34 ci(): Remove unnecessary secrets 2025-02-11 15:37:49 +01:00
YannC
17693c7380 ci(): Missing GOOGLE_SERVICE_ACCOUNT secrets 2025-02-11 15:27:33 +01:00
YannC
806135e3e9 ci(): Remove GITHUB_TOKEN from secrets definition 2025-02-11 15:25:27 +01:00
YannC
7381b32e16 ci(): rewrite release part 2025-02-11 15:22:33 +01:00
YannC
7b175dac04 ci(): trigger main ci on dispatch 2025-02-11 14:19:48 +01:00
YannC
2c3de87dcc ci(): trigger main ci also on develop branch 2025-02-11 14:19:19 +01:00
YannC
7b9efa08a6 fix(h2): remove indenting in sql file (#7306) 2025-02-11 13:52:38 +01:00
Piyush Bhaskar
1a96fe4b4b chore(ui): improve styling of main filter tags (#7305)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-11 13:51:24 +01:00
YannC
938de9b222 fix(core): do not validate subflow if namespace or id is pebble (#7294)
close #7271
2025-02-11 13:44:30 +01:00
Florian Hussonnois
9505c48e55 chore: remove some compilation warnings 2025-02-11 13:04:38 +01:00
YannC
fe197ebd21 ci(): run translations only if necessary 2025-02-11 12:52:40 +01:00
YannC
c5b6d6b2a7 ci(): moved multiple actions to reusable workflows (#7299) 2025-02-11 11:47:56 +01:00
dependabot[bot]
90da5f7cf7 build(deps): bump org.opensearch.client:opensearch-java
Bumps [org.opensearch.client:opensearch-java](https://github.com/opensearch-project/opensearch-java) from 2.19.0 to 2.20.0.
- [Release notes](https://github.com/opensearch-project/opensearch-java/releases)
- [Changelog](https://github.com/opensearch-project/opensearch-java/blob/v2.20.0/CHANGELOG.md)
- [Commits](https://github.com/opensearch-project/opensearch-java/compare/v2.19.0...v2.20.0)

---
updated-dependencies:
- dependency-name: org.opensearch.client:opensearch-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-11 11:45:27 +01:00
dependabot[bot]
4c5638d957 build(deps): bump io.micronaut.platform:micronaut-platform
Bumps [io.micronaut.platform:micronaut-platform](https://github.com/micronaut-projects/micronaut-platform) from 4.7.4 to 4.7.5.
- [Release notes](https://github.com/micronaut-projects/micronaut-platform/releases)
- [Commits](https://github.com/micronaut-projects/micronaut-platform/compare/v4.7.4...v4.7.5)

---
updated-dependencies:
- dependency-name: io.micronaut.platform:micronaut-platform
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-11 11:45:09 +01:00
dependabot[bot]
8260dc6033 build(deps): bump io.pebbletemplates:pebble from 3.2.2 to 3.2.3
Bumps [io.pebbletemplates:pebble](https://github.com/PebbleTemplates/pebble) from 3.2.2 to 3.2.3.
- [Release notes](https://github.com/PebbleTemplates/pebble/releases)
- [Commits](https://github.com/PebbleTemplates/pebble/compare/3.2.2...3.2.3)

---
updated-dependencies:
- dependency-name: io.pebbletemplates:pebble
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-11 11:44:52 +01:00
dependabot[bot]
48320a89d8 build(deps): bump com.microsoft.playwright:playwright
Bumps [com.microsoft.playwright:playwright](https://github.com/microsoft/playwright-java) from 1.49.0 to 1.50.0.
- [Release notes](https://github.com/microsoft/playwright-java/releases)
- [Commits](https://github.com/microsoft/playwright-java/compare/v1.49.0...v1.50.0)

---
updated-dependencies:
- dependency-name: com.microsoft.playwright:playwright
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-11 11:44:16 +01:00
dependabot[bot]
55bdfac7f9 build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.33.11 to 0.34.0.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.33.11...v0.34.0)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-11 11:43:25 +01:00
Loïc Mathieu
f599f25755 chore(core): refactor file related Pebble function to share code 2025-02-11 11:39:20 +01:00
Loïc Mathieu
8de839f326 chore(core): rename fileEmpty Pebble function to isFileEmpty 2025-02-11 11:39:20 +01:00
Ludovic DEHON
25370d10bd fix(core): http client was not using deprecated setter 2025-02-11 11:21:26 +01:00
Nicolas K.
b59d36a90e fix(core): flacky trigger with backfile test (#7295)
* fix(core): flacky trigger with backfile test

* fix(core): flacky schedule test

---------

Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-11 11:07:38 +01:00
Nicolas K.
a5483a0d4c fix(makefile): build plugin now build main branch too (#7297)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-11 11:02:28 +01:00
ben8t
9863e88230 Add core plugin sanity checks (#6904)
* [feature] add core plugin sanity checks

* feat: namespace files task test

* fix: formatting

* fix: sanity checks to use assert

* fix: core sanity flows

* tests: add runner class to run sanity checks flows

* [refactor] remove request sanity check as not a core task (already in fs plugin)

* [refactor] rename file and namespace for sanity checks

* [refactor] update path

---------

Co-authored-by: Will Russell <wrussell@kestra.io>
Co-authored-by: Will Russell <will@wrussell.co.uk>
Co-authored-by: Mathieu Gabelle <mgabelle@kestra.io>
2025-02-11 10:51:59 +01:00
ByronBlaze
88a5cd69e3 feat(core): Added FileEmpty & FileExists Pebble Functions 2025-02-11 10:49:31 +01:00
yuri
ec73d58649 fix(ui): match chart colors (#7290)
Co-authored-by: Barthélémy Ledoux <ledouxb@me.com>
2025-02-11 09:38:33 +01:00
Bart Ledoux
0becf74339 fix: collapsed menu colors in light mode 2025-02-11 09:33:54 +01:00
Miloš Paunović
bfcfb2ac16 chore(ui): make action columns always visible on executions and flows (#7291) 2025-02-11 08:17:29 +01:00
Anna Geller
27f9a505af fix: force run docs (#7289) 2025-02-10 20:31:24 +01:00
Nicolas K.
99db3f35b7 fix(core): #7227 cron schedule with timezone and backfile not triggering (#7285)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-10 17:56:05 +01:00
YannC
c53239470d fix(ci): inputs instead of secrets 2025-02-10 16:27:55 +01:00
YannC
c3e830c2c7 fix(ci): Workflow test ouptputs + changes for codecov 2025-02-10 16:25:00 +01:00
Ludovic DEHON
9790f02376 fix(cicd): add npm install on vulnerabilities check 2025-02-10 15:56:53 +01:00
Miloš Paunović
899a1f6f17 chore(ui): improve flow description coloring (#7282) 2025-02-10 15:51:59 +01:00
YannC
9455643660 fix(ci): pass correctly secret + cleanup 2025-02-10 15:17:31 +01:00
dependabot[bot]
8a3f413236 build(deps): bump software.amazon.awssdk:bom from 2.30.11 to 2.30.16
Bumps software.amazon.awssdk:bom from 2.30.11 to 2.30.16.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-10 15:01:51 +01:00
YannC
4986d68f57 fix(ci): removed useless name in action files 2025-02-10 14:37:17 +01:00
YannC
09ff9d4054 fix(ci): missing checkout in release step 2025-02-10 13:13:48 +01:00
aeSouid
e910685d57 Fix search bar: (#7279)
-fix date picker layout
-change the not_in operation traduction

Fix search bar:
-fix search bar when closing date picker
2025-02-10 13:12:06 +01:00
dependabot[bot]
5aa73ec48a build(deps): bump com.google.cloud:libraries-bom from 26.53.0 to 26.54.0
Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.53.0 to 26.54.0.
- [Release notes](https://github.com/googleapis/java-cloud-bom/releases)
- [Changelog](https://github.com/googleapis/java-cloud-bom/blob/main/release-please-config.json)
- [Commits](https://github.com/googleapis/java-cloud-bom/compare/v26.53.0...v26.54.0)

---
updated-dependencies:
- dependency-name: com.google.cloud:libraries-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-10 12:49:01 +01:00
YannC
5dcd27c064 fix(ci): disabled previous generate translations CI 2025-02-10 12:36:00 +01:00
YannC
9ba27b9bd5 fix(ci): workflow test correct previous job status usage 2025-02-10 12:35:06 +01:00
YannC
549da52afa ci(): simplification (#7202)
* ci():
- split in multiples actions
- simplify translation script
- simplify front build
- run test for back/front only when needed

* fix(): token prop

* ui(translation): new word

* fix(): correct usage of paths-filter action

* fix(): first iteration of new main workflow

* chore(translations): auto generate values for languages other than english

---------

Co-authored-by: GitHub Action <actions@github.com>
2025-02-10 12:24:25 +01:00
Miloš Paunović
d486d78200 chore(ui): disable saving flow actions if there are errors (#7278) 2025-02-10 11:59:51 +01:00
aeSouid
a7a134a050 Fix search flow by query: (#7277)
add new condition of the field is query
2025-02-10 11:54:39 +01:00
GitHub Action
e0f4ab735d chore(translations): auto generate values for languages other than english 2025-02-10 10:49:34 +00:00
Barthélémy Ledoux
b052925b04 feat: give blueprint pages a distinct name (#7254)
* feat(translations): add new blueprint labels for flows, apps, and dashboard

* feat(translations): update left menu titles for blueprints

* chore(translations): auto generate values for languages other than english

* chore(translations): auto generate values for languages other than english

---------

Co-authored-by: GitHub Action <actions@github.com>
2025-02-10 11:48:24 +01:00
Piyush Bhaskar
c3443ab99a feat(ui): center view when switching between topology modes (#7257) 2025-02-10 11:11:39 +01:00
Bart Ledoux
ec967a57fb fix: use the udpated labelsFromQuery in labels 2025-02-10 10:56:35 +01:00
Piyush Bhaskar
9ba13ae994 chore(ui): tweak the video container ratio in the docs sidebar (#7260) 2025-02-10 10:35:54 +01:00
Loïc Mathieu
be1871430e feat(core, jdbc): directly process WorkerTaskResult from flowable tasks
And other places in the ExecutorService when we create them when processing the execution queue.

Fixes #5521
2025-02-10 10:03:00 +01:00
Dheeraj_R_Gowda
39257bf3bb chore(ui): align chart duration label with switch toggle (#7258) 2025-02-10 09:35:10 +01:00
Aabhas Sao
346100c86c chore(ui): align chart duration label with switch toggle (#7259) 2025-02-10 09:24:36 +01:00
Ludovic DEHON
0c3ed3b751 feat(cicd): restore codeql weekly 2025-02-08 00:48:11 +01:00
Ludovic DEHON
1bbe0e659e feat(cicd): don't mark failed for front end test 2025-02-07 20:28:31 +01:00
Ludovic DEHON
123f748030 feat(cicd): add test analysis on frontend 2025-02-07 18:30:27 +01:00
Ludovic DEHON
d0b53d217d feat(cicd): add unit test for js (#7246) 2025-02-07 18:23:08 +01:00
Bart Ledoux
6f7bb80c6d fix: make dashboard tables the right color 2025-02-07 16:39:29 +01:00
Loïc Mathieu
1eacb447d5 feat(core): simplify Pebble error messages 2025-02-07 14:59:10 +01:00
Ludovic DEHON
3f429ef0a8 feat(cicd): add codecov flags 2025-02-07 14:27:59 +01:00
Florian Hussonnois
dc8576afed fix(ci): fix and remove unecessary setps in set version workflows 2025-02-07 11:51:07 +01:00
Bart Ledoux
c31609a12e chore: update theme colors 2025-02-07 11:44:25 +01:00
Loïc Mathieu
84c07ef011 fix(webserver,core): move the LogStreamService in core so EE can use it for apps 2025-02-07 11:40:21 +01:00
MilosPaunovic
e335b76b3c chore(ui): check for missing property on update metadata 2025-02-07 11:27:14 +01:00
Bart Ledoux
95b94f3961 fix: labels should not be purple if inactive 2025-02-07 10:46:33 +01:00
Bart Ledoux
e0ee26a9c0 fix: add comment on i18n code 2025-02-07 10:07:15 +01:00
Florian Hussonnois
480fc75897 feat(ci): add workflows for release process
* move scripts to folder dev-tools
* add new workflow gradle-release.yml
* add new workflow setversion-tag.yml
* rename existing workflow
2025-02-07 09:57:06 +01:00
Nicolas K.
c8d7e5f357 feat(core): add sanity check to request task (#7230)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-07 09:34:09 +01:00
咬轮猫
792f7b6911 fix(ui): amend the language switching issue (#7235) 2025-02-07 09:15:52 +01:00
Loïc Mathieu
2a95aee968 feat(webserver): use a shared queue consumer from the log follow endpoint 2025-02-07 09:15:42 +01:00
amartknez
129ba6abe4 chore(ui): initial work on filter persistency (#7234)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-07 09:09:06 +01:00
Ludovic DEHON
ab2a0108a9 feat(cicd): add codecov bundle analysis 2025-02-07 09:08:37 +01:00
Ludovic DEHON
03dccd1444 feat(cicd): codecov + tests report 2025-02-07 08:45:26 +01:00
Loïc Mathieu
4c401ce0c0 feat(webserver): improvement to ExecutionStreaminService 2025-02-06 17:30:14 +01:00
malay.worldref@gmail.com
13be8b8125 feat(webserver): optimize queue usage for follow endpoints 2025-02-06 17:30:14 +01:00
GitHub Action
440faa7cd1 chore(translations): auto generate values for languages other than english 2025-02-06 16:10:14 +00:00
aeSouid
5a8a4e99d7 Change the filter params (#7051)
* Feat Filter:
Change the filters to the new search format
2025-02-06 17:09:32 +01:00
aeSouid
296e1a67a5 Feat Search: (#6910)
New search with operation for all /search endpoints :
- define list  of operation and fields.
- Add custom parser for filter[field][operation]=value format
- create new Annotation @QueryFilterFormat to bind to specfic pojo
- Implement new abstract repo search method
2025-02-06 17:08:15 +01:00
Loïc Mathieu
ecd36ec2a4 fix(core): possible NPE on LabelService.containsAll 2025-02-06 16:26:11 +01:00
rajatsingh23
ca9b90a0ec chore(ui): show executions per namespace only if there are 2 or more items for chart (#7216)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-06 15:22:23 +01:00
Piyush Bhaskar
37e337a657 chore(ui): only show warning on bulk execution deletion if nonTerminated is true (#7211)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-06 15:12:16 +01:00
Miloš Paunović
157f78fae0 chore(ui): generate random flow ID using combination of animal names and numbers (#7223)
* chore(ui): creating a flow from the namespace view should use it's ID as the designated value

* chore(ui): generate random flow ID using combination of animal names and numbers
2025-02-06 14:55:50 +01:00
Pravesh-Sudha
7158a0c98f chore(ui): amended global pagination coloring (#7201)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-06 13:11:00 +01:00
Miloš Paunović
753308a341 fix(ui): prevent doubling the executions chart on flow overview (#7219) 2025-02-06 11:58:05 +01:00
Loïc Mathieu
21aebe4e77 feat(core): allow loading secrets from a different namspace
Fixes #5970
2025-02-06 10:42:17 +01:00
Miloš Paunović
959e80efa8 chore(ui): re-order the list of optional columns (#7213) 2025-02-06 09:32:56 +01:00
Miloš Paunović
bba612ef9b feat(ui): add script to help with creation of release notes (#7212) 2025-02-06 09:18:08 +01:00
nKwiatkowski
a0483dc201 fix(core): request option doesn't initialize properly 2025-02-05 19:03:06 +01:00
Miloš Paunović
2f3d7f573b chore(ui): replace the visual for no tabs opened on namespace editor (#7204) 2025-02-05 18:37:13 +01:00
Nicolas K.
2e09777790 fix(core): #7181 log level rendered as string (#7198)
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
2025-02-05 15:35:29 +01:00
Miloš Paunović
0b1eda0f03 chore(ui): consolidate markdown files (#7197) 2025-02-05 12:51:31 +01:00
Miloš Paunović
1ee6402fa2 chore(translations): remove extra keys from translation files (#7193) 2025-02-05 12:23:34 +01:00
Miloš Paunović
71933352b6 chore(ui): remove the option to change editor theme separately (#7192)
* chore(ui): remove obsolete log statements from console

* chore(ui): remove the option to change editor theme separately
2025-02-05 12:18:29 +01:00
GitHub Action
cb7ed73bea chore(translations): auto generate values for languages other than english 2025-02-05 09:56:50 +00:00
Miloš Paunović
5f13d5b2aa chore(ui): rename advanced properties to other in no code (#7190) 2025-02-05 10:56:11 +01:00
GitHub Action
e4973c3319 chore(translations): auto generate values for languages other than english 2025-02-05 09:53:47 +00:00
Miloš Paunović
abc3372161 chore(ui): rename advanced properties to other in no code (#7189) 2025-02-05 10:53:14 +01:00
Bart Ledoux
9619dca764 fix(theme): make plugin rendering reactive to theme switch 2025-02-05 10:53:02 +01:00
Barthélémy Ledoux
e80b2011de feat: theme switch to "theme switch" the charts (#7151)
* chore: store the theme in the store

* use the new theme in charts

* use the theme value in more places

* create a useTheme composable

* create the useScheme composable

* restore nodata
2025-02-05 10:43:04 +01:00
GitHub Action
86f3acace3 chore(translations): auto generate values for languages other than english 2025-02-05 09:33:27 +00:00
Piyush Bhaskar
b6c9f50503 feat(ui): add option to choose visible columns in flow and execution listings (#6932)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-05 10:32:37 +01:00
Piyush Bhaskar
2e9938806a feat(ui): Docs markdown alert styled based on alert level in product. (#6818)
* feat(ui): Align the CSS or style configuration so that all documentation components (Docs, plugin docs, blueprints) use the same markdown style.

* Remove alert styling from DocsLayout since it has been handled within Alert.vue using ---ks variables.

* revert Input_Count.

---------

Co-authored-by: Barthélémy Ledoux <ledouxb@me.com>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-05 10:13:30 +01:00
Piyush Bhaskar
13bc0de9dc chore(ui): improve the states options list inside filter values (#7176)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-05 09:03:25 +01:00
Piyush Bhaskar
da7509ce30 chore(ui): update the visual of no data component (#7179)
Co-authored-by: MilosPaunovic <paun992@hotmail.com>
2025-02-05 08:45:52 +01:00
Shruti Mantri
15aeee3d8b chore(ui): improve the example for not condition (#6820) 2025-02-05 08:35:21 +01:00
dependabot[bot]
42920508b7 build(deps-dev): bump @shikijs/markdown-it from 2.3.0 to 2.3.1 in /ui (#7184)
Bumps [@shikijs/markdown-it](https://github.com/shikijs/shiki/tree/HEAD/packages/markdown-it) from 2.3.0 to 2.3.1.
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v2.3.1/packages/markdown-it)

---
updated-dependencies:
- dependency-name: "@shikijs/markdown-it"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-05 08:32:26 +01:00
dependabot[bot]
f45c703fd9 build(deps): bump shiki from 2.3.0 to 2.3.1 in /ui (#7183)
Bumps [shiki](https://github.com/shikijs/shiki/tree/HEAD/packages/shiki) from 2.3.0 to 2.3.1.
- [Release notes](https://github.com/shikijs/shiki/releases)
- [Commits](https://github.com/shikijs/shiki/commits/v2.3.1/packages/shiki)

---
updated-dependencies:
- dependency-name: shiki
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-05 08:31:04 +01:00
dependabot[bot]
872c70fe37 build(deps): bump posthog-js from 1.215.2 to 1.215.3 in /ui (#7182)
Bumps [posthog-js](https://github.com/PostHog/posthog-js) from 1.215.2 to 1.215.3.
- [Release notes](https://github.com/PostHog/posthog-js/releases)
- [Changelog](https://github.com/PostHog/posthog-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/PostHog/posthog-js/compare/v1.215.2...v1.215.3)

---
updated-dependencies:
- dependency-name: posthog-js
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-05 08:30:53 +01:00
brian.mulier
bbb03b288b fix(ui): global plugin doc with new redesign + auto-expand properties initially 2025-02-04 21:58:08 +01:00
Florian Hussonnois
8d8b7e7a67 fix(core): make flow/namespace variables available for input expr
related-to: kestra-io/kestra-ee#2826
2025-02-04 16:55:49 +01:00
GitHub Action
a9042fb27b chore(translations): auto generate values for languages other than english 2025-02-04 15:00:55 +00:00
YannC
5ff5667231 feat(ui): now display an error when SSE failed (#7177)
* feat(ui): now display an error when SSE failed

close #1303

* chore(translations): auto generate values for languages other than english

---------

Co-authored-by: GitHub Action <actions@github.com>
2025-02-04 16:00:27 +01:00
Florian Hussonnois
42ad09d79f chore: add utility script to check for plugin artifacts 2025-02-04 15:58:12 +01:00
Loïc Mathieu
30db740de8 fix(core): retry flaky test AbstractRunnerTest.restartForEachItem
It is flaky on CI but never fail locally even with 100 repetitions
2025-02-04 14:22:20 +01:00
Loïc Mathieu
20d55122e7 fix(core): retry test HttpClientTest.getText
It is flaky in CI but never fail locally even with 100 repetitions
2025-02-04 14:22:20 +01:00
Miloš Paunović
3294c358fa fix(ui): align dashboard button label to icon (#7175) 2025-02-04 14:14:58 +01:00
Karthik73965
42899d30d1 chore(ui): add skeleton loaders to dashboard cards (#6602)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-04 13:56:22 +01:00
GitHub Action
6ba71712a8 chore(translations): auto generate values for languages other than english 2025-02-04 12:04:11 +00:00
Miloš Paunović
36b8ef866d fix(ui): amend translation string for no results (#7172) 2025-02-04 13:03:45 +01:00
GitHub Action
fff8d630c5 chore(translations): auto generate values for languages other than english 2025-02-04 11:41:26 +00:00
Aabhas Sao
bc630684b1 chore(ui): enable command palette for monaco editor (#6944)
Signed-off-by: Aabhas Sao <aabhassao0@gmail.com>
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-04 12:40:59 +01:00
Loïc Mathieu
f8b64cfbdf fix(core): retry flaky test AbstractRunnereTest.concurrencyQueuePause
It is flaky on CI but cannot be fail locally even with 100 repetitions.
2025-02-04 12:34:43 +01:00
Miloš Paunović
4ee8b33aeb chore(deps): regular dependency update (#7171) 2025-02-04 12:27:22 +01:00
Piyush Bhaskar
68074723c4 chore(ui): amend color of the input length counter (#6990) 2025-02-04 12:09:32 +01:00
GitHub Action
52159ee647 chore(translations): auto generate values for languages other than english 2025-02-04 11:02:44 +00:00
Piyush Bhaskar
d697e5dc67 feat(ui): add keyboard shortcuts dialog to editor (#6628)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-04 12:01:04 +01:00
brian.mulier
21733d8497 fix(ui): null-safe search filters 2025-02-04 11:40:44 +01:00
rajatsingh23
0501d29255 chore(ui): show each plugin deprecation warning in new line (#6839)
Co-authored-by: Miloš Paunović <paun992@hotmail.com>
2025-02-04 11:22:19 +01:00
Loïc Mathieu
560703fde4 fix(core): retry flaky test AbstractRunnerTest.restartFailedThenFailureWithGlobalErrors
This test is flaky on CI but never locally even with 100 repetitions.
2025-02-04 10:38:42 +01:00
Loïc Mathieu
671eb2b57e feat(jdbc-*): delete subflow_executions table
Fixes #6831
2025-02-04 10:34:39 +01:00
Miloš Paunović
5cc30558c2 fix(ui): amend pagination on namespace flows listing (#7163) 2025-02-04 10:18:04 +01:00
Loïc Mathieu
5a46204394 chore(core): small perf improvements to MapUtils 2025-02-04 10:06:58 +01:00
Ludovic DEHON
c918276103 fix(core): process runner are not serialized correctly on worker
close #7053
2025-02-03 21:27:04 +01:00
YannC
04dd038b0f feat(core): new Publish task for metrics (#7122)
* feat(core): new Publish task for metrics

* fix(): review fixes
2025-02-03 16:37:16 +01:00
Bart Ledoux
1bbe48fe5b fix: bring back hover in main menu 2025-02-03 15:40:12 +01:00
Bart Ledoux
9499ee2c27 fix: setup docId for blueprints 2025-02-03 15:15:34 +01:00
Piyush Bhaskar
05bbaa4f00 fix(ui): Custom Dashboard name overflows. (#7124)
* fix(ui): Custom Dashboard name overflows.

* fix(ui): avoid dashboard button being too long

---------

Co-authored-by: YannC <ycoornaert@kestra.io>
2025-02-03 14:37:02 +01:00
brian.mulier
fa5ef3b4c6 fix(ui): switching from custom Flow blueprints tab to dashboard was not working 2025-02-03 14:00:13 +01:00
Miloš Paunović
e927a1f784 fix(ui): get the string fields in no code to use editor and have auto completion back (#7150) 2025-02-03 12:19:31 +01:00
dependabot[bot]
36306495e4 build(deps): bump org.owasp.dependencycheck from 12.0.1 to 12.0.2
Bumps org.owasp.dependencycheck from 12.0.1 to 12.0.2.

---
updated-dependencies:
- dependency-name: org.owasp.dependencycheck
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 12:18:02 +01:00
MilosPaunovic
594d005168 chore(ui): properly pass a prop related to saved searches 2025-02-03 11:52:40 +01:00
Miloš Paunović
c992496fb0 feat(ui): multiple improvements of no code editor (#7146)
* refactor(ui): prevent multiple warning in console by adding inheritAttrs properly

* chore(ui): make plugin selector field not clearable

* feat(ui): allow re-ordering of array items

* fix(ui): remove concurrency when limit set to 0
2025-02-03 11:35:03 +01:00
Bart Ledoux
e15eb55877 fix: sidemenu bring back the gray hover 2025-02-03 11:33:20 +01:00
Bart Ledoux
85ebf49b68 fix: enterprise edition tag in light mode 2025-02-03 11:30:48 +01:00
dependabot[bot]
3a0beabe4f build(deps): bump com.gradleup.shadow from 8.3.5 to 8.3.6
Bumps [com.gradleup.shadow](https://github.com/GradleUp/shadow) from 8.3.5 to 8.3.6.
- [Release notes](https://github.com/GradleUp/shadow/releases)
- [Commits](https://github.com/GradleUp/shadow/compare/8.3.5...8.3.6)

---
updated-dependencies:
- dependency-name: com.gradleup.shadow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 11:28:29 +01:00
dependabot[bot]
d769b7aad7 build(deps): bump flyingSaucerVersion from 9.11.2 to 9.11.3
Bumps `flyingSaucerVersion` from 9.11.2 to 9.11.3.

Updates `org.xhtmlrenderer:flying-saucer-core` from 9.11.2 to 9.11.3
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.11.2...v9.11.3)

Updates `org.xhtmlrenderer:flying-saucer-pdf` from 9.11.2 to 9.11.3
- [Release notes](https://github.com/flyingsaucerproject/flyingsaucer/releases)
- [Changelog](https://github.com/flyingsaucerproject/flyingsaucer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/flyingsaucerproject/flyingsaucer/compare/v9.11.2...v9.11.3)

---
updated-dependencies:
- dependency-name: org.xhtmlrenderer:flying-saucer-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.xhtmlrenderer:flying-saucer-pdf
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 11:28:22 +01:00
dependabot[bot]
f1f0d31864 build(deps): bump software.amazon.awssdk.crt:aws-crt
Bumps [software.amazon.awssdk.crt:aws-crt](https://github.com/awslabs/aws-crt-java) from 0.33.9 to 0.33.11.
- [Release notes](https://github.com/awslabs/aws-crt-java/releases)
- [Commits](https://github.com/awslabs/aws-crt-java/compare/v0.33.9...v0.33.11)

---
updated-dependencies:
- dependency-name: software.amazon.awssdk.crt:aws-crt
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 11:26:57 +01:00
dependabot[bot]
bd0e4a5f48 build(deps): bump com.azure:azure-sdk-bom from 1.2.30 to 1.2.31
Bumps [com.azure:azure-sdk-bom](https://github.com/azure/azure-sdk-for-java) from 1.2.30 to 1.2.31.
- [Release notes](https://github.com/azure/azure-sdk-for-java/releases)
- [Commits](https://github.com/azure/azure-sdk-for-java/compare/azure-sdk-bom_1.2.30...azure-sdk-bom_1.2.31)

---
updated-dependencies:
- dependency-name: com.azure:azure-sdk-bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 11:11:33 +01:00
dependabot[bot]
d50f631fb6 build(deps): bump software.amazon.awssdk:bom from 2.30.6 to 2.30.11
Bumps software.amazon.awssdk:bom from 2.30.6 to 2.30.11.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-02-03 11:09:32 +01:00
yuri
26c15a444d fix(ui): amend log lines in Firefox (#7133) 2025-02-03 10:18:19 +01:00
Bart Ledoux
97d12f9bc3 chore: add a test for firefox weird wrap 2025-02-03 10:15:53 +01:00
yuri1969
ae05de4a1c fix(cli): print help on missing parameters
Missing required positional parameters caused just a stacktrace print.

Now such cases print the error message acompanied by command's help.

Examples:

* `./kestra flow namespace update`
* `./kestra flow dot`
2025-02-03 10:06:48 +01:00
yuri1969
625135959c chore(core): fix various compilation warnings
Fixed clear Java compilation warnings.
2025-02-03 09:36:50 +01:00
Loïc Mathieu
e6827f2736 fix(core): remove the dynamic property patterns 2025-02-03 09:31:32 +01:00
Anna Geller
91bf50e837 Update dashboard_home.md 2025-02-02 14:11:14 +01:00
Piyush Bhaskar
64c8e0295c feat(ui): Add a link to the filtered Executions table. (#7129) 2025-02-01 23:01:02 +01:00
brian-mulier-p
1565f3361a fix(ui): restore namespace filter manual typing & various improvements (#7127) 2025-02-01 10:08:14 +01:00
Florian Hussonnois
e2b67f253a ci: fix release workflows 2025-01-31 19:34:18 +01:00
brian.mulier
db0422b142 chore(deps): bump ui-libs to v0.0.125 2025-01-31 18:13:48 +01:00
Nicolas K.
b1b80acbca feat(build): add script to makefile (#7125)
---------
Co-authored-by: nKwiatkowski <nkwiatkowski@kestra.io>
Co-authored-by: Ludovic DEHON <tchiot.ludo@gmail.com>
2025-01-31 16:34:11 +01:00
Barthélémy Ledoux
4f9b7f7417 fix: make table links primary instead of purple (#7106) 2025-01-31 16:23:05 +01:00
Barthélémy Ledoux
83c7b2e663 fix: use the proper variable for select header in table (#7107) 2025-01-31 16:22:43 +01:00
GitHub Action
dee76c038a chore(translations): auto generate values for languages other than english 2025-01-31 12:51:00 +00:00
Miloš Paunović
310210286d feat(ui): add finally block to no code editor (#7123) 2025-01-31 13:49:56 +01:00
Florian Hussonnois
555b769aa2 fix(ci): update scripts/workflows for plugins 2025-01-31 12:05:05 +01:00
YannC
e527f0133a fix(ui): Remove useless double click row action
close #7000
2025-01-31 11:57:18 +01:00
Bart Ledoux
631a016e06 fix: avoid clearing selected value on every error
closes #7115
2025-01-31 11:48:49 +01:00
Bart Ledoux
3a8b24edae fix: remove Labels tag type warning 2025-01-31 11:19:15 +01:00
Bart Ledoux
65dd4f9fd6 fix: remove FE warnings on flow edition 2025-01-31 11:17:34 +01:00
Bart Ledoux
a4dcb8dcd6 fix: protect axios JSON parsing 2025-01-31 11:08:33 +01:00
Miloš Paunović
00aa28fc41 feat(ui): allow task re-ordering from no code editor (#7120) 2025-01-31 10:55:05 +01:00
Bart Ledoux
3c7e26c88b build: prevent corepack crash 2025-01-31 10:02:27 +01:00
Loïc Mathieu
4e753c7b98 fix(core): retry flaky test TimeoutTest.timeout()
As its failure cannot be reproduced locally even with 100 repetitions, there is no other choice than retrying it.
2025-01-31 09:47:00 +01:00
Bart Ledoux
1fc0b21071 fix: make sure normal single line table dont' push the build #7018 2025-01-30 22:23:25 +01:00
Bart Ledoux
f17b5802b5 fix: plugin header icon for ticket #4252 2025-01-30 22:22:04 +01:00
Barthélémy Ledoux
5389ec78a1 fix: use the new charts in the flows page (#6970) 2025-01-30 21:24:23 +01:00
Loïc Mathieu
6190d91134 fix(core): subflow labels must not be overriden by parent flow ones 2025-01-30 17:18:28 +01:00
Loïc Mathieu
96f4466f1d fix(cli): retry flaky test FlowCreateOrUpdateCommandTest.runWithDelete.
Adding a retry because it cannot be reproduce to fail locally even with 100 repetitions.
2025-01-30 17:14:01 +01:00
Ludovic DEHON
6ee0e86ca3 fix(core): http request with head and 404 and sending the wrong exception
close #4658
2025-01-30 17:04:51 +01:00
Loïc Mathieu
db4f186bdc fix(core): retry flaky test AbstractRunnerTest.multipleConditionTriggerFailed()
As the failure cannot be reproduce locally even with 100 repetitions of the test.
2025-01-30 16:43:50 +01:00
Loïc Mathieu
9b2c4c9a13 fix(core): subflow validation didn't work anymore 2025-01-30 16:17:30 +01:00
Bart Ledoux
7a37a950bf chore: update palette 2025-01-30 15:58:01 +01:00
Barthélémy Ledoux
a7df59c8bf feat: show a lock on EE only pages (#7093) 2025-01-30 15:54:23 +01:00
Miloš Paunović
5b6fbe9609 feat(ui): improve the task array component (#7095)
* feat(ui): improve the task array component

* chore(ui): replace existing task on editing during creation instead of re-adding them
2025-01-30 14:25:30 +01:00
Florian Hussonnois
d80de4410d fix(ui): fix missing param kind for blueprint in flow editor (#7087)
fix: #7087
2025-01-30 14:16:19 +01:00
Bart Ledoux
0e09f6821c fix: transfer utm parameters correctly 2025-01-30 12:48:53 +01:00
Bart Ledoux
157566300d fix: unlock audit logs in execution 2025-01-30 12:43:54 +01:00
AJ Emerich
e1a4f2e2fc fix(docs): remove custom dashboard website component
https://github.com/kestra-io/kestra/issues/7085
2025-01-30 12:13:55 +01:00
Miloš Paunović
cd822bd349 chore(ui): properly check the existence of fields inside schema
* chore(ui): remove unnecessary binding of listeners

* chore(ui): check the existence of fields
2025-01-30 11:41:15 +01:00
Ludovic DEHON
84447fd428 feat(ui): display attempts status on taskrun on left
close #7030
2025-01-30 11:39:22 +01:00
Miloš Paunović
78060ec2f8 chore(ui): move apps link in left menu just below the flows (#7063) 2025-01-30 11:34:10 +01:00
Miloš Paunović
646c8fa773 fix(ui): amend no code editor breadcrumbs issue (#7054)
* chore(ui): task array component to have margins between lines

* fix(ui): amend no code editor breadcrumbs issue
2025-01-30 11:33:57 +01:00
Bart Ledoux
7a299e51f8 fix: remove some warnings 2025-01-30 11:10:44 +01:00
Bart Ledoux
accbafe132 fix: remove the topbar from namepasce/flows 2025-01-30 11:10:32 +01:00
Bart Ledoux
4915465573 fix(demo): make button clickable with utm link 2025-01-30 11:09:39 +01:00
Bart Ledoux
4234c76b03 build: try and fix FE CI 2025-01-30 10:56:32 +01:00
Loïc Mathieu
03caf38251 fix(cli): repeate flaky tests FileChangedEventListenerTest
This is inherently racy as it's async and watch the filesystem which cannot be done reliabily.
2025-01-30 10:55:17 +01:00
Loïc Mathieu
1931300e98 chore(deps): downgrade Protobuf to 3.25.5
3.25.6 is not compatible with 3.25.5 and Orc still uses 3.25.5
2025-01-30 10:40:43 +01:00
Miloš Paunović
450310b6a5 feat(ui): multiple improvements of no code editor (#7076)
* fix(ui): allow creation of multiple tasks from the no code editor

* chore(ui): make input text be of textarea type for resizability

* chore(ui): allow to add task from topology either before or after the target one
2025-01-30 10:31:09 +01:00
Loïc Mathieu
2777b34385 fix(script): AbstractExecScript.injectDefaults should throw IllegalVariableEvaluationException 2025-01-30 09:57:51 +01:00
Florian Hussonnois
d4d8e326e4 ci: fix workflow docker for all plugins 2025-01-29 17:45:50 +01:00
Florian Hussonnois
fde739a3be ci: fix workflow docker 2025-01-29 17:35:24 +01:00
Florian Hussonnois
f96cbf1ad3 ci: update workflow docker 2025-01-29 17:28:31 +01:00
Florian Hussonnois
84a30d4002 ci: update workflow docker 2025-01-29 17:25:42 +01:00
Loïc Mathieu
6afdbb01fa fix(core): restartForEachItem() is flaky
With this test change, running 100 tests with MySQL pass!
2025-01-29 17:10:50 +01:00
Barthélémy Ledoux
b91336f446 fix: color of tags in executions list (#7035) 2025-01-29 16:57:47 +01:00
Bart Ledoux
f1b2940652 fix: remove default variables for box-shadow 2025-01-29 16:56:22 +01:00
Barthélémy Ledoux
75b8b4a483 fix: use proper css variables for table colors (#7049)
* fix: use proper css variables for table colors

* update ui-libs
2025-01-29 16:54:29 +01:00
Barthélémy Ledoux
c9cb2593bc chore: update "cluster" into "instance" in side menu (#6996)
* chore: update "cluster" into "instance" in side menu

* chore(translations): auto generate values for languages other than english

---------

Co-authored-by: GitHub Action <actions@github.com>
2025-01-29 16:51:52 +01:00
Barthélémy Ledoux
e23f34e53f fix(styles): remove background color from edit buttons (#7037) 2025-01-29 16:51:28 +01:00
Barthélémy Ledoux
2461e8cda9 feat: add a shadow to cards (#7038) 2025-01-29 16:50:08 +01:00
Bart Ledoux
b8fac95e1e fix: namespaces should have the card background
closes #6828
2025-01-29 16:48:48 +01:00
YannC
d17fe2548a fix(ui): dynamic format date
close #7015
2025-01-29 15:55:57 +01:00
YannC
b86b4bb160 feat(webserver): if no date provided for dashboard, then use default timewindow 2025-01-29 15:55:57 +01:00
Loïc Mathieu
64108ae24d chore(deps): downgrade protobug
Orc uses an older version.
And probably also other libs that we're using are still in 3.x
2025-01-29 15:51:48 +01:00
brian.mulier
8d00c8a557 feat(ui): don't show deprecated tasks in the plugins list
closes #4526
2025-01-29 15:42:22 +01:00
Loïc Mathieu
07dfaada9c fix(cli): flow watcher should compute plugin defaults
fixes #6908
2025-01-29 15:42:12 +01:00
Florian Hussonnois
0b82b25a48 chore(ci): update release workflows 2025-01-29 15:10:45 +01:00
Florian Hussonnois
5c8c2a5c42 chore(version): update snapshot version 'v0.22.0-SNAPSHOT'. 2025-01-29 14:27:00 +01:00
759 changed files with 44365 additions and 25730 deletions

67
.devcontainer/Dockerfile Normal file
View File

@@ -0,0 +1,67 @@
FROM ubuntu:24.04
ARG DEBIAN_FRONTEND=noninteractive
USER root
WORKDIR /root
RUN apt update && apt install -y \
apt-transport-https ca-certificates gnupg curl wget git zip unzip less zsh net-tools iputils-ping jq lsof
ENV HOME="/root"
# --------------------------------------
# Git
# --------------------------------------
# Need to add the devcontainer workspace folder as a safe directory to enable git
# version control system to be enabled in the containers file system.
RUN git config --global --add safe.directory "/workspaces/kestra"
# --------------------------------------
# --------------------------------------
# Oh my zsh
# --------------------------------------
RUN sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" -- \
-t robbyrussell \
-p git -p node -p npm
ENV SHELL=/bin/zsh
# --------------------------------------
# --------------------------------------
# Java
# --------------------------------------
RUN wget https://download.oracle.com/java/21/latest/jdk-21_linux-x64_bin.deb
RUN dpkg -i ./jdk-21_linux-x64_bin.deb
ENV JAVA_HOME=/usr/java/jdk-21-oracle-x64
ENV PATH="$PATH:$JAVA_HOME/bin"
# Will load a custom configuration file for Micronaut
ENV MICRONAUT_ENVIRONMENTS=local,override
# Sets the path where you save plugins as Jar and is loaded during the startup process
ENV KESTRA_PLUGINS_PATH="/workspaces/kestra/local/plugins"
# --------------------------------------
# --------------------------------------
# Node.js
# --------------------------------------
RUN curl -fsSL https://deb.nodesource.com/setup_22.x -o nodesource_setup.sh \
&& bash nodesource_setup.sh && apt install -y nodejs
# Increases JavaScript heap memory to 4GB to prevent heap out of error during startup
ENV NODE_OPTIONS=--max-old-space-size=4096
# --------------------------------------
# --------------------------------------
# Python
# --------------------------------------
RUN apt install -y python3 pip python3-venv
# --------------------------------------
# --------------------------------------
# SSH
# --------------------------------------
RUN mkdir -p ~/.ssh
RUN touch ~/.ssh/config
RUN echo "Host github.com" >> ~/.ssh/config \
&& echo " IdentityFile ~/.ssh/id_ed25519" >> ~/.ssh/config
RUN touch ~/.ssh/id_ed25519
# --------------------------------------

149
.devcontainer/README.md Normal file
View File

@@ -0,0 +1,149 @@
# Kestra Devcontainer
This devcontainer provides a quick and easy setup for anyone using VSCode to get up and running quickly with this project to start development on either the frontend or backend. It bootstraps a docker container for you to develop inside of without the need to manually setup the environment.
---
## INSTRUCTIONS
### Setup:
Take a look at this guide to get an idea of what the setup is like as this devcontainer setup follows this approach: https://kestra.io/docs/getting-started/contributing
Once you have this repo cloned to your local system, you will need to install the VSCode extension [Remote Development](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack).
Then run the following command from the command palette:
`Dev Containers: Open Folder in Container...` and select your Kestra root folder.
This will then put you inside a docker container ready for development.
NOTE: you'll need to wait for the gradle build to finish and compile Java files but this process should happen automatically within VSCode.
In the meantime, you can move onto the next step...
---
### Development:
- Create a `.env.development.local` file in the `ui` folder and paste the following:
```bash
# This lets the frontend know what the backend URL is but you are free to change this to your actual server URL e.g. hosted version of Kestra.
VITE_APP_API_URL=http://localhost:8080
```
- Navigate into the `ui` folder and run `npm install` to install the dependencies for the frontend project.
- Now go to the `cli/src/main/resources` folder and create a `application-override.yml` file.
Now you have two choices:
`Local mode`:
Runs the Kestra server in local mode which uses a H2 database, so this is the only config you'd need:
```yaml
micronaut:
server:
cors:
enabled: true
configurations:
all:
allowedOrigins:
- http://localhost:5173
```
You can then open a new terminal and run the following command to start the backend server: `./gradlew runLocal`
`Standalone mode`:
Runs in standalone mode which uses Postgres. Make sure to have a local Postgres instance already running on localhost:
```yaml
kestra:
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
anonymous-usage-report:
enabled: false
server:
basic-auth:
enabled: false
datasources:
postgres:
# It is important to note that you must use the "host.docker.internal" host when connecting to a docker container outside of your devcontainer as attempting to use localhost will only point back to this devcontainer.
url: jdbc:postgresql://host.docker.internal:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
flyway:
datasources:
postgres:
enabled: true
locations:
- classpath:migrations/postgres
# We must ignore missing migrations as we may delete the wrong ones or delete those that are not used anymore.
ignore-migration-patterns: "*:missing,*:future"
out-of-order: true
micronaut:
server:
cors:
enabled: true
configurations:
all:
allowedOrigins:
- http://localhost:5173
```
Then add the following settings to the `.vscode/launch.json` file:
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "java",
"name": "Kestra Standalone",
"request": "launch",
"mainClass": "io.kestra.cli.App",
"projectName": "cli",
"args": "server standalone"
}
]
}
```
You can then use the VSCode `Run and Debug` extension to start the Kestra server.
Additionally, if you're doing frontend development, you can run `npm run dev` from the `ui` folder after having the above running (which will provide a backend) to access your application from `localhost:5173`. This has the benefit to watch your changes and hot-reload upon doing frontend changes.
#### Plugins
If you want your plugins to be loaded inside your devcontainer, point the `source` field to a folder containing jars of the plugins you want to embed in the following snippet in `devcontainer.json`:
```
"mounts": [
{
"source": "/absolute/path/to/your/local/jar/plugins/folder",
"target": "/workspaces/kestra/local/plugins",
"type": "bind"
}
],
```
---
### GIT
If you want to commit to GitHub, make sure to navigate to the `~/.ssh` folder and either create a new SSH key or override the existing `id_ed25519` file and paste an existing SSH key from your local machine into this file. You will then need to change the permissions of the file by running: `chmod 600 id_ed25519`. This will allow you to then push to GitHub.
---

View File

@@ -0,0 +1,46 @@
{
"name": "kestra",
"build": {
"context": ".",
"dockerfile": "Dockerfile"
},
"workspaceFolder": "/workspaces/kestra",
"forwardPorts": [5173, 8080],
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.profiles.linux": {
"zsh": {
"path": "/bin/zsh"
}
},
"workbench.iconTheme": "vscode-icons",
"editor.tabSize": 4,
"editor.formatOnSave": true,
"files.insertFinalNewline": true,
"editor.defaultFormatter": "esbenp.prettier-vscode",
"telemetry.telemetryLevel": "off",
"editor.bracketPairColorization.enabled": true,
"editor.guides.bracketPairs": "active"
},
"extensions": [
"redhat.vscode-yaml",
"dbaeumer.vscode-eslint",
"vscode-icons-team.vscode-icons",
"eamodio.gitlens",
"esbenp.prettier-vscode",
"aaron-bond.better-comments",
"codeandstuff.package-json-upgrade",
"andys8.jest-snippets",
"oderwat.indent-rainbow",
"evondev.indent-rainbow-palettes",
"formulahendry.auto-rename-tag",
"IronGeek.vscode-env",
"yoavbls.pretty-ts-errors",
"github.vscode-github-actions",
"vscjava.vscode-java-pack",
"ms-azuretools.vscode-docker"
]
}
}
}

View File

@@ -1,117 +0,0 @@
name: Daily Core check
on:
schedule:
- cron: '0 5 * * *'
jobs:
check:
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
name: Check & Publish
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
caches-enabled: true
# Services
- name: Build the docker-compose stack
run: docker compose -f docker-compose-ci.yml up -d
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
# Gradle check
- name: Build with Gradle
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
env:
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
run: |
echo $GOOGLE_SERVICE_ACCOUNT | base64 -d > ~/.gcp-service-account.json
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp-service-account.json
./gradlew check javadoc --parallel --refresh-dependencies
# Sonar
- name: Analyze with Sonar
if: ${{ env.SONAR_TOKEN != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: ./gradlew sonar --info
# Allure check
- name: Auth to Google Cloud
id: auth
if: ${{ always() && env.GOOGLE_SERVICE_ACCOUNT != 0 }}
uses: 'google-github-actions/auth@v2'
with:
credentials_json: '${{ secrets.GOOGLE_SERVICE_ACCOUNT }}'
- uses: rlespinasse/github-slug-action@v5
- name: Publish allure report
uses: andrcuns/allure-publish-action@v2.9.0
if: ${{ always() && env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
env:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
JAVA_HOME: /usr/lib/jvm/default-jvm/
with:
storageType: gcs
resultsGlob: "**/build/allure-results"
bucket: internal-kestra-host
baseUrl: "https://internal.dev.kestra.io"
prefix: ${{ format('{0}/{1}', github.repository, 'allure/java') }}
copyLatest: true
ignoreMissingResults: true
# Jacoco
- name: 'Set up Cloud SDK'
if: ${{ github.ref == 'refs/heads/develop' && env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
uses: 'google-github-actions/setup-gcloud@v2'
- name: 'Copy jacoco files'
if: ${{ github.ref == 'refs/heads/develop' && env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
run: |
mv build/reports/jacoco/testCodeCoverageReport build/reports/jacoco/test/
mv build/reports/jacoco/test/testCodeCoverageReport.xml build/reports/jacoco/test/jacocoTestReport.xml
gsutil -m rsync -d -r build/reports/jacoco/test/ gs://internal-kestra-host/${{ format('{0}/{1}', github.repository, 'jacoco') }}
# report test
- name: Test Report
uses: mikepenz/action-junit-report@v5
if: success() || failure()
with:
report_paths: '**/build/test-results/**/TEST-*.xml'
# Codecov
- uses: codecov/codecov-action@v5
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
- name: Notify failed CI
id: send-ci-failed
if: always() && job.status != 'success'
uses: kestra-io/actions/.github/actions/send-ci-failed@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

View File

@@ -6,11 +6,11 @@
name: "CodeQL"
on:
push:
branches: [develop]
schedule:
- cron: '0 5 * * 1'
workflow_dispatch: {}
jobs:
analyze:
name: Analyze
@@ -51,13 +51,23 @@ jobs:
# Set up JDK
- name: Set up JDK
uses: actions/setup-java@v4
if: ${{ matrix.language == 'java' }}
with:
distribution: 'temurin'
java-version: 21
- name: Setup gradle
if: ${{ matrix.language == 'java' }}
uses: gradle/actions/setup-gradle@v4
- name: Build with Gradle
if: ${{ matrix.language == 'java' }}
run: ./gradlew testClasses -x :ui:installFrontend -x :ui:assembleFrontend
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
if: ${{ matrix.language != 'java' }}
uses: github/codeql-action/autobuild@v3
# Command-line programs to run using the OS shell.

View File

@@ -1,4 +1,4 @@
name: Create Docker images on tag
name: Create Docker images on Release
on:
workflow_dispatch:
@@ -11,6 +11,10 @@ on:
options:
- "true"
- "false"
release-tag:
description: 'Kestra Release Tag'
required: false
type: string
plugin-version:
description: 'Plugin version'
required: false
@@ -38,17 +42,16 @@ jobs:
name: Publish Docker
needs: [ plugins ]
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
strategy:
matrix:
image:
- name: "-no-plugins"
plugins: ""
packages: ""
packages: jattach
python-libs: ""
- name: ""
plugins: ${{needs.plugins.outputs.plugins}}
packages: python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip
packages: python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip jattach
python-libs: kestra
steps:
- uses: actions/checkout@v4
@@ -57,13 +60,22 @@ jobs:
- name: Set image name
id: vars
run: |
TAG=${GITHUB_REF#refs/*/}
echo "tag=${TAG}" >> $GITHUB_OUTPUT
echo "plugins=${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
if [[ "${{ inputs.release-tag }}" == "" ]]; then
TAG=${GITHUB_REF#refs/*/}
echo "tag=${TAG}" >> $GITHUB_OUTPUT
else
TAG="${{ inputs.release-tag }}"
echo "tag=${TAG}" >> $GITHUB_OUTPUT
fi
if [[ "${{ env.PLUGIN_VERSION }}" == *"-SNAPSHOT" ]]; then
echo "plugins=--repositories=https://s01.oss.sonatype.org/content/repositories/snapshots ${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT;
else
echo "plugins=${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
fi
# Download release
- name: Download release
uses: robinraju/release-downloader@v1.11
uses: robinraju/release-downloader@v1.12
with:
tag: ${{steps.vars.outputs.tag}}
fileName: 'kestra-*'
@@ -77,6 +89,11 @@ jobs:
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Docker - Fix Qemu
shell: bash
run: |
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

View File

@@ -42,19 +42,13 @@ jobs:
with:
path: kestra
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
- uses: kestra-io/actions/.github/actions/setup-build@main
id: build
with:
java-enabled: true
caches-enabled: true
node-enabled: true
python-enabled: true
# Get Docker Image
- name: Download Kestra Image

View File

@@ -0,0 +1,67 @@
name: Auto-Translate UI keys and create PR
on:
schedule:
- cron: "0 9-21 * * *" # Every hour from 9 AM to 9 PM
workflow_dispatch:
inputs:
retranslate_modified_keys:
description: "Whether to re-translate modified keys even if they already have translations."
type: choice
options:
- "false"
- "true"
default: "false"
required: false
jobs:
translations:
name: Translations
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
name: Checkout
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install Python dependencies
run: pip install gitpython openai
- name: Generate translations
run: python ui/src/translations/generate_translations.py ${{ github.event.inputs.retranslate_modified_keys }}
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: "20.x"
- name: Check keys matching
run: node ui/src/translations/check.js
- name: Set up Git
run: |
git config --global user.name "GitHub Action"
git config --global user.email "actions@github.com"
- name: Commit and create PR
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
BRANCH_NAME="chore/update-translations-$(date +%s)"
git checkout -b $BRANCH_NAME
git add ui/src/translations/*.json
if git diff --cached --quiet; then
echo "No changes to commit. Exiting with success."
exit 0
fi
git commit -m "chore(translations): localize to languages other than English"
git push -u origin $BRANCH_NAME || (git push origin --delete $BRANCH_NAME && git push -u origin $BRANCH_NAME)
gh pr create --title "Translations from en.json" --body "This PR was created automatically by a GitHub Action." --base develop --head $BRANCH_NAME --assignee anna-geller --reviewer anna-geller

View File

@@ -1,111 +0,0 @@
name: Generate Translations
on:
pull_request:
types: [opened, synchronize]
paths:
- "ui/src/translations/en.json"
push:
branches:
- develop
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
jobs:
commit:
name: Commit directly to PR
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == false }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 50
ref: ${{ github.head_ref }}
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install Python dependencies
run: pip install gitpython openai
- name: Generate translations
run: python ui/src/translations/generate_translations.py
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: "20.x"
- name: Check keys matching
run: node ui/src/translations/check.js
- name: Set up Git
run: |
git config --global user.name "GitHub Action"
git config --global user.email "actions@github.com"
- name: Check for changes and commit
env:
GH_TOKEN: ${{ github.token }}
run: |
git add ui/src/translations/*.json
if git diff --cached --quiet; then
echo "No changes to commit. Exiting with success."
exit 0
fi
git commit -m "chore(translations): auto generate values for languages other than english"
git push origin ${{ github.head_ref }}
pull_request:
name: Open PR for a forked repository
runs-on: ubuntu-latest
if: ${{ github.event.pull_request.head.repo.fork == true }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 10
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install Python dependencies
run: pip install gitpython openai
- name: Generate translations
run: python ui/src/translations/generate_translations.py
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: "20.x"
- name: Check keys matching
run: node ui/src/translations/check.js
- name: Set up Git
run: |
git config --global user.name "GitHub Action"
git config --global user.email "actions@github.com"
- name: Create and push a new branch
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
BRANCH_NAME="generated-translations-${{ github.event.pull_request.head.repo.name }}"
git checkout -b $BRANCH_NAME
git add ui/src/translations/*.json
if git diff --cached --quiet; then
echo "No changes to commit. Exiting with success."
exit 0
fi
git commit -m "chore(translations): auto generate values for languages other than english"
git push origin $BRANCH_NAME

View File

@@ -4,7 +4,7 @@ on:
workflow_dispatch:
inputs:
releaseVersion:
description: 'The release version (e.g., 0.21.0)'
description: 'The release version (e.g., 0.21.0-rc1)'
required: true
type: string
nextVersion:
@@ -18,29 +18,50 @@ on:
jobs:
release:
name: Release plugins
runs-on: kestra-private-standard
runs-on: ubuntu-latest
steps:
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
caches-enabled: true
# Get Plugins List
- name: Get Plugins List
uses: ./.github/actions/plugins-list
if: "!startsWith(github.ref, 'refs/tags/v')"
id: plugins-list
with:
plugin-version: 'LATEST'
- name: 'Configure Git'
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
# Execute
- name: Run Gradle Release
if: ${{ github.event.inputs.dryRun == 'false' }}
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./release-plugins.sh;
./release-plugins.sh \
chmod +x ./dev-tools/release-plugins.sh;
./dev-tools/release-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--next-version=${{github.event.inputs.nextVersion}} \
--yes \
@@ -51,8 +72,9 @@ jobs:
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./release-plugins.sh;
./release-plugins.sh \
chmod +x ./dev-tools/release-plugins.sh;
./dev-tools/release-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--next-version=${{github.event.inputs.nextVersion}} \
--dry-run \

89
.github/workflows/gradle-release.yml vendored Normal file
View File

@@ -0,0 +1,89 @@
name: Run Gradle Release
run-name: "Releasing Kestra ${{ github.event.inputs.releaseVersion }} 🚀"
on:
workflow_dispatch:
inputs:
releaseVersion:
description: 'The release version (e.g., 0.21.0-rc1)'
required: true
type: string
nextVersion:
description: 'The next version (e.g., 0.22.0-SNAPSHOT)'
required: true
type: string
env:
RELEASE_VERSION: "${{ github.event.inputs.releaseVersion }}"
NEXT_VERSION: "${{ github.event.inputs.nextVersion }}"
jobs:
release:
name: Release Kestra
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/develop'
steps:
# Checks
- name: Check Inputs
run: |
if ! [[ "$RELEASE_VERSION" =~ ^[0-9]+(\.[0-9]+)\.0-rc[01](-SNAPSHOT)?$ ]]; then
echo "Invalid release version. Must match regex: ^[0-9]+(\.[0-9]+)\.0-rc[01](-SNAPSHOT)?$"
exit 1
fi
if ! [[ "$NEXT_VERSION" =~ ^[0-9]+(\.[0-9]+)\.0-SNAPSHOT$ ]]; then
echo "Invalid next version. Must match regex: ^[0-9]+(\.[0-9]+)\.0-SNAPSHOT$"
exit 1;
fi
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
caches-enabled: true
- name: Configure Git
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
# Execute
- name: Run Gradle Release
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
# Extract the major and minor versions
BASE_VERSION=$(echo "$RELEASE_VERSION" | sed -E 's/^([0-9]+\.[0-9]+)\..*/\1/')
PUSH_RELEASE_BRANCH="releases/v${BASE_VERSION}.x"
# Create and push release branch
git checkout -b "$PUSH_RELEASE_BRANCH";
git push -u origin "$PUSH_RELEASE_BRANCH";
# Run gradle release
git checkout develop;
if [[ "$RELEASE_VERSION" == *"-SNAPSHOT" ]]; then
# -SNAPSHOT qualifier maybe used to test release-candidates
./gradlew release -Prelease.useAutomaticVersion=true \
-Prelease.releaseVersion="${RELEASE_VERSION}" \
-Prelease.newVersion="${NEXT_VERSION}" \
-Prelease.pushReleaseVersionBranch="${PUSH_RELEASE_BRANCH}" \
-Prelease.failOnSnapshotDependencies=false
else
./gradlew release -Prelease.useAutomaticVersion=true \
-Prelease.releaseVersion="${RELEASE_VERSION}" \
-Prelease.newVersion="${NEXT_VERSION}" \
-Prelease.pushReleaseVersionBranch="${PUSH_RELEASE_BRANCH}"
fi

View File

@@ -1,529 +1,58 @@
name: Main
name: Main Workflow
on:
workflow_dispatch:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: true
type: string
push:
branches:
- master
- main
- releases/*
- develop
tags:
- v*
pull_request:
branches:
- develop
repository_dispatch:
types: [rebuild]
workflow_dispatch:
inputs:
skip-test:
description: "Skip test"
type: choice
required: true
default: "false"
options:
- "true"
- "false"
plugin-version:
description: "Plugin version"
required: false
type: string
default: "LATEST"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
group: ${{ github.workflow }}-${{ github.ref }}-main
cancel-in-progress: true
env:
JAVA_VERSION: "21"
DOCKER_APT_PACKAGES: python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip
DOCKER_PYTHON_LIBRARIES: kestra
PLUGIN_VERSION: ${{ github.event.inputs.plugin-version != null && github.event.inputs.plugin-version || 'LATEST' }}
jobs:
build-artifacts:
name: Build Artifacts
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
docker-tag: ${{ steps.vars.outputs.tag }}
docker-artifact-name: ${{ steps.vars.outputs.artifact }}
plugins: ${{ steps.plugins-list.outputs.plugins }}
steps:
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Npm
- name: Npm install
shell: bash
working-directory: ui
run: npm ci
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
caches-enabled: true
# Get Plugins List
- name: Get Plugins List
uses: ./.github/actions/plugins-list
if: "!startsWith(github.ref, 'refs/tags/v')"
id: plugins-list
with:
plugin-version: ${{ env.PLUGIN_VERSION }}
# Set Plugins List
- name: Set Plugin List
id: plugins
if: "!startsWith(github.ref, 'refs/tags/v')"
run: |
PLUGINS="${{ steps.plugins-list.outputs.plugins }}"
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" || $TAG == v* ]]; then
echo "plugins=$PLUGINS" >> $GITHUB_OUTPUT
else
echo "plugins=--repositories=https://s01.oss.sonatype.org/content/repositories/snapshots $PLUGINS" >> $GITHUB_OUTPUT
fi
# Build
- name: Build with Gradle
run: |
./gradlew executableJar
- name: Copy exe to image
run: |
cp build/executable/* docker/app/kestra && chmod +x docker/app/kestra
# Docker Tag
- name: Set up Vars
id: vars
run: |
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" ]]
then
TAG="latest";
elif [[ $TAG = "develop" ]]
then
TAG="develop";
elif [[ $TAG = v* ]]
then
TAG="${TAG}";
else
TAG="build-${{ github.run_id }}";
fi
echo "tag=${TAG}" >> $GITHUB_OUTPUT
echo "artifact=docker-kestra-${TAG}" >> $GITHUB_OUTPUT
# Docker setup
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# Docker Build
- name: Build & Export Docker Image
uses: docker/build-push-action@v6
if: "!startsWith(github.ref, 'refs/tags/v')"
with:
context: .
push: false
file: Dockerfile
tags: |
kestra/kestra:${{ steps.vars.outputs.tag }}
build-args: |
KESTRA_PLUGINS=${{ steps.plugins.outputs.plugins }}
APT_PACKAGES=${{ env.DOCKER_APT_PACKAGES }}
PYTHON_LIBRARIES=${{ env.DOCKER_PYTHON_LIBRARIES }}
outputs: type=docker,dest=/tmp/${{ steps.vars.outputs.artifact }}.tar
# Upload artifacts
- name: Upload JAR
uses: actions/upload-artifact@v4
with:
name: jar
path: build/libs/
- name: Upload Executable
uses: actions/upload-artifact@v4
with:
name: exe
path: build/executable/
- name: Upload Docker
uses: actions/upload-artifact@v4
if: "!startsWith(github.ref, 'refs/tags/v')"
with:
name: ${{ steps.vars.outputs.artifact }}
path: /tmp/${{ steps.vars.outputs.artifact }}.tar
check-frontend:
name: Run frontend tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Npm install
shell: bash
working-directory: ui
run: npm ci
- name: Npm lint
uses: reviewdog/action-eslint@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-pr-review # Change reporter.
workdir: "ui"
- name: Run front-end unit tests
shell: bash
working-directory: ui
run: npm run test:unit
- name: Install Playwright
shell: bash
working-directory: ui
run: npx playwright install --with-deps
- name: Build Storybook
shell: bash
working-directory: ui
run: npm run build-storybook --quiet
- name: Serve Storybook and run tests
shell: bash
working-directory: ui
run: |
npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
"npx http-server storybook-static --port 6006 --silent" \
"npx wait-on tcp:127.0.0.1:6006 && npm run test-storybook"
# Run Reusable Workflow from QA repository
check-e2e:
name: Check E2E Tests
needs: build-artifacts
if: ${{ (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') && !startsWith(github.ref, 'refs/tags/v') }}
uses: ./.github/workflows/e2e.yml
strategy:
fail-fast: false
matrix:
backends: ["postgres"]
tests:
name: Execute tests
uses: ./.github/workflows/workflow-test.yml
with:
tags: oss
docker-artifact-name: ${{ needs.build-artifacts.outputs.docker-artifact-name }}
docker-image-tag: kestra/kestra:${{ needs.build-artifacts.outputs.docker-tag }}
backend: ${{ matrix.backends }}
secrets:
GITHUB_AUTH_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
check:
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
name: Check & Publish
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
caches-enabled: true
# Services
- name: Build the docker-compose stack
run: docker compose -f docker-compose-ci.yml up -d
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
# Gradle check
- name: Build with Gradle
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
env:
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
run: |
echo $GOOGLE_SERVICE_ACCOUNT | base64 -d > ~/.gcp-service-account.json
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp-service-account.json
./gradlew check javadoc --parallel
# Sonar
- name: Analyze with Sonar
if: ${{ env.SONAR_TOKEN != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: ./gradlew sonar --info
# Allure check
- name: Auth to Google Cloud
id: auth
if: ${{ always() && env.GOOGLE_SERVICE_ACCOUNT != 0 }}
uses: "google-github-actions/auth@v2"
with:
credentials_json: "${{ secrets.GOOGLE_SERVICE_ACCOUNT }}"
- uses: rlespinasse/github-slug-action@v5
- name: Publish allure report
uses: andrcuns/allure-publish-action@v2.9.0
if: ${{ always() && env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
env:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
JAVA_HOME: /usr/lib/jvm/default-jvm/
with:
storageType: gcs
resultsGlob: "**/build/allure-results"
bucket: internal-kestra-host
baseUrl: "https://internal.dev.kestra.io"
prefix: ${{ format('{0}/{1}', github.repository, 'allure/java') }}
copyLatest: true
ignoreMissingResults: true
# Jacoco
- name: "Set up Cloud SDK"
if: ${{ env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
uses: "google-github-actions/setup-gcloud@v2"
- name: "Copy jacoco files"
if: ${{ env.GOOGLE_SERVICE_ACCOUNT != 0 && (github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '') }}
run: |
mv build/reports/jacoco/testCodeCoverageReport build/reports/jacoco/test/
mv build/reports/jacoco/test/testCodeCoverageReport.xml build/reports/jacoco/test/jacocoTestReport.xml
gsutil -m rsync -d -r build/reports/jacoco/test/ gs://internal-kestra-host/${{ format('{0}/{1}', github.repository, 'jacoco') }}
# report test
- name: Test Report
uses: mikepenz/action-junit-report@v5
if: success() || failure()
with:
report_paths: "**/build/test-results/**/TEST-*.xml"
# Codecov
- uses: codecov/codecov-action@v5
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
report-status: false
release:
name: Github Release
runs-on: ubuntu-latest
needs: [check, check-e2e]
if: |
always() &&
startsWith(github.ref, 'refs/tags/v') &&
needs.check.result == 'success' &&
(
github.event.inputs.skip-test == 'true' ||
(needs.check-e2e.result == 'skipped' || needs.check-e2e.result == 'success')
)
steps:
# Download Exec
- name: Download executable
uses: actions/download-artifact@v4
if: startsWith(github.ref, 'refs/tags/v')
with:
name: exe
path: build/executable
name: Release
needs: [tests]
uses: ./.github/workflows/workflow-release.yml
with:
plugin-version: ${{ github.event.inputs.plugin-version != null && github.event.inputs.plugin-version || 'LATEST' }}
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_PASSWORD: ${{ secrets.DOCKERHUB_PASSWORD }}
SONATYPE_USER: ${{ secrets.SONATYPE_USER }}
SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_GPG_KEYID: ${{ secrets.SONATYPE_GPG_KEYID }}
SONATYPE_GPG_PASSWORD: ${{ secrets.SONATYPE_GPG_PASSWORD }}
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE }}
# GitHub Release
- name: Create GitHub release
id: create_github_release
uses: "marvinpinto/action-automatic-releases@latest"
if: startsWith(github.ref, 'refs/tags/v')
continue-on-error: true
with:
repo_token: "${{ secrets.GITHUB_TOKEN }}"
prerelease: false
files: |
build/executable/*
# Trigger gha workflow to bump helm chart version
- name: trigger the Helm chart version bump
uses: peter-evans/repository-dispatch@v3
if: steps.create_github_release.conclusion == 'success'
with:
token: ${{ secrets.GH_PERSONAL_TOKEN }}
repository: kestra-io/helm-charts
event-type: update-helm-chart-version
client-payload: |-
{
"new_version": "${{ github.ref_name }}",
"github_repository": "${{ github.repository }}",
"github_actor": "${{ github.actor }}"
}
docker:
name: Publish Docker
runs-on: ubuntu-latest
needs: [build-artifacts, check, check-e2e]
if: |
always() &&
github.ref == 'refs/heads/develop' &&
needs.check.result == 'success' &&
(
github.event.inputs.skip-test == 'true' ||
(needs.check-e2e.result == 'skipped' || needs.check-e2e.result == 'success')
)
strategy:
matrix:
image:
- tag: ${{needs.build-artifacts.outputs.docker-tag}}-no-plugins
packages: ""
python-libraries: ""
- tag: ${{needs.build-artifacts.outputs.docker-tag}}
plugins: ${{needs.build-artifacts.outputs.plugins}}
packages: python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip
python-libraries: kestra
steps:
- uses: actions/checkout@v4
# Docker setup
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
# Docker Login
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
# Vars
- name: Set image name
id: vars
run: |
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" || $TAG == v* ]]; then
echo "plugins=${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
else
echo "plugins=--repositories=https://s01.oss.sonatype.org/content/repositories/snapshots ${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
fi
# Build Docker Image
- name: Download executable
uses: actions/download-artifact@v4
with:
name: exe
path: build/executable
- name: Copy exe to image
run: |
cp build/executable/* docker/app/kestra && chmod +x docker/app/kestra
# Docker Build and push
- name: Build Docker Image
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: kestra/kestra:${{ matrix.image.tag }}
platforms: linux/amd64,linux/arm64
build-args: |
KESTRA_PLUGINS=${{ steps.vars.outputs.plugins }}
APT_PACKAGES=${{matrix.image.packages}}
PYTHON_LIBRARIES=${{matrix.image.python-libraries}}
maven:
name: Publish to Maven
runs-on: ubuntu-latest
needs: [check, check-e2e]
if: |
always() &&
(github.ref == 'refs/heads/develop' || startsWith(github.ref, 'refs/tags/v')) &&
needs.check.result == 'success' &&
(
github.event.inputs.skip-test == 'true' ||
(needs.check-e2e.result == 'skipped' || needs.check-e2e.result == 'success')
)
steps:
- uses: actions/checkout@v4
# Checkout GitHub Actions
- uses: actions/checkout@v4
with:
repository: kestra-io/actions
path: actions
ref: main
# Setup build
- uses: ./actions/.github/actions/setup-build
id: build
with:
java-enabled: true
node-enabled: true
caches-enabled: true
# Publish
- name: Publish package to Sonatype
if: github.ref == 'refs/heads/develop'
env:
ORG_GRADLE_PROJECT_sonatypeUsername: ${{ secrets.SONATYPE_USER }}
ORG_GRADLE_PROJECT_sonatypePassword: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_GPG_KEYID: ${{ secrets.SONATYPE_GPG_KEYID }}
SONATYPE_GPG_PASSWORD: ${{ secrets.SONATYPE_GPG_PASSWORD }}
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE }}
run: |
mkdir -p ~/.gradle/
echo "signing.keyId=${SONATYPE_GPG_KEYID}" > ~/.gradle/gradle.properties
echo "signing.password=${SONATYPE_GPG_PASSWORD}" >> ~/.gradle/gradle.properties
echo "signing.secretKeyRingFile=${HOME}/.gradle/secring.gpg" >> ~/.gradle/gradle.properties
echo ${SONATYPE_GPG_FILE} | base64 -d > ~/.gradle/secring.gpg
./gradlew publishToSonatype
# Release
- name: Release package to Maven Central
if: startsWith(github.ref, 'refs/tags/v')
env:
ORG_GRADLE_PROJECT_sonatypeUsername: ${{ secrets.SONATYPE_USER }}
ORG_GRADLE_PROJECT_sonatypePassword: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_GPG_KEYID: ${{ secrets.SONATYPE_GPG_KEYID }}
SONATYPE_GPG_PASSWORD: ${{ secrets.SONATYPE_GPG_PASSWORD }}
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE }}
run: |
echo "signing.keyId=${SONATYPE_GPG_KEYID}" > ~/.gradle/gradle.properties
echo "signing.password=${SONATYPE_GPG_PASSWORD}" >> ~/.gradle/gradle.properties
echo "signing.secretKeyRingFile=${HOME}/.gradle/secring.gpg" >> ~/.gradle/gradle.properties
echo ${SONATYPE_GPG_FILE} | base64 -d > ~/.gradle/secring.gpg
./gradlew publishToSonatype closeAndReleaseSonatypeStagingRepository
end:
runs-on: ubuntu-latest
needs:
- check-e2e
- check
- maven
- docker
- release
if: always()
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
steps:
# Update
- name: Update internal
- name: Github - Update internal
uses: benc-uk/workflow-dispatch@v1
if: github.ref == 'refs/heads/develop' && needs.docker.result == 'success'
with:
@@ -533,7 +62,7 @@ jobs:
token: ${{ secrets.GH_PERSONAL_TOKEN }}
# Slack
- name: Slack notification
- name: Slack - Notification
uses: Gamesight/slack-workflow-status@master
if: ${{ always() && env.SLACK_WEBHOOK_URL != 0 }}
with:

67
.github/workflows/pull-request.yml vendored Normal file
View File

@@ -0,0 +1,67 @@
name: Pull Request Workflow
on:
pull_request:
branches:
- develop
concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}-pr
cancel-in-progress: true
jobs:
file-changes:
name: File changes detection
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
ui: ${{ steps.changes.outputs.ui }}
translations: ${{ steps.changes.outputs.translations }}
backend: ${{ steps.changes.outputs.backend }}
steps:
- uses: dorny/paths-filter@v3
id: changes
with:
filters: |
ui:
- 'ui/**'
backend:
- '!{ui,.github}/**'
token: ${{ secrets.GITHUB_TOKEN }}
frontend:
name: Frontend - Tests
needs: [file-changes]
if: "needs.file-changes.outputs.ui == 'true'"
uses: ./.github/workflows/workflow-frontend-test.yml
secrets:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
backend:
name: Backend - Tests
needs: file-changes
if: "needs.file-changes.outputs.backend == 'true'"
uses: ./.github/workflows/workflow-backend-test.yml
secrets:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
end:
name: End
runs-on: ubuntu-latest
if: always()
needs: [frontend, backend]
steps:
# Slack
- name: Slack notification
uses: Gamesight/slack-workflow-status@master
if: ${{ always() && env.SLACK_WEBHOOK_URL != 0 }}
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
slack_webhook_url: ${{ secrets.SLACK_WEBHOOK_URL }}
name: GitHub Actions
icon_emoji: ":github-actions:"
channel: "C02DQ1A7JLR"

View File

@@ -1,4 +1,4 @@
name: Update and Tag Kestra Plugins
name: Set Version and Tag Plugins
on:
workflow_dispatch:
@@ -14,7 +14,7 @@ on:
jobs:
tag:
name: Release plugins
runs-on: kestra-private-standard
runs-on: ubuntu-latest
steps:
# Checkout
- uses: actions/checkout@v4
@@ -24,30 +24,36 @@ jobs:
# Get Plugins List
- name: Get Plugins List
uses: ./.github/actions/plugins-list
if: "!startsWith(github.ref, 'refs/tags/v')"
id: plugins-list
with:
plugin-version: 'LATEST'
- name: 'Configure Git'
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
# Execute
- name: Tag Plugins
- name: Set Version and Tag Plugins
if: ${{ github.event.inputs.dryRun == 'false' }}
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./tag-release-plugins.sh;
./tag-release-plugins.sh \
chmod +x ./dev-tools/setversion-tag-plugins.sh;
./dev-tools/setversion-tag-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--yes \
${{ steps.plugins-list.outputs.repositories }}
- name: Run Gradle Release (DRY_RUN)
- name: Set Version and Tag Plugins (DRY_RUN)
if: ${{ github.event.inputs.dryRun == 'true' }}
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
chmod +x ./tag-release-plugins.sh;
./tag-release-plugins.sh \
chmod +x ./dev-tools/setversion-tag-plugins.sh;
./dev-tools/setversion-tag-plugins.sh \
--release-version=${{github.event.inputs.releaseVersion}} \
--dry-run \
--yes \

58
.github/workflows/setversion-tag.yml vendored Normal file
View File

@@ -0,0 +1,58 @@
name: Set Version and Tag
run-name: "Set version and Tag Kestra to ${{ github.event.inputs.releaseVersion }} 🚀"
on:
workflow_dispatch:
inputs:
releaseVersion:
description: 'The release version (e.g., 0.21.1)'
required: true
type: string
env:
RELEASE_VERSION: "${{ github.event.inputs.releaseVersion }}"
jobs:
release:
name: Release Kestra
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/releases/v')
steps:
# Checks
- name: Check Inputs
run: |
if ! [[ "$RELEASE_VERSION" =~ ^[0-9]+(\.[0-9]+)(\.[0-9]+)(-rc[0-9])?(-SNAPSHOT)?$ ]]; then
echo "Invalid release version. Must match regex: ^[0-9]+(\.[0-9]+)(\.[0-9]+)-(rc[0-9])?(-SNAPSHOT)?$"
exit 1
fi
CURRENT_BRANCH="{{ github.ref }}"
# Extract the major and minor versions
BASE_VERSION=$(echo "$RELEASE_VERSION" | sed -E 's/^([0-9]+\.[0-9]+)\..*/\1/')
RELEASE_BRANCH="refs/heads/releases/v${BASE_VERSION}.x"
if ! [[ "$CURRENT_BRANCH" == "$RELEASE_BRANCH" ]]; then
echo "Invalid release branch. Expected $RELEASE_BRANCH, was $CURRENT_BRANCH"
exit 1
fi
# Checkout
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Configure Git
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
# Execute
- name: Run Gradle Release
env:
GITHUB_PAT: ${{ secrets.GH_PERSONAL_TOKEN }}
run: |
# Update version
sed -i "s/^version=.*/version=$RELEASE_VERSION/" ./gradle.properties
git add ./gradle.properties
git commit -m"chore(version): update to version '$RELEASE_VERSION'"
git push
git tag -a "v$RELEASE_VERSION" -m"v$RELEASE_VERSION"
git push origin "v$RELEASE_VERSION"

View File

@@ -33,6 +33,12 @@ jobs:
node-enabled: true
caches-enabled: true
# Npm
- name: Npm - Install
shell: bash
working-directory: ui
run: npm ci
# Run OWASP dependency check plugin
- name: Gradle Dependency Check
env:
@@ -74,7 +80,7 @@ jobs:
# Run Trivy image scan for Docker vulnerabilities, see https://github.com/aquasecurity/trivy-action
- name: Docker Vulnerabilities Check
uses: aquasecurity/trivy-action@0.29.0
uses: aquasecurity/trivy-action@0.30.0
with:
image-ref: kestra/kestra:develop
format: table
@@ -107,7 +113,7 @@ jobs:
# Run Trivy image scan for Docker vulnerabilities, see https://github.com/aquasecurity/trivy-action
- name: Docker Vulnerabilities Check
uses: aquasecurity/trivy-action@0.29.0
uses: aquasecurity/trivy-action@0.30.0
with:
image-ref: kestra/kestra:latest
format: table

View File

@@ -0,0 +1,139 @@
name: Backend - Tests
on:
workflow_call:
secrets:
GITHUB_AUTH_TOKEN:
description: "The GitHub Token."
required: true
CODECOV_TOKEN:
description: 'Codecov Token'
required: true
SONAR_TOKEN:
description: 'Sonar Token'
required: true
GOOGLE_SERVICE_ACCOUNT:
description: 'Google Service Account'
required: true
permissions:
contents: write
checks: write
actions: read
jobs:
test:
name: Backend - Tests
runs-on: ubuntu-latest
env:
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
steps:
- uses: actions/checkout@v4
name: Checkout - Current ref
# Setup build
- uses: kestra-io/actions/.github/actions/setup-build@main
name: Setup - Build
id: build
with:
java-enabled: true
node-enabled: true
python-enabled: true
# Services
- name: Setup - Start docker compose
shell: bash
run: docker compose -f docker-compose-ci.yml up -d
# Gradle check
- name: Gradle - Build
if: ${{ github.event.inputs.skip-test == 'false' || github.event.inputs.skip-test == '' }}
env:
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
shell: bash
run: |
echo $GOOGLE_SERVICE_ACCOUNT | base64 -d > ~/.gcp-service-account.json
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.gcp-service-account.json
./gradlew check javadoc --parallel
# report test
- name: Test - Publish Test Results
uses: dorny/test-reporter@v2
if: always()
with:
name: Java Tests Report
reporter: java-junit
path: '**/build/test-results/test/TEST-*.xml'
list-suites: 'failed'
list-tests: 'failed'
fail-on-error: 'false'
# Sonar
- name: Test - Analyze with Sonar
if: env.SONAR_TOKEN != ''
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_AUTH_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
shell: bash
run: ./gradlew sonar --info
# GCP
- name: GCP - Auth with unit test account
id: auth
if: always() && env.GOOGLE_SERVICE_ACCOUNT != ''
continue-on-error: true
uses: "google-github-actions/auth@v2"
with:
credentials_json: "${{ secrets.GOOGLE_SERVICE_ACCOUNT }}"
- name: GCP - Setup Cloud SDK
if: env.GOOGLE_SERVICE_ACCOUNT != ''
uses: "google-github-actions/setup-gcloud@v2"
# Allure check
- uses: rlespinasse/github-slug-action@v5
name: Allure - Generate slug variables
- name: Allure - Publish report
uses: andrcuns/allure-publish-action@v2.9.0
if: always() && env.GOOGLE_SERVICE_ACCOUNT != ''
continue-on-error: true
env:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_AUTH_TOKEN }}
JAVA_HOME: /usr/lib/jvm/default-jvm/
with:
storageType: gcs
resultsGlob: "**/build/allure-results"
bucket: internal-kestra-host
baseUrl: "https://internal.dev.kestra.io"
prefix: ${{ format('{0}/{1}', github.repository, 'allure/java') }}
copyLatest: true
ignoreMissingResults: true
# Jacoco
- name: Jacoco - Copy reports
if: env.GOOGLE_SERVICE_ACCOUNT != ''
continue-on-error: true
shell: bash
run: |
mv build/reports/jacoco/testCodeCoverageReport build/reports/jacoco/test/
mv build/reports/jacoco/test/testCodeCoverageReport.xml build/reports/jacoco/test/jacocoTestReport.xml
gsutil -m rsync -d -r build/reports/jacoco/test/ gs://internal-kestra-host/${{ format('{0}/{1}', github.repository, 'jacoco') }}
# Codecov
- name: Codecov - Upload coverage reports
uses: codecov/codecov-action@v5
if: ${{ !cancelled() }}
continue-on-error: true
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: backend
- name: Codecov - Upload test results
uses: codecov/test-results-action@v1
if: ${{ !cancelled() }}
continue-on-error: true
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: backend

View File

@@ -0,0 +1,152 @@
name: Build Artifacts
on:
workflow_call:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: true
type: string
outputs:
docker-tag:
value: ${{ jobs.build.outputs.docker-tag }}
description: "The Docker image Tag for Kestra"
docker-artifact-name:
value: ${{ jobs.build.outputs.docker-artifact-name }}
description: "The GitHub artifact containing the Kestra docker image name."
plugins:
value: ${{ jobs.build.outputs.plugins }}
description: "The Kestra plugins list used for the build."
jobs:
build:
name: Build - Artifacts
runs-on: ubuntu-latest
outputs:
docker-tag: ${{ steps.vars.outputs.tag }}
docker-artifact-name: ${{ steps.vars.outputs.artifact }}
plugins: ${{ steps.plugins.outputs.plugins }}
env:
PLUGIN_VERSION: ${{ github.event.inputs.plugin-version != null && github.event.inputs.plugin-version || 'LATEST' }}
steps:
- name: Checkout - Current ref
uses: actions/checkout@v4
with:
fetch-depth: 0
# Npm
- name: Setup - Npm install
shell: bash
working-directory: ui
run: npm ci
# Setup build
- uses: kestra-io/actions/.github/actions/setup-build@main
name: Setup - Build
id: build
with:
java-enabled: true
node-enabled: true
# Get Plugins List
- name: Plugins - Get List
uses: ./.github/actions/plugins-list
if: "!startsWith(github.ref, 'refs/tags/v')"
id: plugins-list
with:
plugin-version: ${{ env.PLUGIN_VERSION }}
# Set Plugins List
- name: Plugins - Set List
id: plugins
if: "!startsWith(github.ref, 'refs/tags/v')"
shell: bash
run: |
PLUGINS="${{ steps.plugins-list.outputs.plugins }}"
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" || $TAG == v* ]]; then
echo "plugins=$PLUGINS" >> $GITHUB_OUTPUT
else
echo "plugins=--repositories=https://s01.oss.sonatype.org/content/repositories/snapshots $PLUGINS" >> $GITHUB_OUTPUT
fi
# Build
- name: Gradle - Build
shell: bash
run: |
./gradlew executableJar
- name: Artifacts - Copy exe to image
shell: bash
run: |
cp build/executable/* docker/app/kestra && chmod +x docker/app/kestra
# Docker Tag
- name: Setup - Docker vars
id: vars
shell: bash
run: |
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" ]]
then
TAG="latest";
elif [[ $TAG = "develop" ]]
then
TAG="develop";
elif [[ $TAG = v* ]]
then
TAG="${TAG}";
else
TAG="build-${{ github.run_id }}";
fi
echo "tag=${TAG}" >> $GITHUB_OUTPUT
echo "artifact=docker-kestra-${TAG}" >> $GITHUB_OUTPUT
# Docker setup
- name: Docker - Setup QEMU
uses: docker/setup-qemu-action@v3
- name: Docker - Fix Qemu
shell: bash
run: |
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes
- name: Docker - Setup Buildx
uses: docker/setup-buildx-action@v3
# Docker Build
- name: Docker - Build & export image
uses: docker/build-push-action@v6
if: "!startsWith(github.ref, 'refs/tags/v')"
with:
context: .
push: false
file: Dockerfile
tags: |
kestra/kestra:${{ steps.vars.outputs.tag }}
build-args: |
KESTRA_PLUGINS=${{ steps.plugins.outputs.plugins }}
APT_PACKAGES=${{ env.DOCKER_APT_PACKAGES }}
PYTHON_LIBRARIES=${{ env.DOCKER_PYTHON_LIBRARIES }}
outputs: type=docker,dest=/tmp/${{ steps.vars.outputs.artifact }}.tar
# Upload artifacts
- name: Artifacts - Upload JAR
uses: actions/upload-artifact@v4
with:
name: jar
path: build/libs/
- name: Artifacts - Upload Executable
uses: actions/upload-artifact@v4
with:
name: exe
path: build/executable/
- name: Artifacts - Upload Docker
uses: actions/upload-artifact@v4
if: "!startsWith(github.ref, 'refs/tags/v')"
with:
name: ${{ steps.vars.outputs.artifact }}
path: /tmp/${{ steps.vars.outputs.artifact }}.tar

View File

@@ -0,0 +1,84 @@
name: Frontend - Tests
on:
workflow_call:
secrets:
GITHUB_AUTH_TOKEN:
description: "The GitHub Token."
required: true
CODECOV_TOKEN:
description: 'Codecov Token'
required: true
env:
# to save corepack from itself
COREPACK_INTEGRITY_KEYS: 0
jobs:
test:
name: Frontend - Tests
runs-on: ubuntu-latest
steps:
- id: checkout
name: Checkout - Current ref
uses: actions/checkout@v4
with:
ref: ${{ github.head_ref }}
- name: Npm - install
shell: bash
working-directory: ui
run: npm ci
- name: Npm - lint
uses: reviewdog/action-eslint@v1
with:
github_token: ${{ secrets.GITHUB_AUTH_TOKEN }}
reporter: github-pr-review
workdir: ui
- name: Npm - Run build
shell: bash
working-directory: ui
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: npm run build
- name: Run front-end unit tests
shell: bash
working-directory: ui
run: npm run test:cicd
- name: Storybook - Install Playwright
shell: bash
working-directory: ui
run: npx playwright install --with-deps
- name: Storybook - Build
shell: bash
working-directory: ui
run: npm run build-storybook --quiet
- name: Storybook - Run tests
shell: bash
working-directory: ui
run: |
npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
"npx http-server storybook-static --port 6006 --silent" \
"npx wait-on tcp:127.0.0.1:6006 && npm run test:storybook"
- name: Codecov - Upload coverage reports
uses: codecov/codecov-action@v5
if: ${{ !cancelled() && github.event.pull_request.head.repo.full_name == github.repository }}
continue-on-error: true
with:
token: ${{ secrets.CODECOV_TOKEN }}
flags: frontend
- name: Codecov - Upload test results
uses: codecov/test-results-action@v1
if: ${{ !cancelled() }}
continue-on-error: true
with:
token: ${{ secrets.CODECOV_TOKEN && github.event.pull_request.head.repo.full_name == github.repository }}
flags: frontend

View File

@@ -0,0 +1,48 @@
name: Github - Release
on:
workflow_call:
secrets:
GH_PERSONAL_TOKEN:
description: "The Github personal token."
required: true
jobs:
publish:
name: Github - Release
runs-on: ubuntu-latest
steps:
# Download Exec
- name: Artifacts - Download executable
uses: actions/download-artifact@v4
if: startsWith(github.ref, 'refs/tags/v')
with:
name: exe
path: build/executable
# GitHub Release
- name: GitHub - Create release
id: create_github_release
uses: "marvinpinto/action-automatic-releases@latest"
if: startsWith(github.ref, 'refs/tags/v')
continue-on-error: true
with:
repo_token: "${{ secrets.GITHUB_TOKEN }}"
prerelease: false
files: |
build/executable/*
# Trigger gha workflow to bump helm chart version
- name: GitHub - Trigger the Helm chart version bump
uses: peter-evans/repository-dispatch@v3
if: steps.create_github_release.conclusion == 'success'
with:
token: ${{ secrets.GH_PERSONAL_TOKEN }}
repository: kestra-io/helm-charts
event-type: update-helm-chart-version
client-payload: |-
{
"new_version": "${{ github.ref_name }}",
"github_repository": "${{ github.repository }}",
"github_actor": "${{ github.actor }}"
}

View File

@@ -0,0 +1,100 @@
name: Publish - Docker
on:
workflow_dispatch:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: false
type: string
workflow_call:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: false
type: string
secrets:
DOCKERHUB_USERNAME:
description: "The Dockerhub username."
required: true
DOCKERHUB_PASSWORD:
description: "The Dockerhub password."
required: true
jobs:
publish:
name: Publish - Docker
runs-on: ubuntu-latest
strategy:
matrix:
image:
- tag: ${{ needs.build-artifacts.outputs.docker-tag }}-no-plugins
packages: jattach
python-libraries: ""
- tag: ${{ needs.build-artifacts.outputs.docker-tag }}
plugins: ${{ needs.build-artifacts.outputs.plugins }}
packages: python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip jattach
python-libraries: kestra
steps:
- name: Checkout - Current ref
uses: actions/checkout@v4
# Docker setup
- name: Docker - Setup QEMU
uses: docker/setup-qemu-action@v3
- name: Docker - Fix Qemu
shell: bash
run: |
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes
- name: Docker - Setup Docker Buildx
uses: docker/setup-buildx-action@v3
# Docker Login
- name: Docker - Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
# Vars
- name: Docker - Set image name
shell: bash
id: vars
run: |
TAG=${GITHUB_REF#refs/*/}
if [[ $TAG = "master" || $TAG == v* ]]; then
echo "plugins=${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
else
echo "plugins=--repositories=https://s01.oss.sonatype.org/content/repositories/snapshots ${{ matrix.image.plugins }}" >> $GITHUB_OUTPUT
fi
# Build Docker Image
- name: Artifacts - Download executable
uses: actions/download-artifact@v4
with:
name: exe
path: build/executable
- name: Docker - Copy exe to image
shell: bash
run: |
cp build/executable/* docker/app/kestra && chmod +x docker/app/kestra
# Docker Build and push
- name: Docker - Build image
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: kestra/kestra:${{ matrix.image.tag }}
platforms: linux/amd64,linux/arm64
build-args: |
KESTRA_PLUGINS=${{ steps.vars.outputs.plugins }}
APT_PACKAGES=${{ matrix.image.packages }}
PYTHON_LIBRARIES=${{ matrix.image.python-libraries }}

View File

@@ -0,0 +1,57 @@
name: Publish - Maven
on:
workflow_call:
secrets:
SONATYPE_USER:
description: "The Sonatype username."
required: true
SONATYPE_PASSWORD:
description: "The Sonatype password."
required: true
SONATYPE_GPG_KEYID:
description: "The Sonatype GPG key id."
required: true
SONATYPE_GPG_PASSWORD:
description: "The Sonatype GPG password."
required: true
SONATYPE_GPG_FILE:
description: "The Sonatype GPG file."
required: true
jobs:
publish:
name: Publish - Maven
runs-on: ubuntu-latest
steps:
- name: Checkout - Current ref
uses: actions/checkout@v4
# Setup build
- name: Setup - Build
uses: kestra-io/actions/.github/actions/setup-build@main
id: build
with:
java-enabled: true
node-enabled: true
# Publish
- name: Publish - Release package to Maven Central
shell: bash
env:
ORG_GRADLE_PROJECT_sonatypeUsername: ${{ secrets.SONATYPE_USER }}
ORG_GRADLE_PROJECT_sonatypePassword: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_GPG_KEYID: ${{ secrets.SONATYPE_GPG_KEYID }}
SONATYPE_GPG_PASSWORD: ${{ secrets.SONATYPE_GPG_PASSWORD }}
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE}}
run: |
mkdir -p ~/.gradle/
echo "signing.keyId=${SONATYPE_GPG_KEYID}" > ~/.gradle/gradle.properties
echo "signing.password=${SONATYPE_GPG_PASSWORD}" >> ~/.gradle/gradle.properties
echo "signing.secretKeyRingFile=${HOME}/.gradle/secring.gpg" >> ~/.gradle/gradle.properties
echo ${SONATYPE_GPG_FILE} | base64 -d > ~/.gradle/secring.gpg
./gradlew publishToSonatype ${{ startsWith(github.ref, 'refs/tags/v') && 'closeAndReleaseSonatypeStagingRepository' || '' }}
# Gradle dependency
- name: Java - Gradle dependency graph
uses: gradle/actions/dependency-submission@v4

71
.github/workflows/workflow-release.yml vendored Normal file
View File

@@ -0,0 +1,71 @@
name: Release
on:
workflow_dispatch:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: false
type: string
workflow_call:
inputs:
plugin-version:
description: "Kestra version"
default: 'LATEST'
required: false
type: string
secrets:
DOCKERHUB_USERNAME:
description: "The Dockerhub username."
required: true
DOCKERHUB_PASSWORD:
description: "The Dockerhub password."
required: true
SONATYPE_USER:
description: "The Sonatype username."
required: true
SONATYPE_PASSWORD:
description: "The Sonatype password."
required: true
SONATYPE_GPG_KEYID:
description: "The Sonatype GPG key id."
required: true
SONATYPE_GPG_PASSWORD:
description: "The Sonatype GPG password."
required: true
SONATYPE_GPG_FILE:
description: "The Sonatype GPG file."
required: true
jobs:
build-artifacts:
name: Build - Artifacts
uses: ./.github/workflows/workflow-build-artifacts.yml
with:
plugin-version: ${{ github.event.inputs.plugin-version != null && github.event.inputs.plugin-version || 'LATEST' }}
Docker:
name: Publish Docker
needs: build-artifacts
uses: ./.github/workflows/workflow-publish-docker.yml
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_PASSWORD: ${{ secrets.DOCKERHUB_PASSWORD }}
Maven:
name: Publish Maven
uses: ./.github/workflows/workflow-publish-maven.yml
secrets:
SONATYPE_USER: ${{ secrets.SONATYPE_USER }}
SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
SONATYPE_GPG_KEYID: ${{ secrets.SONATYPE_GPG_KEYID }}
SONATYPE_GPG_PASSWORD: ${{ secrets.SONATYPE_GPG_PASSWORD }}
SONATYPE_GPG_FILE: ${{ secrets.SONATYPE_GPG_FILE }}
Github:
name: Github Release
needs: build-artifacts
if: startsWith(github.ref, 'refs/tags/v')
uses: ./.github/workflows/workflow-github-release.yml
secrets:
GH_PERSONAL_TOKEN: ${{ secrets.GH_PERSONAL_TOKEN }}

97
.github/workflows/workflow-test.yml vendored Normal file
View File

@@ -0,0 +1,97 @@
name: Tests
on:
schedule:
- cron: '0 4 * * 1,2,3,4,5'
workflow_call:
inputs:
report-status:
description: "Report status of the jobs in outputs"
type: string
required: false
default: false
outputs:
frontend_status:
description: "Status of the frontend job"
value: ${{ jobs.set-frontend-status.outputs.frontend_status }}
backend_status:
description: "Status of the backend job"
value: ${{ jobs.set-backend-status.outputs.backend_status }}
jobs:
file-changes:
name: File changes detection
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
ui: ${{ steps.changes.outputs.ui }}
backend: ${{ steps.changes.outputs.backend }}
steps:
- uses: actions/checkout@v4
if: "!startsWith(github.ref, 'refs/tags/v')"
- uses: dorny/paths-filter@v3
if: "!startsWith(github.ref, 'refs/tags/v')"
id: changes
with:
filters: |
ui:
- 'ui/**'
backend:
- '!{ui,.github}/**'
token: ${{ secrets.GITHUB_TOKEN }}
frontend:
name: Frontend - Tests
needs: file-changes
if: "needs.file-changes.outputs.ui == 'true' || startsWith(github.ref, 'refs/tags/v')"
uses: ./.github/workflows/workflow-frontend-test.yml
secrets:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
backend:
name: Backend - Tests
needs: file-changes
if: "needs.file-changes.outputs.backend == 'true' || startsWith(github.ref, 'refs/tags/v')"
uses: ./.github/workflows/workflow-backend-test.yml
secrets:
GITHUB_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
GOOGLE_SERVICE_ACCOUNT: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
# Output every job status
# To be used in other workflows
report-status:
name: Report Status
runs-on: ubuntu-latest
needs: [ frontend, backend ]
if: always() && (inputs.report-status == 'true')
outputs:
frontend_status: ${{ steps.set-frontend-status.outputs.frontend_status }}
backend_status: ${{ steps.set-backend-status.outputs.backend_status }}
steps:
- id: set-frontend-status
name: Set frontend job status
run: echo "::set-output name=frontend_status::${{ needs.frontend.result }}"
- id: set-backend-status
name: Set backend job status
run: echo "::set-output name=backend_status::${{ needs.backend.result }}"
notify:
name: Notify - Slack
runs-on: ubuntu-latest
needs: [ frontend, backend ]
if: github.event_name == 'schedule'
steps:
- name: Notify failed CI
id: send-ci-failed
if: |
always() && (needs.frontend.result != 'success' ||
needs.backend.result != 'success')
uses: kestra-io/actions/.github/actions/send-ci-failed@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

3
.gitignore vendored
View File

@@ -36,6 +36,8 @@ yarn.lock
ui/coverage
ui/stats.html
ui/.frontend-gradle-plugin
ui/utils/CHANGELOG.md
ui/test-report.junit.xml
### Docker
/.env
@@ -56,3 +58,4 @@ core/src/main/resources/gradle.properties
**/allure-results/*
*storybook.log
storybook-static

View File

@@ -32,6 +32,7 @@
#plugin-git:io.kestra.plugin:plugin-git:LATEST
#plugin-github:io.kestra.plugin:plugin-github:LATEST
#plugin-googleworkspace:io.kestra.plugin:plugin-googleworkspace:LATEST
#plugin-graalvm:io.kestra.plugin:plugin-graalvm:LATEST
#plugin-hightouch:io.kestra.plugin:plugin-hightouch:LATEST
#plugin-hubspot:io.kestra.plugin:plugin-hubspot:LATEST
#plugin-huggingface:io.kestra.plugin:plugin-huggingface:LATEST
@@ -40,6 +41,7 @@
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-db2:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-duckdb:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-druid:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-mariadb:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-mysql:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-oracle:LATEST
#plugin-jdbc:io.kestra.plugin:plugin-jdbc-pinot:LATEST

View File

@@ -17,6 +17,8 @@ VERSION := $(shell ./gradlew properties -q | awk '/^version:/ {print $$2}')
GIT_COMMIT := $(shell git rev-parse --short HEAD)
GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)
DATE := $(shell date --rfc-3339=seconds)
PLUGIN_GIT_DIR ?= $(pwd)/..
PLUGIN_JARS_DIR ?= $(pwd)/locals/plugins
DOCKER_IMAGE = kestra/kestra
DOCKER_PATH = ./
@@ -87,7 +89,7 @@ build-docker: build-exec
--compress \
--rm \
-f ./Dockerfile \
--build-arg="APT_PACKAGES=python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip" \
--build-arg="APT_PACKAGES=python3 python3-venv python-is-python3 python3-pip nodejs npm curl zip unzip jattach" \
--build-arg="PYTHON_LIBRARIES=kestra" \
-t ${DOCKER_IMAGE}:${VERSION} ${DOCKER_PATH} || exit 1 ;
@@ -174,3 +176,72 @@ start-standalone-postgres: kill --private-start-standalone-postgres health
start-standalone-local: kill --private-start-standalone-local health
#checkout all plugins
clone-plugins:
@echo "Using PLUGIN_GIT_DIR: $(PLUGIN_GIT_DIR)"
@mkdir -p "$(PLUGIN_GIT_DIR)"
@echo "Fetching repository list from GitHub..."
@REPOS=$(gh repo list kestra-io -L 1000 --json name | jq -r .[].name | sort | grep "^plugin-") \
for repo in $$REPOS; do \
if [[ $$repo == plugin-* ]]; then \
if [ -d "$(PLUGIN_GIT_DIR)/$$repo" ]; then \
echo "Skipping: $$repo (Already cloned)"; \
else \
echo "Cloning: $$repo using SSH..."; \
git clone "git@github.com:kestra-io/$$repo.git" "$(PLUGIN_GIT_DIR)/$$repo"; \
fi; \
fi; \
done
@echo "Done!"
# Update all plugins jar
build-plugins:
@echo "🔍 Scanning repositories in '$(PLUGIN_GIT_DIR)'..."
@MASTER_REPOS=(); \
for repo in "$(PLUGIN_GIT_DIR)"/*; do \
if [ -d "$$repo/.git" ]; then \
branch=$$(git -C "$$repo" rev-parse --abbrev-ref HEAD); \
if [[ "$$branch" == "master" || "$$branch" == "main" ]]; then \
MASTER_REPOS+=("$$repo"); \
else \
echo "❌ Skipping: $$(basename "$$repo") (Not on master or main branch)"; \
fi; \
fi; \
done; \
\
# === STEP 2: Update Repos on Master or Main Branch === \
echo "⬇️ Updating repositories on master or main branch..."; \
for repo in "$${MASTER_REPOS[@]}"; do \
echo "🔄 Updating: $$(basename "$$repo")"; \
git -C "$$repo" pull --rebase; \
done; \
\
# === STEP 3: Build with Gradle === \
echo "⚙️ Building repositories with Gradle..."; \
for repo in "$${MASTER_REPOS[@]}"; do \
echo "🔨 Building: $$(basename "$$repo")"; \
gradle clean build -x test shadowJar -p "$$repo"; \
done; \
\
# === STEP 4: Copy Latest JARs (Ignoring javadoc & sources) === \
echo "📦 Organizing built JARs..."; \
mkdir -p "$(PLUGIN_JARS_DIR)"; \
for repo in "$${MASTER_REPOS[@]}"; do \
REPO_NAME=$$(basename "$$repo"); \
\
JARS=($$(find "$$repo" -type f -name "plugin-*.jar" ! -name "*-javadoc.jar" ! -name "*-sources.jar")); \
if [ $${#JARS[@]} -eq 0 ]; then \
echo "⚠️ Warning: No valid plugin JARs found for $$REPO_NAME"; \
continue; \
fi; \
\
for jar in "$${JARS[@]}"; do \
JAR_NAME=$$(basename "$$jar"); \
BASE_NAME=$$(echo "$$JAR_NAME" | sed -E 's/(-[0-9]+.*)?\.jar$$//'); \
rm -f "$(PLUGIN_JARS_DIR)/$$BASE_NAME"-[0-9]*.jar; \
cp "$$jar" "$(PLUGIN_JARS_DIR)/"; \
echo "✅ Copied JAR: $$JAR_NAME"; \
done; \
done; \
\
echo "🎉 Done! All master and main branch repos updated, built, and organized."

View File

@@ -24,6 +24,13 @@
<a href="https://www.youtube.com/@kestra-io"><img height="25" src="https://kestra.io/youtube.svg" alt="youtube" /></a> &nbsp;
</p>
<p align="center">
<a href="https://trendshift.io/repositories/2714" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/2714" alt="kestra-io%2Fkestra | Trendshift" width="250" height="55"/>
</a>
<a href="https://www.producthunt.com/posts/kestra?embed=true&utm_source=badge-top-post-badge&utm_medium=badge&utm_souce=badge-kestra" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/top-post-badge.svg?post_id=612077&theme=light&period=daily&t=1740737506162" alt="Kestra - All&#0045;in&#0045;one&#0032;automation&#0032;&#0038;&#0032;orchestration&#0032;platform | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
</p>
<p align="center">
<a href="https://go.kestra.io/video/product-overview" target="_blank">
<img src="https://kestra.io/startvideo.png" alt="Get started in 4 minutes with Kestra" width="640px" />
@@ -47,7 +54,7 @@ Kestra is an open-source, event-driven orchestration platform that makes both **
- **Structure & Resilience**: tame chaos and bring resilience to your workflows with **namespaces**, **labels**, **subflows**, **retries**, **timeout**, **error handling**, **inputs**, **outputs** that generate artifacts in the UI, **variables**, **conditional branching**, **advanced scheduling**, **event triggers**, **backfills**, **dynamic tasks**, **sequential and parallel tasks**, and skip tasks or triggers when needed by setting the flag `disabled` to `true`.
🧑‍💻 The YAML definition gets automatically adjusted any time you make changes to a workflow from the UI or via an API call. Therefore, the orchestration logic is **always managed declaratively in code**, even if you modify your workflows in other ways (UI, CI/CD, Terraform, API calls).
🧑‍💻 The YAML definition gets automatically adjusted any time you make changes to a workflow from the UI or via an API call. Therefore, the orchestration logic is **always managed declaratively in code**, even if you modify your workflows in other ways (UI, CI/CD, Terraform, API calls).
<p align="center">

View File

@@ -16,7 +16,7 @@ plugins {
id "java"
id 'java-library'
id "idea"
id "com.gradleup.shadow" version "8.3.5"
id "com.gradleup.shadow" version "8.3.6"
id "application"
// test
@@ -28,18 +28,18 @@ plugins {
id "com.github.ben-manes.versions" version "0.52.0"
// front
id 'org.siouan.frontend-jdk21' version '10.0.0' apply false
id 'com.github.node-gradle.node' version '7.1.0'
// release
id "io.github.gradle-nexus.publish-plugin" version "2.0.0"
id 'net.researchgate.release' version '3.1.0'
id "com.gorylenko.gradle-git-properties" version "2.4.2"
id "com.gorylenko.gradle-git-properties" version "2.5.0"
id 'signing'
id 'ru.vyarus.pom' version '3.0.0' apply false
id 'ru.vyarus.github-info' version '2.0.0' apply false
// OWASP dependency check
id "org.owasp.dependencycheck" version "12.0.1" apply false
id "org.owasp.dependencycheck" version "12.1.0" apply false
}
idea {
@@ -74,7 +74,7 @@ dependencies {
**********************************************************************************************************************/
allprojects {
if (it.name != 'platform') {
group "io.kestra"
group = "io.kestra"
java {
sourceCompatibility = targetJavaVersion
@@ -121,7 +121,6 @@ allprojects {
micronaut "io.micronaut:micronaut-management"
micronaut "io.micrometer:micrometer-core"
micronaut "io.micronaut.micrometer:micronaut-micrometer-registry-prometheus"
micronaut "io.micronaut.micrometer:micronaut-micrometer-registry-otlp"
micronaut "io.micronaut:micronaut-http-client"
micronaut "io.micronaut.reactor:micronaut-reactor-http-client"
micronaut "io.micronaut.tracing:micronaut-tracing-opentelemetry-http"
@@ -280,7 +279,7 @@ subprojects {
}
dependencies {
agent "org.aspectj:aspectjweaver:1.9.22.1"
agent "org.aspectj:aspectjweaver:1.9.23"
}
test {

View File

@@ -12,18 +12,9 @@ dependencies {
implementation 'ch.qos.logback.contrib:logback-json-classic'
implementation 'ch.qos.logback.contrib:logback-jackson'
// plugins
implementation 'org.eclipse.aether:aether-api'
implementation 'org.eclipse.aether:aether-spi'
implementation 'org.eclipse.aether:aether-util'
implementation 'org.eclipse.aether:aether-impl'
implementation 'org.eclipse.aether:aether-connector-basic'
implementation 'org.eclipse.aether:aether-transport-file'
implementation 'org.eclipse.aether:aether-transport-http'
implementation('org.apache.maven:maven-aether-provider') {
// sisu dependency injector is not used
exclude group: 'org.eclipse.sisu'
}
// OTLP metrics
implementation "io.micronaut.micrometer:micronaut-micrometer-registry-otlp"
// aether still use javax.inject
compileOnly 'javax.inject:javax.inject:1'
@@ -43,4 +34,7 @@ dependencies {
implementation project(":storage-local")
implementation project(":webserver")
//test
testImplementation "org.wiremock:wiremock"
}

View File

@@ -4,16 +4,17 @@ import ch.qos.logback.classic.LoggerContext;
import com.google.common.collect.ImmutableMap;
import io.kestra.cli.commands.servers.ServerCommandInterface;
import io.kestra.cli.services.StartupHookInterface;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.plugins.PluginManager;
import io.kestra.core.plugins.PluginRegistry;
import io.kestra.webserver.services.FlowAutoLoaderService;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.yaml.YamlPropertySourceLoader;
import io.micronaut.core.annotation.Introspected;
import io.micronaut.http.uri.UriBuilder;
import io.micronaut.management.endpoint.EndpointDefaultConfiguration;
import io.micronaut.runtime.server.EmbeddedServer;
import jakarta.inject.Provider;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.utils.URIBuilder;
import io.kestra.core.utils.Rethrow;
import picocli.CommandLine;
@@ -26,10 +27,13 @@ import java.nio.file.Paths;
import java.text.MessageFormat;
import java.time.temporal.ChronoUnit;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.Callable;
import jakarta.inject.Inject;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
@CommandLine.Command(
@Command(
versionProvider = VersionProvider.class,
mixinStandardHelpOptions = true,
showDefaultValues = true
@@ -49,22 +53,28 @@ abstract public class AbstractCommand implements Callable<Integer> {
@Inject
private io.kestra.core.utils.VersionProvider versionProvider;
@Inject
protected Provider<PluginRegistry> pluginRegistryProvider;
@Inject
protected Provider<PluginManager> pluginManagerProvider;
private PluginRegistry pluginRegistry;
@CommandLine.Option(names = {"-v", "--verbose"}, description = "Change log level. Multiple -v options increase the verbosity.", showDefaultValue = CommandLine.Help.Visibility.NEVER)
@Option(names = {"-v", "--verbose"}, description = "Change log level. Multiple -v options increase the verbosity.", showDefaultValue = CommandLine.Help.Visibility.NEVER)
private boolean[] verbose = new boolean[0];
@CommandLine.Option(names = {"-l", "--log-level"}, description = "Change log level (values: ${COMPLETION-CANDIDATES})")
@Option(names = {"-l", "--log-level"}, description = "Change log level (values: ${COMPLETION-CANDIDATES})")
private LogLevel logLevel = LogLevel.INFO;
@CommandLine.Option(names = {"--internal-log"}, description = "Change also log level for internal log")
@Option(names = {"--internal-log"}, description = "Change also log level for internal log")
private boolean internalLog = false;
@CommandLine.Option(names = {"-c", "--config"}, description = "Path to a configuration file")
@Option(names = {"-c", "--config"}, description = "Path to a configuration file")
private Path config = Paths.get(System.getProperty("user.home"), ".kestra/config.yml");
@CommandLine.Option(names = {"-p", "--plugins"}, description = "Path to plugins directory")
protected Path pluginsPath = System.getenv("KESTRA_PLUGINS_PATH") != null ? Paths.get(System.getenv("KESTRA_PLUGINS_PATH")) : null;
@Option(names = {"-p", "--plugins"}, description = "Path to plugins directory")
protected Path pluginsPath = Optional.ofNullable(System.getenv("KESTRA_PLUGINS_PATH")).map(Paths::get).orElse(null);
public enum LogLevel {
TRACE,
@@ -76,7 +86,7 @@ abstract public class AbstractCommand implements Callable<Integer> {
@Override
public Integer call() throws Exception {
Thread.currentThread().setName(this.getClass().getDeclaredAnnotation(CommandLine.Command.class).name());
Thread.currentThread().setName(this.getClass().getDeclaredAnnotation(Command.class).name());
startLogger();
sendServerLog();
if (this.startupHook != null) {
@@ -84,8 +94,14 @@ abstract public class AbstractCommand implements Callable<Integer> {
}
if (this.pluginsPath != null && loadExternalPlugins()) {
pluginRegistry = pluginRegistry();
pluginRegistry = pluginRegistryProvider.get();
pluginRegistry.registerIfAbsent(pluginsPath);
// PluginManager mus only be initialized if a registry is also instantiated
if (isPluginManagerEnabled()) {
PluginManager manager = pluginManagerProvider.get();
manager.start();
}
}
startWebserver();
@@ -102,8 +118,15 @@ abstract public class AbstractCommand implements Callable<Integer> {
return true;
}
protected PluginRegistry pluginRegistry() {
return KestraContext.getContext().getPluginRegistry(); // Lazy init
/**
* Specifies whether the {@link PluginManager} service must be initialized.
* <p>
* This method can be overridden by concrete commands.
*
* @return {@code true} if the {@link PluginManager} service must be initialized.
*/
protected boolean isPluginManagerEnabled() {
return true;
}
private static String message(String message, Object... format) {
@@ -157,7 +180,6 @@ abstract public class AbstractCommand implements Callable<Integer> {
logger.getName().startsWith("io.kestra") &&
!logger.getName().startsWith("io.kestra.ee.runner.kafka.services"))
)
|| logger.getName().startsWith("flow")
)
.forEach(
logger -> logger.setLevel(ch.qos.logback.classic.Level.valueOf(this.logLevel.name()))
@@ -183,9 +205,9 @@ abstract public class AbstractCommand implements Callable<Integer> {
if (this.endpointConfiguration.getPort().isPresent()) {
URI endpoint = null;
try {
endpoint = new URIBuilder(server.getURL().toURI())
.setPort(this.endpointConfiguration.getPort().get())
.setPath("/health")
endpoint = UriBuilder.of(server.getURL().toURI())
.port(this.endpointConfiguration.getPort().get())
.path("/health")
.build();
} catch (URISyntaxException e) {
e.printStackTrace();
@@ -207,10 +229,12 @@ abstract public class AbstractCommand implements Callable<Integer> {
return false;
}
protected void shutdownHook(Rethrow.RunnableChecked<Exception> run) {
protected void shutdownHook(boolean logShutdown, Rethrow.RunnableChecked<Exception> run) {
Runtime.getRuntime().addShutdownHook(new Thread(
() -> {
log.warn("Receiving shutdown ! Try to graceful exit");
if (logShutdown) {
log.warn("Receiving shutdown ! Try to graceful exit");
}
try {
run.run();
} catch (Exception e) {

View File

@@ -88,11 +88,12 @@ public class App implements Callable<Integer> {
.environments(Environment.CLI);
CommandLine cmd = new CommandLine(mainClass, CommandLine.defaultFactory());
continueOnParsingErrors(cmd);
CommandLine.ParseResult parseResult = cmd.parseArgs(args);
List<CommandLine> parsedCommands = parseResult.asCommandLineList();
CommandLine commandLine = parsedCommands.get(parsedCommands.size() - 1);
CommandLine commandLine = parsedCommands.getLast();
Class<?> cls = commandLine.getCommandSpec().userObject().getClass();
if (AbstractCommand.class.isAssignableFrom(cls)) {
@@ -114,15 +115,17 @@ public class App implements Callable<Integer> {
.stream()
.filter(argSpec -> ((Field) argSpec.userObject()).getName().equals("serverPort"))
.findFirst()
.ifPresent(argSpec -> {
properties.put("micronaut.server.port", argSpec.getValue());
});
.ifPresent(argSpec -> properties.put("micronaut.server.port", argSpec.getValue()));
builder.properties(properties);
}
return builder.build();
}
private static void continueOnParsingErrors(CommandLine cmd) {
cmd.getCommandSpec().parser().collectErrors(true);
}
@SuppressWarnings("unchecked")
private static <T> T getPropertiesFromMethod(Class<?> cls, String methodName, Object instance) {
try {

View File

@@ -33,6 +33,9 @@ public class FlowUpdatesCommand extends AbstractApiCommand {
@CommandLine.Option(names = {"--delete"}, negatable = true, description = "Whether missing should be deleted")
public boolean delete = false;
@CommandLine.Option(names = {"--namespace"}, description = "The parent namespace of the flows, if not set, every namespace are allowed.")
public String namespace;
@SuppressWarnings("deprecation")
@Override
public Integer call() throws Exception {
@@ -58,8 +61,12 @@ public class FlowUpdatesCommand extends AbstractApiCommand {
body = String.join("\n---\n", flows);
}
try(DefaultHttpClient client = client()) {
String namespaceQuery = "";
if (namespace != null) {
namespaceQuery = "&namespace=" + namespace;
}
MutableHttpRequest<String> request = HttpRequest
.POST(apiUri("/flows/bulk") + "?delete=" + delete, body).contentType(MediaType.APPLICATION_YAML);
.POST(apiUri("/flows/bulk") + "?allowNamespaceChild=true&delete=" + delete + namespaceQuery, body).contentType(MediaType.APPLICATION_YAML);
List<UpdateResult> updated = client.toBlocking().retrieve(
this.requestOptions(request),

View File

@@ -39,7 +39,7 @@ public class FlowValidateCommand extends AbstractValidateCommand {
Flow flow = (Flow) object;
List<String> warnings = new ArrayList<>();
warnings.addAll(flowService.deprecationPaths(flow).stream().map(deprecation -> deprecation + " is deprecated").toList());
warnings.addAll(flowService.warnings(flow));
warnings.addAll(flowService.warnings(flow, this.tenantId));
return warnings;
},
(Object object) -> {

View File

@@ -1,31 +1,36 @@
package io.kestra.cli.commands.plugins;
import io.micronaut.configuration.picocli.PicocliRunner;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import io.kestra.cli.AbstractCommand;
import io.kestra.cli.App;
import picocli.CommandLine;
import io.micronaut.configuration.picocli.PicocliRunner;
import lombok.SneakyThrows;
import picocli.CommandLine.Command;
@CommandLine.Command(
@Command(
name = "plugins",
description = "Manage plugins",
mixinStandardHelpOptions = true,
subcommands = {
PluginInstallCommand.class,
PluginListCommand.class,
PluginDocCommand.class
PluginDocCommand.class,
PluginSearchCommand.class
}
)
@Slf4j
public class PluginCommand extends AbstractCommand {
@SneakyThrows
@Override
public Integer call() throws Exception {
super.call();
PicocliRunner.call(App.class, "plugins", "--help");
PicocliRunner.call(App.class, "plugins", "--help");
return 0;
}
@Override
protected boolean loadExternalPlugins() {
return false;
}
}

View File

@@ -1,9 +1,9 @@
package io.kestra.cli.commands.plugins;
import com.google.common.base.Charsets;
import com.google.common.io.Files;
import io.kestra.cli.AbstractCommand;
import io.kestra.core.docs.DocumentationGenerator;
import io.kestra.core.plugins.PluginRegistry;
import io.kestra.core.plugins.RegisteredPlugin;
import io.kestra.core.serializers.JacksonMapper;
import io.micronaut.context.ApplicationContext;
@@ -43,8 +43,10 @@ public class PluginDocCommand extends AbstractCommand {
super.call();
DocumentationGenerator documentationGenerator = applicationContext.getBean(DocumentationGenerator.class);
List<RegisteredPlugin> plugins = core ? pluginRegistry().plugins() : pluginRegistry().externalPlugins();
PluginRegistry registry = pluginRegistryProvider.get();
List<RegisteredPlugin> plugins = core ? registry.plugins() : registry.externalPlugins();
boolean hasFailures = false;
for (RegisteredPlugin registeredPlugin : plugins) {
try {
documentationGenerator
@@ -61,7 +63,7 @@ public class PluginDocCommand extends AbstractCommand {
Files
.asCharSink(
file,
Charsets.UTF_8
StandardCharsets.UTF_8
).write(s.getBody());
stdOut("Generate doc in: {0}", file);
@@ -101,4 +103,10 @@ public class PluginDocCommand extends AbstractCommand {
return hasFailures ? 1 : 0;
}
/** {@inheritDoc} **/
@Override
protected boolean isPluginManagerEnabled() {
return false;
}
}

View File

@@ -1,98 +1,123 @@
package io.kestra.cli.commands.plugins;
import org.apache.commons.io.FilenameUtils;
import io.kestra.core.contexts.MavenPluginRepositoryConfig;
import io.kestra.core.plugins.LocalPluginManager;
import io.kestra.core.plugins.MavenPluginDownloader;
import io.kestra.core.plugins.PluginArtifact;
import io.kestra.core.plugins.PluginCatalogService;
import io.kestra.core.plugins.PluginManager;
import io.micronaut.http.client.HttpClient;
import io.micronaut.http.client.annotation.Client;
import io.micronaut.http.uri.UriBuilder;
import io.kestra.cli.AbstractCommand;
import io.kestra.cli.plugins.PluginDownloader;
import io.kestra.cli.plugins.RepositoryConfig;
import io.kestra.core.utils.IdUtils;
import org.apache.http.client.utils.URIBuilder;
import jakarta.inject.Provider;
import picocli.CommandLine;
import java.net.URI;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
import jakarta.inject.Inject;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;
import picocli.CommandLine.Option;
import picocli.CommandLine.Spec;
import static io.kestra.core.utils.Rethrow.throwConsumer;
@CommandLine.Command(
@Command(
name = "install",
description = "Install plugins"
)
public class PluginInstallCommand extends AbstractCommand {
@CommandLine.Parameters(index = "0..*", description = "Plugins to install. Represented as Maven artifact coordinates.")
@Option(names = {"--locally"}, description = "Specifies if plugins must be installed locally. If set to false the installation depends on your Kestra configuration.")
boolean locally = true;
@Option(names = {"--all"}, description = "Install all available plugins")
boolean all = false;
@Parameters(index = "0..*", description = "Plugins to install. Represented as Maven artifact coordinates (i.e., <groupId>:<artifactId>:(<version>|LATEST)")
List<String> dependencies = new ArrayList<>();
@CommandLine.Option(names = {"--repositories"}, description = "URL to additional Maven repositories")
@Option(names = {"--repositories"}, description = "URL to additional Maven repositories")
private URI[] repositories;
@CommandLine.Spec
@Spec
CommandLine.Model.CommandSpec spec;
@Inject
private PluginDownloader pluginDownloader;
Provider<MavenPluginDownloader> mavenPluginRepositoryProvider;
@Inject
@Client("api") HttpClient httpClient;
@Override
public Integer call() throws Exception {
super.call();
if (this.pluginsPath == null) {
if (this.locally && this.pluginsPath == null) {
throw new CommandLine.ParameterException(this.spec.commandLine(), "Missing required options '--plugins' " +
"or environment variable 'KESTRA_PLUGINS_PATH"
);
}
if (!pluginsPath.toFile().exists()) {
if (!pluginsPath.toFile().mkdir()) {
throw new RuntimeException("Cannot create directory: " + pluginsPath.toFile().getAbsolutePath());
}
}
List<MavenPluginRepositoryConfig> repositoryConfigs = List.of();
if (repositories != null) {
Arrays.stream(repositories)
.forEach(throwConsumer(s -> {
URIBuilder uriBuilder = new URIBuilder(s);
RepositoryConfig.RepositoryConfigBuilder builder = RepositoryConfig.builder()
repositoryConfigs = Arrays.stream(repositories)
.map(uri -> {
MavenPluginRepositoryConfig.MavenPluginRepositoryConfigBuilder builder = MavenPluginRepositoryConfig
.builder()
.id(IdUtils.create());
if (uriBuilder.getUserInfo() != null) {
int index = uriBuilder.getUserInfo().indexOf(":");
builder.basicAuth(new RepositoryConfig.BasicAuth(
uriBuilder.getUserInfo().substring(0, index),
uriBuilder.getUserInfo().substring(index + 1)
String userInfo = uri.getUserInfo();
if (userInfo != null) {
String[] userInfoParts = userInfo.split(":");
builder = builder.basicAuth(new MavenPluginRepositoryConfig.BasicAuth(
userInfoParts[0],
userInfoParts[1]
));
uriBuilder.setUserInfo(null);
}
builder.url(uriBuilder.build().toString());
pluginDownloader.addRepository(builder.build());
}));
builder.url(UriBuilder.of(uri).userInfo(null).build().toString());
return builder.build();
}).toList();
}
List<URL> resolveUrl = pluginDownloader.resolve(dependencies);
stdOut("Resolved Plugin(s) with {0}", resolveUrl);
if (all) {
PluginCatalogService service = new PluginCatalogService(httpClient, false, true);
dependencies = service.get().stream().map(Objects::toString).toList();
}
for (URL url: resolveUrl) {
Files.copy(
Paths.get(url.toURI()),
Paths.get(pluginsPath.toString(), FilenameUtils.getName(url.toString())),
StandardCopyOption.REPLACE_EXISTING
if (dependencies.isEmpty()) {
stdErr("Error: No plugin to install.");
return CommandLine.ExitCode.OK;
}
final List<PluginArtifact> pluginArtifacts;
try {
pluginArtifacts = dependencies.stream().map(PluginArtifact::fromCoordinates).toList();
} catch (IllegalArgumentException e) {
stdErr(e.getMessage());
return CommandLine.ExitCode.USAGE;
}
try (final PluginManager pluginManager = getPluginManager()) {
List<PluginArtifact> installed = pluginManager.install(
pluginArtifacts,
repositoryConfigs,
false,
pluginsPath
);
List<URI> uris = installed.stream().map(PluginArtifact::uri).toList();
stdOut("Successfully installed plugins {0} into {1}", dependencies, uris);
return CommandLine.ExitCode.OK;
}
}
stdOut("Successfully installed plugins {0} into {1}", dependencies, pluginsPath);
return 0;
private PluginManager getPluginManager() {
return locally ? new LocalPluginManager(mavenPluginRepositoryProvider.get()) : this.pluginManagerProvider.get();
}
@Override

View File

@@ -1,22 +1,31 @@
package io.kestra.cli.commands.plugins;
import io.kestra.cli.AbstractCommand;
import io.kestra.core.plugins.PluginRegistry;
import io.kestra.core.plugins.RegisteredPlugin;
import jakarta.inject.Inject;
import jakarta.inject.Provider;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Spec;
import java.util.List;
@CommandLine.Command(
@Command(
name = "list",
description = "List all plugins already installed"
)
public class PluginListCommand extends AbstractCommand {
@CommandLine.Spec
@Spec
CommandLine.Model.CommandSpec spec;
@CommandLine.Option(names = {"--core"}, description = "Also write core tasks plugins")
@Option(names = {"--core"}, description = "Also write core tasks plugins")
private boolean core = false;
@Inject
private PluginRegistry registry;
@Override
public Integer call() throws Exception {
super.call();
@@ -27,7 +36,8 @@ public class PluginListCommand extends AbstractCommand {
);
}
List<RegisteredPlugin> plugins = core ? pluginRegistry().plugins() : pluginRegistry().externalPlugins();
List<RegisteredPlugin> plugins = core ? registry.plugins() : registry.externalPlugins();
plugins.forEach(registeredPlugin -> stdOut(registeredPlugin.toString()));
return 0;

View File

@@ -0,0 +1,149 @@
package io.kestra.cli.commands.plugins;
import io.kestra.cli.AbstractCommand;
import io.micronaut.core.type.Argument;
import io.micronaut.http.HttpRequest;
import io.micronaut.http.client.HttpClient;
import io.micronaut.http.client.annotation.Client;
import jakarta.inject.Inject;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.ArrayList;
import java.util.List;
@Command(
name = "search",
description = "Search for available Kestra plugins"
)
public class PluginSearchCommand extends AbstractCommand {
@Inject
@Client("api")
private HttpClient httpClient;
private static final ObjectMapper MAPPER = new ObjectMapper();
private static final char SPACE = ' ';
@Parameters(index = "0", description = "Search term (optional)", defaultValue = "")
private String searchTerm;
@Override
public Integer call() throws Exception {
super.call();
try {
JsonNode root = fetchPlugins();
List<PluginInfo> plugins = findPlugins(root);
printResults(plugins);
return 0;
} catch (Exception e) {
stdOut("Error processing plugins: {0}", e.getMessage());
return 1;
}
}
private JsonNode fetchPlugins() throws Exception {
String response = httpClient.toBlocking()
.retrieve(
HttpRequest.GET("/v1/plugins")
.header("Accept", "application/json")
);
return MAPPER.readTree(response);
}
private List<PluginInfo> findPlugins(JsonNode root) {
String searchTermLower = searchTerm.toLowerCase();
List<PluginInfo> plugins = new ArrayList<>();
for (JsonNode plugin : root) {
if (matchesSearch(plugin, searchTermLower)) {
plugins.add(new PluginInfo(
plugin.path("name").asText(),
plugin.path("title").asText(),
plugin.path("group").asText(),
plugin.path("version").asText("")
));
}
}
plugins.sort((p1, p2) -> p1.name.compareToIgnoreCase(p2.name));
return plugins;
}
private boolean matchesSearch(JsonNode plugin, String term) {
if (term.isEmpty()) {
return true;
}
return plugin.path("name").asText().toLowerCase().contains(term) ||
plugin.path("title").asText().toLowerCase().contains(term) ||
plugin.path("group").asText().toLowerCase().contains(term);
}
private void printResults(List<PluginInfo> plugins) {
if (plugins.isEmpty()) {
stdOut("No plugins found{0}",
searchTerm.isEmpty() ? "" : " matching '" + searchTerm + "'");
return;
}
stdOut("\nFound {0} plugins{1}",
plugins.size(),
searchTerm.isEmpty() ? "" : " matching '" + searchTerm + "'"
);
printPluginsTable(plugins);
}
private void printPluginsTable(List<PluginInfo> plugins) {
int maxName = 4, maxTitle = 5, maxGroup = 5;
for (PluginInfo plugin : plugins) {
maxName = Math.max(maxName, plugin.name.length());
maxTitle = Math.max(maxTitle, plugin.title.length());
maxGroup = Math.max(maxGroup, plugin.group.length());
}
StringBuilder namePad = new StringBuilder(maxName);
StringBuilder titlePad = new StringBuilder(maxTitle);
StringBuilder groupPad = new StringBuilder(maxGroup);
stdOut("");
printRow(namePad, titlePad, groupPad, "NAME", "TITLE", "GROUP", "VERSION",
maxName, maxTitle, maxGroup);
for (PluginInfo plugin : plugins) {
printRow(namePad, titlePad, groupPad, plugin.name, plugin.title, plugin.group, plugin.version,
maxName, maxTitle, maxGroup);
}
stdOut("");
}
private void printRow(StringBuilder namePad, StringBuilder titlePad, StringBuilder groupPad,
String name, String title, String group, String version,
int maxName, int maxTitle, int maxGroup) {
stdOut("{0} {1} {2} {3}",
pad(namePad, name, maxName),
pad(titlePad, title, maxTitle),
pad(groupPad, group, maxGroup),
version
);
}
private String pad(StringBuilder sb, String str, int length) {
sb.setLength(0);
sb.append(str);
while (sb.length() < length) {
sb.append(SPACE);
}
return sb.toString();
}
private record PluginInfo(String name, String title, String group, String version) {}
@Override
protected boolean loadExternalPlugins() {
return false;
}
}

View File

@@ -0,0 +1,69 @@
package io.kestra.cli.commands.plugins;
import io.kestra.cli.AbstractCommand;
import io.kestra.core.plugins.LocalPluginManager;
import io.kestra.core.plugins.MavenPluginDownloader;
import io.kestra.core.plugins.PluginArtifact;
import io.kestra.core.plugins.PluginManager;
import jakarta.inject.Inject;
import jakarta.inject.Provider;
import picocli.CommandLine;
import picocli.CommandLine.Parameters;
import picocli.CommandLine.Spec;
import java.net.URI;
import java.util.ArrayList;
import java.util.List;
@CommandLine.Command(
name = "uninstall",
description = "uninstall a plugin"
)
public class PluginUninstallCommand extends AbstractCommand {
@Parameters(index = "0..*", description = "the plugins to uninstall")
List<String> dependencies = new ArrayList<>();
@Spec
CommandLine.Model.CommandSpec spec;
@Inject
Provider<MavenPluginDownloader> mavenPluginRepositoryProvider;
@Override
public Integer call() throws Exception {
super.call();
List<PluginArtifact> pluginArtifacts;
try {
pluginArtifacts = dependencies.stream().map(PluginArtifact::fromCoordinates).toList();
} catch (IllegalArgumentException e) {
stdErr(e.getMessage());
return CommandLine.ExitCode.USAGE;
}
final PluginManager pluginManager;
// If a PLUGIN_PATH is provided, then use the LocalPluginManager
if (pluginsPath != null) {
pluginManager = new LocalPluginManager(mavenPluginRepositoryProvider.get());
} else {
// Otherwise, we delegate to the configured plugin-manager.
pluginManager = this.pluginManagerProvider.get();
}
List<PluginArtifact> uninstalled = pluginManager.uninstall(
pluginArtifacts,
false,
pluginsPath
);
List<URI> uris = uninstalled.stream().map(PluginArtifact::uri).toList();
stdOut("Successfully uninstalled plugins {0} from {1}", dependencies, uris);
return CommandLine.ExitCode.OK;
}
@Override
protected boolean loadExternalPlugins() {
return false;
}
}

View File

@@ -1,12 +1,20 @@
package io.kestra.cli.commands.servers;
import io.kestra.cli.AbstractCommand;
import io.kestra.core.contexts.KestraContext;
import jakarta.annotation.PostConstruct;
import picocli.CommandLine;
abstract public class AbstractServerCommand extends AbstractCommand implements ServerCommandInterface {
@CommandLine.Option(names = {"--port"}, description = "The port to bind")
Integer serverPort;
@Override
public Integer call() throws Exception {
this.shutdownHook(true, () -> KestraContext.getContext().shutdown());
return super.call();
}
protected static int defaultWorkerThread() {
return Runtime.getRuntime().availableProcessors() * 4;
}

View File

@@ -1,7 +1,6 @@
package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.ExecutorInterface;
import io.kestra.core.services.SkipExecutionService;
@@ -9,7 +8,6 @@ import io.kestra.core.services.StartExecutorService;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;
import picocli.CommandLine;
import java.util.Collections;
@@ -20,7 +18,6 @@ import java.util.Map;
name = "executor",
description = "Start the Kestra executor"
)
@Slf4j
public class ExecutorCommand extends AbstractServerCommand {
@Inject
private ApplicationContext applicationContext;
@@ -66,13 +63,10 @@ public class ExecutorCommand extends AbstractServerCommand {
this.startExecutorService.applyOptions(startExecutors, notStartExecutors);
super.call();
this.shutdownHook(() -> KestraContext.getContext().shutdown());
ExecutorInterface executorService = applicationContext.getBean(ExecutorInterface.class);
executorService.run();
log.info("Executor started");
Await.until(() -> !this.applicationContext.isRunning());
return 0;

View File

@@ -1,13 +1,11 @@
package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.IndexerInterface;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;
import picocli.CommandLine;
import java.util.Map;
@@ -16,7 +14,6 @@ import java.util.Map;
name = "indexer",
description = "Start the Kestra indexer"
)
@Slf4j
public class IndexerCommand extends AbstractServerCommand {
@Inject
private ApplicationContext applicationContext;
@@ -31,13 +28,10 @@ public class IndexerCommand extends AbstractServerCommand {
@Override
public Integer call() throws Exception {
super.call();
this.shutdownHook(() -> KestraContext.getContext().shutdown());
IndexerInterface indexer = applicationContext.getBean(IndexerInterface.class);
indexer.run();
log.info("Indexer started");
Await.until(() -> !this.applicationContext.isRunning());
return 0;

View File

@@ -1,7 +1,6 @@
package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.schedulers.AbstractScheduler;
import io.kestra.core.utils.Await;
@@ -31,12 +30,10 @@ public class SchedulerCommand extends AbstractServerCommand {
@Override
public Integer call() throws Exception {
super.call();
this.shutdownHook(() -> KestraContext.getContext().shutdown());
AbstractScheduler scheduler = applicationContext.getBean(AbstractScheduler.class);
scheduler.run();
log.info("Scheduler started");
Await.until(() -> !this.applicationContext.isRunning());
return 0;

View File

@@ -2,7 +2,6 @@ package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.cli.services.FileChangedEventListener;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.repositories.LocalFlowRepositoryLoader;
import io.kestra.core.runners.StandAloneRunner;
@@ -12,7 +11,6 @@ import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.annotation.Nullable;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;
import picocli.CommandLine;
import java.io.File;
@@ -25,7 +23,6 @@ import java.util.Map;
name = "standalone",
description = "Start the standalone all-in-one server"
)
@Slf4j
public class StandAloneCommand extends AbstractServerCommand {
@CommandLine.Spec
CommandLine.Model.CommandSpec spec;
@@ -95,7 +92,6 @@ public class StandAloneCommand extends AbstractServerCommand {
this.startExecutorService.applyOptions(startExecutors, notStartExecutors);
super.call();
this.shutdownHook(() -> KestraContext.getContext().shutdown());
if (flowPath != null) {
try {
@@ -124,8 +120,6 @@ public class StandAloneCommand extends AbstractServerCommand {
fileWatcher.startListeningFromConfig();
}
this.shutdownHook(standAloneRunner::close);
Await.until(() -> !this.applicationContext.isRunning());
return 0;

View File

@@ -1,9 +1,7 @@
package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.ExecutorInterface;
import io.kestra.core.runners.IndexerInterface;
import io.kestra.core.utils.Await;
import io.kestra.core.utils.ExecutorsUtils;
@@ -57,20 +55,11 @@ public class WebServerCommand extends AbstractServerCommand {
log.info("Starting an embedded indexer, this can be disabled by using `--no-indexer`.");
poolExecutor = executorsUtils.cachedThreadPool("webserver-indexer");
poolExecutor.execute(applicationContext.getBean(IndexerInterface.class));
shutdownHook(false, () -> poolExecutor.shutdown());
}
log.info("Webserver started");
this.shutdownHook(() -> {
this.close();
KestraContext.getContext().shutdown();
});
Await.until(() -> !this.applicationContext.isRunning());
return 0;
}
private void close() {
if (this.poolExecutor != null) {
this.poolExecutor.shutdown();
}
}
}

View File

@@ -1,13 +1,11 @@
package io.kestra.cli.commands.servers;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.contexts.KestraContext;
import io.kestra.core.models.ServerType;
import io.kestra.core.runners.Worker;
import io.kestra.core.utils.Await;
import io.micronaut.context.ApplicationContext;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;
import picocli.CommandLine;
import picocli.CommandLine.Option;
@@ -18,7 +16,6 @@ import java.util.UUID;
name = "worker",
description = "Start the Kestra worker"
)
@Slf4j
public class WorkerCommand extends AbstractServerCommand {
@Inject
@@ -40,7 +37,6 @@ public class WorkerCommand extends AbstractServerCommand {
@Override
public Integer call() throws Exception {
super.call();
this.shutdownHook(() -> KestraContext.getContext().shutdown());
if (this.workerGroupKey != null && !this.workerGroupKey.matches("[a-zA-Z0-9_-]+")) {
throw new IllegalArgumentException("The --worker-group option must match the [a-zA-Z0-9_-]+ pattern");
}
@@ -52,13 +48,6 @@ public class WorkerCommand extends AbstractServerCommand {
worker.run();
if (this.workerGroupKey != null) {
log.info("Worker started with {} thread(s) in group '{}'", this.thread, this.workerGroupKey);
}
else {
log.info("Worker started with {} thread(s)", this.thread);
}
Await.until(() -> !this.applicationContext.isRunning());
return 0;

View File

@@ -1,153 +0,0 @@
package io.kestra.cli.plugins;
import com.google.common.collect.ImmutableList;
import io.micronaut.context.annotation.Value;
import io.micronaut.core.annotation.Nullable;
import jakarta.inject.Inject;
import jakarta.inject.Singleton;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;
import org.apache.maven.repository.internal.MavenRepositorySystemUtils;
import org.eclipse.aether.DefaultRepositorySystemSession;
import org.eclipse.aether.RepositorySystem;
import org.eclipse.aether.RepositorySystemSession;
import org.eclipse.aether.artifact.Artifact;
import org.eclipse.aether.artifact.DefaultArtifact;
import org.eclipse.aether.connector.basic.BasicRepositoryConnectorFactory;
import org.eclipse.aether.impl.DefaultServiceLocator;
import org.eclipse.aether.repository.LocalRepository;
import org.eclipse.aether.repository.RemoteRepository;
import org.eclipse.aether.resolution.*;
import org.eclipse.aether.spi.connector.RepositoryConnectorFactory;
import org.eclipse.aether.spi.connector.transport.TransporterFactory;
import org.eclipse.aether.transport.file.FileTransporterFactory;
import org.eclipse.aether.transport.http.HttpTransporterFactory;
import org.eclipse.aether.util.repository.AuthenticationBuilder;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
@Singleton
@Slf4j
public class PluginDownloader {
private final List<RepositoryConfig> repositoryConfigs;
private final RepositorySystem system;
private final RepositorySystemSession session;
@Inject
public PluginDownloader(
List<RepositoryConfig> repositoryConfigs,
@Nullable @Value("${kestra.plugins.local-repository-path}") String localRepositoryPath
) {
this.repositoryConfigs = repositoryConfigs;
this.system = repositorySystem();
this.session = repositorySystemSession(system, localRepositoryPath);
}
public void addRepository(RepositoryConfig repositoryConfig) {
this.repositoryConfigs.add(repositoryConfig);
}
public List<URL> resolve(List<String> dependencies) throws MalformedURLException, ArtifactResolutionException, VersionRangeResolutionException {
List<RemoteRepository> repositories = remoteRepositories();
List<ArtifactResult> artifactResults = resolveArtifacts(repositories, dependencies);
List<URL> localUrls = resolveUrls(artifactResults);
log.debug("Resolved Plugin {} with {}", dependencies, localUrls);
return localUrls;
}
private List<RemoteRepository> remoteRepositories() {
return repositoryConfigs
.stream()
.map(repositoryConfig -> {
var build = new RemoteRepository.Builder(
repositoryConfig.getId(),
"default",
repositoryConfig.getUrl()
);
if (repositoryConfig.getBasicAuth() != null) {
var authenticationBuilder = new AuthenticationBuilder();
authenticationBuilder.addUsername(repositoryConfig.getBasicAuth().getUsername());
authenticationBuilder.addPassword(repositoryConfig.getBasicAuth().getPassword());
build.setAuthentication(authenticationBuilder.build());
}
return build.build();
})
.toList();
}
private static RepositorySystem repositorySystem() {
DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator();
locator.addService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class);
locator.addService(TransporterFactory.class, FileTransporterFactory.class);
locator.addService(TransporterFactory.class, HttpTransporterFactory.class);
return locator.getService(RepositorySystem.class);
}
private RepositorySystemSession repositorySystemSession(RepositorySystem system, String localRepositoryPath) {
DefaultRepositorySystemSession session = MavenRepositorySystemUtils.newSession();
if (localRepositoryPath == null) {
try {
final String tempDirectory = Files.createTempDirectory(this.getClass().getSimpleName().toLowerCase())
.toAbsolutePath()
.toString();
localRepositoryPath = tempDirectory;
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
try {
FileUtils.deleteDirectory(new File(tempDirectory));
} catch (IOException e) {
throw new RuntimeException(e);
}
}));
} catch (IOException e) {
throw new RuntimeException(e);
}
}
LocalRepository localRepo = new LocalRepository(localRepositoryPath);
session.setLocalRepositoryManager(system.newLocalRepositoryManager(session, localRepo));
return session;
}
private List<ArtifactResult> resolveArtifacts(List<RemoteRepository> repositories, List<String> dependencies) throws ArtifactResolutionException, VersionRangeResolutionException {
List<ArtifactResult> results = new ArrayList<>(dependencies.size());
for (String dependency: dependencies) {
var artifact = new DefaultArtifact(dependency);
var version = system.resolveVersionRange(session, new VersionRangeRequest(artifact, repositories, null));
var artifactRequest = new ArtifactRequest(
new DefaultArtifact(artifact.getGroupId(), artifact.getArtifactId(), "jar", version.getHighestVersion().toString()),
repositories,
null
);
var artifactResult = system.resolveArtifact(session, artifactRequest);
results.add(artifactResult);
}
return results;
}
private List<URL> resolveUrls(List<ArtifactResult> artifactResults) throws MalformedURLException {
ImmutableList.Builder<URL> urls = ImmutableList.builder();
for (ArtifactResult artifactResult : artifactResults) {
URL url;
url = artifactResult.getArtifact().getFile().toPath().toUri().toURL();
urls.add(url);
}
return urls.build();
}
}

View File

@@ -1,30 +0,0 @@
package io.kestra.cli.plugins;
import io.micronaut.context.annotation.EachProperty;
import io.micronaut.context.annotation.Parameter;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
@EachProperty("kestra.plugins.repositories")
@Getter
@AllArgsConstructor
@Builder
public class RepositoryConfig {
String id;
String url;
BasicAuth basicAuth;
@Getter
@AllArgsConstructor
public static class BasicAuth {
private String username;
private String password;
}
public RepositoryConfig(@Parameter String id) {
this.id = id;
}
}

View File

@@ -7,6 +7,7 @@ import io.kestra.core.models.validations.ModelValidator;
import io.kestra.core.repositories.FlowRepositoryInterface;
import io.kestra.core.serializers.YamlParser;
import io.kestra.core.services.FlowListenersInterface;
import io.kestra.core.services.PluginDefaultService;
import io.micronaut.context.annotation.Requires;
import io.micronaut.context.annotation.Value;
import io.micronaut.scheduling.io.watch.FileWatchConfiguration;
@@ -36,6 +37,9 @@ public class FileChangedEventListener {
@Inject
private FlowRepositoryInterface flowRepositoryInterface;
@Inject
private PluginDefaultService pluginDefaultService;
@Inject
private YamlParser yamlParser;
@@ -64,7 +68,7 @@ public class FileChangedEventListener {
public void startListeningFromConfig() throws IOException, InterruptedException {
if (fileWatchConfiguration != null && fileWatchConfiguration.isEnabled()) {
this.flowFilesManager = new LocalFlowFileWatcher(flowRepositoryInterface);
this.flowFilesManager = new LocalFlowFileWatcher(flowRepositoryInterface, pluginDefaultService);
List<Path> paths = fileWatchConfiguration.getPaths();
this.setup(paths);
@@ -107,7 +111,6 @@ public class FileChangedEventListener {
} else {
log.info("File watching is disabled.");
}
}
public void startListening(List<Path> paths) throws IOException, InterruptedException {
@@ -118,60 +121,64 @@ public class FileChangedEventListener {
WatchKey key;
while ((key = watchService.take()) != null) {
for (WatchEvent<?> watchEvent : key.pollEvents()) {
WatchEvent.Kind<?> kind = watchEvent.kind();
Path entry = (Path) watchEvent.context();
try {
WatchEvent.Kind<?> kind = watchEvent.kind();
Path entry = (Path) watchEvent.context();
if (entry.toString().endsWith(".yml") || entry.toString().endsWith(".yaml")) {
if (entry.toString().endsWith(".yml") || entry.toString().endsWith(".yaml")) {
if (kind == StandardWatchEventKinds.ENTRY_CREATE || kind == StandardWatchEventKinds.ENTRY_MODIFY) {
if (kind == StandardWatchEventKinds.ENTRY_CREATE || kind == StandardWatchEventKinds.ENTRY_MODIFY) {
Path filePath = ((Path) key.watchable()).resolve(entry);
if (Files.isDirectory(filePath)) {
loadFlowsFromFolder(filePath);
} else {
Path filePath = ((Path) key.watchable()).resolve(entry);
if (Files.isDirectory(filePath)) {
loadFlowsFromFolder(filePath);
} else {
try {
String content = Files.readString(filePath, Charset.defaultCharset());
try {
String content = Files.readString(filePath, Charset.defaultCharset());
Optional<Flow> flow = parseFlow(content, entry);
if (flow.isPresent()) {
if (kind == StandardWatchEventKinds.ENTRY_MODIFY) {
// Check if we already have a file with the given path
if (flows.stream().anyMatch(flowWithPath -> flowWithPath.getPath().equals(filePath.toString()))) {
Optional<FlowWithPath> previous = flows.stream().filter(flowWithPath -> flowWithPath.getPath().equals(filePath.toString())).findFirst();
// Check if Flow from file has id/namespace updated
if (previous.isPresent() && !previous.get().uidWithoutRevision().equals(flow.get().uidWithoutRevision())) {
flows.removeIf(flowWithPath -> flowWithPath.getPath().equals(filePath.toString()));
flowFilesManager.deleteFlow(previous.get().getTenantId(), previous.get().getNamespace(), previous.get().getId());
Optional<Flow> flow = parseFlow(content, entry);
if (flow.isPresent()) {
if (kind == StandardWatchEventKinds.ENTRY_MODIFY) {
// Check if we already have a file with the given path
if (flows.stream().anyMatch(flowWithPath -> flowWithPath.getPath().equals(filePath.toString()))) {
Optional<FlowWithPath> previous = flows.stream().filter(flowWithPath -> flowWithPath.getPath().equals(filePath.toString())).findFirst();
// Check if Flow from file has id/namespace updated
if (previous.isPresent() && !previous.get().uidWithoutRevision().equals(flow.get().uidWithoutRevision())) {
flows.removeIf(flowWithPath -> flowWithPath.getPath().equals(filePath.toString()));
flowFilesManager.deleteFlow(previous.get().getTenantId(), previous.get().getNamespace(), previous.get().getId());
flows.add(FlowWithPath.of(flow.get(), filePath.toString()));
}
} else {
flows.add(FlowWithPath.of(flow.get(), filePath.toString()));
}
} else {
flows.add(FlowWithPath.of(flow.get(), filePath.toString()));
}
} else {
flows.add(FlowWithPath.of(flow.get(), filePath.toString()));
flowFilesManager.createOrUpdateFlow(flow.get(), content);
log.info("Flow {} from file {} has been created or modified", flow.get().getId(), entry);
}
flowFilesManager.createOrUpdateFlow(flow.get(), content);
log.info("Flow {} from file {} has been created or modified", flow.get().getId(), entry);
} catch (NoSuchFileException e) {
log.error("File not found: {}", entry, e);
} catch (IOException e) {
log.error("Error reading file: {}", entry, e);
}
} catch (NoSuchFileException e) {
log.error("File not found: {}", entry, e);
} catch (IOException e) {
log.error("Error reading file: {}", entry, e);
}
} else {
Path filePath = ((Path) key.watchable()).resolve(entry);
flows.stream()
.filter(flow -> flow.getPath().equals(filePath.toString()))
.findFirst()
.ifPresent(flowWithPath -> {
flowFilesManager.deleteFlow(flowWithPath.getTenantId(), flowWithPath.getNamespace(), flowWithPath.getId());
this.flows.removeIf(fwp -> fwp.uidWithoutRevision().equals(flowWithPath.uidWithoutRevision()));
});
}
} else {
Path filePath = ((Path) key.watchable()).resolve(entry);
flows.stream()
.filter(flow -> flow.getPath().equals(filePath.toString()))
.findFirst()
.ifPresent(flowWithPath -> {
flowFilesManager.deleteFlow(flowWithPath.getTenantId(), flowWithPath.getNamespace(), flowWithPath.getId());
this.flows.removeIf(fwp -> fwp.uidWithoutRevision().equals(flowWithPath.uidWithoutRevision()));
});
}
} catch (Exception e) {
log.error("Unexpected error while watching flows", e);
}
}
key.reset();
@@ -230,7 +237,8 @@ public class FileChangedEventListener {
private Optional<Flow> parseFlow(String content, Path entry) {
try {
Flow flow = yamlParser.parse(content, Flow.class);
modelValidator.validate(flow);
FlowWithSource withPluginDefault = pluginDefaultService.injectDefaults(FlowWithSource.of(flow, content));
modelValidator.validate(withPluginDefault);
return Optional.of(flow);
} catch (ConstraintViolationException e) {
log.warn("Error while parsing flow: {}", entry, e);

View File

@@ -3,32 +3,36 @@ package io.kestra.cli.services;
import io.kestra.core.models.flows.Flow;
import io.kestra.core.models.flows.FlowWithSource;
import io.kestra.core.repositories.FlowRepositoryInterface;
import io.micronaut.context.annotation.Requires;
import io.kestra.core.services.PluginDefaultService;
import lombok.extern.slf4j.Slf4j;
@Requires(property = "micronaut.io.watch.enabled", value = "true")
@Slf4j
public class LocalFlowFileWatcher implements FlowFilesManager {
private FlowRepositoryInterface flowRepositoryInterface;
private final FlowRepositoryInterface flowRepository;
private final PluginDefaultService pluginDefaultService;
public LocalFlowFileWatcher(FlowRepositoryInterface flowRepositoryInterface) {
this.flowRepositoryInterface = flowRepositoryInterface;
public LocalFlowFileWatcher(FlowRepositoryInterface flowRepository, PluginDefaultService pluginDefaultService) {
this.flowRepository = flowRepository;
this.pluginDefaultService = pluginDefaultService;
}
@Override
public FlowWithSource createOrUpdateFlow(Flow flow, String content) {
return flowRepositoryInterface.findById(null, flow.getNamespace(), flow.getId())
.map(previous -> flowRepositoryInterface.update(flow, previous, content, flow))
.orElseGet(() -> flowRepositoryInterface.create(flow, content, flow));
FlowWithSource withDefault = pluginDefaultService.injectDefaults(FlowWithSource.of(flow, content));
return flowRepository.findById(null, flow.getNamespace(), flow.getId())
.map(previous -> flowRepository.update(flow, previous, content, withDefault))
.orElseGet(() -> flowRepository.create(flow, content, withDefault));
}
@Override
public void deleteFlow(FlowWithSource toDelete) {
flowRepositoryInterface.findByIdWithSource(toDelete.getTenantId(), toDelete.getNamespace(), toDelete.getId()).ifPresent(flowRepositoryInterface::delete);
log.error("Flow {} has been deleted", toDelete.getId());
flowRepository.findByIdWithSource(toDelete.getTenantId(), toDelete.getNamespace(), toDelete.getId()).ifPresent(flowRepository::delete);
log.info("Flow {} has been deleted", toDelete.getId());
}
@Override
public void deleteFlow(String tenantId, String namespace, String id) {
flowRepositoryInterface.findByIdWithSource(tenantId, namespace, id).ifPresent(flowRepositoryInterface::delete);
log.error("Flow {} has been deleted", id);
flowRepository.findByIdWithSource(tenantId, namespace, id).ifPresent(flowRepository::delete);
log.info("Flow {} has been deleted", id);
}
}

View File

@@ -1,6 +1,9 @@
micronaut:
application:
name: kestra
# Disable Micronaut Open Telemetry
otel:
enabled: false
router:
static-resources:
swagger:
@@ -71,6 +74,13 @@ micronaut:
type: scheduled
core-pool-size: 1
# Disable OpenTelemetry metrics by default, users that need it must enable it and configure the collector URL.
metrics:
export:
otlp:
enabled: false
# url: http://localhost:4318/v1/metrics
jackson:
serialization:
writeDatesAsTimestamps: false
@@ -135,6 +145,11 @@ kestra:
initial-delay: 1h
fixed-delay: 1h
retention: 7d
types:
- type : io.kestra.core.models.executions.LogEntry
retention: 1h
- type: io.kestra.core.models.executions.MetricEntry
retention: 1h
plugins:
repositories:

View File

@@ -45,4 +45,20 @@ class AppTest {
assertThat(out.toString(), startsWith("Usage: kestra server " + serverType));
}
}
@Test
void missingRequiredParamsPrintHelpInsteadOfException() {
final ByteArrayOutputStream out = new ByteArrayOutputStream();
System.setErr(new PrintStream(out));
final String[] argsWithMissingParams = new String[]{"flow", "namespace", "update"};
try (ApplicationContext ctx = App.applicationContext(App.class, argsWithMissingParams)) {
new CommandLine(App.class, new MicronautFactory(ctx)).execute(argsWithMissingParams);
assertThat(out.toString(), startsWith("Missing required parameters: "));
assertThat(out.toString(), containsString("Usage: kestra flow namespace update "));
assertThat(out.toString(), not(containsString("MissingParameterException: ")));
}
}
}

View File

@@ -5,6 +5,7 @@ import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.Environment;
import io.micronaut.runtime.server.EmbeddedServer;
import org.junit.jupiter.api.Test;
import org.junitpioneer.jupiter.RetryingTest;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
@@ -15,7 +16,7 @@ import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.StringContains.containsString;
class FlowCreateOrUpdateCommandTest {
@Test
@RetryingTest(5) // flaky on CI but cannot be reproduced even with 100 repetitions
void runWithDelete() {
URL directory = FlowCreateOrUpdateCommandTest.class.getClassLoader().getResource("flows");
ByteArrayOutputStream out = new ByteArrayOutputStream();

View File

@@ -109,6 +109,33 @@ class FlowUpdatesCommandTest {
}
}
@Test
void invalidWithNamespace() {
URL directory = FlowUpdatesCommandTest.class.getClassLoader().getResource("flows");
ByteArrayOutputStream out = new ByteArrayOutputStream();
System.setErr(new PrintStream(out));
try (ApplicationContext ctx = ApplicationContext.run(Environment.CLI, Environment.TEST)) {
EmbeddedServer embeddedServer = ctx.getBean(EmbeddedServer.class);
embeddedServer.start();
String[] args = {
"--server",
embeddedServer.getURL().toString(),
"--user",
"myuser:pass:word",
"--namespace",
"io.kestra.cli",
"--delete",
directory.getPath(),
};
PicocliRunner.call(FlowUpdatesCommand.class, ctx, args);
assertThat(out.toString(), containsString("Invalid entity: flow.namespace: io.kestra.outsider_quattro_-1 - flow namespace is invalid"));
}
}
@Test
void helper() {
URL directory = FlowUpdatesCommandTest.class.getClassLoader().getResource("helper");

View File

@@ -0,0 +1,27 @@
package io.kestra.cli.commands.plugins;
import io.micronaut.configuration.picocli.PicocliRunner;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.Environment;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.StringContains.containsString;
class PluginCommandTest {
@Test
void shouldGetHelps() {
ByteArrayOutputStream out = new ByteArrayOutputStream();
System.setOut(new PrintStream(out));
try (ApplicationContext ctx = ApplicationContext.run(Environment.CLI, Environment.TEST)) {
PicocliRunner.call(PluginCommand.class, ctx);
assertThat(out.toString(), containsString("Usage: kestra plugins"));
}
}
}

View File

@@ -9,7 +9,6 @@ import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.stream.Collectors;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;
@@ -17,7 +16,7 @@ import static org.hamcrest.Matchers.*;
class PluginInstallCommandTest {
@Test
void fixedVersion() throws IOException {
void shouldInstallPluginLocallyGivenFixedVersion() throws IOException {
Path pluginsPath = Files.createTempDirectory(PluginInstallCommandTest.class.getSimpleName());
pluginsPath.toFile().deleteOnExit();
@@ -28,12 +27,12 @@ class PluginInstallCommandTest {
List<Path> files = Files.list(pluginsPath).toList();
assertThat(files.size(), is(1));
assertThat(files.getFirst().getFileName().toString(), is("plugin-notifications-0.6.0.jar"));
assertThat(files.getFirst().getFileName().toString(), is("io_kestra_plugin__plugin-notifications__0_6_0.jar"));
}
}
@Test
void latestVersion() throws IOException {
void shouldInstallPluginLocallyGivenLatestVersion() throws IOException {
Path pluginsPath = Files.createTempDirectory(PluginInstallCommandTest.class.getSimpleName());
pluginsPath.toFile().deleteOnExit();
@@ -44,13 +43,13 @@ class PluginInstallCommandTest {
List<Path> files = Files.list(pluginsPath).toList();
assertThat(files.size(), is(1));
assertThat(files.getFirst().getFileName().toString(), startsWith("plugin-notifications"));
assertThat(files.getFirst().getFileName().toString(), startsWith("io_kestra_plugin__plugin-notifications__"));
assertThat(files.getFirst().getFileName().toString(), not(containsString("LATEST")));
}
}
@Test
void rangeVersion() throws IOException {
void shouldInstallPluginLocallyGivenRangeVersion() throws IOException {
Path pluginsPath = Files.createTempDirectory(PluginInstallCommandTest.class.getSimpleName());
pluginsPath.toFile().deleteOnExit();
@@ -62,7 +61,7 @@ class PluginInstallCommandTest {
List<Path> files = Files.list(pluginsPath).toList();
assertThat(files.size(), is(1));
assertThat(files.getFirst().getFileName().toString(), is("storage-s3-0.12.1.jar"));
assertThat(files.getFirst().getFileName().toString(), is("io_kestra_storage__storage-s3__0_12_1.jar"));
}
}
}

View File

@@ -4,7 +4,6 @@ import io.micronaut.configuration.picocli.PicocliRunner;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.Environment;
import org.apache.commons.io.FileUtils;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayOutputStream;
@@ -25,7 +24,7 @@ class PluginListCommandTest {
private static final String PLUGIN_TEMPLATE_TEST = "plugin-template-test-0.18.0-SNAPSHOT.jar";
@Test
void run() throws IOException, URISyntaxException {
void shouldListPluginsInstalledLocally() throws IOException, URISyntaxException {
Path pluginsPath = Files.createTempDirectory(PluginListCommandTest.class.getSimpleName());
pluginsPath.toFile().deleteOnExit();

View File

@@ -0,0 +1,105 @@
package io.kestra.cli.commands.plugins;
import com.github.tomakehurst.wiremock.junit5.WireMockTest;
import io.micronaut.configuration.picocli.PicocliRunner;
import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.Environment;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import java.util.Map;
import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;
@WireMockTest(httpPort = 28181)
class PluginSearchCommandTest {
private ByteArrayOutputStream outputStreamCaptor;
private final PrintStream originalOut = System.out;
@BeforeEach
void setUp() {
outputStreamCaptor = new ByteArrayOutputStream();
System.setOut(new PrintStream(outputStreamCaptor));
}
@AfterEach
void tearDown() {
System.setOut(originalOut);
}
@Test
void searchWithExactMatch() {
stubFor(get(urlEqualTo("/v1/plugins"))
.willReturn(aResponse()
.withHeader("Content-Type", "application/json")
.withBody("""
[
{
"name": "plugin-notifications",
"title": "Notifications",
"group": "io.kestra.plugin",
"version": "0.6.0"
},
{
"name": "plugin-scripts",
"title": "Scripts",
"group": "io.kestra.plugin",
"version": "0.5.0"
}
]
""")));
try (ApplicationContext ctx = ApplicationContext.builder(Environment.CLI, Environment.TEST)
.properties(Map.of("micronaut.http.services.api.url", "http://localhost:28181"))
.start()) {
String[] args = {"notifications"};
PicocliRunner.call(PluginSearchCommand.class, ctx, args);
String output = outputStreamCaptor.toString().trim();
assertThat(output, containsString("Found 1 plugins matching 'notifications'"));
assertThat(output, containsString("plugin-notifications"));
assertThat(output, not(containsString("plugin-scripts")));
}
}
@Test
void searchWithEmptyQuery() {
stubFor(get(urlEqualTo("/v1/plugins"))
.willReturn(aResponse()
.withHeader("Content-Type", "application/json")
.withBody("""
[
{
"name": "plugin-notifications",
"title": "Notifications",
"group": "io.kestra.plugin",
"version": "0.6.0"
},
{
"name": "plugin-scripts",
"title": "Scripts",
"group": "io.kestra.plugin",
"version": "0.5.0"
}
]
""")));
try (ApplicationContext ctx = ApplicationContext.builder(Environment.CLI, Environment.TEST)
.properties(Map.of("micronaut.http.services.api.url", "http://localhost:28181"))
.start()) {
String[] args = {""};
PicocliRunner.call(PluginSearchCommand.class, ctx, args);
String output = outputStreamCaptor.toString().trim();
assertThat(output, containsString("Found 2 plugins"));
assertThat(output, containsString("plugin-notifications"));
assertThat(output, containsString("plugin-scripts"));
}
}
}

View File

@@ -0,0 +1,131 @@
package io.kestra.cli.services;
import io.kestra.core.models.flows.Flow;
import io.kestra.core.repositories.FlowRepositoryInterface;
import io.kestra.core.utils.Await;
import io.micronaut.test.extensions.junit5.annotation.MicronautTest;
import jakarta.inject.Inject;
import org.apache.commons.io.FileUtils;
import org.junit.jupiter.api.*;
import org.junitpioneer.jupiter.RetryingTest;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import static io.kestra.core.utils.Rethrow.throwRunnable;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;
@MicronautTest(environments = {"test", "file-watch"}, transactional = false)
class FileChangedEventListenerTest {
public static final String FILE_WATCH = "build/file-watch";
@Inject
private FileChangedEventListener fileWatcher;
@Inject
private FlowRepositoryInterface flowRepository;
private final ExecutorService executorService = Executors.newSingleThreadExecutor();
private final AtomicBoolean started = new AtomicBoolean(false);
@BeforeAll
static void setup() throws IOException {
if (!Files.exists(Path.of(FILE_WATCH))) {
Files.createDirectories(Path.of(FILE_WATCH));
}
}
@AfterAll
static void tearDown() throws IOException {
if (Files.exists(Path.of(FILE_WATCH))) {
FileUtils.deleteDirectory(Path.of(FILE_WATCH).toFile());
}
}
@BeforeEach
void beforeEach() throws Exception {
if (started.compareAndSet(false, true)) {
executorService.execute(throwRunnable(() -> fileWatcher.startListeningFromConfig()));
}
}
@RetryingTest(5) // Flaky on CI but always pass locally
void test() throws IOException, TimeoutException {
// remove the flow if it already exists
flowRepository.findByIdWithSource(null, "io.kestra.tests.watch", "myflow").ifPresent(flow -> flowRepository.delete(flow));
// create a basic flow
String flow = """
id: myflow
namespace: io.kestra.tests.watch
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World! 🚀
""";
Files.write(Path.of(FILE_WATCH + "/myflow.yaml"), flow.getBytes());
Await.until(
() -> flowRepository.findById(null, "io.kestra.tests.watch", "myflow").isPresent(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
Flow myflow = flowRepository.findById(null, "io.kestra.tests.watch", "myflow").orElseThrow();
assertThat(myflow.getTasks(), hasSize(1));
assertThat(myflow.getTasks().getFirst().getId(), is("hello"));
assertThat(myflow.getTasks().getFirst().getType(), is("io.kestra.plugin.core.log.Log"));
// delete the flow
Files.delete(Path.of(FILE_WATCH + "/myflow.yaml"));
Await.until(
() -> flowRepository.findById(null, "io.kestra.tests.watch", "myflow").isEmpty(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
}
@RetryingTest(5) // Flaky on CI but always pass locally
void testWithPluginDefault() throws IOException, TimeoutException {
// remove the flow if it already exists
flowRepository.findByIdWithSource(null, "io.kestra.tests.watch", "pluginDefault").ifPresent(flow -> flowRepository.delete(flow));
// create a flow with plugin default
String pluginDefault = """
id: pluginDefault
namespace: io.kestra.tests.watch
tasks:
- id: helloWithDefault
type: io.kestra.plugin.core.log.Log
pluginDefaults:
- type: io.kestra.plugin.core.log.Log
values:
message: Hello World!
""";
Files.write(Path.of(FILE_WATCH + "/plugin-default.yaml"), pluginDefault.getBytes());
Await.until(
() -> flowRepository.findById(null, "io.kestra.tests.watch", "pluginDefault").isPresent(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
Flow pluginDefaultFlow = flowRepository.findById(null, "io.kestra.tests.watch", "pluginDefault").orElseThrow();
assertThat(pluginDefaultFlow.getTasks(), hasSize(1));
assertThat(pluginDefaultFlow.getTasks().getFirst().getId(), is("helloWithDefault"));
assertThat(pluginDefaultFlow.getTasks().getFirst().getType(), is("io.kestra.plugin.core.log.Log"));
// delete both files
Files.delete(Path.of(FILE_WATCH + "/plugin-default.yaml"));
Await.until(
() -> flowRepository.findById(null, "io.kestra.tests.watch", "pluginDefault").isEmpty(),
Duration.ofMillis(100),
Duration.ofSeconds(10)
);
}
}

View File

@@ -0,0 +1,12 @@
micronaut:
io:
watch:
enabled: true
paths:
- build/file-watch
kestra:
repository:
type: memory
queue:
type: memory

76
codecov.yml Normal file
View File

@@ -0,0 +1,76 @@
component_management:
individual_components:
- component_id: cli
name: Cli
paths:
- cli/**
- component_id: core
name: Core
paths:
- core/**
- component_id: e2e-tests
name: End to End
paths:
- e2e-tests/**
- component_id: jdbc
name: Jdbc
paths:
- jdbc/**
- component_id: jdbc-h2
name: Jdbc H2
paths:
- jdbc-h2/**
- component_id: jdbc-mysql
name: Jdbc Mysql
paths:
- jdbc-mysql/**
- component_id: jdbc-postgres
name: Jdbc Postgres
paths:
- jdbc-postgres/**
- component_id: model
name: Model
paths:
- model/**
- component_id: processor
name: Processor
paths:
- processor/**
- component_id: repository-memory
name: Repository Memory
paths:
- repository-memory/**
- component_id: runner-memory
name: Runner Memory
paths:
- runner-memory/**
- component_id: script
name: Script
paths:
- script/**
- component_id: storage-local
name: Storage Local
paths:
- storage-local/**
- component_id: tests
name: Tests
paths:
- tests/**
- component_id: ui
name: Ui
paths:
- ui/**
- component_id: webserver
name: Webserver
paths:
- webserver/**
flag_management:
default_rules:
carryforward: true
statuses:
- type: project
target: 80%
threshold: 1%
- type: patch
target: 90%

View File

@@ -38,6 +38,13 @@ dependencies {
implementation group: 'dev.failsafe', name: 'failsafe'
api 'org.apache.httpcomponents.client5:httpclient5'
// plugins
implementation 'org.apache.maven.resolver:maven-resolver-impl'
implementation 'org.apache.maven.resolver:maven-resolver-supplier'
implementation 'org.apache.maven.resolver:maven-resolver-connector-basic'
implementation 'org.apache.maven.resolver:maven-resolver-transport-file'
implementation 'org.apache.maven.resolver:maven-resolver-transport-http'
// scheduler
implementation group: 'com.cronutils', name: 'cron-utils'
@@ -66,7 +73,7 @@ dependencies {
testImplementation "io.micronaut:micronaut-http-server-netty"
testImplementation "io.micronaut:micronaut-management"
testImplementation "org.testcontainers:testcontainers:1.20.4"
testImplementation "org.testcontainers:junit-jupiter:1.20.4"
testImplementation "org.testcontainers:testcontainers:1.20.6"
testImplementation "org.testcontainers:junit-jupiter:1.20.6"
testImplementation "org.bouncycastle:bcpkix-jdk18on:1.80"
}

View File

@@ -3,7 +3,6 @@ package io.kestra.core.contexts;
import io.kestra.core.exceptions.KestraRuntimeException;
import io.kestra.core.plugins.DefaultPluginRegistry;
import io.kestra.core.plugins.PluginRegistry;
import io.kestra.core.plugins.serdes.PluginDeserializer;
import io.kestra.core.storages.StorageInterface;
import io.kestra.core.storages.StorageInterfaceFactory;
import io.micronaut.context.annotation.Bean;
@@ -34,7 +33,7 @@ public class KestraBeansFactory {
StorageConfig storageConfig;
@Value("${kestra.storage.type}")
Optional<String> storageType;
protected Optional<String> storageType;
@Requires(missingBeans = PluginRegistry.class)
@Singleton
@@ -42,16 +41,25 @@ public class KestraBeansFactory {
return DefaultPluginRegistry.getOrCreate();
}
@Singleton
public StorageInterfaceFactory storageInterfaceFactory(final PluginRegistry pluginRegistry){
return new StorageInterfaceFactory(pluginRegistry, validator);
}
@Requires(missingBeans = StorageInterface.class)
@Singleton
@Bean(preDestroy = "close")
public StorageInterface storageInterface(final PluginRegistry pluginRegistry) throws IOException {
String pluginId = storageType.orElseThrow(() -> new KestraRuntimeException(String.format(
public StorageInterface storageInterface(final StorageInterfaceFactory storageInterfaceFactory) throws IOException {
String pluginId = getStoragePluginId(storageInterfaceFactory);
return storageInterfaceFactory.make(null, pluginId, storageConfig.getStorageConfig(pluginId));
}
public String getStoragePluginId(StorageInterfaceFactory storageInterfaceFactory) {
return storageType.orElseThrow(() -> new KestraRuntimeException(String.format(
"No storage configured through the application property '%s'. Supported types are: %s"
, KESTRA_STORAGE_TYPE_CONFIG,
StorageInterfaceFactory.getLoggableStorageIds(pluginRegistry)
storageInterfaceFactory.getLoggableStorageIds()
)));
return StorageInterfaceFactory.make(pluginRegistry, pluginId, storageConfig.getStorageConfig(pluginId), validator);
}
@ConfigurationProperties("kestra")
@@ -67,7 +75,7 @@ public class KestraBeansFactory {
* @return the configuration.
*/
@SuppressWarnings("unchecked")
private Map<String, Object> getStorageConfig(String type) {
public Map<String, Object> getStorageConfig(String type) {
return (Map<String, Object>) storage.get(StringConvention.CAMEL_CASE.format(type));
}
}

View File

@@ -25,7 +25,7 @@ public abstract class KestraContext {
private static final AtomicReference<KestraContext> INSTANCE = new AtomicReference<>();
// Properties
private static final String KESTRA_SERVER_TYPE = "kestra.server-type";
public static final String KESTRA_SERVER_TYPE = "kestra.server-type";
/**
* Gets the current {@link KestraContext}.

View File

@@ -0,0 +1,27 @@
package io.kestra.core.contexts;
import io.micronaut.context.annotation.ConfigurationProperties;
import io.micronaut.context.annotation.EachProperty;
import io.micronaut.context.annotation.Parameter;
import io.micronaut.core.annotation.Nullable;
import lombok.Builder;
@Builder
@EachProperty("kestra.plugins.repositories")
public record MavenPluginRepositoryConfig(
@Parameter
String id,
String url,
@Nullable
BasicAuth basicAuth
) {
@Builder
@ConfigurationProperties("basic-auth")
public record BasicAuth(
String username,
String password
) {
}
}

View File

@@ -1,6 +1,7 @@
package io.kestra.core.docs;
import com.google.common.base.CaseFormat;
import io.kestra.core.models.Plugin;
import io.kestra.core.models.tasks.retrys.AbstractRetry;
import io.kestra.core.models.tasks.runners.TaskRunner;
import lombok.AllArgsConstructor;

View File

@@ -1,10 +1,9 @@
package io.kestra.core.docs;
import io.kestra.core.plugins.RegisteredPlugin;
import io.kestra.core.plugins.PluginClassAndMetadata;
import lombok.*;
import java.util.*;
import java.util.stream.Collectors;
@Getter
@EqualsAndHashCode
@@ -21,16 +20,18 @@ public class ClassPluginDocumentation<T> extends AbstractClassDocumentation<T> {
private Map<String, Object> outputsSchema;
@SuppressWarnings("unchecked")
private ClassPluginDocumentation(JsonSchemaGenerator jsonSchemaGenerator, RegisteredPlugin plugin, Class<? extends T> cls, Class<T> baseCls, String alias) {
super(jsonSchemaGenerator, cls, baseCls);
private ClassPluginDocumentation(JsonSchemaGenerator jsonSchemaGenerator, PluginClassAndMetadata<T> plugin, boolean allProperties) {
super(jsonSchemaGenerator, plugin.type(), allProperties ? null : plugin.baseClass());
// plugins metadata
this.cls = alias == null ? cls.getName() : alias;
Class<? extends T> cls = plugin.type();
this.cls = plugin.alias() == null ? cls.getName() : plugin.alias();
this.group = plugin.group();
this.docLicense = plugin.license();
this.pluginTitle = plugin.title();
this.icon = plugin.icon(cls);
if (alias != null) {
this.icon = plugin.icon();
if (plugin.alias() != null) {
replacement = cls.getName();
}
@@ -38,10 +39,10 @@ public class ClassPluginDocumentation<T> extends AbstractClassDocumentation<T> {
this.subGroup = cls.getPackageName().substring(this.group.length() + 1);
}
this.shortName = alias == null ? cls.getSimpleName() : alias.substring(alias.lastIndexOf('.') + 1);
this.shortName = plugin.alias() == null ? cls.getSimpleName() : plugin.alias().substring(plugin.alias().lastIndexOf('.') + 1);
// outputs
this.outputsSchema = jsonSchemaGenerator.outputs(baseCls, cls);
this.outputsSchema = jsonSchemaGenerator.outputs(allProperties ? null : plugin.baseClass(), cls);
if (this.outputsSchema.containsKey("$defs")) {
this.defs.putAll((Map<String, Object>) this.outputsSchema.get("$defs"));
@@ -67,17 +68,13 @@ public class ClassPluginDocumentation<T> extends AbstractClassDocumentation<T> {
.toList();
}
if (alias != null) {
if (plugin.alias() != null) {
this.deprecated = true;
}
}
public static <T> ClassPluginDocumentation<T> of(JsonSchemaGenerator jsonSchemaGenerator, RegisteredPlugin plugin, Class<? extends T> cls, Class<T> baseCls) {
return new ClassPluginDocumentation<>(jsonSchemaGenerator, plugin, cls, baseCls, null);
}
public static <T> ClassPluginDocumentation<T> of(JsonSchemaGenerator jsonSchemaGenerator, RegisteredPlugin plugin, Class<? extends T> cls, Class<T> baseCls, String alias) {
return new ClassPluginDocumentation<>(jsonSchemaGenerator, plugin, cls, baseCls, alias);
public static <T> ClassPluginDocumentation<T> of(JsonSchemaGenerator jsonSchemaGenerator, PluginClassAndMetadata<T> plugin, boolean allProperties) {
return new ClassPluginDocumentation<>(jsonSchemaGenerator, plugin, allProperties);
}
@AllArgsConstructor

View File

@@ -1,6 +1,5 @@
package io.kestra.core.docs;
import com.google.common.base.Charsets;
import com.google.common.collect.ImmutableMap;
import io.kestra.core.models.annotations.PluginSubGroup;
import io.kestra.core.models.conditions.Condition;
@@ -8,6 +7,7 @@ import io.kestra.core.models.tasks.logs.LogExporter;
import io.kestra.core.models.tasks.runners.TaskRunner;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.triggers.AbstractTrigger;
import io.kestra.core.plugins.PluginClassAndMetadata;
import io.kestra.core.plugins.RegisteredPlugin;
import io.kestra.core.runners.pebble.Extension;
import io.kestra.core.runners.pebble.JsonWriter;
@@ -29,6 +29,7 @@ import org.apache.commons.io.IOUtils;
import java.io.IOException;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -37,7 +38,7 @@ import static io.kestra.core.utils.Rethrow.throwFunction;
@Singleton
public class DocumentationGenerator {
private static PebbleEngine pebbleEngine;
private static final PebbleEngine PEBBLE_ENGINE;
@Inject
JsonSchemaGenerator jsonSchemaGenerator;
@@ -46,7 +47,7 @@ public class DocumentationGenerator {
ClasspathLoader classpathLoader = new ClasspathLoader();
classpathLoader.setPrefix("docs/");
pebbleEngine = new PebbleEngine.Builder()
PEBBLE_ENGINE = new PebbleEngine.Builder()
.newLineTrimming(false)
.loader(classpathLoader)
.extension(new AbstractExtension() {
@@ -62,6 +63,7 @@ public class DocumentationGenerator {
.build();
}
@SuppressWarnings({"unchecked", "rawtypes"})
public List<Document> generate(RegisteredPlugin registeredPlugin) throws Exception {
ArrayList<Document> result = new ArrayList<>();
@@ -72,7 +74,7 @@ public class DocumentationGenerator {
result.addAll(this.generate(registeredPlugin, registeredPlugin.getConditions(), Condition.class, "conditions"));
//noinspection unchecked
result.addAll(this.generate(registeredPlugin, registeredPlugin.getTaskRunners(), (Class) TaskRunner.class, "task-runners"));
result.addAll(this.generate(registeredPlugin, registeredPlugin.getLogExporters(), LogExporter.class, "log-exporters"));
result.addAll(this.generate(registeredPlugin, registeredPlugin.getLogExporters(), (Class) LogExporter.class, "log-exporters"));
result.addAll(guides(registeredPlugin));
@@ -216,7 +218,15 @@ public class DocumentationGenerator {
private <T> List<Document> generate(RegisteredPlugin registeredPlugin, List<Class<? extends T>> cls, Class<T> baseCls, String type) {
return cls
.stream()
.map(r -> ClassPluginDocumentation.of(jsonSchemaGenerator, registeredPlugin, r, baseCls))
.map(pluginClass -> {
PluginClassAndMetadata<T> metadata = PluginClassAndMetadata.create(
registeredPlugin,
pluginClass,
baseCls,
null
);
return ClassPluginDocumentation.of(jsonSchemaGenerator, metadata, true);
})
.map(pluginDocumentation -> {
try {
return new Document(
@@ -246,21 +256,21 @@ public class DocumentationGenerator {
classPluginDocumentation.getCls() + ".md";
}
public static <T> String render(ClassPluginDocumentation<T> classPluginDocumentation) throws IOException {
public static String render(ClassPluginDocumentation<?> classPluginDocumentation) throws IOException {
return render("task", JacksonMapper.toMap(classPluginDocumentation));
}
public static <T> String render(AbstractClassDocumentation<T> classInputDocumentation) throws IOException {
public static String render(AbstractClassDocumentation classInputDocumentation) throws IOException {
return render("task", JacksonMapper.toMap(classInputDocumentation));
}
public static <T> String render(String templateName, Map<String, Object> vars) throws IOException {
public static String render(String templateName, Map<String, Object> vars) throws IOException {
String pebbleTemplate = IOUtils.toString(
Objects.requireNonNull(DocumentationGenerator.class.getClassLoader().getResourceAsStream("docs/" + templateName + ".peb")),
Charsets.UTF_8
StandardCharsets.UTF_8
);
PebbleTemplate compiledTemplate = pebbleEngine.getLiteralTemplate(pebbleTemplate);
PebbleTemplate compiledTemplate = PEBBLE_ENGINE.getLiteralTemplate(pebbleTemplate);
Writer writer = new JsonWriter();
compiledTemplate.evaluate(writer, vars);

View File

@@ -0,0 +1,65 @@
package io.kestra.core.docs;
import io.kestra.core.models.dashboards.Dashboard;
import io.kestra.core.models.flows.Flow;
import io.kestra.core.models.flows.PluginDefault;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.templates.Template;
import io.kestra.core.models.triggers.AbstractTrigger;
import jakarta.inject.Singleton;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
/**
* Service for getting schemas.
*/
@Singleton
public class JsonSchemaCache {
private final JsonSchemaGenerator jsonSchemaGenerator;
private final ConcurrentMap<CacheKey, Map<String, Object>> schemaCache = new ConcurrentHashMap<>();
private final Map<SchemaType, Class<?>> classesBySchemaType = new HashMap<>();
/**
* Creates a new {@link JsonSchemaCache} instance.
*
* @param jsonSchemaGenerator The {@link JsonSchemaGenerator}.
*/
public JsonSchemaCache(final JsonSchemaGenerator jsonSchemaGenerator) {
this.jsonSchemaGenerator = Objects.requireNonNull(jsonSchemaGenerator, "JsonSchemaGenerator cannot be null");
registerClassForType(SchemaType.FLOW, Flow.class);
registerClassForType(SchemaType.TEMPLATE, Template.class);
registerClassForType(SchemaType.TASK, Task.class);
registerClassForType(SchemaType.TRIGGER, AbstractTrigger.class);
registerClassForType(SchemaType.PLUGINDEFAULT, PluginDefault.class);
registerClassForType(SchemaType.DASHBOARD, Dashboard.class);
}
public Map<String, Object> getSchemaForType(final SchemaType type,
final boolean arrayOf) {
return schemaCache.computeIfAbsent(new CacheKey(type, arrayOf), (key) -> {
Class<?> cls = Optional.ofNullable(classesBySchemaType.get(type))
.orElseThrow(() -> new IllegalArgumentException("Cannot found schema for type '" + type + "'"));
return jsonSchemaGenerator.schemas(cls, arrayOf);
});
}
public void registerClassForType(final SchemaType type, final Class<?> clazz) {
classesBySchemaType.put(type, clazz);
}
public void clear() {
schemaCache.clear();
}
private record CacheKey(SchemaType type, boolean arrayOf) {
}
}

View File

@@ -82,6 +82,7 @@ public class JsonSchemaGenerator {
}
replaceAnyOfWithOneOf(objectNode);
pullOfDefaultFromOneOf(objectNode);
removeRequiredOnPropsWithDefaults(objectNode);
return JacksonMapper.toMap(objectNode);
} catch (IllegalArgumentException e) {
@@ -89,6 +90,27 @@ public class JsonSchemaGenerator {
}
}
private void removeRequiredOnPropsWithDefaults(ObjectNode objectNode) {
objectNode.findParents("required").forEach(jsonNode -> {
if (jsonNode instanceof ObjectNode clazzSchema && clazzSchema.get("required") instanceof ArrayNode requiredPropsNode && clazzSchema.get("properties") instanceof ObjectNode properties) {
List<String> requiredFieldValues = StreamSupport.stream(requiredPropsNode.spliterator(), false)
.map(JsonNode::asText)
.toList();
properties.fields().forEachRemaining(e -> {
int indexInRequiredArray = requiredFieldValues.indexOf(e.getKey());
if (indexInRequiredArray != -1 && e.getValue() instanceof ObjectNode valueNode && valueNode.has("default")) {
requiredPropsNode.remove(indexInRequiredArray);
}
});
if (requiredPropsNode.isEmpty()) {
clazzSchema.remove("required");
}
}
});
}
private void replaceAnyOfWithOneOf(ObjectNode objectNode) {
objectNode.findParents("anyOf").forEach(jsonNode -> {
if (jsonNode instanceof ObjectNode oNode) {
@@ -296,6 +318,9 @@ public class JsonSchemaGenerator {
if (pluginPropertyAnnotation.beta()) {
memberAttributes.put("$beta", true);
}
if (pluginPropertyAnnotation.internalStorageURI()) {
memberAttributes.put("$internalStorageURI", true);
}
}
Schema schema = member.getAnnotationConsideringFieldAndGetter(Schema.class);
@@ -311,10 +336,12 @@ public class JsonSchemaGenerator {
if (member.getDeclaredType().isInstanceOf(Property.class)) {
memberAttributes.put("$dynamic", true);
// if we are in the String definition of a Property but the target type is not String: we configure the pattern
Class<?> targetType = member.getDeclaredType().getTypeParameters().getFirst().getErasedType();
if (!String.class.isAssignableFrom(targetType) && String.class.isAssignableFrom(member.getType().getErasedType())) {
memberAttributes.put("pattern", ".*{{.*}}.*");
}
// TODO this was a good idea but their is too much cases where it didn't work like in List or Map so if we want it we need to make it more clever
// I keep it for now commented but at some point we may want to re-do and improve it or remove these commented lines
// Class<?> targetType = member.getDeclaredType().getTypeParameters().getFirst().getErasedType();
// if (!String.class.isAssignableFrom(targetType) && String.class.isAssignableFrom(member.getType().getErasedType())) {
// memberAttributes.put("pattern", ".*{{.*}}.*");
// }
} else if (member.getDeclaredType().isInstanceOf(Data.class)) {
memberAttributes.put("$dynamic", false);
}
@@ -603,6 +630,7 @@ public class JsonSchemaGenerator {
ObjectNode objectNode = generator.generateSchema(cls);
replaceAnyOfWithOneOf(objectNode);
pullOfDefaultFromOneOf(objectNode);
removeRequiredOnPropsWithDefaults(objectNode);
return JacksonMapper.toMap(extractMainRef(objectNode));
} catch (IllegalArgumentException e) {

View File

@@ -7,6 +7,7 @@ import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.*;
import java.util.function.Predicate;
import java.util.stream.Collectors;
import static java.util.function.Predicate.not;
@@ -40,15 +41,22 @@ public class Plugin {
private String subGroup;
public static Plugin of(RegisteredPlugin registeredPlugin, @Nullable String subgroup) {
return Plugin.of(registeredPlugin, subgroup, true);
}
public static Plugin of(RegisteredPlugin registeredPlugin, @Nullable String subgroup, boolean includeDeprecated) {
Plugin plugin = new Plugin();
plugin.name = registeredPlugin.name();
PluginSubGroup subGroupInfos = null;
if (subgroup == null) {
plugin.title = registeredPlugin.title();
} else {
subGroupInfos = registeredPlugin.allClass().stream().filter(c -> c.getName().contains(subgroup)).map(clazz -> clazz.getPackage().getDeclaredAnnotation(PluginSubGroup.class)).toList().getFirst();
plugin.title = !subGroupInfos.title().isEmpty() ? subGroupInfos.title() : subgroup.substring(subgroup.lastIndexOf('.') + 1);;
subGroupInfos = registeredPlugin.allClass().stream()
.filter(c -> c.getPackageName().contains(subgroup))
.min(Comparator.comparingInt(a -> a.getPackageName().length()))
.map(clazz -> clazz.getPackage().getDeclaredAnnotation(PluginSubGroup.class))
.orElseThrow();
plugin.title = !subGroupInfos.title().isEmpty() ? subGroupInfos.title() : subgroup.substring(subgroup.lastIndexOf('.') + 1);
}
plugin.group = registeredPlugin.group();
plugin.description = subGroupInfos != null && !subGroupInfos.description().isEmpty() ? subGroupInfos.description() : registeredPlugin.description();
@@ -70,27 +78,28 @@ public class Plugin {
plugin.categories = subGroupInfos != null ?
Arrays.stream(subGroupInfos.categories()).toList() :
registeredPlugin
.allClass()
.stream()
.map(clazz -> clazz.getPackage().getDeclaredAnnotation(PluginSubGroup.class))
.filter(Objects::nonNull)
.flatMap(r -> Arrays.stream(r.categories()))
.distinct()
.toList();
.allClass()
.stream()
.map(clazz -> clazz.getPackage().getDeclaredAnnotation(PluginSubGroup.class))
.filter(Objects::nonNull)
.flatMap(r -> Arrays.stream(r.categories()))
.distinct()
.toList();
plugin.subGroup = subgroup;
plugin.tasks = filterAndGetClassName(registeredPlugin.getTasks()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.triggers = filterAndGetClassName(registeredPlugin.getTriggers()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.conditions = filterAndGetClassName(registeredPlugin.getConditions()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.storages = filterAndGetClassName(registeredPlugin.getStorages()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.secrets = filterAndGetClassName(registeredPlugin.getSecrets()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.taskRunners = filterAndGetClassName(registeredPlugin.getTaskRunners()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.apps = filterAndGetClassName(registeredPlugin.getApps()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.appBlocks = filterAndGetClassName(registeredPlugin.getAppBlocks()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.charts = filterAndGetClassName(registeredPlugin.getCharts()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.dataFilters = filterAndGetClassName(registeredPlugin.getDataFilters()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
plugin.logExporters = filterAndGetClassName(registeredPlugin.getLogExporters()).stream().filter(c -> subgroup == null || c.startsWith(subgroup)).toList();
Predicate<Class<?>> packagePredicate = c -> subgroup == null || c.getPackageName().equals(subgroup);
plugin.tasks = filterAndGetClassName(registeredPlugin.getTasks(), includeDeprecated, packagePredicate).stream().toList();
plugin.triggers = filterAndGetClassName(registeredPlugin.getTriggers(), includeDeprecated, packagePredicate).stream().toList();
plugin.conditions = filterAndGetClassName(registeredPlugin.getConditions(), includeDeprecated, packagePredicate).stream().toList();
plugin.storages = filterAndGetClassName(registeredPlugin.getStorages(), includeDeprecated, packagePredicate).stream().toList();
plugin.secrets = filterAndGetClassName(registeredPlugin.getSecrets(), includeDeprecated, packagePredicate).stream().toList();
plugin.taskRunners = filterAndGetClassName(registeredPlugin.getTaskRunners(), includeDeprecated, packagePredicate).stream().toList();
plugin.apps = filterAndGetClassName(registeredPlugin.getApps(), includeDeprecated, packagePredicate).stream().toList();
plugin.appBlocks = filterAndGetClassName(registeredPlugin.getAppBlocks(), includeDeprecated, packagePredicate).stream().toList();
plugin.charts = filterAndGetClassName(registeredPlugin.getCharts(), includeDeprecated, packagePredicate).stream().toList();
plugin.dataFilters = filterAndGetClassName(registeredPlugin.getDataFilters(), includeDeprecated, packagePredicate).stream().toList();
plugin.logExporters = filterAndGetClassName(registeredPlugin.getLogExporters(), includeDeprecated, packagePredicate).stream().toList();
return plugin;
}
@@ -99,13 +108,16 @@ public class Plugin {
* Filters the given list of class all internal Plugin, as well as, all legacy org.kestra classes.
* Those classes are only filtered from the documentation to ensure backward compatibility.
*
* @param list The list of classes?
* @return a filtered streams.
* @param list The list of classes?
* @param includeDeprecated whether to include deprecated plugins or not
* @return a filtered streams.
*/
private static List<String> filterAndGetClassName(final List<? extends Class<?>> list) {
private static List<String> filterAndGetClassName(final List<? extends Class<?>> list, boolean includeDeprecated, Predicate<Class<?>> clazzFilter) {
return list
.stream()
.filter(not(io.kestra.core.models.Plugin::isInternal))
.filter(p -> includeDeprecated || !io.kestra.core.models.Plugin.isDeprecated(p))
.filter(clazzFilter)
.map(Class::getName)
.filter(c -> !c.startsWith("org.kestra."))
.toList();

View File

@@ -1,11 +1,20 @@
package io.kestra.core.docs;
import com.fasterxml.jackson.annotation.JsonCreator;
import io.kestra.core.utils.Enums;
public enum SchemaType {
flow,
template,
task,
trigger,
plugindefault,
apps,
dashboard
FLOW,
TEMPLATE,
TASK,
TRIGGER,
PLUGINDEFAULT,
APPS,
DASHBOARD;
@JsonCreator
public static SchemaType fromString(final String value) {
return Enums.getForNameIgnoreCase(value, SchemaType.class);
}
}

View File

@@ -7,6 +7,7 @@ public enum CrudEventType {
DELETE,
LOGIN,
LOGOUT,
IMPERSONATE
IMPERSONATE,
LOGIN_FAILURE
}

View File

@@ -142,12 +142,22 @@ public class HttpRequest {
public abstract static class RequestBody {
public abstract HttpEntity to() throws IOException;
public abstract Object getContent() throws IOException;
public abstract Charset getCharset() throws IOException;
public abstract String getContentType() throws IOException;
protected ContentType entityContentType() throws IOException {
return this.getCharset() != null ? ContentType.create(this.getContentType(), this.getCharset()) : ContentType.create(this.getContentType());
}
public static RequestBody from(HttpEntity entity) throws IOException {
if (entity == null) {
return null;
}
Charset charset = Charset.forName(entity.getContentEncoding());
Charset charset = entity.getContentEncoding() != null ? Charset.forName(entity.getContentEncoding()) : StandardCharsets.UTF_8;
if (entity.getContentType().equals(ContentType.APPLICATION_OCTET_STREAM.getMimeType())) {
return ByteArrayRequestBody.builder()
@@ -172,71 +182,80 @@ public class HttpRequest {
.build();
}
throw new IllegalArgumentException("Unsupported Content-Type: " + entity.getContentType());
return ByteArrayRequestBody.builder()
.charset(charset)
.contentType(entity.getContentType())
.content(entity.getContent().readAllBytes())
.build();
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class InputStreamRequestBody extends RequestBody {
@Builder.Default
private String contentType = ContentType.APPLICATION_OCTET_STREAM.getMimeType();
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private InputStream content;
public HttpEntity to() {
return new InputStreamEntity(content, ContentType.create(contentType, charset));
public HttpEntity to() throws IOException {
return new InputStreamEntity(content, this.entityContentType());
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class StringRequestBody extends RequestBody {
@Builder.Default
private String contentType = ContentType.TEXT_PLAIN.getMimeType();
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private String content;
public HttpEntity to() {
return new StringEntity(this.content, ContentType.create(contentType, charset));
public HttpEntity to() throws IOException {
return new StringEntity(this.content, this.entityContentType());
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class ByteArrayRequestBody extends RequestBody {
@Builder.Default
private String contentType = ContentType.APPLICATION_OCTET_STREAM.getMimeType();
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private byte[] content;
public HttpEntity to() {
return new ByteArrayEntity(content, ContentType.create(contentType, charset));
public HttpEntity to() throws IOException {
return new ByteArrayEntity(content, this.entityContentType());
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class JsonRequestBody extends RequestBody {
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private Object content;
@Override
public String getContentType() throws IOException {
return ContentType.APPLICATION_JSON.getMimeType();
}
public HttpEntity to() throws IOException {
try {
return new StringEntity(
JacksonMapper.ofJson().writeValueAsString(content),
ContentType.APPLICATION_JSON.withCharset(this.charset)
this.charset != null ? ContentType.APPLICATION_JSON.withCharset(this.charset) : ContentType.APPLICATION_JSON
);
} catch (JsonProcessingException e) {
throw new IOException(e);
@@ -244,37 +263,49 @@ public class HttpRequest {
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class UrlEncodedRequestBody extends RequestBody {
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private Map<String, Object> content;
@Override
public String getContentType() throws IOException {
return ContentType.APPLICATION_FORM_URLENCODED.getMimeType();
}
public HttpEntity to() throws IOException {
return new UrlEncodedFormEntity(
this.content .entrySet()
.stream()
.map(e -> new BasicNameValuePair(e.getKey(), e.getValue().toString()))
.toList(),
this.charset
);
List<BasicNameValuePair> list = this.content.entrySet()
.stream()
.map(e -> new BasicNameValuePair(e.getKey(), e.getValue().toString()))
.toList();
return this.charset != null ? new UrlEncodedFormEntity(list, this.charset) : new UrlEncodedFormEntity(list);
}
}
@Getter
@AllArgsConstructor
@SuperBuilder
public static class MultipartRequestBody extends RequestBody {
@Builder.Default
private Charset charset = StandardCharsets.UTF_8;
private Charset charset;
private Map<String, Object> content;
@Override
public String getContentType() throws IOException {
return ContentType.MULTIPART_MIXED.getMimeType();
}
public HttpEntity to() throws IOException {
MultipartEntityBuilder builder = MultipartEntityBuilder
.create()
.setCharset(this.charset);
.create();
if (this.charset != null) {
builder.setCharset(this.charset);
}
content.forEach((key, value) -> {
switch (value) {

View File

@@ -54,7 +54,7 @@ public class HttpResponse<T> {
null
)
.headers(HttpService.toHttpHeaders(response.getHeaders()))
.body(response instanceof ClassicHttpResponse classicHttpResponse ?
.body(response instanceof ClassicHttpResponse classicHttpResponse && classicHttpResponse.getEntity() != null ?
IOUtils.toByteArray(classicHttpResponse.getEntity().getContent()) :
null
)

View File

@@ -29,6 +29,7 @@ import org.apache.hc.core5.http.io.HttpClientResponseHandler;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.ssl.SSLContexts;
import org.apache.hc.core5.util.Timeout;
import org.codehaus.plexus.util.StringUtils;
import java.io.Closeable;
import java.io.IOException;
@@ -87,47 +88,49 @@ public class HttpClient implements Closeable {
// Timeout
if (this.configuration.getTimeout() != null) {
var connectTiemout = runContext.render(this.configuration.getTimeout().getConnectTimeout()).as(Duration.class);
connectTiemout.ifPresent(duration -> connectionConfig.setConnectTimeout(Timeout.of(duration)));
var connectTimeout = runContext.render(this.configuration.getTimeout().getConnectTimeout()).as(Duration.class);
connectTimeout.ifPresent(duration -> connectionConfig.setConnectTimeout(Timeout.of(duration)));
var readIdleTiemout = runContext.render(this.configuration.getTimeout().getReadIdleTimeout()).as(Duration.class);
readIdleTiemout.ifPresent(duration -> connectionConfig.setSocketTimeout(Timeout.of(duration)));
var readIdleTimeout = runContext.render(this.configuration.getTimeout().getReadIdleTimeout()).as(Duration.class);
readIdleTimeout.ifPresent(duration -> connectionConfig.setSocketTimeout(Timeout.of(duration)));
}
// proxy
if (this.configuration.getProxy() != null) {
SocketAddress proxyAddr = new InetSocketAddress(
runContext.render(configuration.getProxy().getAddress()).as(String.class).orElse(null),
runContext.render(configuration.getProxy().getPort()).as(Integer.class).orElse(null)
);
if (this.configuration.getProxy() != null && configuration.getProxy().getAddress() != null) {
String proxyAddress = runContext.render(configuration.getProxy().getAddress()).as(String.class).orElse(null);
Proxy proxy = new Proxy(runContext.render(configuration.getProxy().getType()).as(Proxy.Type.class).orElse(null), proxyAddr);
builder.setProxySelector(new ProxySelector() {
@Override
public void connectFailed(URI uri, SocketAddress sa, IOException e) {
/* ignore */
}
@Override
public List<Proxy> select(URI uri) {
return List.of(proxy);
}
});
if (this.configuration.getProxy().getUsername() != null && this.configuration.getProxy().getPassword() != null) {
builder.setProxyAuthenticationStrategy(new DefaultAuthenticationStrategy());
credentialsStore.setCredentials(
new AuthScope(
runContext.render(this.configuration.getProxy().getAddress()).as(String.class).orElse(null),
runContext.render(this.configuration.getProxy().getPort()).as(Integer.class).orElse(null)
),
new UsernamePasswordCredentials(
runContext.render(this.configuration.getProxy().getUsername()).as(String.class).orElseThrow(),
runContext.render(this.configuration.getProxy().getPassword()).as(String.class).orElseThrow().toCharArray()
)
if (StringUtils.isNotEmpty(proxyAddress)) {
int port = runContext.render(configuration.getProxy().getPort()).as(Integer.class).orElseThrow();
SocketAddress proxyAddr = new InetSocketAddress(
proxyAddress,
port
);
Proxy proxy = new Proxy(runContext.render(configuration.getProxy().getType()).as(Proxy.Type.class).orElse(null), proxyAddr);
builder.setProxySelector(new ProxySelector() {
@Override
public void connectFailed(URI uri, SocketAddress sa, IOException e) {
/* ignore */
}
@Override
public List<Proxy> select(URI uri) {
return List.of(proxy);
}
});
if (this.configuration.getProxy().getUsername() != null && this.configuration.getProxy().getPassword() != null) {
builder.setProxyAuthenticationStrategy(new DefaultAuthenticationStrategy());
credentialsStore.setCredentials(
new AuthScope(proxyAddress, port),
new UsernamePasswordCredentials(
runContext.render(this.configuration.getProxy().getUsername()).as(String.class).orElseThrow(),
runContext.render(this.configuration.getProxy().getPassword()).as(String.class).orElseThrow().toCharArray()
)
);
}
}
}

View File

@@ -3,8 +3,8 @@ package io.kestra.core.http.client.configurations;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import io.kestra.core.exceptions.IllegalVariableEvaluationException;
import io.kestra.core.models.property.Property;
import io.kestra.core.runners.RunContext;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
@@ -14,8 +14,9 @@ import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
@JsonSubTypes.Type(value = BearerAuthConfiguration.class, name = "BEARER")
})
@SuperBuilder(toBuilder = true)
@NoArgsConstructor
public abstract class AbstractAuthConfiguration {
public abstract Property<AuthType> getType();
public abstract AuthType getType();
public abstract void configure(HttpClientBuilder builder, RunContext runContext) throws IllegalVariableEvaluationException;

View File

@@ -6,8 +6,7 @@ import io.kestra.core.models.property.Property;
import io.kestra.core.runners.RunContext;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotNull;
import lombok.Builder;
import lombok.Getter;
import lombok.*;
import lombok.experimental.SuperBuilder;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
import org.apache.hc.core5.http.HttpHeaders;
@@ -16,19 +15,20 @@ import org.apache.hc.core5.http.message.BasicHeader;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
@Getter
@SuperBuilder(toBuilder = true)
@Getter
@NoArgsConstructor
public class BasicAuthConfiguration extends AbstractAuthConfiguration {
@NotNull
@JsonInclude
@Builder.Default
protected Property<AuthType> type = Property.of(AuthType.BASIC);
protected AuthType type = AuthType.BASIC;
@Schema(title = "The username for HTTP basic authentication.")
private final Property<String> username;
private Property<String> username;
@Schema(title = "The password for HTTP basic authentication.")
private final Property<String> password;
private Property<String> password;
@Override
public void configure(HttpClientBuilder builder, RunContext runContext) throws IllegalVariableEvaluationException {

View File

@@ -8,21 +8,23 @@ import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotNull;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.experimental.SuperBuilder;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
import org.apache.hc.core5.http.HttpHeaders;
import org.apache.hc.core5.http.message.BasicHeader;
@Getter
@SuperBuilder(toBuilder = true)
@Getter
@NoArgsConstructor
public class BearerAuthConfiguration extends AbstractAuthConfiguration {
@NotNull
@JsonInclude
@Builder.Default
protected Property<AuthType> type = Property.of(AuthType.BEARER);
protected AuthType type = AuthType.BEARER;
@Schema(title = "The token for bearer token authentication.")
private final Property<String> token;
private Property<String> token;
@Override
public void configure(HttpClientBuilder builder, RunContext runContext) throws IllegalVariableEvaluationException {

View File

@@ -2,20 +2,21 @@ package io.kestra.core.http.client.configurations;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.property.Property;
import io.micronaut.http.client.HttpClientConfiguration;
import io.micronaut.logging.LogLevel;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.jackson.Jacksonized;
import java.net.Proxy;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.temporal.ChronoUnit;
@Builder(toBuilder = true)
@Getter
@Jacksonized
public class HttpConfiguration {
@Schema(title = "The timeout configuration.")
@PluginProperty
@@ -28,6 +29,7 @@ public class HttpConfiguration {
@Schema(title = "The authentification to use.")
private AbstractAuthConfiguration auth;
@Setter
@Schema(title = "The SSL request options")
private SslOptions ssl;
@@ -35,6 +37,7 @@ public class HttpConfiguration {
@Builder.Default
private Property<Boolean> followRedirects = Property.of(true);
@Setter
@Schema(title = "If true, allow a failed response code (response code >= 400)")
@Builder.Default
private Property<Boolean> allowFailed = Property.of(false);
@@ -55,261 +58,212 @@ public class HttpConfiguration {
}
// Deprecated properties
/**
* @deprecated
*/
@Schema(title = "The time allowed to establish a connection to the server before failing.")
@Deprecated
private final Property<Duration> connectTimeout;
private final Duration connectTimeout;
/**
* @deprecated
*/
@Deprecated
public void setConnectTimeout(Property<Duration> connectTimeout) {
if (this.timeout == null) {
this.timeout = TimeoutConfiguration.builder()
.build();
}
this.timeout = this.timeout.toBuilder()
.connectTimeout(connectTimeout)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The maximum time allowed for reading data from the server before failing.")
@Builder.Default
@Deprecated
private final Property<Duration> readTimeout = Property.of(Duration.ofSeconds(HttpClientConfiguration.DEFAULT_READ_TIMEOUT_SECONDS));
private final Duration readTimeout;
/**
* @deprecated
*/
@Deprecated
public void setReadTimeout(Property<Duration> readTimeout) {
if (this.timeout == null) {
this.timeout = TimeoutConfiguration.builder()
.build();
}
this.timeout = this.timeout.toBuilder()
.readIdleTimeout(readTimeout)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The type of proxy to use.")
@Builder.Default
@Deprecated
private final Property<Proxy.Type> proxyType = Property.of(Proxy.Type.DIRECT);
private final Proxy.Type proxyType;
/**
* @deprecated
*/
@Deprecated
public void setProxyType(Property<Proxy.Type> proxyType) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.type(proxyType)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The address of the proxy server.")
@Deprecated
private final Property<String> proxyAddress;
private final String proxyAddress;
/**
* @deprecated
*/
@Deprecated
public void setProxyAddress(Property<String> proxyAddress) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.address(proxyAddress)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The port of the proxy server.")
@Deprecated
private final Property<Integer> proxyPort;
private final Integer proxyPort;
/**
* @deprecated
*/
@Deprecated
public void setProxyPort(Property<Integer> proxyPort) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.port(proxyPort)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The username for proxy authentication.")
@Deprecated
private final Property<String> proxyUsername;
private final String proxyUsername;
/**
* @deprecated
*/
@Deprecated
public void setProxyUsername(Property<String> proxyUsername) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.username(proxyUsername)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The password for proxy authentication.")
@Deprecated
private final Property<String> proxyPassword;
private final String proxyPassword;
/**
* @deprecated
*/
@Deprecated
public void setProxyPassword(Property<String> proxyPassword) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.password(proxyPassword)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The username for HTTP basic authentication.")
@Deprecated
private final Property<String> basicAuthUser;
private final String basicAuthUser;
/**
* @deprecated
*/
@Deprecated
public void setBasicAuthUser(Property<String> basicAuthUser) {
if (this.auth == null || !(this.auth instanceof BasicAuthConfiguration)) {
this.auth = BasicAuthConfiguration.builder()
.build();
}
this.auth = ((BasicAuthConfiguration) this.auth).toBuilder()
.username(basicAuthUser)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The password for HTTP basic authentication.")
@Deprecated
private final Property<String> basicAuthPassword;
private final String basicAuthPassword;
/**
* @deprecated
*/
@Deprecated
private void setBasicAuthPassword(Property<String> basicAuthPassword) {
if (this.auth == null || !(this.auth instanceof BasicAuthConfiguration)) {
this.auth = BasicAuthConfiguration.builder()
.build();
}
this.auth = ((BasicAuthConfiguration) this.auth).toBuilder()
.password(basicAuthPassword)
.build();
}
/**
* @deprecated
*/
@Schema(title = "The log level for the HTTP client.")
@PluginProperty
@Deprecated
private final LogLevel logLevel;
/**
* @deprecated
*/
// Deprecated properties with no equivalent value to be kept, silently ignore
@Schema(title = "The time allowed for a read connection to remain idle before closing it.")
@Deprecated
private void setLogLevel(LogLevel logLevel) {
if (logLevel == LogLevel.TRACE) {
this.logs = new LoggingType[]{
LoggingType.REQUEST_HEADERS,
LoggingType.REQUEST_BODY,
LoggingType.RESPONSE_HEADERS,
LoggingType.RESPONSE_BODY
};
} else if (logLevel == LogLevel.DEBUG) {
this.logs = new LoggingType[]{
LoggingType.REQUEST_HEADERS,
LoggingType.RESPONSE_HEADERS,
};
} else if (logLevel == LogLevel.INFO) {
this.logs = new LoggingType[]{
LoggingType.RESPONSE_HEADERS,
};
private final Duration readIdleTimeout;
@Schema(title = "The time an idle connection can remain in the client's connection pool before being closed.")
@Deprecated
private final Duration connectionPoolIdleTimeout;
@Schema(title = "The maximum content length of the response.")
@Deprecated
private final Integer maxContentLength;
public static class HttpConfigurationBuilder {
@Deprecated
public HttpConfigurationBuilder connectTimeout(Duration connectTimeout) {
if (this.timeout == null) {
this.timeout = TimeoutConfiguration.builder()
.build();
}
this.timeout = this.timeout.toBuilder()
.connectTimeout(Property.of(connectTimeout))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder readTimeout(Duration readTimeout) {
if (this.timeout == null) {
this.timeout = TimeoutConfiguration.builder()
.build();
}
this.timeout = this.timeout.toBuilder()
.readIdleTimeout(Property.of(readTimeout))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder proxyType(Proxy.Type proxyType) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.type(Property.of(proxyType))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder proxyAddress(String proxyAddress) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.address(Property.of(proxyAddress))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder proxyPort(Integer proxyPort) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.port(Property.of(proxyPort))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder proxyUsername(String proxyUsername) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.username(Property.of(proxyUsername))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder proxyPassword(String proxyPassword) {
if (this.proxy == null) {
this.proxy = ProxyConfiguration.builder()
.build();
}
this.proxy = this.proxy.toBuilder()
.password(Property.of(proxyPassword))
.build();
return this;
}
@SuppressWarnings("DeprecatedIsStillUsed")
@Deprecated
public HttpConfigurationBuilder basicAuthUser(String basicAuthUser) {
if (this.auth == null || !(this.auth instanceof BasicAuthConfiguration)) {
this.auth = BasicAuthConfiguration.builder()
.build();
}
this.auth = ((BasicAuthConfiguration) this.auth).toBuilder()
.username(Property.of(basicAuthUser))
.build();
return this;
}
@SuppressWarnings("DeprecatedIsStillUsed")
@Deprecated
public HttpConfigurationBuilder basicAuthPassword(String basicAuthPassword) {
if (this.auth == null || !(this.auth instanceof BasicAuthConfiguration)) {
this.auth = BasicAuthConfiguration.builder()
.build();
}
this.auth = ((BasicAuthConfiguration) this.auth).toBuilder()
.password(Property.of(basicAuthPassword))
.build();
return this;
}
@Deprecated
public HttpConfigurationBuilder logLevel(LogLevel logLevel) {
if (logLevel == LogLevel.TRACE) {
this.logs = new LoggingType[]{
LoggingType.REQUEST_HEADERS,
LoggingType.REQUEST_BODY,
LoggingType.RESPONSE_HEADERS,
LoggingType.RESPONSE_BODY
};
} else if (logLevel == LogLevel.DEBUG) {
this.logs = new LoggingType[]{
LoggingType.REQUEST_HEADERS,
LoggingType.RESPONSE_HEADERS,
};
} else if (logLevel == LogLevel.INFO) {
this.logs = new LoggingType[]{
LoggingType.RESPONSE_HEADERS,
};
}
return this;
}
}
// Deprecated properties with no real value to be kept, silently ignore
/**
* @deprecated
*/
@Schema(title = "The time allowed for a read connection to remain idle before closing it.")
@Builder.Default
@Deprecated
private final Property<Duration> readIdleTimeout = Property.of(Duration.of(HttpClientConfiguration.DEFAULT_READ_IDLE_TIMEOUT_MINUTES, ChronoUnit.MINUTES));
/**
* @deprecated
*/
@Schema(title = "The time an idle connection can remain in the client's connection pool before being closed.")
@Builder.Default
@Deprecated
private final Property<Duration> connectionPoolIdleTimeout = Property.of(Duration.ofSeconds(HttpClientConfiguration.DEFAULT_CONNECTION_POOL_IDLE_TIMEOUT_SECONDS));
/**
* @deprecated
*/
@Schema(title = "The maximum content length of the response.")
@Builder.Default
@Deprecated
private final Property<Integer> maxContentLength = Property.of(HttpClientConfiguration.DEFAULT_MAX_CONTENT_LENGTH);
}

View File

@@ -4,11 +4,13 @@ import io.kestra.core.models.property.Property;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.Builder;
import lombok.Getter;
import lombok.extern.jackson.Jacksonized;
import java.net.Proxy;
@Getter
@Builder(toBuilder = true)
@Jacksonized
public class ProxyConfiguration {
@Schema(title = "The type of proxy to use.")
@Builder.Default

View File

@@ -0,0 +1,17 @@
package io.kestra.core.log;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.boolex.EvaluationException;
import ch.qos.logback.core.boolex.EventEvaluatorBase;
public class KestraLogFilter extends EventEvaluatorBase<ILoggingEvent> {
@Override
public boolean evaluate(ILoggingEvent event) throws NullPointerException, EvaluationException {
var message = event.getMessage();
// as this filter is called very often, for perf,
// we use startWith and do all checks successfully instead of using a more elegant construct like Stream...
return message.startsWith("outOfOrder mode is active. Migration of schema") ||
message.startsWith("Version mismatch : Database version is older than what dialect POSTGRES supports") ||
message.startsWith("Failed to bind as java.util.concurrent.Executors$AutoShutdownDelegatedExecutorService is unsupported.");
}
}

View File

@@ -3,6 +3,7 @@ package io.kestra.core.models;
import io.kestra.core.utils.MapUtils;
import jakarta.validation.constraints.NotNull;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@@ -46,4 +47,19 @@ public record Label(@NotNull String key, @NotNull String value) {
.map(entry -> new Label(entry.getKey(), entry.getValue()))
.toList();
}
/**
* Static helper method for converting a label string to a map.
*
* @param label The label string.
* @return The map of key/value labels.
*/
public static Map<String, String> from(String label) {
Map<String, String> map = new HashMap<>();
String[] keyValueArray = label.split(":");
if (keyValueArray.length == 2) {
map.put(keyValueArray[0], keyValueArray[1]);
}
return map;
}
}

View File

@@ -0,0 +1,16 @@
package io.kestra.core.models;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.Pattern;
/**
* Interface that can be implemented by classes supporting plugin versioning.
*
* @see Plugin
*/
public interface PluginVersioning {
@Pattern(regexp="\\d+\\.\\d+\\.\\d+(-[a-zA-Z0-9]+)?|([a-zA-Z0-9]+)")
@Schema(title = "The version of the plugin to use.")
String getVersion();
}

View File

@@ -0,0 +1,292 @@
package io.kestra.core.models;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonValue;
import io.kestra.core.models.dashboards.filters.*;
import io.kestra.core.utils.Enums;
import lombok.Builder;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
@Builder
public record QueryFilter(
Field field,
Op operation,
Object value
) {
@JsonCreator
public QueryFilter(
@JsonProperty("field") Field field,
@JsonProperty("operation") Op operation,
@JsonProperty("value") Object value
) {
this.field = field;
this.operation = operation;
this.value = value;
}
public enum Op {
EQUALS,
NOT_EQUALS,
GREATER_THAN,
LESS_THAN,
GREATER_THAN_OR_EQUAL_TO,
LESS_THAN_OR_EQUAL_TO,
IN,
NOT_IN,
STARTS_WITH,
ENDS_WITH,
CONTAINS,
REGEX;
}
@SuppressWarnings("unchecked")
public <T extends Enum<T>> AbstractFilter<T> toDashboardFilterBuilder(T field, Object value) {
switch (this.operation) {
case EQUALS:
return EqualTo.<T>builder().field(field).value(value).build();
case NOT_EQUALS:
return NotEqualTo.<T>builder().field(field).value(value).build();
case GREATER_THAN:
return GreaterThan.<T>builder().field(field).value(value).build();
case LESS_THAN:
return LessThan.<T>builder().field(field).value(value).build();
case GREATER_THAN_OR_EQUAL_TO:
return GreaterThanOrEqualTo.<T>builder().field(field).value(value).build();
case LESS_THAN_OR_EQUAL_TO:
return LessThanOrEqualTo.<T>builder().field(field).value(value).build();
case IN:
return In.<T>builder().field(field).values((List<Object>) value).build();
case NOT_IN:
return NotIn.<T>builder().field(field).values((List<Object>) value).build();
case STARTS_WITH:
return StartsWith.<T>builder().field(field).value(value.toString()).build();
case ENDS_WITH:
return EndsWith.<T>builder().field(field).value(value.toString()).build();
case CONTAINS:
return Contains.<T>builder().field(field).value(value.toString()).build();
case REGEX:
return Regex.<T>builder().field(field).value(value.toString()).build();
default:
throw new IllegalArgumentException("Unsupported operation: " + this.operation);
}
}
public enum Field {
QUERY("q") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.REGEX);
}
},
SCOPE("scope") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS);
}
},
NAMESPACE("namespace") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.STARTS_WITH, Op.ENDS_WITH, Op.REGEX);
}
},
LABELS("labels") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS);
}
},
FLOW_ID("flowId") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.IN, Op.NOT_IN);
}
},
START_DATE("startDate") {
@Override
public List<Op> supportedOp() {
return List.of(Op.GREATER_THAN, Op.LESS_THAN, Op.EQUALS, Op.NOT_EQUALS);
}
},
END_DATE("endDate") {
@Override
public List<Op> supportedOp() {
return List.of(Op.GREATER_THAN, Op.LESS_THAN, Op.EQUALS, Op.NOT_EQUALS);
}
},
STATE("state") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.IN, Op.NOT_IN);
}
},
TIME_RANGE("timeRange") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.STARTS_WITH,
Op.ENDS_WITH, Op.IN, Op.NOT_IN, Op.REGEX);
}
},
TRIGGER_EXECUTION_ID("triggerExecutionId") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.STARTS_WITH, Op.ENDS_WITH, Op.IN, Op.NOT_IN);
}
},
TRIGGER_ID("triggerId") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.STARTS_WITH, Op.ENDS_WITH, Op.IN, Op.NOT_IN);
}
},
CHILD_FILTER("childFilter") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS);
}
},
WORKER_ID("workerId") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS, Op.CONTAINS, Op.STARTS_WITH, Op.ENDS_WITH, Op.IN, Op.NOT_IN);
}
},
EXISTING_ONLY("existingOnly") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS);
}
},
MIN_LEVEL("level") {
@Override
public List<Op> supportedOp() {
return List.of(Op.EQUALS, Op.NOT_EQUALS);
}
};
private static final Map<String, Field> BY_VALUE = Arrays.stream(values())
.collect(Collectors.toMap(Field::value, Function.identity()));
public abstract List<Op> supportedOp();
private final String value;
Field(String value) {
this.value = value;
}
@JsonCreator
public static Field fromString(String value) {
return Enums.fromString(value, BY_VALUE, "field");
}
@JsonValue
public String value() {
return value;
}
}
public enum Resource {
FLOW {
@Override
public List<Field> supportedField() {
return List.of(Field.LABELS, Field.NAMESPACE, Field.QUERY, Field.SCOPE);
}
},
NAMESPACE {
@Override
public List<Field> supportedField() {
return List.of(Field.EXISTING_ONLY);
}
},
EXECUTION {
@Override
public List<Field> supportedField() {
return List.of(
Field.QUERY, Field.SCOPE, Field.FLOW_ID, Field.START_DATE, Field.END_DATE, Field.TIME_RANGE,
Field.STATE, Field.LABELS, Field.TRIGGER_EXECUTION_ID, Field.CHILD_FILTER,
Field.NAMESPACE
);
}
},
LOG {
@Override
public List<Field> supportedField() {
return List.of(Field.NAMESPACE, Field.START_DATE, Field.END_DATE,
Field.FLOW_ID, Field.TRIGGER_ID, Field.MIN_LEVEL
);
}
},
TASK {
@Override
public List<Field> supportedField() {
return List.of(Field.NAMESPACE, Field.QUERY, Field.END_DATE, Field.FLOW_ID, Field.START_DATE,
Field.STATE, Field.LABELS, Field.TRIGGER_EXECUTION_ID, Field.CHILD_FILTER
);
}
},
TEMPLATE {
@Override
public List<Field> supportedField() {
return List.of(Field.NAMESPACE, Field.QUERY);
}
},
TRIGGER {
@Override
public List<Field> supportedField() {
return List.of(Field.QUERY, Field.NAMESPACE, Field.WORKER_ID, Field.FLOW_ID
);
}
};
public abstract List<Field> supportedField();
/**
* Converts {@code Resource} enums to a list of {@code ResourceField},
* including fields and their supported operations.
*
* @return List of {@code ResourceField} with resource names, fields, and operations.
*/
public static List<ResourceField> asResourceList() {
return Arrays.stream(values())
.map(Resource::toResourceField)
.toList();
}
private static ResourceField toResourceField(Resource resource) {
List<FieldOp> fieldOps = resource.supportedField().stream()
.map(Resource::toFieldInfo)
.toList();
return new ResourceField(resource.name().toLowerCase(), fieldOps);
}
private static FieldOp toFieldInfo(Field field) {
List<Operation> operations = field.supportedOp().stream()
.map(Resource::toOperation)
.toList();
return new FieldOp(field.name().toLowerCase(), field.value(), operations);
}
private static Operation toOperation(Op op) {
return new Operation(op.name(), op.name());
}
}
public record ResourceField(String name, List<FieldOp> fields) {
}
public record FieldOp(String name, String value, List<Operation> operations) {
}
public record Operation(String name, String value) {
}
}

View File

@@ -4,6 +4,7 @@ import com.google.common.annotations.VisibleForTesting;
import io.kestra.core.repositories.ServiceInstanceRepositoryInterface;
import io.kestra.core.server.Service;
import io.kestra.core.server.ServiceInstance;
import io.kestra.core.server.ServiceType;
import java.math.BigDecimal;
import java.math.RoundingMode;
@@ -60,7 +61,7 @@ public record ServiceUsage(
final Duration interval) {
List<DailyServiceStatistics> statistics = Arrays
.stream(Service.ServiceType.values())
.stream(ServiceType.values())
.map(type -> of(from, to, repository, type, interval))
.toList();
return new ServiceUsage(statistics);
@@ -69,13 +70,13 @@ public record ServiceUsage(
private static DailyServiceStatistics of(final Instant from,
final Instant to,
final ServiceInstanceRepositoryInterface repository,
final Service.ServiceType serviceType,
final ServiceType serviceType,
final Duration interval) {
return of(serviceType, interval, repository.findAllInstancesBetween(serviceType, from, to));
}
@VisibleForTesting
static DailyServiceStatistics of(final Service.ServiceType serviceType,
static DailyServiceStatistics of(final ServiceType serviceType,
final Duration interval,
final List<ServiceInstance> instances) {
// Compute the number of running service per time-interval.

View File

@@ -1,13 +1,9 @@
package io.kestra.core.models.dashboards;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.annotation.JsonTypeName;
import io.kestra.core.models.QueryFilter;
import io.kestra.core.models.annotations.Plugin;
import io.kestra.core.models.dashboards.filters.AbstractFilter;
import io.kestra.core.repositories.QueryBuilderInterface;
import io.kestra.plugin.core.dashboard.data.Executions;
import io.kestra.plugin.core.dashboard.data.Logs;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Pattern;
@@ -17,6 +13,7 @@ import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.experimental.SuperBuilder;
import java.time.ZonedDateTime;
import java.util.Collections;
import java.util.List;
import java.util.Map;
@@ -47,6 +44,6 @@ public abstract class DataFilter<F extends Enum<F>, C extends ColumnDescriptor<F
public abstract Class<? extends QueryBuilderInterface<F>> repositoryClass();
public abstract void setGlobalFilter(GlobalFilter globalFilter);
public abstract void setGlobalFilter(List<QueryFilter> queryFilterList, ZonedDateTime startDate, ZonedDateTime endDate);
}

View File

@@ -122,6 +122,10 @@ public class Execution implements DeletedInterface, TenantInterface {
return newExecution(flow, null, labels, Optional.empty());
}
public List<Label> getLabels() {
return Optional.ofNullable(this.labels).orElse(new ArrayList<>());
}
/**
* Factory method for constructing a new {@link Execution} object for the given {@link Flow} and
* inputs.
@@ -816,11 +820,20 @@ public class Execution implements DeletedInterface, TenantInterface {
));
Map<String, Object> result = new HashMap<>();
for (TaskRun current : this.taskRunList) {
if (current.getOutputs() != null) {
result = MapUtils.merge(result, outputs(current, byIds));
}
}
this.taskRunList.stream()
.filter(taskRun -> taskRun.getOutputs() != null)
.collect(Collectors.groupingBy(taskRun -> taskRun.getTaskId()))
.forEach((taskId, taskRuns) -> {
Map<String, Object> taskOutputs = new HashMap<>();
for (TaskRun current : taskRuns) {
if (current.getIteration() != null) {
taskOutputs = MapUtils.merge(taskOutputs, outputs(current, byIds));
} else {
taskOutputs.putAll(outputs(current, byIds));
}
}
result.put(taskId, taskOutputs);
});
return result;
}
@@ -833,18 +846,17 @@ public class Execution implements DeletedInterface, TenantInterface {
if (parents.isEmpty()) {
if (taskRun.getValue() == null) {
return Map.of(taskRun.getTaskId(), taskRun.getOutputs());
return taskRun.getOutputs();
} else {
return Map.of(taskRun.getTaskId(),
Map.of(taskRun.getValue(), taskRun.getOutputs()));
return Map.of(taskRun.getValue(), taskRun.getOutputs());
}
}
Map<String, Object> result = MapUtils.newHashMap(1);
Map<String, Object> result = HashMap.newHashMap(1);
Map<String, Object> current = result;
for (TaskRun t : parents) {
HashMap<String, Object> item = MapUtils.newHashMap(1);
HashMap<String, Object> item = HashMap.newHashMap(1);
current.put(t.getValue(), item);
current = item;
}
@@ -857,7 +869,7 @@ public class Execution implements DeletedInterface, TenantInterface {
}
}
return Map.of(taskRun.getTaskId(), result);
return result;
}

View File

@@ -21,12 +21,14 @@ import io.kestra.core.models.tasks.FlowableTask;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.tasks.retrys.AbstractRetry;
import io.kestra.core.models.triggers.AbstractTrigger;
import io.kestra.core.models.triggers.Trigger;
import io.kestra.core.models.validations.ManualConstraintViolation;
import io.kestra.core.serializers.JacksonMapper;
import io.kestra.core.serializers.ListOrMapOfLabelDeserializer;
import io.kestra.core.serializers.ListOrMapOfLabelSerializer;
import io.kestra.core.services.FlowService;
import io.kestra.core.utils.IdUtils;
import io.kestra.core.utils.ListUtils;
import io.kestra.core.validations.FlowValidation;
import io.micronaut.core.annotation.Introspected;
import io.swagger.v3.oas.annotations.media.Schema;
@@ -36,8 +38,6 @@ import jakarta.validation.Valid;
import jakarta.validation.constraints.NotEmpty;
import lombok.*;
import lombok.experimental.SuperBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.*;
import java.util.stream.Collectors;
@@ -94,6 +94,9 @@ public class Flow extends AbstractFlow implements HasUID {
@Deprecated
List<Listener> listeners;
@Valid
List<Task> afterExecution;
@Valid
List<AbstractTrigger> triggers;
@@ -133,11 +136,6 @@ public class Flow extends AbstractFlow implements HasUID {
List<SLA> sla;
public Logger logger() {
return LoggerFactory.getLogger("flow." + this.id);
}
/** {@inheritDoc **/
@Override
@JsonIgnore
@@ -176,6 +174,14 @@ public class Flow extends AbstractFlow implements HasUID {
);
}
public static String uid(Trigger trigger) {
return IdUtils.fromParts(
trigger.getTenantId(),
trigger.getNamespace(),
trigger.getFlowId()
);
}
public static String uidWithoutRevision(Execution execution) {
return IdUtils.fromParts(
execution.getTenantId(),
@@ -195,10 +201,10 @@ public class Flow extends AbstractFlow implements HasUID {
public Stream<Task> allTasks() {
return Stream.of(
this.tasks != null ? this.tasks : new ArrayList<Task>(),
this.errors != null ? this.errors : new ArrayList<Task>(),
this._finally != null ? this._finally : new ArrayList<Task>(),
this.listenersTasks()
this.tasks != null ? this.tasks : Collections.<Task>emptyList(),
this.errors != null ? this.errors : Collections.<Task>emptyList(),
this._finally != null ? this._finally : Collections.<Task>emptyList(),
this.afterExecutionTasks()
)
.flatMap(Collection::stream);
}
@@ -278,6 +284,14 @@ public class Flow extends AbstractFlow implements HasUID {
.orElse(null);
}
public AbstractTrigger findTriggerByTriggerId(String triggerId) {
return this.triggers
.stream()
.filter(trigger -> trigger.getId().equals(triggerId))
.findFirst()
.orElse(null);
}
/**
* @deprecated should not be used
*/
@@ -320,15 +334,11 @@ public class Flow extends AbstractFlow implements HasUID {
}
}
private List<Task> listenersTasks() {
if (this.getListeners() == null) {
return new ArrayList<>();
}
return this.getListeners()
.stream()
.flatMap(listener -> listener.getTasks().stream())
.toList();
private List<Task> afterExecutionTasks() {
return ListUtils.concat(
ListUtils.emptyOnNull(this.getListeners()).stream().flatMap(listener -> listener.getTasks().stream()).toList(),
this.getAfterExecution()
);
}
public boolean equalsWithoutRevision(Flow o) {

View File

@@ -1,6 +1,7 @@
package io.kestra.core.models.flows;
import com.fasterxml.jackson.annotation.JsonProperty;
import io.kestra.core.models.tasks.Task;
import io.kestra.core.models.tasks.TaskForExecution;
import io.kestra.core.models.triggers.AbstractTriggerForExecution;
import io.kestra.core.utils.ListUtils;
@@ -29,6 +30,9 @@ public class FlowForExecution extends AbstractFlow {
@JsonProperty("finally")
List<TaskForExecution> _finally;
@Valid
List<TaskForExecution> afterExecution;
@Valid
List<AbstractTriggerForExecution> triggers;
@@ -42,6 +46,7 @@ public class FlowForExecution extends AbstractFlow {
.tasks(flow.getTasks().stream().map(TaskForExecution::of).toList())
.errors(ListUtils.emptyOnNull(flow.getErrors()).stream().map(TaskForExecution::of).toList())
._finally(ListUtils.emptyOnNull(flow.getFinally()).stream().map(TaskForExecution::of).toList())
.afterExecution(ListUtils.emptyOnNull(flow.getAfterExecution()).stream().map(TaskForExecution::of).toList())
.triggers(ListUtils.emptyOnNull(flow.getTriggers()).stream().map(AbstractTriggerForExecution::of).toList())
.disabled(flow.isDisabled())
.deleted(flow.isDeleted())

View File

@@ -31,6 +31,7 @@ public class FlowWithSource extends Flow implements HasSource {
.errors(this.errors)
._finally(this._finally)
.listeners(this.listeners)
.afterExecution(this.afterExecution)
.triggers(this.triggers)
.pluginDefaults(this.pluginDefaults)
.disabled(this.disabled)
@@ -72,6 +73,7 @@ public class FlowWithSource extends Flow implements HasSource {
.tasks(flow.tasks)
.errors(flow.errors)
._finally(flow._finally)
.afterExecution(flow.afterExecution)
.listeners(flow.listeners)
.triggers(flow.triggers)
.pluginDefaults(flow.pluginDefaults)

View File

@@ -55,6 +55,7 @@ public abstract class AbstractGraph {
public enum BranchType {
ERROR,
FINALLY
FINALLY,
AFTER_EXECUTION
}
}

View File

@@ -33,6 +33,9 @@ public class GraphCluster extends AbstractGraph {
return _finally;
}
@JsonIgnore
private final GraphClusterAfterExecution afterExecution;
@JsonIgnore
private final GraphClusterEnd end;
@@ -50,14 +53,17 @@ public class GraphCluster extends AbstractGraph {
this.relationType = null;
this.root = new GraphClusterRoot();
this._finally = new GraphClusterFinally();
this.afterExecution = new GraphClusterAfterExecution();
this.end = new GraphClusterEnd();
this.taskNode = null;
this.addNode(this.root);
this.addNode(this._finally);
this.addNode(this.afterExecution);
this.addNode(this.end);
this.addEdge(this.getFinally(), this.getEnd(), new Relation());
this.addEdge(this.getFinally(), this.getAfterExecution(), new Relation());
this.addEdge(this.getAfterExecution(), this.getEnd(), new Relation());
}
public GraphCluster(Task task, TaskRun taskRun, List<String> values, RelationType relationType) {
@@ -74,14 +80,17 @@ public class GraphCluster extends AbstractGraph {
this.relationType = relationType;
this.root = new GraphClusterRoot();
this._finally = new GraphClusterFinally();
this.afterExecution = new GraphClusterAfterExecution();
this.end = new GraphClusterEnd();
this.taskNode = taskNode;
this.addNode(this.root);
this.addNode(this._finally);
this.addNode(this.afterExecution);
this.addNode(this.end);
this.addEdge(this.getFinally(), this.getEnd(), new Relation());
this.addEdge(this.getFinally(), this.getAfterExecution(), new Relation());
this.addEdge(this.getAfterExecution(), this.getEnd(), new Relation());
}
public void addNode(AbstractGraph node) {

View File

@@ -0,0 +1,11 @@
package io.kestra.core.models.hierarchies;
import io.kestra.core.utils.IdUtils;
import lombok.Getter;
@Getter
public class GraphClusterAfterExecution extends AbstractGraph {
public GraphClusterAfterExecution() {
super("after-execution-" + IdUtils.create());
}
}

View File

@@ -5,6 +5,7 @@ public enum RelationType {
CHOICE,
ERROR,
FINALLY,
AFTER_EXECUTION,
PARALLEL,
DYNAMIC
}

View File

@@ -90,5 +90,10 @@ public class SubflowGraphTask extends AbstractGraphTask {
public String getType() {
return ((TaskInterface) subflowTask).getType();
}
@Override
public String getVersion() {
return ((TaskInterface) subflowTask).getVersion();
}
}
}

View File

@@ -22,6 +22,8 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
import static io.kestra.core.utils.Rethrow.throwFunction;
/**
* Define a plugin properties that will be rendered and converted to a target type at use time.
*
@@ -136,12 +138,31 @@ public class Property<T> {
*
* @see io.kestra.core.runners.RunContextProperty#asList(Class, Map)
*/
@SuppressWarnings("unchecked")
public static <T, I> T asList(Property<T> property, RunContext runContext, Class<I> itemClazz, Map<String, Object> variables) throws IllegalVariableEvaluationException {
if (property.value == null) {
String rendered = runContext.render(property.expression, variables);
JavaType type = MAPPER.getTypeFactory().constructCollectionLikeType(List.class, itemClazz);
try {
property.value = MAPPER.readValue(rendered, type);
String trimmedExpression = property.expression.trim();
// We need to detect if the expression is already a list or if it's a pebble expression (for eg. referencing a variable containing a list).
// Doing that allows us to, if it's an expression, first render then read it as a list.
if (trimmedExpression.startsWith("{{") && trimmedExpression.endsWith("}}")) {
property.value = MAPPER.readValue(runContext.render(property.expression, variables), type);
}
// Otherwise, if it's already a list, we read it as a list first then render it from run context which handle list rendering by rendering each item of the list
else {
List<?> asRawList = MAPPER.readValue(property.expression, List.class);
property.value = (T) asRawList.stream()
.map(throwFunction(item -> {
if (item instanceof String str) {
return MAPPER.convertValue(runContext.render(str, variables), itemClazz);
} else if (item instanceof Map map) {
return MAPPER.convertValue(runContext.render(map, variables), itemClazz);
}
return item;
}))
.toList();
}
} catch (JsonProcessingException e) {
throw new IllegalVariableEvaluationException(e);
}
@@ -169,6 +190,7 @@ public class Property<T> {
*
* @see io.kestra.core.runners.RunContextProperty#asMap(Class, Class, Map)
*/
@SuppressWarnings({"rawtypes", "unchecked"})
public static <T, K,V> T asMap(Property<T> property, RunContext runContext, Class<K> keyClass, Class<V> valueClass, Map<String, Object> variables) throws IllegalVariableEvaluationException {
if (property.value == null) {
JavaType targetMapType = MAPPER.getTypeFactory().constructMapType(Map.class, keyClass, valueClass);

View File

@@ -0,0 +1,5 @@
package io.kestra.core.models.tasks;
public enum FileExistComportment {
OVERWRITE, FAIL, WARN, IGNORE
}

View File

@@ -1,6 +1,5 @@
package io.kestra.core.models.tasks;
import io.kestra.core.models.annotations.PluginProperty;
import io.kestra.core.models.property.Property;
import io.micronaut.core.annotation.Introspected;
import io.swagger.v3.oas.annotations.media.Schema;
@@ -11,12 +10,14 @@ import lombok.NoArgsConstructor;
import java.util.List;
import jakarta.validation.Valid;
import lombok.extern.jackson.Jacksonized;
@Builder
@Getter
@NoArgsConstructor
@AllArgsConstructor
@Introspected
@Jacksonized
public class NamespaceFiles {
@Schema(
title = "Whether to enable namespace files to be loaded into the working directory. If explicitly set to `true` in a task, it will load all [Namespace Files](https://kestra.io/docs/developer-guide/namespace-files) into the task's working directory. Note that this property is by default set to `true` so that you can specify only the `include` and `exclude` properties to filter the files to load without having to explicitly set `enabled` to `true`."
@@ -35,4 +36,16 @@ public class NamespaceFiles {
)
@Valid
private Property<List<String>> exclude;
@Schema(
title = "A list of namespaces in which searching files. The files are loaded in the namespace order, and only the latest version of a file is kept. Meaning if a file is present in the first and second namespace, only the file present on the second namespace will be loaded."
)
@Builder.Default
private Property<List<String>> namespaces = Property.of(List.of("{{flow.namespace}}"));
@Schema(
title = "Comportment of the task if a file already exist in the working directory."
)
@Builder.Default
private Property<FileExistComportment> ifExists = Property.of(FileExistComportment.OVERWRITE);
}

Some files were not shown because too many files have changed in this diff Show More