jprdonnelly/kestra
1
0
Fork 0
mirror of https://github.com/kestra-io/kestra.git synced 2025-12-19 18:05:41 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,807 Commits 164 Branches 427 Tags
develop
Commit Graph

149 Commits

Author SHA1 Message Date
Ludovic DEHON
80bba59c7b fix(core): prevent FlowableTask to hit Workers (#238) 
close #223
 2020-12-03 19:05:01 +01:00
Ludovic DEHON
e3a4219d56 fix(tasks): fix bash docs & renaming files to be consistent (#235) 2020-12-03 17:21:27 +01:00
tchiotludo
b93f098890 fix(core): Remove trigger last execution when we remove the trigger 
close #229
 2020-11-27 15:23:43 +01:00
tchiotludo
213317cb1c feat(core): make scheduler independent from ElasticSearch and use Kafka 2020-11-27 15:22:47 +01:00
tchiotludo
273d817cc2 feat(core): make scheduler scalable 
Now scheduler can be scalable and deploy as many time as you want.
We use kafka to dispatch flow between different scheduler.
feat(sftp): add proxy options to sftp (#3)

chore(docs): update README with docs link
 2020-11-27 15:21:24 +01:00
tchiotludo
fa4610d8ba feat(runner-memory): try to be more resilient for unit test 2020-10-29 10:08:24 +01:00
tchiotludo
6d33b89025 feat(tasks): add a EachParallel task 2020-10-28 07:53:27 +01:00
tchiotludo
9e89e150be feat(core): add metrics on all executors 2020-10-28 07:52:15 +01:00
tchiotludo
5a3acef783 feat(core): move all documentation to JsonSchema 2020-10-27 11:23:57 +01:00
tchiotludo
55e78b1d7a feat(core): condition available for all triggers and log for trigger 
- now conditions are usable on every trigger (not only flow)
- Log from trigger will be save to LogRepository
- TriggerId is saved on execution to know which trigger start this execution
 2020-10-23 13:23:20 +02:00
tchiotludo
6340a757b6 fix(core): fix template on other server than executor 
Kafka is enabled but we don't want KafkaTemplateExecutor on webserver
 2020-09-25 22:00:25 +02:00
Ludovic DEHON
818921500e feat(core): handle worker task with java ThreadPool (#165) 
This allow to handle parallel processing on the same kafka partition.
We need to detect in this case dead worker, this is done with Executor and WorkerInstance declaration
 2020-09-25 15:56:09 +02:00
eric regnier
6d467bc8a4 feat(core): add a template task (#161) 
Add a task type template that will allow to have some template defined and reuse on other flow. 
 
Co-authored-by: tchiotludo <tchiot.ludo@gmail.com>
 2020-09-25 15:25:30 +02:00
Ludovic DEHON
0df248abd6 feat(core): add flow trigger (#162) 
Allow to trigger an execution at the end of another one
Also reintroduce Kafka only for KafkaRunner (removing FlowRepository)
Move Listeners conditions to a generic condition usable for Trigger & Listener
 2020-09-22 09:16:51 +02:00
Ludovic DEHON
b4b44f6a73 feat(core): introduce kill a running execution (#151) 
Allow to kill a running execution. 
All tasks killed will be in KILLING state waiting the completion of the kill and go to KILLED state after that
 2020-09-10 09:50:53 +02:00
Ludovic DEHON
86e0b79e64 feat(core): remove log from Execution and use a dedicated Queue / Repository (#97) 
Large log can lead to  the Execution object.
Now we put the log line by line on a new queue and save it to dedicated repository.
This also allow log to be real time.

close #74
close #34

Co-authored-by: Eric Régnier <utopman@gmail.com>
 2020-08-20 09:50:42 +02:00
tchiotludo
0af8eba474 fix(runner-memory): Index out of bound on MemoryQueue 2020-08-04 12:08:08 +02:00
tchiotludo
80ee74fee6 fix(runner-memory): try to fix error index bound must be positive during unit test 2020-07-09 14:10:11 +02:00
tchiotludo
220390eaff feat(core): introduce RunContextFactory 
RunContextFactory simplify unit test and avoid depending on applicationContext
 2020-07-09 14:10:11 +02:00
tchiotludo
4d531e6757 feat(kafka-runner): add a flow queue that allow kafka executor to run without repository 
Kafka Executor now is fully independent and run with only Kafka & KafkaStream
 2020-06-18 14:39:18 +02:00
tchiotludo
5ba02fc87f feat(core): add some missing metrics on Executor 2020-06-03 10:41:15 +02:00
Ludovic DEHON
361b1d18cf fix(core): fix possible duplicate workerTask submitted (#118) 
On high concurrency, some WorkerTask can be send multiple time lead to error : Can't find taskRun.
We need to add a deduplication and flatten the KafkaStream tree for KafkaExecutor.
 2020-05-28 17:22:05 +02:00
tchiotludo
c5cf812fc7 feat(core): handle restart with parallel tasks 2020-05-05 22:05:43 +02:00
tchiotludo
8afa001b73 feat(core): handle parallel Task on memory runner 2020-05-05 22:05:43 +02:00
tchiotludo
6eedab1a92 fix(core): better handling of MemoryRunner startup 2020-04-21 21:34:35 +02:00
tchiotludo
2c93232b6d fix(core): Runner error handling on internal failure 
some borderline error are not catched for now on runner
 2020-03-18 14:47:02 +01:00
Ludovic DEHON
7800d448be fix(core): handle case executor will failed but invalid flow (#82) 
mostly when invalid variable, the executor will raise an exception, the executor will crash and the flow was keep running.
Now we catch the exception and pass the execution to failed
 2020-03-06 11:56:57 +01:00
tchiotludo
f655bad01e test(core): replace fail fast ThreadUncaughtExceptionHandlers in unit test in order to keep retry feature 2020-03-02 20:46:48 +01:00
tchiotludo
3db80adf4e feat(core): add a fail fast for all thread pool 2020-02-27 11:32:30 +01:00
tchiotludo
f013062bed feat(core): add server metrics 
add metrics for Worker, Indexer & Executor
 2020-02-12 21:18:20 +01:00
tchiotludo
49f829e3d3 fix(core): Execution status randomly stuck in "created" 
The reason is that the ExecutionController try to fetch current execution on repository.
But since it was async call, the ExecutionController can return an `Flowable.empty()` on this case that will stop the webui.
Also fix infinite consume in case the execution is terminated, Now return the last execution if terminated.

close #48

close #48
 2020-02-10 12:22:23 +01:00
tchiotludo
7d73de3123 fix(core): add serialization on MemoryQueue 
this allow to debug de ClassLoader and have serialization like other queue
 2020-01-26 21:24:16 +01:00
tchiotludo
e886b07740 feat(core): RunContext vars for envs and globals vars 
* Allow to pass globals vars from configuration files
* Filter env vars passed with configurable prefix
* Always pass ApplicationContext to RunContext to avoid weird case
 2020-01-17 22:30:41 +01:00
tchiotludo
c710ae524d feat(cicd): Introduce github action 
* Github action main workflow
* Publish package & main jar to Bintray
* Slack notification
 2020-01-02 17:30:16 +01:00
tchiotludo
e40014b0b9 chore(name): rename to final name 2019-12-27 13:50:15 +01:00
Ludovic DEHON
1fef7509bb feat(ui): introduce web ui (#12) 
Introduce a web ui with initial support for :
* Flows list  (optionally filter by namespace) 
* Flow Detail : 
  * Initial topology view
  * Flow Executions 
  * Edit flow 
  * Trigger flow
* Executions List  
* Executions Detail : 
  * Overview 
  * Gantt view 
  * Logs 
  * Initial topology view
 2019-12-26 14:55:33 +01:00
tchiotludo
3af0ef4ae9 fix(core): introduce Listeners 
Listeners allow to execute some task after the flow. They are still considered as a TaskRun and save in the execution. They can have condition.

- add SlackIncomingWebhook notifications task
- add SlackExecution task with a message about the result of the execution
 2019-12-19 10:34:59 +01:00
tchiotludo
7ae56e3a0b chore(deps): update all dependencies 2019-12-02 21:41:19 +01:00
tchiotludo
fff742130c refactor(core): change namespace of repository & runner test 2019-12-02 18:09:45 +01:00
tchiotludo
3519e44563 feat(core): introduce EachSequential flow tasks 
Also introduced ResolvedTask to old tasks dynamic value
Filter taskRun execution by parentTaskRun
 2019-11-27 18:48:42 +01:00
tchiotludo
2008cfd221 fix(core): Fix complex flow with nested child tasks 
- refactor the whole flow handler
- remove old Executor class and merge it with ExecutionState
 2019-11-19 13:36:02 +01:00
tchiotludo
331ba5d78e feat(repository): add a configuration to enabled to right repository 
- also move flow test to AbstractFlowRepositoryTest to have a consistent behaviour between repository
 2019-11-04 13:09:16 +01:00
tchiotludo
20cd6761b9 feat(flow): handle correctly namespace & revision 2019-11-03 20:53:37 +01:00
tchiotludo
20a53d9279 fix(runner): fix the ExecutionState 
ExecutionState revamp to try to avoid missing taskRun
 2019-10-27 21:19:20 +01:00
tchiotludo
bcc5798d7f feat(storage): initial commit of storage 2019-10-19 13:43:16 +02:00
tchiotludo
05f1e20a3c feat(execution): vars & template everywhere 
- add handlebars formatting
- add output from tasks
- fill variables for handlebars with many value (env vars, inputs, outputs, ...)
 2019-10-05 15:09:15 +02:00
tchiotludo
5d3d3cc53e feat(worker): save log to executions 2019-10-03 09:42:38 +02:00
tchiotludo
b4d026574c feat(queue): introduce Kafka Queue 2019-10-03 07:44:27 +02:00
tchiotludo
83c11ee119 refactor(core): separate to different module for runner-memory & local repository 2019-09-23 20:55:18 +02:00