Skip to content

Releases: kestra-io/kestra

v0.9.8

11 Jul 08:32
@github-actions github-actions
v0.9.8
9260817
Compare
Choose a tag to compare
v0.9.8

Bug Fixes

  • CI: Set fixed plugin version to latest 0.9.X (YannC)

Chores

  • version: update to version 'v0.9.8'. (YannC)
Assets 3
Loading

v0.9.7

10 Jul 14:26
@github-actions github-actions
v0.9.7
32ce62c
Compare
Choose a tag to compare
v0.9.7

Bug Fixes

  • jdbc: DateTimeFormatter can be reused in the JdbcMapper (Loïc Mathieu)

Chores

  • version: update to version 'v0.9.7'. (YannC)

Commits

  • fix: Updated count variable (YannC)
Assets 3
Loading

v0.10.0

06 Jul 12:58
@github-actions github-actions
v0.10.0
bd93937
Compare
Choose a tag to compare
v0.10.0

Kestra Release Note - 0.10.0

This new release brings important new core features such as Blueprints, new Script plugins, integration of basic authentication, and a secret function making the Open Source Edition even more robust.

Features

Blueprints

Blueprints are a curated, organized, and searchable catalog of ready-to-use examples designed to help you kick-start your workflow.

It’s available as a new section on the left bar menu and directly within the editor view**. All blueprints are validated and documented so that they just work.

You can easily customize and integrate them into your new or existing flows with a single click on the “Use” button.

Check out the documentation.

Untitled

Untitled2

Enterprise Edition users could write their own organization blueprints so they can share and grow a library of internal blueprints within teams.
Untitled3

Improved Support for Scripting

In this release, we revamped scripting tasks to bring more flexibility and control.
Each script task is now, by default, running in its own dedicated container.You can attach any Docker image you want if you need a specific one.
New tasks are available for Shell, Python, R, and Node.js:

  • Script: to write and run ad-hoc scripts. It comes with a beforeCommands property to execute any instruction needed before running the main script (installing dependencies for example)
    id: script
    namespace: release
    
    tasks:
      - id: run_python
        type: io.kestra.plugin.scripts.python.Script
        beforeCommands:
          - pip install requests
        warningOnStdErr: false
        script: | 
          import requests
          import json
    
          response = requests.get("https://api.github.com")
          data = response.json()
          print(data)
  • Command: to run arbitrary commands in a single task configuration. This task can be very powerful and associated with the WorkingDirectory task. For example, to clone a Git repository and then execute the corresponding scripts.
    id: command
    namespace: release
    
    tasks:
      - id: working
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
    
          - id: clone_repo
            type: io.kestra.plugin.git.Clone
            url: https://github.com/your_repository
    
          - id: run_python
            type: io.kestra.plugin.scripts.python.Command
            beforeCommands:
              - pip install -r requirements.txt
        dockerOptions:
        	    image: ghcr.io/kestra-io/pydata:latest
            commands:
              - python main.py

  • LocalFile: to create files in the local filesystem or to send files from the local filesystem to the internal storage. This task allows to uncouple inline scripts to their execution.

        id: "local-file"
    namespace: release
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: inputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          inputs:
            hello.txt: "Hello World"
            address.json: "{{ outputs.myTaskId.uri }}"
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - cat hello.txt
    	- sed -n 's/.*"country":"\([^"]*\)".*/\1/p' address.json

It also allows to expose outputs to internal storage. In the following example, we create two files with Bash commands and expose those into Kestra internal storage with the outputs property LocalFiles

    id: "local-files"
    namespace: release
    
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - mkdir -p sub/dir
            - echo "Hello from Bash" >> sub/dir/bash1.txt
            - echo "Hello from Bash" >> sub/dir/bash2.txt
        - id: outputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          outputs:
            - sub/**

Those tasks run by default on DOCKER but you can use the runner: PROCESS property to run it as a process on the Kestra host.

Note: the old scripting tasks w ill be deprecated, removed from the core and being retro compatible within the new plugins.

DAG task

Creating directed acyclic graphs, a common pattern in data orchestration, was already possible in Kestra by through Flow dependencies. With a brand new DAG task, it’s now even easier to do; directly between tasks at the Flow level.

id: magicDAG
namespace: dev
tasks:
  - id: dag
    type: io.kestra.core.tasks.flows.Dag
    tasks:
      - task:
          id: customers
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_customers.csv
      - task:
          id: orders
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_orders.csv
      - task:
          id: payments
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_payments.csv
      - task:
          id: transform
          type: io.kestra.core.tasks.log.Log
          message: |
            Transforming data from: 
              {{outputs.customers.uri}}
              {{outputs.orders.uri}}
              {{outputs.payments.uri}}
        dependsOn:
          - customers
          - orders
          - payments

image

Label on executions

So far, it was only possible to add labels on a flow level by adjusting the workflow code. This release adds the ability to set custom labels for specific Executions.

Also, the labels added on a flow level will be automatically propagated to Execution labels.

For example you can add labels for “experiment” executions so you can retrieve them in the UI easily.
Untitled4

Screenshot 2023-07-05 at 15 43 45

Basic Authentication

The community was asking about authentication in the open source to secure their instance in production. We heard you ! You can now add basic authentication to your Kestra instance with username/password values in the Kestra configuration.

server:
  basic-auth:
    enabled: true
    username: admin
    password: *****

Secret Function

We introduced a secret() function allowing to read base64 secret value from environment variables.

id: secret
namespace: release
tasks:
  - id: get_secret
    type: io.kestra.core.tasks.debugs.Return
    format: '{{ secret("my_secret")}}'

Worker Group

This release introduce worker groups, a ****set of workers that can be targeted specifically for a task execution or a polling trigger evaluation. For this, the task or the polling trigger must define the workerGroup.key property with the key of the worker group to target. A default worker group can also be configured at the namespace level.

Here are common use cases in which Worker Groups can be beneficial:

  • Execute tasks and polling triggers on specific compute instances (e.g., a VM with a GPU and preconfigured CUDA drivers).
  • Execute tasks and polling triggers on a worker with a specific Operating System (e.g., a Windows server).
  • Restrict backend access to a set of workers (firewall rules, private networks, ...).
  • Execute tasks and polling triggers close to a remote backend (region selection).

Here is how you can ensure that a task is executed only by specific worker instances:

id: gpuTask
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: |
      This task will be executed on a specific remote worker that has access to a GPU
    workerGroup:
    key: gpuWorkerGroupKey

Polling Triggers

We have made significant improvements to the Polling trigger in the latest version of Kestra to enhance performance, strengthen security measures, simplify maintenance, and clarify the system architecture.
In previous versions, the trigger evaluation process was directly handled by the Scheduler. This meant that both the Worker and the Scheduler potentially accessed external systems, leading to manage the security and scalability of two services.

From now on, The Scheduler delegates the evaluation of polling triggers to the Worker. This architectural change has several advantages. Firstly, it simpli...

Read more
Assets 3
Loading

v0.9.5

20 Jun 13:12
@github-actions github-actions
v0.9.5
4a7dbbb
Compare
Choose a tag to compare
v0.9.5

Bug Fixes

  • ui: fix query issue in home component (YannC)

Chores

  • version: update snapshot version 'v0.9.5-SNAPSHOT'. (Ludovic DEHON)
  • version: update to version 'v0.9.5'. (YannC)
Assets 3
Loading

v0.9.4

12 Jun 14:30
@github-actions github-actions
v0.9.4
830c7cc
Compare
Choose a tag to compare
v0.9.4

Bug Fixes

Chores

Assets 3
Loading

v0.9.2

05 Jun 14:13
@github-actions github-actions
v0.9.2
cd25424
Compare
Choose a tag to compare
v0.9.2

Features

Bug Fixes

  • ui: don't load task type and use flow instead (Ludovic DEHON)
  • core: TestUtils.mockTrigger() must call runContext.forScheduler() (Loïc Mathieu)
  • ui: Flow with dependencies can now be deleted (YannC)
  • core: multiple potential deadlocks and races on the AbstractScheduler (Loïc Mathieu)
  • core: redundant synchronization on the FlowListeners (Loïc Mathieu)
  • ui: validation of trigger on lowcode failed (#1440) (Ludovic DEHON)
  • tests: falling test depending on the backend (Ludovic DEHON)

Chores

Assets 3
Loading

v0.9.1

30 May 20:31
@github-actions github-actions
v0.9.1
c761c2f
Compare
Choose a tag to compare
v0.9.1

Features

Bug Fixes

Chores

Assets 3
Loading

v0.9.0

24 May 07:23
@github-actions github-actions
v0.9.0
519aa8e
Compare
Choose a tag to compare
v0.9.0

Kestra Release Note - 0.9.0

This new release focused on user experience improvements and bugs resolving.

New Features

  • ⭐ Flow Metrics Dashboard : since previous versions, any flows execution exposed Metrics. However there were no useful UI to explore and see those metrics. In this new release we created a new "Metrics tab" accessible at the Flow level. It may be useful to track specific metrics (count rows processed, explore third-party integration billing, see time of processes, etc.) and improve the observability of users tasks.
  • Save Flow as drafts: users are now able to save Flows as draft, even in the presence of errors in the Flow declaration. Useful when drafting flows and allows users to conveniently return to their work at a later time.

User Experiences Improvements

  • Manage editor windows size: editor can shows both source and topology or source and documentation. We added the ability to resize those two windows with a slider.
  • Ability to re-execute a Flow with the last inputs.
  • Task required properties are now listed first in the documentation pages.
  • Several improvements and fixes : improved flow and inputs validation, improved page redirections, fixed gantt display, improved error messages, fixed theme discrepancies etc.

Plugins

Documentation

Enterprise Edition

  • New License system.
  • Inherited variables are now available even if the user doesn’t have access to the namespace.
  • Namespaces now have a proper dashboard.
  • Fix stability issue when multiples conditions generate too many logs.
  • Improve login and authentication.
  • Fix bug on Gantt when using Each like tasks.

All Changes

Features

Bug Fixes

Read more
Assets 3
Loading

v0.8.1

10 May 14:27
@github-actions github-actions
v0.8.1
cef20cc
Compare
Choose a tag to compare
v0.8.1

Chores

Assets 3
Loading

v0.8.0

12 Apr 14:14
@github-actions github-actions
v0.8.0
4d23111
Compare
Choose a tag to compare
v0.8.0

Features

Bug Fixes

Read more

Contributors

Cacheable
Assets 3
Loading