Getting started with Kestra Modern engine for business automation

In this article, I’ll share my experience with Kestra. It’s a modern workflow orchestration platform that’s become popular among developers. If you’ve worked with Apache Airflow or simlar tools, Kestra feels different. It offers a fresh take on workflow automation

Introduction

I’ve worked with many workflow engines. I now value tools that enhance the developer experience while still being functional. Kestra grabbed my attention. It aims to simplify complex business process automation. Plus. it keeps the flexibility needed for real-world use.

Many of my projects use Spring Boot microservices that need orchestration. I wondered how Kestra would fit into this setup. The results have been quite impressive, and I think you’ll find is as compelling as I do.

What is Kestra?

Kestra is open-source platform. It helps you build, schedule, and monitor complex data pipelines and business processes. Consider it a simple, modern choice compared to traditional workflow engines. It prioritizes ease and boosts developer productivity.

The platform is built around several key principles:

• YAML-First approach: Unlike some tools that require you to write code in specific programming languages, Kestra uses YAML to define workflows. This makes it accessible to both developers and non-technical team members.
• Plugin ecosystem: Kestra comes with a rich set of plugins for various integrations - databases, APIs, file systems, cloud services and more. If you need something that doesn’t exist, creating custom plugin is straightforward.
• Real-time monitoring: The web interface provides gives your clear insight into your workflows. It offers detailed logs, metrics, and easy debugging features.
• API-First design: Everything in Kestra can be managed via API, making it perfect for GitOps workflows and automated deployments

Why choose Kestra?

Before diving into implementation, let me explain why you may choose Kestra over other alternatives:

• Developer experience: Setting up workflows is intuitive. If you can write YAML, you can create workflows in Kestra.
• Scalability: Kestra manages everything, from basic scheduled tasks to complex workflows with many executions.
• Modern architecture: Kestra uses modern tech and design. It feels fresh next to older workflow tools.
• Community and documentation: The community is lively. The documentation is thorough and includes practical examples.

Setting up your development environment

Let’s start by setting up Kestra locally. I’ll use Docker Compose since it’s the simplest way to get everything running.

First let’s create a directory for our Kestra project:

mkdir kestra-demo
cd kestra-demo

Now let’s create a docker-compose.yml file to run Kestra with PostgreSQL:

version: "3.8"

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: kestra
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - kestra

  kestra:
    image: kestra/kestra:latest
    pull_policy: always
    user: "root"
    command: server standalone --worker-thread=128
    volumes:
      - kestra_data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd:rw
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: kestra
        kestra:
          server:
            basic-auth:
              enabled: false
          repository:
            type: postgres
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
        micronaut:
          application:
            name: kestra
          server:
            port: 8080        
    ports:
      - "8080:8080"
    depends_on:
      - postgres
    networks:
      - kestra

volumes:
  postgres_data:
    driver: local
  kestra_data:
    driver: local

networks:
  kestra:
    driver: bridge

Start the services

docker compose up -d

After a few minutes, you should be able to access the Kestra UI at http://localhost:8080

Your first workflow

Let’s create a simple workflow to understand how Kestra works. In the Kestra UI, navigate to the «Flows» section and create a new flow.

Here’s a basic workflow that demonstrated several key concepts:

id: hello-world-demo
namespace: dev.vrnsky

description: A simple demonstration workflow

inputs:
  - id: name
    type: STRING
    defaults: "World"
    description: Name to greet

tasks:
  - id: greeting
    type: io.kestra.core.tasks.log.Log
    message: "Hello, {{ inputs.name }}!"

  - id: current-time
    type: io.kestra.core.tasks.scripts.Bash
    commands:
      - echo "Current time is: $(date)"

  - id: generate-report
    type: io.kestra.core.tasks.scripts.Python
    beforeCommands:
      - pip install requests
    script: |
      import json
      import requests
      from datetime import datetime

      # Simulate API call
      response = {
          "timestamp": datetime.now().isoformat(),
          "greeting": "{{ inputs.name }}",
          "status": "success"
      }

      print(json.dumps(response, indent=2))

      # Save to output
      with open('report.json', 'w') as f:
          json.dump(response, f, indent=2)      
    outputs:
      - id: report
        from: report.json

triggers:
  - id: daily-schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "0 9 * * *"  # Daily at 9 AM

Access full article