Re: kdevops-ng: graduating kdevops beyond Ansible

[Jeff Layton <jlayton@kernel.org>]

On Thu, 2026-06-18 at 11:30 +0200, Daniel Gomez wrote:
> kdevops is a framework for Linux kernel development and test automation.
> Its core features, namely workflow reproducibility, variability, and
> scalability, are delivered through Kconfig, the variability language,
> and Ansible, which provides host and guest idempotency along with
> workflow orchestration at scale, whether on baremetal, local VMs, or
> the cloud.
> 
> kdevops supports rolling distributions such as Debian testing, Fedora,
> and openSUSE. Recently we extended Nix support, which raised the
> question: how do we drive Nix's declarative language from Ansible? We
> answered by wiring Nix in under Ansible and its templates, as one more
> way to declare host and guest environments. But that was the wrong
> framing: we had bolted Nix onto today's toolkit instead of rethinking
> it. Reproducibility and idempotency now come from Nix by construction,
> so I think Ansible's original reason for being in kdevops falls away.
> The better question is: how do we keep kdevops's core principles, lean
> on Nix, and drop Ansible?
> 
> What remains once you do is not the configuration management plane. It
> is development workflow orchestration: build QEMU, build the kernel,
> build a guest rootfs/closure, boot it, run a test, collect results,
> diff against a baseline. That work is imperative and sequenced, work
> for a workflow engine, which is where tools like Windmill [1] come in.
> Windmill calls itself as "the fastest workflow engine" and an
> "open-source developer platform to power your entire infra and turn
> scripts into webhooks, workflows and UIs." Choosing to move kdevops
> onto Windmill would keep what made kdevops kdevops, namely workflows,
> quick bring-ups, baselines, and A/B regression detection, while trading
> Kconfig, Make, Ansible, and host-distro provisioning for typed
> run-forms, flows as code, and a worker queue. Nix supplies the
> environment, much like a container or venv/poetry, along with the guest
> OS system closure: declarative and portable. Windmill orchestrates the
> whole pipeline end to end, graduating kdevops into a fully reproducible,
> scalable, and configurable kernel-development framework, with both a UI
> and a CLI, that runs locally or in the cloud. Defined as code and driven
> by schedules and triggers, the same flows also make it a continuous
> integration pipeline. Because steps can be written in any language
> Windmill supports, including Ansible, Bash, Go, Python, and Rust,
> developers can not only use kdevops but extend it with their own
> scripts, turning it into a workflow hub. Note that choosing this path
> does not mean NixOS is required on the controller node; Nix is simply a
> runtime dependency that can be installed alongside your distro of
> choice.
> 
> It'd be good to know what folks think about the possibility of evolving
> kdevops in this direction, deprecating Ansible along with Kconfig and
> Makefiles in favor of the new approach. To that end, I suggest a demo
> day where I can show why I think this is the next step worth taking, and
> whether it's a tradeoff users and maintainers are willing to make.
>
> If this is of interest and you'd like a look, I've ported equivalents of
> bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
> plus an fstests run for XFS in the proof-of-concept demo project [2].
> You can also find some screenshots in [3].
> 
> A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
> format and client libraries are Apache-2.0. kdevops-ng runs Windmill
> unmodified and self-hosted as a separate service, and the flows and
> scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
> Windmill rather than derived from it, so there shouldn't be any
> licensing concerns.
> 
> [1] https://windmill.dev
> [2] https://github.com/dagomez137/kdevops-ng
> [3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots

Some thoughts:

I'd be interested to see the demo.  It's a little hard to make a
judgment about moving it in this direction without knowing specifically
what it would look like.  I took a quick look at the git repo and the
windmill site, but I don't really "get it" yet.

I do agree that kconfig/makefiles are not really suited to this task.
We've made it work, but it's a bit of a square peg in a round hole.

One of the things I liked is that kdevops spawns a normal (familiar)
distro, and that makes it easy to get in and troubleshoot when things
are broken. If I have to learn how to operate in yet another new
distro, I suppose I can, but it doesn't excite me.

OTOH, the goal here is kernel testing, so userland really doesn't
matter too much.
-- 
Jeff Layton <jlayton@kernel.org>