Re: kdevops-ng: graduating kdevops beyond Ansible

[Daniel Gomez <da.gomez@kernel.org>]

On 18/06/2026 14.31, Jeff Layton wrote:
> On Thu, 2026-06-18 at 11:30 +0200, Daniel Gomez wrote:
> Some thoughts:
> 
> I'd be interested to see the demo.  

Great! I will send an invite if this thread moves forward.

> It's a little hard to make a
> judgment about moving it in this direction without knowing specifically
> what it would look like.  I took a quick look at the git repo and the
> windmill site, but I don't really "get it" yet.

100%.

AFAIK, even Windmill itself struggles a bit to position what it is in
the market. But I think they get some obviously useful things right for
developer workflow orchestration that nobody else does. BTW, before
landing on this project I also looked at others: cijoe [1], Kestra,
and, at a different level, task.dev and just.systems. And these are the
alternatives Windmill compares itself against [2], in case any of them
are familiar to you (none were to me).

For a quick sense of what Windmill is, here's their architecture
diagram: https://www.windmill.dev/docs/misc/architecture

The way I think of it: a GitLab runner plus the YAML pipeline in
.gitlab-ci.yml (or the GitHub Actions equivalent), but for your
development workflows, not your project's CI. Though it can obviously be
used for project's CI too. All in one package, with far more pipeline
("flow", in Windmill terms) features [3]. That last part is one that I'm
also interested in: we could turn kdevops into a kernel regression
workflow engine with git-bisect and generate reports we can submit
directly to the mailing list and maintainers (similar to 0-day). Yes, we
can already do this today with Luis's patches from last year (I tried
them and they work), but it's exactly the kind of workflow where you can
see Ansible isn't the right tool.

Link: https://cijoe.readthedocs.io/en/latest/# [1]
Link: https://www.windmill.dev/docs/compared_to/peers [2]
Link: https://www.windmill.dev/docs/core_concepts#flow-specific-features [3]

> 
> I do agree that kconfig/makefiles are not really suited to this task.
> We've made it work, but it's a bit of a square peg in a round hole.
> 
> One of the things I liked is that kdevops spawns a normal (familiar)
> distro, and that makes it easy to get in and troubleshoot when things
> are broken. If I have to learn how to operate in yet another new
> distro, I suppose I can, but it doesn't excite me.

I understand, and I'm 100% with you: Nix (+ Windmill) is simply too
many new things at once.

The real benefit here is that the guest and the controller run the exact
same environment. You could compare it to virtme-ng with overlayfs,
where, AFAIK, you boot a guest off the host's rootfs (read-only, with
an overlay for writes), so the guest mirrors the host. The difference is
that with virtme-ng that shared environment is still whatever Debian or
Fedora you happen to be running, so it isn't pinned or reproducible;
with Nix it's the same shared environment (same /nix/store).

A way forward could be to use Windmill as the Ansible, Makefile, Kconfig
replacement or the Nix-based paths, and keep the current toolkit for the
libvirt/libguestfs/Terraform paths. Both can coexist.

> 
> OTOH, the goal here is kernel testing, so userland really doesn't
> matter too much.

But we also need changes on userland test suites so I think it matters
too. I use Nix for exactly that, to build and pin custom versions
of the test suites and tools (e.g. fstests, fio, bcc, etc.) that I can
run either baremetal or in a VM. The benefit is that the host and guest
share the same /nix/store (the guest mounts it over virtiofs), so a tool
you build on the host is the exact same build the guest runs.