Re: kdevops-ng: graduating kdevops beyond Ansible

[Jeff Layton <jlayton@kernel.org>]

On Thu, 2026-06-18 at 09:22 -0400, Chuck Lever wrote:
> On Thu, Jun 18, 2026, at 5:30 AM, Daniel Gomez wrote:
> > kdevops is a framework for Linux kernel development and test automation.
> > Its core features, namely workflow reproducibility, variability, and
> > scalability, are delivered through Kconfig, the variability language,
> > and Ansible, which provides host and guest idempotency along with
> > workflow orchestration at scale, whether on baremetal, local VMs, or
> > the cloud.
> > 
> > kdevops supports rolling distributions such as Debian testing, Fedora,
> > and openSUSE. Recently we extended Nix support, which raised the
> > question: how do we drive Nix's declarative language from Ansible? We
> > answered by wiring Nix in under Ansible and its templates, as one more
> > way to declare host and guest environments. But that was the wrong
> > framing: we had bolted Nix onto today's toolkit instead of rethinking
> > it. Reproducibility and idempotency now come from Nix by construction,
> > so I think Ansible's original reason for being in kdevops falls away.
> > The better question is: how do we keep kdevops's core principles, lean
> > on Nix, and drop Ansible?
> > 
> > What remains once you do is not the configuration management plane. It
> > is development workflow orchestration: build QEMU, build the kernel,
> > build a guest rootfs/closure, boot it, run a test, collect results,
> > diff against a baseline. That work is imperative and sequenced, work
> > for a workflow engine, which is where tools like Windmill [1] come in.
> > Windmill calls itself as "the fastest workflow engine" and an
> > "open-source developer platform to power your entire infra and turn
> > scripts into webhooks, workflows and UIs." Choosing to move kdevops
> > onto Windmill would keep what made kdevops kdevops, namely workflows,
> > quick bring-ups, baselines, and A/B regression detection, while trading
> > Kconfig, Make, Ansible, and host-distro provisioning for typed
> > run-forms, flows as code, and a worker queue. Nix supplies the
> > environment, much like a container or venv/poetry, along with the guest
> > OS system closure: declarative and portable. Windmill orchestrates the
> > whole pipeline end to end, graduating kdevops into a fully reproducible,
> > scalable, and configurable kernel-development framework, with both a UI
> > and a CLI, that runs locally or in the cloud. Defined as code and driven
> > by schedules and triggers, the same flows also make it a continuous
> > integration pipeline. Because steps can be written in any language
> > Windmill supports, including Ansible, Bash, Go, Python, and Rust,
> > developers can not only use kdevops but extend it with their own
> > scripts, turning it into a workflow hub. Note that choosing this path
> > does not mean NixOS is required on the controller node; Nix is simply a
> > runtime dependency that can be installed alongside your distro of
> > choice.
> > 
> > It'd be good to know what folks think about the possibility of evolving
> > kdevops in this direction, deprecating Ansible along with Kconfig and
> > Makefiles in favor of the new approach. To that end, I suggest a demo
> > day where I can show why I think this is the next step worth taking, and
> > whether it's a tradeoff users and maintainers are willing to make.
> > 
> > If this is of interest and you'd like a look, I've ported equivalents of
> > bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
> > plus an fstests run for XFS in the proof-of-concept demo project [2].
> > You can also find some screenshots in [3].
> > 
> > A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
> > format and client libraries are Apache-2.0. kdevops-ng runs Windmill
> > unmodified and self-hosted as a separate service, and the flows and
> > scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
> > Windmill rather than derived from it, so there shouldn't be any
> > licensing concerns.
> > 
> > [1] https://windmill.dev
> > [2] https://github.com/dagomez137/kdevops-ng
> > [3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots
> 
> Agreed that Ansible is evolving faster than is convenient for us
> and kdevops' use of it seems to be laden with technical debt.
> 
> I don't object at all to creative forward thinking, and I am not
> outright objecting to the proposal, but my concerns are:
> 
> + How much development effort is this? I have nearly zero time to
>   work on kdevops these days. Even with AI assistance, this seems
>   like an enormous task.
> 

Ditto. I'm mostly a user of kdevops these days and don't have as much
time to work on it.

> + Also as a user of kdevops, this means I would have to spend a
>   lot of time I don't have learning the new configuration shapes
>   and figuring out how to migrate my current CI to them. That's
>   not inviting.
> 
> + What would cloud support look like in a post-Ansible world? I
>   don't think Nix has impact on the cloud aspect of kdevops, and this
>   proposal seems to shift Nix into the role of the primary kdevops
>   virtualization method. Should we simply split kdevops into an "all
>   local virtualization" project and an "all cloud virtualization"
>   project that go their separate ways? Tying cloud and local together
>   seems to be a frequent source of chafing.
> 

It does.

OTOH, some parts of kdevops (the test running bits, in particular) are
mostly agnostic to the backend where it's being run. I don't think we
want to have to replicate that part.

Maybe we need to split the whole thing into two -- one part that does
the bringup and setup, and one part that does the test running?

> + I'd rather see a focus on addressing the technical debt of
>   continuing to support distributions that no one uses any more,
>   and moving away from requiring root to run local virtualization
>   by installing kdevops from a package. Those are narrower work
>   items that offer a bigger bang-for-buck. (You might argue that
>   Nix solves the latter problem already).

I already don't run as root for local virtualization. I thought
everyone did? Some of the initial setup requires root, but that's sort
of a separate problem.
-- 
Jeff Layton <jlayton@kernel.org>