kdevops-ng: graduating kdevops beyond Ansible

[Daniel Gomez <da.gomez@kernel.org>]

kdevops is a framework for Linux kernel development and test automation.
Its core features, namely workflow reproducibility, variability, and
scalability, are delivered through Kconfig, the variability language,
and Ansible, which provides host and guest idempotency along with
workflow orchestration at scale, whether on baremetal, local VMs, or
the cloud.

kdevops supports rolling distributions such as Debian testing, Fedora,
and openSUSE. Recently we extended Nix support, which raised the
question: how do we drive Nix's declarative language from Ansible? We
answered by wiring Nix in under Ansible and its templates, as one more
way to declare host and guest environments. But that was the wrong
framing: we had bolted Nix onto today's toolkit instead of rethinking
it. Reproducibility and idempotency now come from Nix by construction,
so I think Ansible's original reason for being in kdevops falls away.
The better question is: how do we keep kdevops's core principles, lean
on Nix, and drop Ansible?

What remains once you do is not the configuration management plane. It
is development workflow orchestration: build QEMU, build the kernel,
build a guest rootfs/closure, boot it, run a test, collect results,
diff against a baseline. That work is imperative and sequenced, work
for a workflow engine, which is where tools like Windmill [1] come in.
Windmill calls itself as "the fastest workflow engine" and an
"open-source developer platform to power your entire infra and turn
scripts into webhooks, workflows and UIs." Choosing to move kdevops
onto Windmill would keep what made kdevops kdevops, namely workflows,
quick bring-ups, baselines, and A/B regression detection, while trading
Kconfig, Make, Ansible, and host-distro provisioning for typed
run-forms, flows as code, and a worker queue. Nix supplies the
environment, much like a container or venv/poetry, along with the guest
OS system closure: declarative and portable. Windmill orchestrates the
whole pipeline end to end, graduating kdevops into a fully reproducible,
scalable, and configurable kernel-development framework, with both a UI
and a CLI, that runs locally or in the cloud. Defined as code and driven
by schedules and triggers, the same flows also make it a continuous
integration pipeline. Because steps can be written in any language
Windmill supports, including Ansible, Bash, Go, Python, and Rust,
developers can not only use kdevops but extend it with their own
scripts, turning it into a workflow hub. Note that choosing this path
does not mean NixOS is required on the controller node; Nix is simply a
runtime dependency that can be installed alongside your distro of
choice.

It'd be good to know what folks think about the possibility of evolving
kdevops in this direction, deprecating Ansible along with Kconfig and
Makefiles in favor of the new approach. To that end, I suggest a demo
day where I can show why I think this is the next step worth taking, and
whether it's a tradeoff users and maintainers are willing to make.

If this is of interest and you'd like a look, I've ported equivalents of
bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
plus an fstests run for XFS in the proof-of-concept demo project [2].
You can also find some screenshots in [3].

A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
format and client libraries are Apache-2.0. kdevops-ng runs Windmill
unmodified and self-hosted as a separate service, and the flows and
scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
Windmill rather than derived from it, so there shouldn't be any
licensing concerns.

[1] https://windmill.dev
[2] https://github.com/dagomez137/kdevops-ng
[3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots