kdevops-ng: graduating kdevops beyond Ansible

kdevops-ng: graduating kdevops beyond Ansible

[Daniel Gomez]

kdevops is a framework for Linux kernel development and test automation.
Its core features, namely workflow reproducibility, variability, and
scalability, are delivered through Kconfig, the variability language,
and Ansible, which provides host and guest idempotency along with
workflow orchestration at scale, whether on baremetal, local VMs, or
the cloud.

kdevops supports rolling distributions such as Debian testing, Fedora,
and openSUSE. Recently we extended Nix support, which raised the
question: how do we drive Nix's declarative language from Ansible? We
answered by wiring Nix in under Ansible and its templates, as one more
way to declare host and guest environments. But that was the wrong
framing: we had bolted Nix onto today's toolkit instead of rethinking
it. Reproducibility and idempotency now come from Nix by construction,
so I think Ansible's original reason for being in kdevops falls away.
The better question is: how do we keep kdevops's core principles, lean
on Nix, and drop Ansible?

What remains once you do is not the configuration management plane. It
is development workflow orchestration: build QEMU, build the kernel,
build a guest rootfs/closure, boot it, run a test, collect results,
diff against a baseline. That work is imperative and sequenced, work
for a workflow engine, which is where tools like Windmill [1] come in.
Windmill calls itself as "the fastest workflow engine" and an
"open-source developer platform to power your entire infra and turn
scripts into webhooks, workflows and UIs." Choosing to move kdevops
onto Windmill would keep what made kdevops kdevops, namely workflows,
quick bring-ups, baselines, and A/B regression detection, while trading
Kconfig, Make, Ansible, and host-distro provisioning for typed
run-forms, flows as code, and a worker queue. Nix supplies the
environment, much like a container or venv/poetry, along with the guest
OS system closure: declarative and portable. Windmill orchestrates the
whole pipeline end to end, graduating kdevops into a fully reproducible,
scalable, and configurable kernel-development framework, with both a UI
and a CLI, that runs locally or in the cloud. Defined as code and driven
by schedules and triggers, the same flows also make it a continuous
integration pipeline. Because steps can be written in any language
Windmill supports, including Ansible, Bash, Go, Python, and Rust,
developers can not only use kdevops but extend it with their own
scripts, turning it into a workflow hub. Note that choosing this path
does not mean NixOS is required on the controller node; Nix is simply a
runtime dependency that can be installed alongside your distro of
choice.

It'd be good to know what folks think about the possibility of evolving
kdevops in this direction, deprecating Ansible along with Kconfig and
Makefiles in favor of the new approach. To that end, I suggest a demo
day where I can show why I think this is the next step worth taking, and
whether it's a tradeoff users and maintainers are willing to make.

If this is of interest and you'd like a look, I've ported equivalents of
bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
plus an fstests run for XFS in the proof-of-concept demo project [2].
You can also find some screenshots in [3].

A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
format and client libraries are Apache-2.0. kdevops-ng runs Windmill
unmodified and self-hosted as a separate service, and the flows and
scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
Windmill rather than derived from it, so there shouldn't be any
licensing concerns.

[1] https://windmill.dev
[2] https://github.com/dagomez137/kdevops-ng
[3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots

Re: kdevops-ng: graduating kdevops beyond Ansible

[Jeff Layton]

On Thu, 2026-06-18 at 11:30 +0200, Daniel Gomez wrote:
> kdevops is a framework for Linux kernel development and test automation.
> Its core features, namely workflow reproducibility, variability, and
> scalability, are delivered through Kconfig, the variability language,
> and Ansible, which provides host and guest idempotency along with
> workflow orchestration at scale, whether on baremetal, local VMs, or
> the cloud.
> 
> kdevops supports rolling distributions such as Debian testing, Fedora,
> and openSUSE. Recently we extended Nix support, which raised the
> question: how do we drive Nix's declarative language from Ansible? We
> answered by wiring Nix in under Ansible and its templates, as one more
> way to declare host and guest environments. But that was the wrong
> framing: we had bolted Nix onto today's toolkit instead of rethinking
> it. Reproducibility and idempotency now come from Nix by construction,
> so I think Ansible's original reason for being in kdevops falls away.
> The better question is: how do we keep kdevops's core principles, lean
> on Nix, and drop Ansible?
> 
> What remains once you do is not the configuration management plane. It
> is development workflow orchestration: build QEMU, build the kernel,
> build a guest rootfs/closure, boot it, run a test, collect results,
> diff against a baseline. That work is imperative and sequenced, work
> for a workflow engine, which is where tools like Windmill [1] come in.
> Windmill calls itself as "the fastest workflow engine" and an
> "open-source developer platform to power your entire infra and turn
> scripts into webhooks, workflows and UIs." Choosing to move kdevops
> onto Windmill would keep what made kdevops kdevops, namely workflows,
> quick bring-ups, baselines, and A/B regression detection, while trading
> Kconfig, Make, Ansible, and host-distro provisioning for typed
> run-forms, flows as code, and a worker queue. Nix supplies the
> environment, much like a container or venv/poetry, along with the guest
> OS system closure: declarative and portable. Windmill orchestrates the
> whole pipeline end to end, graduating kdevops into a fully reproducible,
> scalable, and configurable kernel-development framework, with both a UI
> and a CLI, that runs locally or in the cloud. Defined as code and driven
> by schedules and triggers, the same flows also make it a continuous
> integration pipeline. Because steps can be written in any language
> Windmill supports, including Ansible, Bash, Go, Python, and Rust,
> developers can not only use kdevops but extend it with their own
> scripts, turning it into a workflow hub. Note that choosing this path
> does not mean NixOS is required on the controller node; Nix is simply a
> runtime dependency that can be installed alongside your distro of
> choice.
> 
> It'd be good to know what folks think about the possibility of evolving
> kdevops in this direction, deprecating Ansible along with Kconfig and
> Makefiles in favor of the new approach. To that end, I suggest a demo
> day where I can show why I think this is the next step worth taking, and
> whether it's a tradeoff users and maintainers are willing to make.
>
> If this is of interest and you'd like a look, I've ported equivalents of
> bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
> plus an fstests run for XFS in the proof-of-concept demo project [2].
> You can also find some screenshots in [3].
> 
> A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
> format and client libraries are Apache-2.0. kdevops-ng runs Windmill
> unmodified and self-hosted as a separate service, and the flows and
> scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
> Windmill rather than derived from it, so there shouldn't be any
> licensing concerns.
> 
> [1] https://windmill.dev
> [2] https://github.com/dagomez137/kdevops-ng
> [3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots

Some thoughts:

I'd be interested to see the demo.  It's a little hard to make a
judgment about moving it in this direction without knowing specifically
what it would look like.  I took a quick look at the git repo and the
windmill site, but I don't really "get it" yet.

I do agree that kconfig/makefiles are not really suited to this task.
We've made it work, but it's a bit of a square peg in a round hole.

One of the things I liked is that kdevops spawns a normal (familiar)
distro, and that makes it easy to get in and troubleshoot when things
are broken. If I have to learn how to operate in yet another new
distro, I suppose I can, but it doesn't excite me.

OTOH, the goal here is kernel testing, so userland really doesn't
matter too much.
-- 
Jeff Layton <jlayton@kernel.org>

Re: kdevops-ng: graduating kdevops beyond Ansible

[Chuck Lever]

On Thu, Jun 18, 2026, at 5:30 AM, Daniel Gomez wrote:
> kdevops is a framework for Linux kernel development and test automation.
> Its core features, namely workflow reproducibility, variability, and
> scalability, are delivered through Kconfig, the variability language,
> and Ansible, which provides host and guest idempotency along with
> workflow orchestration at scale, whether on baremetal, local VMs, or
> the cloud.
>
> kdevops supports rolling distributions such as Debian testing, Fedora,
> and openSUSE. Recently we extended Nix support, which raised the
> question: how do we drive Nix's declarative language from Ansible? We
> answered by wiring Nix in under Ansible and its templates, as one more
> way to declare host and guest environments. But that was the wrong
> framing: we had bolted Nix onto today's toolkit instead of rethinking
> it. Reproducibility and idempotency now come from Nix by construction,
> so I think Ansible's original reason for being in kdevops falls away.
> The better question is: how do we keep kdevops's core principles, lean
> on Nix, and drop Ansible?
>
> What remains once you do is not the configuration management plane. It
> is development workflow orchestration: build QEMU, build the kernel,
> build a guest rootfs/closure, boot it, run a test, collect results,
> diff against a baseline. That work is imperative and sequenced, work
> for a workflow engine, which is where tools like Windmill [1] come in.
> Windmill calls itself as "the fastest workflow engine" and an
> "open-source developer platform to power your entire infra and turn
> scripts into webhooks, workflows and UIs." Choosing to move kdevops
> onto Windmill would keep what made kdevops kdevops, namely workflows,
> quick bring-ups, baselines, and A/B regression detection, while trading
> Kconfig, Make, Ansible, and host-distro provisioning for typed
> run-forms, flows as code, and a worker queue. Nix supplies the
> environment, much like a container or venv/poetry, along with the guest
> OS system closure: declarative and portable. Windmill orchestrates the
> whole pipeline end to end, graduating kdevops into a fully reproducible,
> scalable, and configurable kernel-development framework, with both a UI
> and a CLI, that runs locally or in the cloud. Defined as code and driven
> by schedules and triggers, the same flows also make it a continuous
> integration pipeline. Because steps can be written in any language
> Windmill supports, including Ansible, Bash, Go, Python, and Rust,
> developers can not only use kdevops but extend it with their own
> scripts, turning it into a workflow hub. Note that choosing this path
> does not mean NixOS is required on the controller node; Nix is simply a
> runtime dependency that can be installed alongside your distro of
> choice.
>
> It'd be good to know what folks think about the possibility of evolving
> kdevops in this direction, deprecating Ansible along with Kconfig and
> Makefiles in favor of the new approach. To that end, I suggest a demo
> day where I can show why I think this is the next step worth taking, and
> whether it's a tradeoff users and maintainers are willing to make.
>
> If this is of interest and you'd like a look, I've ported equivalents of
> bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
> plus an fstests run for XFS in the proof-of-concept demo project [2].
> You can also find some screenshots in [3].
>
> A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
> format and client libraries are Apache-2.0. kdevops-ng runs Windmill
> unmodified and self-hosted as a separate service, and the flows and
> scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
> Windmill rather than derived from it, so there shouldn't be any
> licensing concerns.
>
> [1] https://windmill.dev
> [2] https://github.com/dagomez137/kdevops-ng
> [3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots

Agreed that Ansible is evolving faster than is convenient for us
and kdevops' use of it seems to be laden with technical debt.

I don't object at all to creative forward thinking, and I am not
outright objecting to the proposal, but my concerns are:

+ How much development effort is this? I have nearly zero time to
  work on kdevops these days. Even with AI assistance, this seems
  like an enormous task.

+ Also as a user of kdevops, this means I would have to spend a
  lot of time I don't have learning the new configuration shapes
  and figuring out how to migrate my current CI to them. That's
  not inviting.

+ What would cloud support look like in a post-Ansible world? I
  don't think Nix has impact on the cloud aspect of kdevops, and this
  proposal seems to shift Nix into the role of the primary kdevops
  virtualization method. Should we simply split kdevops into an "all
  local virtualization" project and an "all cloud virtualization"
  project that go their separate ways? Tying cloud and local together
  seems to be a frequent source of chafing.

+ I'd rather see a focus on addressing the technical debt of
  continuing to support distributions that no one uses any more,
  and moving away from requiring root to run local virtualization
  by installing kdevops from a package. Those are narrower work
  items that offer a bigger bang-for-buck. (You might argue that
  Nix solves the latter problem already).

-- 
Chuck Lever

Re: kdevops-ng: graduating kdevops beyond Ansible

[Jeff Layton]

On Thu, 2026-06-18 at 09:22 -0400, Chuck Lever wrote:
> On Thu, Jun 18, 2026, at 5:30 AM, Daniel Gomez wrote:
> > kdevops is a framework for Linux kernel development and test automation.
> > Its core features, namely workflow reproducibility, variability, and
> > scalability, are delivered through Kconfig, the variability language,
> > and Ansible, which provides host and guest idempotency along with
> > workflow orchestration at scale, whether on baremetal, local VMs, or
> > the cloud.
> > 
> > kdevops supports rolling distributions such as Debian testing, Fedora,
> > and openSUSE. Recently we extended Nix support, which raised the
> > question: how do we drive Nix's declarative language from Ansible? We
> > answered by wiring Nix in under Ansible and its templates, as one more
> > way to declare host and guest environments. But that was the wrong
> > framing: we had bolted Nix onto today's toolkit instead of rethinking
> > it. Reproducibility and idempotency now come from Nix by construction,
> > so I think Ansible's original reason for being in kdevops falls away.
> > The better question is: how do we keep kdevops's core principles, lean
> > on Nix, and drop Ansible?
> > 
> > What remains once you do is not the configuration management plane. It
> > is development workflow orchestration: build QEMU, build the kernel,
> > build a guest rootfs/closure, boot it, run a test, collect results,
> > diff against a baseline. That work is imperative and sequenced, work
> > for a workflow engine, which is where tools like Windmill [1] come in.
> > Windmill calls itself as "the fastest workflow engine" and an
> > "open-source developer platform to power your entire infra and turn
> > scripts into webhooks, workflows and UIs." Choosing to move kdevops
> > onto Windmill would keep what made kdevops kdevops, namely workflows,
> > quick bring-ups, baselines, and A/B regression detection, while trading
> > Kconfig, Make, Ansible, and host-distro provisioning for typed
> > run-forms, flows as code, and a worker queue. Nix supplies the
> > environment, much like a container or venv/poetry, along with the guest
> > OS system closure: declarative and portable. Windmill orchestrates the
> > whole pipeline end to end, graduating kdevops into a fully reproducible,
> > scalable, and configurable kernel-development framework, with both a UI
> > and a CLI, that runs locally or in the cloud. Defined as code and driven
> > by schedules and triggers, the same flows also make it a continuous
> > integration pipeline. Because steps can be written in any language
> > Windmill supports, including Ansible, Bash, Go, Python, and Rust,
> > developers can not only use kdevops but extend it with their own
> > scripts, turning it into a workflow hub. Note that choosing this path
> > does not mean NixOS is required on the controller node; Nix is simply a
> > runtime dependency that can be installed alongside your distro of
> > choice.
> > 
> > It'd be good to know what folks think about the possibility of evolving
> > kdevops in this direction, deprecating Ansible along with Kconfig and
> > Makefiles in favor of the new approach. To that end, I suggest a demo
> > day where I can show why I think this is the next step worth taking, and
> > whether it's a tradeoff users and maintainers are willing to make.
> > 
> > If this is of interest and you'd like a look, I've ported equivalents of
> > bootlinux (direct boot), qemu-build, and the systemd/QEMU bringup (QSU),
> > plus an fstests run for XFS in the proof-of-concept demo project [2].
> > You can also find some screenshots in [3].
> > 
> > A note on licensing. Windmill's engine is AGPLv3; its OpenFlow flow
> > format and client libraries are Apache-2.0. kdevops-ng runs Windmill
> > unmodified and self-hosted as a separate service, and the flows and
> > scripts are kdevops-ng's own copyleft-next-0.3.1 code, executed by
> > Windmill rather than derived from it, so there shouldn't be any
> > licensing concerns.
> > 
> > [1] https://windmill.dev
> > [2] https://github.com/dagomez137/kdevops-ng
> > [3] https://github.com/dagomez137/kdevops-ng/tree/main/screenshots
> 
> Agreed that Ansible is evolving faster than is convenient for us
> and kdevops' use of it seems to be laden with technical debt.
> 
> I don't object at all to creative forward thinking, and I am not
> outright objecting to the proposal, but my concerns are:
> 
> + How much development effort is this? I have nearly zero time to
>   work on kdevops these days. Even with AI assistance, this seems
>   like an enormous task.
> 

Ditto. I'm mostly a user of kdevops these days and don't have as much
time to work on it.

> + Also as a user of kdevops, this means I would have to spend a
>   lot of time I don't have learning the new configuration shapes
>   and figuring out how to migrate my current CI to them. That's
>   not inviting.
> 
> + What would cloud support look like in a post-Ansible world? I
>   don't think Nix has impact on the cloud aspect of kdevops, and this
>   proposal seems to shift Nix into the role of the primary kdevops
>   virtualization method. Should we simply split kdevops into an "all
>   local virtualization" project and an "all cloud virtualization"
>   project that go their separate ways? Tying cloud and local together
>   seems to be a frequent source of chafing.
> 

It does.

OTOH, some parts of kdevops (the test running bits, in particular) are
mostly agnostic to the backend where it's being run. I don't think we
want to have to replicate that part.

Maybe we need to split the whole thing into two -- one part that does
the bringup and setup, and one part that does the test running?

> + I'd rather see a focus on addressing the technical debt of
>   continuing to support distributions that no one uses any more,
>   and moving away from requiring root to run local virtualization
>   by installing kdevops from a package. Those are narrower work
>   items that offer a bigger bang-for-buck. (You might argue that
>   Nix solves the latter problem already).

I already don't run as root for local virtualization. I thought
everyone did? Some of the initial setup requires root, but that's sort
of a separate problem.
-- 
Jeff Layton <jlayton@kernel.org>