guestfs storage paths are confusing

guestfs storage paths are confusing

[Chuck Lever]

Hi -

When I run "make defconfig-xxx" and then "make", .extra_vars_auto.yaml
contains:

> kdevops_enable_guestfs: True
> storage_pool_path: "/var/lib/libvirt/images"
> guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/kdevops/guestfs"
> guestfs_base_image_dir: "{{ guestfs_storage_dir }}/base_images"
> libvirt_storage_pool_path: "/var/lib/libvirt/images"
> kdevops_storage_pool_path: "{{ libvirt_storage_pool_path }}/kdevops"
> libvirt_extra_storage_drive_virtio: True


After templating, that makes guestfs_storage_dir:

   /var/lib/libvirt/images/kdevops/kdevops/guestfs

Which seems incorrect to me. Perhaps guestfs_storage_dir
should be defined as simply:

  guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/guestfs" ??

Also, I'd still like to replace the "kdevops" string literal in the
kdevops_storage_pool_path variable with:

  {{ kdevops_storage_pool_user }}

So that multiple users on one system that run kdevops can't clobber each
other's virtual machines or block devices.

-- 
Chuck Lever

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Tue, Apr 08, 2025 at 03:17:29PM -0400, Chuck Lever wrote:
> Hi -
> 
> When I run "make defconfig-xxx" and then "make", .extra_vars_auto.yaml
> contains:
> 
> > kdevops_enable_guestfs: True
> > storage_pool_path: "/var/lib/libvirt/images"
> > guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/kdevops/guestfs"
> > guestfs_base_image_dir: "{{ guestfs_storage_dir }}/base_images"
> > libvirt_storage_pool_path: "/var/lib/libvirt/images"
> > kdevops_storage_pool_path: "{{ libvirt_storage_pool_path }}/kdevops"
> > libvirt_extra_storage_drive_virtio: True
> 
> 
> After templating, that makes guestfs_storage_dir:
> 
>    /var/lib/libvirt/images/kdevops/kdevops/guestfs
> 
> Which seems incorrect to me. Perhaps guestfs_storage_dir
> should be defined as simply:
> 
>   guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/guestfs" ??

Agreed.

> Also, I'd still like to replace the "kdevops" string literal in the
> kdevops_storage_pool_path variable with:
> 
>   {{ kdevops_storage_pool_user }}
> 
> So that multiple users on one system that run kdevops can't clobber each
> other's virtual machines or block devices.

Here's an issue with that too. For libvirt URI system  users they
actually *share* a path with root user, ie, root user has the cache
and the virt-builder cache. Which is also a source of problems
when cleaing up the cache, the clean cache target (I forget what
its called) doesn't clean the root user's virt-builder cache for
libvirt system uri users.

In fact the whole libvirt system uri thing has been such a pain in the
ass over the years it makes me wonder if we should just start requiring
the libvirt system session down users even on debian... We special case
so many times the system session and its what mostly has us requiring
sudo to check pools etc.

  Luis

Re: guestfs storage paths are confusing

[Chuck Lever]

On 4/8/25 3:29 PM, Luis Chamberlain wrote:
> On Tue, Apr 08, 2025 at 03:17:29PM -0400, Chuck Lever wrote:
>> Hi -
>>
>> When I run "make defconfig-xxx" and then "make", .extra_vars_auto.yaml
>> contains:
>>
>>> kdevops_enable_guestfs: True
>>> storage_pool_path: "/var/lib/libvirt/images"
>>> guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/kdevops/guestfs"
>>> guestfs_base_image_dir: "{{ guestfs_storage_dir }}/base_images"
>>> libvirt_storage_pool_path: "/var/lib/libvirt/images"
>>> kdevops_storage_pool_path: "{{ libvirt_storage_pool_path }}/kdevops"
>>> libvirt_extra_storage_drive_virtio: True
>>
>>
>> After templating, that makes guestfs_storage_dir:
>>
>>    /var/lib/libvirt/images/kdevops/kdevops/guestfs
>>
>> Which seems incorrect to me. Perhaps guestfs_storage_dir
>> should be defined as simply:
>>
>>   guestfs_storage_dir: "{{ kdevops_storage_pool_path }}/guestfs" ??
> 
> Agreed.
> 
>> Also, I'd still like to replace the "kdevops" string literal in the
>> kdevops_storage_pool_path variable with:
>>
>>   {{ kdevops_storage_pool_user }}
>>
>> So that multiple users on one system that run kdevops can't clobber each
>> other's virtual machines or block devices.
> 
> Here's an issue with that too. For libvirt URI system  users they
> actually *share* a path with root user, ie, root user has the cache
> and the virt-builder cache. Which is also a source of problems
> when cleaing up the cache, the clean cache target (I forget what
> its called) doesn't clean the root user's virt-builder cache for
> libvirt system uri users.
> 
> In fact the whole libvirt system uri thing has been such a pain in the
> ass over the years it makes me wonder if we should just start requiring
> the libvirt system session down users even on debian... We special case
> so many times the system session and its what mostly has us requiring
> sudo to check pools etc.

One complaint about kdevops I sometimes hear is the requirement for
frequent root access on the control host. kdevops is better off reducing
its privilege footprint, IMHO.

During kdevops installation, I can see the need for root access. During
operation, kdevops should live as a non-privileged agent, if possible.
That is the best common security practice (on the control host,
obviously; Ansible should of course be allowed root on target nodes).


-- 
Chuck Lever

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
> One complaint about kdevops I sometimes hear is the requirement for
> frequent root access on the control host. kdevops is better off reducing
> its privilege footprint, IMHO.

I agree, I think at this point it its clear that although kdevops has
ways to help with a first bringup, that just scares a larger crowd. And
we can simplify our code if we completley move all those requirements
to an *optional* installation phase, done only once.

> During kdevops installation, I can see the need for root access.

If we're going to support "installation" I'm becoming more of a fan of
unifying a way to use libvirt than supporting these two differen qemu
session types. And I think the winner so far is user session. So how
difficult is it to get debian based distros to use user session? Not
sure but I think we should try it and just support it. Supporting both
is just a nighmare and user session is "no root" world".

Any thoughts on this?

I thing adding Nix OS support wil help us get here, as we can just
require Nix OS users to setup libvirt user sesion. Thoughts Andreas?

> During
> operation, kdevops should live as a non-privileged agent, if possible.
> That is the best common security practice (on the control host,
> obviously; Ansible should of course be allowed root on target nodes).

Agreed.

Last low hanging fruit should be selinux / apparmor.

  Luis

Re: guestfs storage paths are confusing

[Chuck Lever]

On 4/9/25 2:10 PM, Luis Chamberlain wrote:
> On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
>> One complaint about kdevops I sometimes hear is the requirement for
>> frequent root access on the control host. kdevops is better off reducing
>> its privilege footprint, IMHO.
> 
> I agree, I think at this point it its clear that although kdevops has
> ways to help with a first bringup, that just scares a larger crowd. And
> we can simplify our code if we completley move all those requirements
> to an *optional* installation phase, done only once.
> 
>> During kdevops installation, I can see the need for root access.
> 
> If we're going to support "installation" I'm becoming more of a fan of
> unifying a way to use libvirt than supporting these two differen qemu
> session types. And I think the winner so far is user session. So how
> difficult is it to get debian based distros to use user session? Not
> sure but I think we should try it and just support it. Supporting both
> is just a nighmare and user session is "no root" world".
> 
> Any thoughts on this?

You are asking about libvirt on Debian specifically, but that's not
something I have much expertise with. But I have a bigger picture
thought:

Make kdevops installation look more like other packaged applications,
with the distro's package manager responsible for pulling in the
dependent packages. Since kdevops lives on GH, create Actions that
can do the packaging for us.

I can envision "sudo dnf install kdevops" and then the dependent
packages are responsible for setting themselves up according to the
distro's needs.

That's pie-in-the-sky. I have no idea how realistic that is. But I
think it would cover libvirt and ansible on all recent distros, and
would help move most of the installation work under a single privilege
escalation.

Things like terraform are downloaded, though; no-one packages that right
now. But downloading terraform seems like something that should happen
during installation, not as a part of "make".

It might also help to have a deprecation schedule for older
distributions. "Debian Stretch no longer supported on the control host".
Also SLES 12, and maybe RHEL/CentOS 6. So then kdevops can drop the
old cruft that used to be necessary to crank-start the thing.

My 2 cents...


> I thing adding Nix OS support will help us get here, as we can just
> require Nix OS users to setup libvirt user session. Thoughts Andreas?
> 
>> During
>> operation, kdevops should live as a non-privileged agent, if possible.
>> That is the best common security practice (on the control host,
>> obviously; Ansible should of course be allowed root on target nodes).
> 
> Agreed.
> 
> Last low hanging fruit should be selinux / apparmor.
> 
>   Luis


-- 
Chuck Lever

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Wed, Apr 09, 2025 at 02:36:56PM -0400, Chuck Lever wrote:
> On 4/9/25 2:10 PM, Luis Chamberlain wrote:
> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
> >> One complaint about kdevops I sometimes hear is the requirement for
> >> frequent root access on the control host. kdevops is better off reducing
> >> its privilege footprint, IMHO.
> > 
> > I agree, I think at this point it its clear that although kdevops has
> > ways to help with a first bringup, that just scares a larger crowd. And
> > we can simplify our code if we completley move all those requirements
> > to an *optional* installation phase, done only once.
> > 
> >> During kdevops installation, I can see the need for root access.
> > 
> > If we're going to support "installation" I'm becoming more of a fan of
> > unifying a way to use libvirt than supporting these two differen qemu
> > session types. And I think the winner so far is user session. So how
> > difficult is it to get debian based distros to use user session? Not
> > sure but I think we should try it and just support it. Supporting both
> > is just a nighmare and user session is "no root" world".
> > 
> > Any thoughts on this?
> 
> You are asking about libvirt on Debian specifically, but that's not
> something I have much expertise with. But I have a bigger picture
> thought:
> 
> Make kdevops installation look more like other packaged applications,
> with the distro's package manager responsible for pulling in the
> dependent packages. Since kdevops lives on GH, create Actions that
> can do the packaging for us.
> 
> I can envision "sudo dnf install kdevops" and then the dependent
> packages are responsible for setting themselves up according to the
> distro's needs.

Works for me.

> That's pie-in-the-sky. I have no idea how realistic that is. But I
> think it would cover libvirt and ansible on all recent distros, and
> would help move most of the installation work under a single privilege
> escalation.
> 
> Things like terraform are downloaded, though; no-one packages that right
> now. But downloading terraform seems like something that should happen
> during installation, not as a part of "make".
> 
> It might also help to have a deprecation schedule for older
> distributions. "Debian Stretch no longer supported on the control host".
> Also SLES 12, and maybe RHEL/CentOS 6. So then kdevops can drop the
> old cruft that used to be necessary to crank-start the thing.

Makes sense too.

 Luis

Re: guestfs storage paths are confusing

[Andreas Hindborg]

"Luis Chamberlain" <mcgrof@kernel.org> writes:

> On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
>> One complaint about kdevops I sometimes hear is the requirement for
>> frequent root access on the control host. kdevops is better off reducing
>> its privilege footprint, IMHO.
>
> I agree, I think at this point it its clear that although kdevops has
> ways to help with a first bringup, that just scares a larger crowd. And
> we can simplify our code if we completley move all those requirements
> to an *optional* installation phase, done only once.

The real question is, why does kdevops even need root? If you can rely
on user space networking with slirp, nothing in the process of spinning
up a VM should require root privileges. If the user has access to kvm,
which is common these days.


Best regards,
Andreas Hindborg

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> 
> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
> >> One complaint about kdevops I sometimes hear is the requirement for
> >> frequent root access on the control host. kdevops is better off reducing
> >> its privilege footprint, IMHO.
> >
> > I agree, I think at this point it its clear that although kdevops has
> > ways to help with a first bringup, that just scares a larger crowd. And
> > we can simplify our code if we completley move all those requirements
> > to an *optional* installation phase, done only once.
> 
> The real question is, why does kdevops even need root? If you can rely
> on user space networking with slirp, 

By default  edora distros use libvirt with user session, debian distros
use the system session. What do you mean by a slirp exactly? Got a
config for it?

> nothing in the process of spinning
> up a VM should require root privileges. If the user has access to kvm,
> which is common these days.

Yeah sure, agreed. Then we just needs docs for what's needed or
an optional installation path.

  Luis

Re: guestfs storage paths are confusing

[Andreas Hindborg]

"Luis Chamberlain" <mcgrof@kernel.org> writes:

> On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
>> "Luis Chamberlain" <mcgrof@kernel.org> writes:
>>
>> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
>> >> One complaint about kdevops I sometimes hear is the requirement for
>> >> frequent root access on the control host. kdevops is better off reducing
>> >> its privilege footprint, IMHO.
>> >
>> > I agree, I think at this point it its clear that although kdevops has
>> > ways to help with a first bringup, that just scares a larger crowd. And
>> > we can simplify our code if we completley move all those requirements
>> > to an *optional* installation phase, done only once.
>>
>> The real question is, why does kdevops even need root? If you can rely
>> on user space networking with slirp,
>
> By default  edora distros use libvirt with user session, debian distros
> use the system session. What do you mean by a slirp exactly? Got a
> config for it?

Slirp is the user mode network stack you get with `-netdev user,...`
[1].

I don't think libvirt is fitting for kdevops use case either. You can
just run qemu directly. libvirt makes sense if you want to spin up
virtual machines across a cluster in an enterprise setting. It is made
to be consumed by the next layer of management software in the stack,
not fore direct user interaction. KDevops deployments are not complex
enough to benefit from libvirt.

>> nothing in the process of spinning
>> up a VM should require root privileges. If the user has access to kvm,
>> which is common these days.
>
> Yeah sure, agreed. Then we just needs docs for what's needed or
> an optional installation path.

You can compose the qemu command from ansible based on your kconfig
values. Everything you need is in the qemu manual/wiki. There are plenty
of qemu wrappers to gather inspiration from, such as vm-ctl [2],
virtme-ng [3], run-kernel [4]. I wrote the last one.


Best regards,
Andreas Hindborg



[1] https://wiki.qemu.org/Documentation/Networking
[2] https://github.com/SamsungDS/vmctl
[3] https://github.com/arighi/virtme-ng
[4] https://github.com/metaspace/run-kernel

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Wed, Apr 23, 2025 at 08:22:50AM +0000, Andreas Hindborg wrote:
> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> 
> > On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
> >> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> >>
> >> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
> >> >> One complaint about kdevops I sometimes hear is the requirement for
> >> >> frequent root access on the control host. kdevops is better off reducing
> >> >> its privilege footprint, IMHO.
> >> >
> >> > I agree, I think at this point it its clear that although kdevops has
> >> > ways to help with a first bringup, that just scares a larger crowd. And
> >> > we can simplify our code if we completley move all those requirements
> >> > to an *optional* installation phase, done only once.
> >>
> >> The real question is, why does kdevops even need root? If you can rely
> >> on user space networking with slirp,
> >
> > By default  edora distros use libvirt with user session, debian distros
> > use the system session. What do you mean by a slirp exactly? Got a
> > config for it?
> 
> Slirp is the user mode network stack you get with `-netdev user,...`
> [1].

That seems like its the same thing as what we refer to as user session.
Ie, you need qemu:///session to use slirp, instead of the debian
default of system session qemu:///system.

> I don't think libvirt is fitting for kdevops use case either. You can
> just run qemu directly. libvirt makes sense if you want to spin up
> virtual machines across a cluster in an enterprise setting. It is made
> to be consumed by the next layer of management software in the stack,
> not fore direct user interaction. KDevops deployments are not complex
> enough to benefit from libvirt.

Just because there is a use case for a simple qemu throw away guest
does not mean there's no reason for users of kdevops to leverage
libvirt. kdevops was built to *scale*. Dozens of guest, not just one
or two. Sometimes over 50. In such cases -- those hosts can benefit
from uber optimizaitons. Which is why we have hypervisor optimizations
we've learned over the years to enable on the host as well, like KSM and
zswap. libvirt by default can leverage a lot of CoW mechanisms if you
share the same guest. 50 guests * simple qemu without optimal libvirt
qemu options to share a base file doesn't scale so well.

So yes, we can have a "I am ok to shoot myself on the foot with my own
virtualization configuration don't worry" option, it does not negate the
goal to also *scale* to about 50 guests even with Nix OS optimally.

That's how we scale for CIs.

> You can compose the qemu command from ansible based on your kconfig
> values. Everything you need is in the qemu manual/wiki. There are plenty
> of qemu wrappers to gather inspiration from, such as vm-ctl [2],
> virtme-ng [3], run-kernel [4]. I wrote the last one.

Until I get to it, patches welcomed :)

  Luis

Re: guestfs storage paths are confusing

[Andreas Hindborg]

"Luis Chamberlain" <mcgrof@kernel.org> writes:

> On Wed, Apr 23, 2025 at 08:22:50AM +0000, Andreas Hindborg wrote:
>> "Luis Chamberlain" <mcgrof@kernel.org> writes:
>>
>> > On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
>> >> "Luis Chamberlain" <mcgrof@kernel.org> writes:
>> >>
>> >> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
>> >> >> One complaint about kdevops I sometimes hear is the requirement for
>> >> >> frequent root access on the control host. kdevops is better off reducing
>> >> >> its privilege footprint, IMHO.
>> >> >
>> >> > I agree, I think at this point it its clear that although kdevops has
>> >> > ways to help with a first bringup, that just scares a larger crowd. And
>> >> > we can simplify our code if we completley move all those requirements
>> >> > to an *optional* installation phase, done only once.
>> >>
>> >> The real question is, why does kdevops even need root? If you can rely
>> >> on user space networking with slirp,
>> >
>> > By default  edora distros use libvirt with user session, debian distros
>> > use the system session. What do you mean by a slirp exactly? Got a
>> > config for it?
>>
>> Slirp is the user mode network stack you get with `-netdev user,...`
>> [1].
>
> That seems like its the same thing as what we refer to as user session.
> Ie, you need qemu:///session to use slirp, instead of the debian
> default of system session qemu:///system.
>
>> I don't think libvirt is fitting for kdevops use case either. You can
>> just run qemu directly. libvirt makes sense if you want to spin up
>> virtual machines across a cluster in an enterprise setting. It is made
>> to be consumed by the next layer of management software in the stack,
>> not fore direct user interaction. KDevops deployments are not complex
>> enough to benefit from libvirt.
>
> Just because there is a use case for a simple qemu throw away guest
> does not mean there's no reason for users of kdevops to leverage
> libvirt. kdevops was built to *scale*. Dozens of guest, not just one
> or two. Sometimes over 50. In such cases -- those hosts can benefit
> from uber optimizaitons. Which is why we have hypervisor optimizations
> we've learned over the years to enable on the host as well, like KSM and
> zswap. libvirt by default can leverage a lot of CoW mechanisms if you
> share the same guest. 50 guests * simple qemu without optimal libvirt
> qemu options to share a base file doesn't scale so well.

It's still just qemu below libvirt and the same optimizations can be
applied if you run qemu directly. Base image sharing with CoW works fine
with bare qemu.

Although I must admit that if you want to spawn VMs across a fleet of
hosts, libvirt is fitting in the sense that it gives you a single remote
management interface to manage each node of the fleet.

> So yes, we can have a "I am ok to shoot myself on the foot with my own
> virtualization configuration don't worry" option, it does not negate the
> goal to also *scale* to about 50 guests even with Nix OS optimally.

Right, I understand.

It's all about perspective. For me and my use case, the scalability
features built into kdevops is a barrier of entry.

>
> That's how we scale for CIs.
>
>> You can compose the qemu command from ansible based on your kconfig
>> values. Everything you need is in the qemu manual/wiki. There are plenty
>> of qemu wrappers to gather inspiration from, such as vm-ctl [2],
>> virtme-ng [3], run-kernel [4]. I wrote the last one.
>
> Until I get to it, patches welcomed :)

I really hope I will be able to do that in a somewhat timely manner. I
also want to contribute the diskless NixOS guest configuration I have
been playing with. That cuts out the entire image creation loop by
mapping the nix store into the VM via virtio-fs.


Best regards,
Andreas Hindborg

Re: guestfs storage paths are confusing

[Luis Chamberlain]

On Fri, Apr 25, 2025 at 07:30:07AM +0000, Andreas Hindborg wrote:
> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> 
> > On Wed, Apr 23, 2025 at 08:22:50AM +0000, Andreas Hindborg wrote:
> >> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> >>
> >> > On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
> >> >> "Luis Chamberlain" <mcgrof@kernel.org> writes:
> >> >>
> >> >> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
> >> >> >> One complaint about kdevops I sometimes hear is the requirement for
> >> >> >> frequent root access on the control host. kdevops is better off reducing
> >> >> >> its privilege footprint, IMHO.
> >> >> >
> >> >> > I agree, I think at this point it its clear that although kdevops has
> >> >> > ways to help with a first bringup, that just scares a larger crowd. And
> >> >> > we can simplify our code if we completley move all those requirements
> >> >> > to an *optional* installation phase, done only once.
> >> >>
> >> >> The real question is, why does kdevops even need root? If you can rely
> >> >> on user space networking with slirp,
> >> >
> >> > By default  edora distros use libvirt with user session, debian distros
> >> > use the system session. What do you mean by a slirp exactly? Got a
> >> > config for it?
> >>
> >> Slirp is the user mode network stack you get with `-netdev user,...`
> >> [1].
> >
> > That seems like its the same thing as what we refer to as user session.
> > Ie, you need qemu:///session to use slirp, instead of the debian
> > default of system session qemu:///system.
> >
> >> I don't think libvirt is fitting for kdevops use case either. You can
> >> just run qemu directly. libvirt makes sense if you want to spin up
> >> virtual machines across a cluster in an enterprise setting. It is made
> >> to be consumed by the next layer of management software in the stack,
> >> not fore direct user interaction. KDevops deployments are not complex
> >> enough to benefit from libvirt.
> >
> > Just because there is a use case for a simple qemu throw away guest
> > does not mean there's no reason for users of kdevops to leverage
> > libvirt. kdevops was built to *scale*. Dozens of guest, not just one
> > or two. Sometimes over 50. In such cases -- those hosts can benefit
> > from uber optimizaitons. Which is why we have hypervisor optimizations
> > we've learned over the years to enable on the host as well, like KSM and
> > zswap. libvirt by default can leverage a lot of CoW mechanisms if you
> > share the same guest. 50 guests * simple qemu without optimal libvirt
> > qemu options to share a base file doesn't scale so well.
> 
> It's still just qemu below libvirt and the same optimizations can be
> applied if you run qemu directly. Base image sharing with CoW works fine
> with bare qemu.

Sure, its possible. It requiers to do the work yourself.

> Although I must admit that if you want to spawn VMs across a fleet of
> hosts, libvirt is fitting in the sense that it gives you a single remote
> management interface to manage each node of the fleet.

That's the point.

Consider buggy CIs, leaving spawned guests around just lingering. If you
don't have a deterministic easy way to query guests a cleanup will bee
hard. Today I can easily clean stale guests up with libvirt.

> It's all about perspective. For me and my use case, the scalability
> features built into kdevops is a barrier of entry.

I get it.

Which is why I also think we should support that use *easier* use case.
We just want a nice deterministic interface, and I hope we can evolve
it to carry the same benefits provided by libvirt. It also doesn't mean
we can't leverage nix os with libvirt too. Both should be supported.

> I also want to contribute the diskless NixOS guest configuration I have
> been playing with. That cuts out the entire image creation loop by
> mapping the nix store into the VM via virtio-fs.

Neatoo!

  Luis