Re: guestfs storage paths are confusing

[Andreas Hindborg <nmi@metaspace.dk>]

"Luis Chamberlain" <mcgrof@kernel.org> writes:

> On Wed, Apr 23, 2025 at 08:22:50AM +0000, Andreas Hindborg wrote:
>> "Luis Chamberlain" <mcgrof@kernel.org> writes:
>>
>> > On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote:
>> >> "Luis Chamberlain" <mcgrof@kernel.org> writes:
>> >>
>> >> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote:
>> >> >> One complaint about kdevops I sometimes hear is the requirement for
>> >> >> frequent root access on the control host. kdevops is better off reducing
>> >> >> its privilege footprint, IMHO.
>> >> >
>> >> > I agree, I think at this point it its clear that although kdevops has
>> >> > ways to help with a first bringup, that just scares a larger crowd. And
>> >> > we can simplify our code if we completley move all those requirements
>> >> > to an *optional* installation phase, done only once.
>> >>
>> >> The real question is, why does kdevops even need root? If you can rely
>> >> on user space networking with slirp,
>> >
>> > By default  edora distros use libvirt with user session, debian distros
>> > use the system session. What do you mean by a slirp exactly? Got a
>> > config for it?
>>
>> Slirp is the user mode network stack you get with `-netdev user,...`
>> [1].
>
> That seems like its the same thing as what we refer to as user session.
> Ie, you need qemu:///session to use slirp, instead of the debian
> default of system session qemu:///system.
>
>> I don't think libvirt is fitting for kdevops use case either. You can
>> just run qemu directly. libvirt makes sense if you want to spin up
>> virtual machines across a cluster in an enterprise setting. It is made
>> to be consumed by the next layer of management software in the stack,
>> not fore direct user interaction. KDevops deployments are not complex
>> enough to benefit from libvirt.
>
> Just because there is a use case for a simple qemu throw away guest
> does not mean there's no reason for users of kdevops to leverage
> libvirt. kdevops was built to *scale*. Dozens of guest, not just one
> or two. Sometimes over 50. In such cases -- those hosts can benefit
> from uber optimizaitons. Which is why we have hypervisor optimizations
> we've learned over the years to enable on the host as well, like KSM and
> zswap. libvirt by default can leverage a lot of CoW mechanisms if you
> share the same guest. 50 guests * simple qemu without optimal libvirt
> qemu options to share a base file doesn't scale so well.

It's still just qemu below libvirt and the same optimizations can be
applied if you run qemu directly. Base image sharing with CoW works fine
with bare qemu.

Although I must admit that if you want to spawn VMs across a fleet of
hosts, libvirt is fitting in the sense that it gives you a single remote
management interface to manage each node of the fleet.

> So yes, we can have a "I am ok to shoot myself on the foot with my own
> virtualization configuration don't worry" option, it does not negate the
> goal to also *scale* to about 50 guests even with Nix OS optimally.

Right, I understand.

It's all about perspective. For me and my use case, the scalability
features built into kdevops is a barrier of entry.

>
> That's how we scale for CIs.
>
>> You can compose the qemu command from ansible based on your kconfig
>> values. Everything you need is in the qemu manual/wiki. There are plenty
>> of qemu wrappers to gather inspiration from, such as vm-ctl [2],
>> virtme-ng [3], run-kernel [4]. I wrote the last one.
>
> Until I get to it, patches welcomed :)

I really hope I will be able to do that in a somewhat timely manner. I
also want to contribute the diskless NixOS guest configuration I have
been playing with. That cuts out the entire image creation loop by
mapping the nix store into the VM via virtio-fs.


Best regards,
Andreas Hindborg