From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-24422.protonmail.ch (mail-24422.protonmail.ch [109.224.244.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0D0C22D7A6 for ; Fri, 25 Apr 2025 07:30:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=109.224.244.22 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745566225; cv=none; b=YcNevVdobjgM0JyUsvMT9ap0MvdMlyf8hwhC5dHl788Pg27G988yZjOyn2IgkwgwdHqRDyMlWaEbPi+yl0smPEP/6IwANf5IrslGkWC1U5rwPZBapTqnMfPd3Y6pUUkb8/rXwuwJGUjLmgZsfJN7+tt7l9AI1wTV685foU8RLew= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745566225; c=relaxed/simple; bh=/kvmBoxXsKxn6pJOyDOx5R7C1VdL+wNEYBV3r6pizmY=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=lOHIgRGU20zba445NUzr4IT6ngJPJYB8AHsSTOT90X/wbpiXbjEV4t8SZCLvPgMyGoG7VZyhVfeBLDkT4Y++b4k9pNuywrt9y+B0NyeD4EWam/hXfMplO7zkwSsFT/oUN1IfhBx5TmhAHhGm9DLyEHegAQZ+tE/JRk+8ulT3rLM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=metaspace.dk; spf=pass smtp.mailfrom=metaspace.dk; dkim=pass (2048-bit key) header.d=metaspace.dk header.i=@metaspace.dk header.b=ku1Z+WMh; arc=none smtp.client-ip=109.224.244.22 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=metaspace.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=metaspace.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=metaspace.dk header.i=@metaspace.dk header.b="ku1Z+WMh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=metaspace.dk; s=protonmail2; t=1745566211; x=1745825411; bh=x1hQUujGTPFUMo9BPkQWhSwDHRopMCZCiVIyuCzB4lQ=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector:List-Unsubscribe:List-Unsubscribe-Post; b=ku1Z+WMhtvoH1tlbKNy+270lRrTt+OjVpacsQUUHk4i/f8qPt86Lu/d6xKZGYqQKZ XasjsyXqayNEAbQ+AerSOHKSF/v35ctQ/Ol62EFfEkdQ+dwBtwLXNXwMdoaZndqMEE v3bd/iZ1x+LqWXAYGycvGUBeX2P19EyHKCRZJYIsjgfI4lbQ528pjTfrpLfq4zoPlO sTeM3MSXMDQfwVRpvuv4GSA9NoFmSfo3IqyLSa3JGAwDTdTfJ4+yO9heDq2uCf9oVI Sx6W4fJzgCaQ9guPcvgfHkR7RNcTABqs+9A+lx19Jw5YNg1v/HtnAZ9n7EtvmdTvSD xePjBWNsOO4Vg== Date: Fri, 25 Apr 2025 07:30:07 +0000 To: Luis Chamberlain From: Andreas Hindborg Cc: Chuck Lever , kdevops@lists.linux.dev Subject: Re: guestfs storage paths are confusing Message-ID: <87v7qsk8b0.fsf@metaspace.dk> In-Reply-To: References: <7adea286-8981-46cd-9751-a7add11d9e85@oracle.com> <87h62wboj8.fsf@metaspace.dk> <8734dznv6z.fsf@metaspace.dk> Feedback-ID: 113830118:user:proton X-Pm-Message-ID: 645224f36b4bba4dce0e571ea590a448f0cfa23e Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable "Luis Chamberlain" writes: > On Wed, Apr 23, 2025 at 08:22:50AM +0000, Andreas Hindborg wrote: >> "Luis Chamberlain" writes: >> >> > On Thu, Apr 10, 2025 at 04:58:59AM +0000, Andreas Hindborg wrote: >> >> "Luis Chamberlain" writes: >> >> >> >> > On Wed, Apr 09, 2025 at 10:55:58AM -0400, Chuck Lever wrote: >> >> >> One complaint about kdevops I sometimes hear is the requirement fo= r >> >> >> frequent root access on the control host. kdevops is better off re= ducing >> >> >> its privilege footprint, IMHO. >> >> > >> >> > I agree, I think at this point it its clear that although kdevops h= as >> >> > ways to help with a first bringup, that just scares a larger crowd.= And >> >> > we can simplify our code if we completley move all those requiremen= ts >> >> > to an *optional* installation phase, done only once. >> >> >> >> The real question is, why does kdevops even need root? If you can rel= y >> >> on user space networking with slirp, >> > >> > By default edora distros use libvirt with user session, debian distro= s >> > use the system session. What do you mean by a slirp exactly? Got a >> > config for it? >> >> Slirp is the user mode network stack you get with `-netdev user,...` >> [1]. > > That seems like its the same thing as what we refer to as user session. > Ie, you need qemu:///session to use slirp, instead of the debian > default of system session qemu:///system. > >> I don't think libvirt is fitting for kdevops use case either. You can >> just run qemu directly. libvirt makes sense if you want to spin up >> virtual machines across a cluster in an enterprise setting. It is made >> to be consumed by the next layer of management software in the stack, >> not fore direct user interaction. KDevops deployments are not complex >> enough to benefit from libvirt. > > Just because there is a use case for a simple qemu throw away guest > does not mean there's no reason for users of kdevops to leverage > libvirt. kdevops was built to *scale*. Dozens of guest, not just one > or two. Sometimes over 50. In such cases -- those hosts can benefit > from uber optimizaitons. Which is why we have hypervisor optimizations > we've learned over the years to enable on the host as well, like KSM and > zswap. libvirt by default can leverage a lot of CoW mechanisms if you > share the same guest. 50 guests * simple qemu without optimal libvirt > qemu options to share a base file doesn't scale so well. It's still just qemu below libvirt and the same optimizations can be applied if you run qemu directly. Base image sharing with CoW works fine with bare qemu. Although I must admit that if you want to spawn VMs across a fleet of hosts, libvirt is fitting in the sense that it gives you a single remote management interface to manage each node of the fleet. > So yes, we can have a "I am ok to shoot myself on the foot with my own > virtualization configuration don't worry" option, it does not negate the > goal to also *scale* to about 50 guests even with Nix OS optimally. Right, I understand. It's all about perspective. For me and my use case, the scalability features built into kdevops is a barrier of entry. > > That's how we scale for CIs. > >> You can compose the qemu command from ansible based on your kconfig >> values. Everything you need is in the qemu manual/wiki. There are plenty >> of qemu wrappers to gather inspiration from, such as vm-ctl [2], >> virtme-ng [3], run-kernel [4]. I wrote the last one. > > Until I get to it, patches welcomed :) I really hope I will be able to do that in a somewhat timely manner. I also want to contribute the diskless NixOS guest configuration I have been playing with. That cuts out the entire image creation loop by mapping the nix store into the VM via virtio-fs. Best regards, Andreas Hindborg