From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C24701DA24 for ; Mon, 13 May 2024 21:01:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715634074; cv=none; b=OkRCOu/Mqm3Z3rfI1lgUjQd2oXDTbmNLFHyecJj6XEEy7E4fy3AWOb+l3bNtilxgpmV7SmWGYD8XVRj1DYRk9/v4URxbCIA7BMjuNt+Xhp1ky31bIUmv4ygvFZ4RdeZs6lfY7BcmhF8vRZhIggxVpkMvYF2D/Iil5OeiPMpJeOQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715634074; c=relaxed/simple; bh=xt9PvI8zR1M4E7ZrUVa6Cf8Tkls+YzVfA7nMKA+5DQM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: In-Reply-To:Content-Type:Content-Disposition; b=NDs506wMmk4xqBR1HSBnOSZ/AP4lxvNhM2wXfwFWJ/gKfkZ8xf3qLBtYm8gSlxnXqmft4oZnNZ9OGod1vfeTHN3BKklVSZduU2yqYxNDvKgzzJlqxCBiK1bcPhWEmuexdbNVzTcrVCWhug2PbEoqn2oGYUcAfXR49pTAHGbmkW4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=SeSqByeW; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SeSqByeW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1715634070; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dKbcR2LcRaCoemASzWumJgf5wIESw94ggwqnB2F1VMk=; b=SeSqByeW7LJ9ShHvg/wLW0M/Ws5traPcCCiWngEwyWeTYGeV2TvtuURY8zJ/aO5Y8aNfbA OsIH9+QHYOPh3jbmkX240AzyO1ia0ut9uuDN19jzWuFh2oKfnEdw+MX6WhxSB2xm4TsElu MeqsS8OJA05Uh2vthDQ4o+rVhI31a/o= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-465-seowgfsPN7yk7IIvPEBlqw-1; Mon, 13 May 2024 16:55:05 -0400 X-MC-Unique: seowgfsPN7yk7IIvPEBlqw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2E9C03C025CB; Mon, 13 May 2024 20:55:05 +0000 (UTC) Received: from aion.redhat.com (unknown [10.22.17.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1501E40C6EB7; Mon, 13 May 2024 20:55:05 +0000 (UTC) Received: by aion.redhat.com (Postfix, from userid 1000) id B25AE14AE4E; Mon, 13 May 2024 16:55:04 -0400 (EDT) Date: Mon, 13 May 2024 16:55:04 -0400 From: Scott Mayhew To: Luis Chamberlain Cc: "Richard W.M. Jones" , kdevops@lists.linux.dev Subject: Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs Message-ID: References: <20240508065039.3408637-1-mcgrof@kernel.org> <20240508065039.3408637-6-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sat, 11 May 2024, Luis Chamberlain wrote: > On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote: > > On Tue, 07 May 2024, Luis Chamberlain wrote: > > > > > debian does not yet provide an index file for virt-builder, but we > > > have URLS with qcow2 and raw files. Using them in guestfs is actually > > > not quite trivial. So we do the handy work to enable others to also > > > use custom URLs and build a virt-builder local source and index file > > > for you. All we need really, is to check the checksums. > > > > > > Sadly this does not yet work, as it seems we get stuck on the grub > > > prompt for some reason. This needs some more investigation. > > > > Yeah when I run 'make bringup' the consoles on all the guests are stuck > > on "Booting `Debian GNU/Linux'". > > Is that with Trixie or also with y9our own custom ISO? That was with the Trixie images from cloud.debian.org. I poked around the image using guestfish, and it looks like it needs to use UEFI boot... so I tried that: $ virt-builder debian-13-genericcloud-amd64-daily -o debian13.qcow2 --format qcow2 --size 20G [ 3.0] Downloading: file:////var/lib/libvirt/images/kdevops/smayhew/kdevops/guestfs/custom_images/debian-13-genericcloud-amd64-daily/debian-13-genericcloud-amd64-daily.raw [ 6.8] Planning how to build this image [ 6.8] Resizing (using virt-resize) to expand the disk to 20.0G [ 30.3] Opening the new disk [ 41.9] Setting a random seed virt-builder: warning: random seed could not be set for this type of guest [ 41.9] Setting the machine ID in /etc/machine-id [ 41.9] Setting passwords virt-builder: Setting random password of root to LLq8WTqQJ4Hs3U2z [ 42.9] SELinux relabelling [ 42.9] Finishing off Output file: debian13.qcow2 Output size: 20.0G Output format: qcow2 Total usable space: 19.5G Free space: 18.9G (96%) $ virt-install --connect qemu:///session --import --name debian13 --vcpus 2 --ram 2048 --disk /home/smayhew/debian13.qcow2 --network bridge=virbr0 --os-variant debian12 --graphics none --noautoconsole --noreboot --boot uefi Starting install... Creating domain... | 0 B 00:00:00 Domain creation completed. You can restart your domain by running: virsh --connect qemu:///session start debian13 $ virsh -c qemu:///session start debian13 Domain 'debian13' started $ virsh -c qemu:///session console debian13 Connected to domain 'debian13' Escape character is ^] (Ctrl + ]) localhost login: root Password: Linux localhost 6.7.12-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1 (2024-04-24) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@localhost:~# cat /etc/os-release PRETTY_NAME="Debian GNU/Linux trixie/sid" NAME="Debian GNU/Linux" VERSION_CODENAME=trixie ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" root@localhost:~# logout So at least the VM booted. Since kdevops using using virsh-define with xml templates instead of virt-install, I guess you'll need a tempate with something like this for those images? $ virsh -c qemu:///session dumpxml --xpath '/domain/os' debian13 hvm /usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2 /home/smayhew/.config/libvirt/qemu/nvram/debian13_VARS.qcow2 $ virsh -c qemu:///session destroy debian13 Domain 'debian13' destroyed And also you need to pass --nvram to get rid of the VMs afterward. $ virsh -c qemu:///session undefine debian13 error: Failed to undefine domain 'debian13' error: Requested operation is not valid: cannot undefine domain with nvram $ virsh -c qemu:///session undefine --nvram debian13 Domain 'debian13' has been undefined -Scott > > > I tried manually running virt-builder + virt-install using the same > > image and I get the same result. Granted, I've only ever used the > > images in the libguestfs repos or images that I've built myself, so I > > could be doing something wrong. > > I'm in hopes we can figure this out, there is likely something special > done on the images hosted by libguestfs.org which regular nocloud images > like debian's trixie is not doing: > > https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw > > Once we figure that out, we should be able to easily do that as a post > up with virt-builder. > > Richard, any hints on what likely should help fix boot? > > > > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs > > > index 5838522908e8..5839fbedfd08 100644 > > > --- a/kconfigs/Kconfig.guestfs > > > +++ b/kconfigs/Kconfig.guestfs > > > @@ -1,5 +1,30 @@ > > > if GUESTFS > > > > > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE > > > + bool > > > > It might be useful for these variables to have prompts so they show up > > in 'make menuconfig'. > > That's a one line change, it just needs a description, for now its auto > because there is no custom raw image URL option, what you descrie seems > for us to want such a new entry and option. So sure, but I think that > can be done in subsequent patch? > > > I have a whole bunch of workflows running for > > RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3 > > images > > I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE > but rather waht we want is just a drop down menu for distro to let you > select "custom", that way you could use a set of defconfigs which would > use the custom image URL, when that is enabled it selects > GUESTFS_HAS_CUSTOM_RAW_IMAGE. > > > and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to > > the latest bits. > > My patch after this "guestfs: add support to infer host distro > mirrororing optimizations" helps to infer when we want to copy a > custom sources list, based on a host's hop count. What you describe > seems to beg for another auto-infererence which could be based on > some target directory and based on on the custom URL. That could be > added and then you wouldn't even need to make any custom .config other > than having a few defconfigs for each distro point release. > > > If I could build my own images and dump them in a > > directory without having to worry about updating the index file that > > would probably speed things up a bit. > > I think this is possible, we could just look for directory we add to > .gitignore, and if the user has it as a symlink, we leverage that as > a key indicator for "hunting season open for custom images, please > populate my Kconfig with these other custom options". > > > > + > > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL > > > + bool > > > + > > > +config GUESTFS_CUSTOM_RAW_IMAGE_URL > > > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE > > > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL > > > + string > > > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64 > > > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64 > > > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64 > > > > You need '.raw' at the end of the URL, right? > > Oops yes sorry. > > > > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh > > > index f90ed499051b..514a26a60436 100755 > > > --- a/scripts/bringup_guestfs.sh > > > +++ b/scripts/bringup_guestfs.sh > > > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs" > > > OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION} > > > BASE_IMAGE_DIR="${STORAGEDIR}/base_images" > > > BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw" > > > + > > > +build_custom_source() > > > +{ > > > + SOURCE_TMP=$(mktemp) > > > + cat <<_EOT >$SOURCE_TMP > > > +[local] > > ^^^ > > You probably want this to be unique based on the flavor. If you switch > > flavors, then you're going to wind up with multiple repo configs with the > > same repo id and virt-builder will only pick up the last one it reads... > > the end result being that it probably won't find the template you want > > to install. > > So this is for the source, not the index, the source is local, can we > not have multiple arbitrary sources with the same name? The indexes for > each file would be different. Here I just wanted to emphasize that the > source is local. > > > > > > +uri=file:///${CUSTOM_INDEX} > > > +proxy=off > > > +_EOT > > See the source, points to a custom index, and so we'd have multiple > local sources, the index files do change per flavor. > > But certainly, it would be good to get this right to make this scale. > For now to start off with trixie, and later to scale to whatever custom > setup we need for random builds. > > Thoughts / advice? > > Anyway, since this still has a big fat warning as "this doesn't work > yet", I've made the changes required so far and pushed these patches in, > we can make changes to scale this, now that at laest its easier to test > these things. > > Luis >