Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Scott Mayhew <smayhew@redhat.com>]

On Sat, 11 May 2024, Luis Chamberlain wrote:

> On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> > On Tue, 07 May 2024, Luis Chamberlain wrote:
> > 
> > > debian does not yet provide an index file for virt-builder, but we
> > > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > > not quite trivial. So we do the handy work to enable others to also
> > > use custom URLs and build a virt-builder local source and index file
> > > for you. All we need really, is to check the checksums.
> > > 
> > > Sadly this does not yet work, as it seems we get stuck on the grub
> > > prompt for some reason. This needs some more investigation.
> > 
> > Yeah when I run 'make bringup' the consoles on all the guests are stuck
> > on "Booting `Debian GNU/Linux'".
> 
> Is that with Trixie or also with y9our own custom ISO?

That was with the Trixie images from cloud.debian.org.

I poked around the image using guestfish, and it looks like it needs to use
UEFI boot... so I tried that:

$ virt-builder debian-13-genericcloud-amd64-daily -o debian13.qcow2 --format qcow2 --size 20G 
[   3.0] Downloading: file:////var/lib/libvirt/images/kdevops/smayhew/kdevops/guestfs/custom_images/debian-13-genericcloud-amd64-daily/debian-13-genericcloud-amd64-daily.raw
[   6.8] Planning how to build this image
[   6.8] Resizing (using virt-resize) to expand the disk to 20.0G
[  30.3] Opening the new disk
[  41.9] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
[  41.9] Setting the machine ID in /etc/machine-id
[  41.9] Setting passwords
virt-builder: Setting random password of root to LLq8WTqQJ4Hs3U2z
[  42.9] SELinux relabelling
[  42.9] Finishing off
                   Output file: debian13.qcow2
                   Output size: 20.0G
                 Output format: qcow2
            Total usable space: 19.5G
                    Free space: 18.9G (96%)

$ virt-install --connect qemu:///session --import --name debian13 --vcpus 2 --ram 2048 --disk /home/smayhew/debian13.qcow2 --network bridge=virbr0 --os-variant debian12 --graphics none --noautoconsole --noreboot --boot uefi

Starting install...
Creating domain...                                                                                                                                                                                                     |    0 B  00:00:00     
Domain creation completed.
You can restart your domain by running:
  virsh --connect qemu:///session start debian13

$ virsh -c qemu:///session start debian13
Domain 'debian13' started

$ virsh -c qemu:///session console debian13
Connected to domain 'debian13'
Escape character is ^] (Ctrl + ])

localhost login: root
Password: 
Linux localhost 6.7.12-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1 (2024-04-24) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@localhost:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@localhost:~# 
logout

So at least the VM booted.

Since kdevops using using virsh-define with xml templates instead of
virt-install, I guess you'll need a tempate with something like this
for those images?

$ virsh -c qemu:///session dumpxml --xpath '/domain/os' debian13
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-8.1">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash" format="qcow2">/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2" format="qcow2">/home/smayhew/.config/libvirt/qemu/nvram/debian13_VARS.qcow2</nvram>
  <boot dev="hd"/>
</os>

$ virsh -c qemu:///session destroy debian13
Domain 'debian13' destroyed

And also you need to pass --nvram to get rid of the VMs afterward.

$ virsh -c qemu:///session undefine debian13
error: Failed to undefine domain 'debian13'
error: Requested operation is not valid: cannot undefine domain with nvram

$ virsh -c qemu:///session undefine --nvram debian13
Domain 'debian13' has been undefined

-Scott
> 
> > I tried manually running virt-builder + virt-install using the same
> > image and I get the same result.  Granted, I've only ever used the
> > images in the libguestfs repos or images that I've built myself, so I
> > could be doing something wrong.
> 
> I'm in hopes we can figure this out, there is likely something special
> done on the images hosted by libguestfs.org which regular nocloud images
> like debian's trixie is not doing:
> 
> https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw
> 
> Once we figure that out, we should be able to easily do that as a post
> up with virt-builder.
> 
> Richard, any hints on what likely should help fix boot?
> 
> > > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > > index 5838522908e8..5839fbedfd08 100644
> > > --- a/kconfigs/Kconfig.guestfs
> > > +++ b/kconfigs/Kconfig.guestfs
> > > @@ -1,5 +1,30 @@
> > >  if GUESTFS
> > >  
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	bool
> > 
> > It might be useful for these variables to have prompts so they show up
> > in 'make menuconfig'. 
> 
> That's a one line change, it just needs a description, for now its auto
> because there is no custom raw image URL option, what you descrie seems
> for us to want such a new entry and option. So sure, but I think that
> can be done in subsequent patch?
> 
> > I have a whole bunch of workflows running for
> > RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> > images 
> 
> I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
> but rather waht we want is just a drop down menu for distro to let you
> select "custom", that way you could use a set of defconfigs which would
> use the custom image URL, when that is enabled it selects
> GUESTFS_HAS_CUSTOM_RAW_IMAGE.
> 
> > and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> > the latest bits.
> 
> My patch after this "guestfs: add support to infer host distro
> mirrororing optimizations" helps to infer when we want to copy a
> custom sources list, based on a host's hop count. What you describe
> seems to beg for another auto-infererence which could be based on
> some target directory and based on on the custom URL. That could be
> added and then you wouldn't even need to make any custom .config other
> than having a few defconfigs for each distro point release.
> 
> > If I could build my own images and dump them in a
> > directory without having to worry about updating the index file that
> > would probably speed things up a bit.
> 
> I think this is possible, we could just look for directory we add to
> .gitignore, and if the user has it as a symlink, we leverage that as
> a key indicator for "hunting season open for custom images, please
> populate my Kconfig with these other custom options".
> 
> > > +
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	bool
> > > +
> > > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	string
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> > 
> > You need '.raw' at the end of the URL, right?
> 
> Oops yes sorry.
> 
> > > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > > index f90ed499051b..514a26a60436 100755
> > > --- a/scripts/bringup_guestfs.sh
> > > +++ b/scripts/bringup_guestfs.sh
> > > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> > >  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> > >  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> > >  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > > +
> > > +build_custom_source()
> > > +{
> > > +	SOURCE_TMP=$(mktemp)
> > > +	cat <<_EOT >$SOURCE_TMP
> > > +[local]
> > ^^^
> > You probably want this to be unique based on the flavor.  If you switch
> > flavors, then you're going to wind up with multiple repo configs with the
> > same repo id and virt-builder will only pick up the last one it reads... 
> > the end result being that it probably won't find the template you want
> > to install.
> 
> So this is for the source, not the index, the source is local, can we
> not have multiple arbitrary sources with the same name? The indexes for
> each file would be different. Here I just wanted to emphasize that the
> source is local.
> 
> > 
> > > +uri=file:///${CUSTOM_INDEX}
> > > +proxy=off
> > > +_EOT
> 
> See the source, points to a custom index, and so we'd have multiple
> local sources, the index files do change per flavor.
> 
> But certainly, it would be good to get this right to make this scale.
> For now to start off with trixie, and later to scale to whatever custom
> setup we need for random builds.
> 
> Thoughts / advice?
> 
> Anyway, since this still has a big fat warning as "this doesn't work
> yet", I've made the changes required so far and pushed these patches in,
> we can make changes to scale this, now that at laest its easier to test
> these things.
> 
>   Luis
>