[PATCH 00/10] guestfs: custom image + mirroring sources.list heuristic

[PATCH 00/10] guestfs: custom image + mirroring sources.list heuristic

[Luis Chamberlain]

A lot of these patches are just cleanups, but there are two patches which took
a bit more time and thought, and that is why I am posting them. The two patches
you should certainly review are:

  5) guestfs: add initial debian trixie support with custom URLs
  6) guestfs: add support to infer host distro mirrororing optimizations

5) adds initial trixie support, but more importantly it makes is super easy
for those of you stuck in vagrant world because you have no virt-builder index
yet available for you. This should get you unstuck and let you start testing
guestfs. Sadly, the debian trixie image doesn't work yet, but there is enough
effort here that I hope we can figure it out eventually. The *real* meat and
bones of that patch is the way to deal with custom local kdevops sources and
indexes for virt-builder in a way that allows us to grow. As hackers we want
bleeding edge, and so this let's us get as close to as bleeding edge as we
can.

6) is just an optimization, if your host is already using mirror we should
simply adopt mirroring for packages on the guest too. This should speed up
guest bring up considerably and best of all, it should be automatic.

Luis Chamberlain (10):
  guestfs: move debian options before image names
  guestfs: modify grub prompt before first bring up
  guestfs: set default root password
  guestfs: check for virt-builder failure
  guestfs: add initial debian trixie support with custom URLs
  guestfs: add support to infer host distro mirrororing optimizations
  guestfs: move rhel activation to its own helper
  guestfs: move copying yum repo to its own routine
  guestfs: move pre-install customizations to its own routine
  guestfs: move debian pre-install hacks to its own routine

 kconfigs/Kconfig.defaults                   |   4 +
 kconfigs/Kconfig.distro                     |  16 ++
 kconfigs/Kconfig.guestfs                    | 155 ++++++++++++-
 scripts/bringup_guestfs.sh                  | 228 +++++++++++++++++---
 scripts/get-distro-has-hop-count-sources.sh |  39 ++++
 scripts/os-debian-version.sh                |  19 ++
 6 files changed, 426 insertions(+), 35 deletions(-)
 create mode 100755 scripts/get-distro-has-hop-count-sources.sh
 create mode 100755 scripts/os-debian-version.sh

-- 
2.43.0

[PATCH 01/10] guestfs: move debian options before image names

[Luis Chamberlain]

Move debian specific options before we define the image filename to look
for, as if we add the option to support other releases other than buster
then the filename will change.

This introduces no functional changes.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 kconfigs/Kconfig.guestfs | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
index 03e0fb865a66..5838522908e8 100644
--- a/kconfigs/Kconfig.guestfs
+++ b/kconfigs/Kconfig.guestfs
@@ -31,6 +31,14 @@ config GUESTFS_DEBIAN
 
 endchoice
 
+if GUESTFS_DEBIAN
+
+config GUESTFS_DEBIAN_BOX_SHORT
+        string
+        default "debian12" if GUESTFS_DEBIAN
+
+endif # GUESTFS_DEBIAN
+
 config VIRT_BUILDER_OS_VERSION
        string "virt-builder os-version"
        default "fedora-39" if GUESTFS_FEDORA
@@ -41,12 +49,4 @@ config VIRT_BUILDER_OS_VERSION
          to get a list of operating systems and versions supported
          by guestfs.
 
-if GUESTFS_DEBIAN
-
-config GUESTFS_DEBIAN_BOX_SHORT
-        string
-        default "debian12" if GUESTFS_DEBIAN
-
-endif
-
 endif # GUESTFS
-- 
2.43.0

[PATCH 02/10] guestfs: modify grub prompt before first bring up

[Luis Chamberlain]

If we're working on new freshly minted images, we want to be able to
debug right away, so setup a console right away.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index dbace8d5d502..e1ce70e256d3 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -62,6 +62,8 @@ _EOT
 install sudo,qemu-guest-agent,python3,bash
 run-command useradd -m kdevops -s /bin/bash
 append-line /etc/sudoers.d/kdevops:kdevops   ALL=(ALL)       NOPASSWD: ALL
+edit /etc/default/grub:s/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"/
+run-command /usr/sbin/update-grub2
 _EOT
 
 	if [ $DO_UNREG -ne 0 ]; then
-- 
2.43.0

[PATCH 03/10] guestfs: set default root password

[Luis Chamberlain]

For guests we used to use vagrant and that had a default root
password. Setting up a custom one makes sense if we're doing
production stuff, but if you're just hacking on stuff on
ephemeral development guests in a private network we want to
be able to log in easily, not scrape random passwords. Use
kdevops for the root password.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index e1ce70e256d3..17fabcecf0d2 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -64,6 +64,7 @@ run-command useradd -m kdevops -s /bin/bash
 append-line /etc/sudoers.d/kdevops:kdevops   ALL=(ALL)       NOPASSWD: ALL
 edit /etc/default/grub:s/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"/
 run-command /usr/sbin/update-grub2
+root-password password:kdevops
 _EOT
 
 	if [ $DO_UNREG -ne 0 ]; then
-- 
2.43.0

[PATCH 04/10] guestfs: check for virt-builder failure

[Luis Chamberlain]

When virt-builder fails we were not capturing the failure and just kept
moving forward with guest initialization. Ensure we go to jail; do not
pass Go, do not collect $200.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 17fabcecf0d2..f90ed499051b 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -86,6 +86,10 @@ _EOT
 
 	echo "Generating new base image for ${OS_VERSION}"
 	virt-builder ${OS_VERSION} --arch `uname -m` -o $BASE_IMAGE --size 20G --format raw --commands-from-file $cmdfile
+	if [[ $? -ne 0 ]]; then
+		echo "Failed to build custom image $BASE_IMAGE"
+		exit 1
+	fi
 fi
 
 # FIXME: is there a yaml equivalent of jq?
-- 
2.43.0

[PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Luis Chamberlain]

debian does not yet provide an index file for virt-builder, but we
have URLS with qcow2 and raw files. Using them in guestfs is actually
not quite trivial. So we do the handy work to enable others to also
use custom URLs and build a virt-builder local source and index file
for you. All we need really, is to check the checksums.

Sadly this does not yet work, as it seems we get stuck on the grub
prompt for some reason. This needs some more investigation.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 kconfigs/Kconfig.defaults    |   4 ++
 kconfigs/Kconfig.distro      |  12 ++++
 kconfigs/Kconfig.guestfs     |  94 ++++++++++++++++++++++++-
 scripts/bringup_guestfs.sh   | 132 +++++++++++++++++++++++++++++++++++
 scripts/os-debian-version.sh |  19 +++++
 5 files changed, 259 insertions(+), 2 deletions(-)
 create mode 100755 scripts/os-debian-version.sh

diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
index 854a92e40ed2..41529dc86ce1 100644
--- a/kconfigs/Kconfig.defaults
+++ b/kconfigs/Kconfig.defaults
@@ -4,6 +4,10 @@ config DEFAULT_BLKTRACE_URL
 	string
 	default "https://git.kernel.dk/blktrace"
 
+config DEFAULT_DATE_ISO8601
+	string
+	default $(shell, date -I)
+
 config DEFAULT_DBENCH_URL
 	string
 	default "https://github.com/linux-kdevops/dbench.git"
diff --git a/kconfigs/Kconfig.distro b/kconfigs/Kconfig.distro
index b064e31a7c29..18e582eda40b 100644
--- a/kconfigs/Kconfig.distro
+++ b/kconfigs/Kconfig.distro
@@ -2,6 +2,18 @@ config DISTRO_DEBIAN
 	bool
 	default $(shell, scripts/os-release-check.sh debian)
 
+if DISTRO_DEBIAN
+
+config DISTRO_DEBIAN_BUSTER
+	bool
+	default $(shell, scripts/os-debian-version.sh buster)
+
+config DISTRO_DEBIAN_TRIXIE
+	bool
+	default $(shell, scripts/os-debian-version.sh trixie)
+
+endif
+
 config DISTRO_FEDORA
 	bool
 	default $(shell, scripts/os-release-check.sh fedora)
diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
index 5838522908e8..5839fbedfd08 100644
--- a/kconfigs/Kconfig.guestfs
+++ b/kconfigs/Kconfig.guestfs
@@ -1,5 +1,30 @@
 if GUESTFS
 
+config GUESTFS_HAS_CUSTOM_RAW_IMAGE
+	bool
+
+config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
+	bool
+
+config GUESTFS_CUSTOM_RAW_IMAGE_URL
+	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
+	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
+	string
+	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
+	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
+	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
+
+config GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
+	bool
+
+config GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL
+	string
+	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
+	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/SHA512SUMS" if GUESTFS_DEBIAN_TRIXIE
+
+config GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING
+	bool
+
 choice
 	prompt "Guestfs Linux distribution to use"
 	default GUESTFS_FEDORA if DISTRO_FEDORA || DISTRO_REDHAT
@@ -33,16 +58,81 @@ endchoice
 
 if GUESTFS_DEBIAN
 
+choice
+	prompt "Debian version"
+	default GUESTFS_DEBIAN_BUSTER
+
+config GUESTFS_DEBIAN_TRIXIE
+	bool "Debian 13 - Trixie"
+	select GUESTFS_HAS_CUSTOM_RAW_IMAGE
+	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
+	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
+	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING
+	help
+	  Select this for debian trixie, debian 13. Since debian does not yet
+	  provie a virt-builder repo, we must download mock up our own URL
+	  for trixie release, as the default guestfs repo only has debian buster.
+	  We download the image for you once, and once its there we don't refresh
+	  it. We can later add support to force a refresh using the latest URL,
+	  as trixie is debian-testing and debian testing always has an new
+	  up to date image release, that is, its a rolling distribution release.
+
+	  This doesn't work yet.. it gets stuck on the grub prompt.
+
+config GUESTFS_DEBIAN_BUSTER
+	bool "Debian 12 - Buster"
+	help
+	  Select this for debian buster, debian 12.
+
+endchoice
+
+if GUESTFS_DEBIAN_TRIXIE
+
+choice
+	prompt "Debian trixie flavor"
+	default GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
+
+config GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
+	bool "debian-13-generic-amd64-daily"
+	help
+	  Should run in any environment using cloud-init, for e.g. OpenStack,
+	  DigitalOcean and also on bare metal.
+
+config GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
+	bool "debian-13-genericcloud-amd64-daily"
+	help
+	  Similar to generic. Should run in any virtualised environment. Is
+	  smaller than generic by excluding drivers for physical hardware.
+
+config GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
+	bool "debian-13-nocloud-amd64-daily"
+	help
+	  Mostly useful for testing the build process itself. Doesn't have
+	  cloud-init installed, but instead allows root login without a
+	  password.
+
+endchoice
+
+endif # GUESTFS_DEBIAN_TRIXIE
+
+config GUESTFS_DEBIAN_IMAGE_NAME
+	string
+	default "debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
+	default "debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
+	default "debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
+	default "debian-12" if GUESTFS_DEBIAN_BUSTER
+
 config GUESTFS_DEBIAN_BOX_SHORT
         string
-        default "debian12" if GUESTFS_DEBIAN
+        default "debian13" if GUESTFS_DEBIAN_TRIXIE
+        default "debian12" if GUESTFS_DEBIAN_BUSTER
 
 endif # GUESTFS_DEBIAN
 
 config VIRT_BUILDER_OS_VERSION
        string "virt-builder os-version"
        default "fedora-39" if GUESTFS_FEDORA
-       default "debian-12" if GUESTFS_DEBIAN
+       default GUESTFS_DEBIAN_IMAGE_NAME if GUESTFS_DEBIAN
        help
          Have virt-builder use this os-version string to
          build a root image for the guest. Run "virt-builder -l"
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index f90ed499051b..514a26a60436 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
 OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
 BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
 BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
+
+build_custom_source()
+{
+	SOURCE_TMP=$(mktemp)
+	cat <<_EOT >$SOURCE_TMP
+[local]
+uri=file:///${CUSTOM_INDEX}
+proxy=off
+_EOT
+	sudo mv $SOURCE_TMP $CUSTOM_SOURCE
+}
+
+build_custom_index()
+{
+	cat <<_EOT >$CUSTOM_INDEX
+[$OS_VERSION]
+file=${OS_VERSION}.raw
+_EOT
+}
+
+fetch_custom_image()
+{
+	wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL
+	if [[ $? -ne 0 ]]; then
+		echo -e "Could not download:\n$CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL"
+		exit 1
+	fi
+}
+
+check_custom_image()
+{
+	SHA512SUMS_FILE="$(basename $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL)"
+	CUSTOM_IMAGE_SHA512SUM="$CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE"
+	if [[ ! -f $CUSTOM_IMAGE_SHA512SUM ]]; then
+		wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL
+		if [[ $? -ne 0 ]]; then
+			echo "Could not get sha512sum file: $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL"
+			exit 1
+		fi
+	fi
+	echo "Checking $CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE"
+
+	# This subshell let's us keep below in the current directory.
+	# sha512sum files are relative to the local directory
+	(cd $CUSTOM_IMAGE_DIR && sha512sum --ignore-missing -c $SHA512SUMS_FILE)
+	if [[ $? -ne 0 ]]; then
+		echo "Invalid SHA512SUM checksum for $CUSTOM_IMAGE as per $SHA512SUMS_FILE"
+		exit 1
+	fi
+	touch $CUSTOM_IMAGE_OK
+}
+
+# Ensure folks are not surprised. If you're using rolling distros you know what
+# you are doing. This gives us the right later to change this at will.
+#
+# In the future we can make this smoother, as we used to have it with vagrant
+# update, but for now downloading *once* for a rolling distro seems ok to start.
+# We give enough information so you can update.
+build_warn_rolling_distro()
+{
+	echo "------------------------------------------------------------------"
+	echo "This is a rolling distribution release! To upgrade just do:"
+	echo
+	echo "rm -rf ${CUSTOM_IMAGE}/*"
+	echo "rm -f  ${CUSTOM_SOURCE}"
+	echo "rm -f  ${CUSTOM_INDEX}"
+	echo
+	echo "Running guests always use their own copy. To rebuild your custom"
+	echo "base image from the custom image, also remov the base image:"
+	echo
+	echo "rm -f  ${BASE_IMAGE}"
+	echo
+	echo "This can always be done safely without affecting running guests."
+	echo "------------------------------------------------------------------"
+}
+
+build_custom_image()
+{
+	CUSTOM_IMAGE_DIR="${STORAGEDIR}/custom_images/${OS_VERSION}"
+	CUSTOM_IMAGE="${CUSTOM_IMAGE_DIR}/${OS_VERSION}.raw"
+	CUSTOM_IMAGE_OK="${CUSTOM_IMAGE_DIR}.ok"
+	CUSTOM_SOURCE="/etc/virt-builder/repos.d/kdevops-custom-images-${OS_VERSION}.conf"
+	CUSTOM_INDEX="${CUSTOM_IMAGE_DIR}/index"
+
+	mkdir -p ${CUSTOM_IMAGE_DIR}
+
+	if [[ ! -f $CUSTOM_IMAGE && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL" == "y" ]]; then
+		fetch_custom_image
+	fi
+
+	if [[ ! -f $CUSTOM_IMAGE_OK && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS" == "y" ]]; then
+		check_custom_image
+	fi
+
+	if [[ ! -f $CUSTOM_IMAGE ]]; then
+		echo "Custom image on path $CUSTOM_IMAGE not found"
+		exit 1
+	fi
+
+	if [[ ! -f $CUSTOM_SOURCE ]]; then
+		build_custom_source
+	fi
+
+	if [[ ! -f $CUSTOM_INDEX ]]; then
+		build_custom_index
+	fi
+
+	echo "Custom virt-builder source: $CUSTOM_SOURCE"
+	echo "Custom virt-builder index:  $CUSTOM_INDEX"
+	echo "Custom image source:        $CUSTOM_IMAGE"
+
+	if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING" == "y" ]]; then
+		build_warn_rolling_distro
+	fi
+
+	echo "Going to build index for $OS_VERSION ..."
+	virt-builder-repository --no-compression $CUSTOM_IMAGE_DIR
+	if [[ $? -ne 0 ]]; then
+		echo "Failed to build repository ..."
+		exit 1
+	fi
+
+	# Note, we don't build $BASE_IMAGE, virt-builder does that later. We
+	# just build $virt-builder, which is the pristine upstream image.
+}
+
 mkdir -p $STORAGEDIR
 mkdir -p $BASE_IMAGE_DIR
+
 if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then
 	sudo chgrp -R $QEMU_GROUP $STORAGEDIR
 	sudo chmod -R g+rw $STORAGEDIR
@@ -41,6 +168,10 @@ fi
 cmdfile=$(mktemp)
 
 if [ ! -f $BASE_IMAGE ]; then
+	if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE" == "y" ]]; then
+		build_custom_image
+	fi
+
 	DO_UNREG=0
 	if echo $OS_VERSION | grep -q '^rhel'; then
 		if [ -n "$CONFIG_RHEL_ORG_ID" -a -n "$CONFIG_RHEL_ACTIVATION_KEY" ]; then
@@ -77,6 +208,7 @@ _EOT
 # Hope we get that interface name right!
 	if echo $OS_VERSION | grep -q '^debian'; then
 		cat <<_EOT >>$cmdfile
+mkdir /etc/network/interfaces.d/
 append-line /etc/network/interfaces.d/enp1s0:auto enp1s0
 append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0
 append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp
diff --git a/scripts/os-debian-version.sh b/scripts/os-debian-version.sh
new file mode 100755
index 000000000000..f6b01d689b87
--- /dev/null
+++ b/scripts/os-debian-version.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+DEBIAN_VERSION_FILE="/etc/debian_version"
+
+if [[ ! -f $DEBIAN_VERSION_FILE ]]; then
+	echo n
+fi
+
+check_debian_version()
+{
+	grep -qi $1 $DEBIAN_VERSION_FILE
+	if [[ $? -eq 0 ]]; then
+		echo y
+		exit
+	fi
+	echo n
+	exit
+}
+
+check_debian_version $1
-- 
2.43.0

[PATCH 06/10] guestfs: add support to infer host distro mirrororing optimizations

[Luis Chamberlain]

For production level uses of kdevops you want to optimize every ounce of
time, and network bandwidth. kdevops already mirrors everything we can
git clone, the next optimization is to ensure both the host and guest
use their own local mirrors for packaging. This is resolved by the
host and we don't care how, but *if* we can detect this optimization
in place, *and* if the host and guest match distros, then it makes
sense to leverage the host's sources file for the repo for the guest.

So, this optimization requires no user input. We start with debian
support, and add enough kconfig magic to let others easily add support
for other distros, all you need is the mapping file and how to detect
when you think it is safe from a host perspective for a guest to use
your sources file.

The first heuristic is a simple traceroute to check how many hops
your host uses for its sources, if its 1 chances are pretty high
the host is using a mirror for its packaging sources. This is also
distros specific as each source file differs, for debian its
/etc/apt/sources.list, and so we traceroute the first entry. Second
heuristic is determining when we can use the host sources on the
guest. For debian we enable this for now when the host is trixie and
the guest is either trixie or buster. That is, if the host is using
trixie we'll upgrade a buster guest to trixie.

We've always wanted trixie for folks working on hacking, its just we
can't use it yet...

If you only want buster, then your host probably only wants to use
buster too.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 kconfigs/Kconfig.distro                     |  4 ++
 kconfigs/Kconfig.guestfs                    | 47 +++++++++++++++++++++
 scripts/bringup_guestfs.sh                  | 13 ++++++
 scripts/get-distro-has-hop-count-sources.sh | 39 +++++++++++++++++
 4 files changed, 103 insertions(+)
 create mode 100755 scripts/get-distro-has-hop-count-sources.sh

diff --git a/kconfigs/Kconfig.distro b/kconfigs/Kconfig.distro
index 18e582eda40b..8289a3ab89e1 100644
--- a/kconfigs/Kconfig.distro
+++ b/kconfigs/Kconfig.distro
@@ -12,6 +12,10 @@ config DISTRO_DEBIAN_TRIXIE
 	bool
 	default $(shell, scripts/os-debian-version.sh trixie)
 
+config DISTRO_DEBIAN_HAS_HOP1_SOURCES
+	bool
+	default $(shell, scripts/get-distro-has-hop-count-sources.sh 1)
+
 endif
 
 config DISTRO_FEDORA
diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
index 5839fbedfd08..dd0b280edbe5 100644
--- a/kconfigs/Kconfig.guestfs
+++ b/kconfigs/Kconfig.guestfs
@@ -25,6 +25,21 @@ config GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL
 config GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING
 	bool
 
+config GUESTFS_HAS_DISTRO_SOURCE_COPY_CAPABILITIES
+	bool
+
+if GUESTFS_HAS_DISTRO_SOURCE_COPY_CAPABILITIES
+
+config GUESTFS_COPY_SOURCES_FROM_HOST_TO_GUEST
+	bool
+
+config GUESTFS_DISTRO_SOURCE_AND_DEST_FILE
+	string
+	depends on GUESTFS_COPY_SOURCES_FROM_HOST_TO_GUEST
+	default "/etc/apt/sources.list" if GUESTFS_DEBIAN
+
+endif
+
 choice
 	prompt "Guestfs Linux distribution to use"
 	default GUESTFS_FEDORA if DISTRO_FEDORA || DISTRO_REDHAT
@@ -51,6 +66,7 @@ config GUESTFS_DEBIAN
 	select HAVE_DISTRO_PREFERS_CUSTOM_HOST_PREFIX
 	select HAVE_DISTRO_PREFERS_FSTESTS_WATCHDOG if KDEVOPS_WORKFLOW_ENABLE_FSTESTS
 	select HAVE_DISTRO_PREFERS_FSTESTS_WATCHDOG_KILL if KDEVOPS_WORKFLOW_ENABLE_FSTESTS
+	select GUESTFS_HAS_DISTRO_SOURCE_COPY_CAPABILITIES
 	help
 	  This option will set the target guest to Debian.
 
@@ -115,6 +131,15 @@ endchoice
 
 endif # GUESTFS_DEBIAN_TRIXIE
 
+config GUESTFS_DEBIAN_HOST_UPDATES_GUEST
+	bool
+	default y if GUESTFS_DEBIAN_BUSTER && DISTRO_DEBIAN_TRIXIE
+
+config GUESTFS_DEBIAN_HOST_GUEST_MATCH
+	bool
+	default y if GUESTFS_DEBIAN_BUSTER && DISTRO_DEBIAN_BUSTER
+	default y if GUESTFS_DEBIAN_TRIXIE && DISTRO_DEBIAN_TRIXIE
+
 config GUESTFS_DEBIAN_IMAGE_NAME
 	string
 	default "debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
@@ -127,6 +152,28 @@ config GUESTFS_DEBIAN_BOX_SHORT
         default "debian13" if GUESTFS_DEBIAN_TRIXIE
         default "debian12" if GUESTFS_DEBIAN_BUSTER
 
+config GUESTFS_DEBIAN_COPY_HOST_SOURCES
+	bool
+	depends on GUESTFS_DEBIAN_HOST_GUEST_MATCH || GUESTFS_DEBIAN_HOST_UPDATES_GUEST
+	default DISTRO_DEBIAN_HAS_HOP1_SOURCES
+	select GUESTFS_COPY_SOURCES_FROM_HOST_TO_GUEST
+	help
+	  We have detected that you are running debian on the host, this option
+	  will be enabled by default if we detect that your /etc/apt/sources.list
+	  is using a source we determine has only one hop via traceroute to it,
+	  for details of that heuristic see the script:
+
+	    ./scripts/get-distro-has-hop-count-sources.sh
+
+	  If your hop distance to your mirror is just 1, you are using debian on
+	  the host, want to use debian guests, both host and guest match the
+	  target release, then its a good assumtion you'd likely want to take
+	  advantage of that same mirror for your guests. So disable this if you
+	  really don't want to take advantage of your sources.list.
+
+	  If this is disabled but you are sure you can use your host's
+	  /etc/apt/sources.list on the guests then enable this.
+
 endif # GUESTFS_DEBIAN
 
 config VIRT_BUILDER_OS_VERSION
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 514a26a60436..c410d0cf725d 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -156,6 +156,15 @@ build_custom_image()
 	# just build $virt-builder, which is the pristine upstream image.
 }
 
+copy_host_sources()
+{
+	TARGET_DIR="$(dirname $CONFIG_GUESTFS_DISTRO_SOURCE_AND_DEST_FILE)"
+	cat <<_EOT >>$cmdfile
+mkdir $TARGET_DIR
+copy-in $CONFIG_GUESTFS_DISTRO_SOURCE_AND_DEST_FILE:$TARGET_DIR
+_EOT
+}
+
 mkdir -p $STORAGEDIR
 mkdir -p $BASE_IMAGE_DIR
 
@@ -188,6 +197,10 @@ copy-in $CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE:/etc/yum.repos.d
 _EOT
 	fi
 
+	if [[ "$CONFIG_GUESTFS_COPY_SOURCES_FROM_HOST_TO_GUEST" == "y" ]]; then
+		copy_host_sources
+	fi
+
 # basic pre-install customization
 	cat <<_EOT >>$cmdfile
 install sudo,qemu-guest-agent,python3,bash
diff --git a/scripts/get-distro-has-hop-count-sources.sh b/scripts/get-distro-has-hop-count-sources.sh
new file mode 100755
index 000000000000..1db975468887
--- /dev/null
+++ b/scripts/get-distro-has-hop-count-sources.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+DEBIAN_VERSION_FILE="/etc/debian_version"
+ACCEPTABLE_HOPS=1
+
+if [[ $# -eq 1 ]]; then
+	ACCEPTABLE_HOPS=$1
+fi
+
+# For now we only support debian. Adding other distros should be easy, its
+# just a matter of mapping a sources file to a column we can use for a server
+# hostname.
+if [[ ! -f $DEBIAN_VERSION_FILE ]]; then
+	echo n
+	exit 0
+fi
+
+SOURCES_FILE="/etc/apt/sources.list"
+
+if [[ ! -f $SOURCES_FILE ]]; then
+	echo n
+	exit 0
+fi
+
+which traceroute > /dev/null
+if [[ $? -ne 0 ]]; then
+	echo n
+	exit 0
+fi
+
+LINE=$(grep -v "^#" $SOURCES_FILE | head -1)
+HOST_URL_LINE=$(echo $LINE | awk '{print $2}')
+HOST=$(echo $HOST_URL_LINE | awk -F[/:] '{print $4}')
+HOP_COUNT=$(traceroute -n -w 1,1,1 $HOST | wc -l)
+HOP_COUNT=$((HOP_COUNT -1))
+
+if [[ $HOP_COUNT -le $ACCEPTABLE_HOPS ]]; then
+	echo y
+	exit 0
+fi
-- 
2.43.0

[PATCH 07/10] guestfs: move rhel activation to its own helper

[Luis Chamberlain]

The cmdfile stuff is an eye-sore, move users of that stuff
which depend on its own kconfig to its own function to make
the main part of the functionality easier to read.

No functional changes.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index c410d0cf725d..2102fd81cf31 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -156,6 +156,23 @@ build_custom_image()
 	# just build $virt-builder, which is the pristine upstream image.
 }
 
+handle_rhel_activation()
+{
+	if [ -n "$CONFIG_RHEL_ORG_ID" -a -n "$CONFIG_RHEL_ACTIVATION_KEY" ]; then
+		DO_UNREG=1
+		cat <<_EOT >>$cmdfile
+run-command subscription-manager register --org=${CONFIG_RHEL_ORG_ID} --activationkey=${CONFIG_RHEL_ACTIVATION_KEY}
+_EOT
+	fi
+}
+
+handle_rhel_unreg()
+{
+	cat <<_EOT >>$cmdfile
+sm-unregister
+_EOT
+}
+
 copy_host_sources()
 {
 	TARGET_DIR="$(dirname $CONFIG_GUESTFS_DISTRO_SOURCE_AND_DEST_FILE)"
@@ -183,12 +200,7 @@ if [ ! -f $BASE_IMAGE ]; then
 
 	DO_UNREG=0
 	if echo $OS_VERSION | grep -q '^rhel'; then
-		if [ -n "$CONFIG_RHEL_ORG_ID" -a -n "$CONFIG_RHEL_ACTIVATION_KEY" ]; then
-			DO_UNREG=1
-			cat <<_EOT >>$cmdfile
-run-command subscription-manager register --org=${CONFIG_RHEL_ORG_ID} --activationkey=${CONFIG_RHEL_ACTIVATION_KEY}
-_EOT
-		fi
+		handle_rhel_activation
 	fi
 
 	if [ -n "$CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE" ]; then
@@ -212,9 +224,7 @@ root-password password:kdevops
 _EOT
 
 	if [ $DO_UNREG -ne 0 ]; then
-		cat <<_EOT >>$cmdfile
-sm-unregister
-_EOT
+		handle_rhel_unreg
 	fi
 
 # Ugh, debian has to be told to bring up the network and regenerate ssh keys
-- 
2.43.0

[PATCH 08/10] guestfs: move copying yum repo to its own routine

[Luis Chamberlain]

Following the same logic as before, move the yum repo copying
to its own routine.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 2102fd81cf31..8210b3cf1b19 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -173,6 +173,13 @@ sm-unregister
 _EOT
 }
 
+copy_yum_repo()
+{
+	cat <<_EOT >>$cmdfile
+copy-in $CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE:/etc/yum.repos.d
+_EOT
+}
+
 copy_host_sources()
 {
 	TARGET_DIR="$(dirname $CONFIG_GUESTFS_DISTRO_SOURCE_AND_DEST_FILE)"
@@ -204,9 +211,7 @@ if [ ! -f $BASE_IMAGE ]; then
 	fi
 
 	if [ -n "$CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE" ]; then
-		cat <<_EOT >>$cmdfile
-copy-in $CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE:/etc/yum.repos.d
-_EOT
+		copy_yum_repo
 	fi
 
 	if [[ "$CONFIG_GUESTFS_COPY_SOURCES_FROM_HOST_TO_GUEST" == "y" ]]; then
-- 
2.43.0

[PATCH 09/10] guestfs: move pre-install customizations to its own routine

[Luis Chamberlain]

As with prior commits, move pre-install customizations to its
own routine.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 8210b3cf1b19..6f96b5c2029a 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -189,6 +189,18 @@ copy-in $CONFIG_GUESTFS_DISTRO_SOURCE_AND_DEST_FILE:$TARGET_DIR
 _EOT
 }
 
+pre_install_customizations()
+{
+	cat <<_EOT >>$cmdfile
+install sudo,qemu-guest-agent,python3,bash
+run-command useradd -m kdevops -s /bin/bash
+append-line /etc/sudoers.d/kdevops:kdevops   ALL=(ALL)       NOPASSWD: ALL
+edit /etc/default/grub:s/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"/
+run-command /usr/sbin/update-grub2
+root-password password:kdevops
+_EOT
+}
+
 mkdir -p $STORAGEDIR
 mkdir -p $BASE_IMAGE_DIR
 
@@ -218,15 +230,7 @@ if [ ! -f $BASE_IMAGE ]; then
 		copy_host_sources
 	fi
 
-# basic pre-install customization
-	cat <<_EOT >>$cmdfile
-install sudo,qemu-guest-agent,python3,bash
-run-command useradd -m kdevops -s /bin/bash
-append-line /etc/sudoers.d/kdevops:kdevops   ALL=(ALL)       NOPASSWD: ALL
-edit /etc/default/grub:s/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"/
-run-command /usr/sbin/update-grub2
-root-password password:kdevops
-_EOT
+	pre_install_customizations
 
 	if [ $DO_UNREG -ne 0 ]; then
 		handle_rhel_unreg
-- 
2.43.0

[PATCH 10/10] guestfs: move debian pre-install hacks to its own routine

[Luis Chamberlain]

As with prior commits. Tidy up.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/bringup_guestfs.sh | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 6f96b5c2029a..b81c9d5c38d1 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -201,6 +201,19 @@ root-password password:kdevops
 _EOT
 }
 
+# Ugh, debian has to be told to bring up the network and regenerate ssh keys
+# Hope we get that interface name right!
+debian_pre_install_hacks()
+{
+	cat <<_EOT >>$cmdfile
+mkdir /etc/network/interfaces.d/
+append-line /etc/network/interfaces.d/enp1s0:auto enp1s0
+append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0
+append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp
+firstboot-command dpkg-reconfigure openssh-server
+_EOT
+}
+
 mkdir -p $STORAGEDIR
 mkdir -p $BASE_IMAGE_DIR
 
@@ -236,16 +249,8 @@ if [ ! -f $BASE_IMAGE ]; then
 		handle_rhel_unreg
 	fi
 
-# Ugh, debian has to be told to bring up the network and regenerate ssh keys
-# Hope we get that interface name right!
 	if echo $OS_VERSION | grep -q '^debian'; then
-		cat <<_EOT >>$cmdfile
-mkdir /etc/network/interfaces.d/
-append-line /etc/network/interfaces.d/enp1s0:auto enp1s0
-append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0
-append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp
-firstboot-command dpkg-reconfigure openssh-server
-_EOT
+		debian_pre_install_hacks
 	fi
 
 	echo "Generating new base image for ${OS_VERSION}"
-- 
2.43.0

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Scott Mayhew]

On Tue, 07 May 2024, Luis Chamberlain wrote:

> debian does not yet provide an index file for virt-builder, but we
> have URLS with qcow2 and raw files. Using them in guestfs is actually
> not quite trivial. So we do the handy work to enable others to also
> use custom URLs and build a virt-builder local source and index file
> for you. All we need really, is to check the checksums.
> 
> Sadly this does not yet work, as it seems we get stuck on the grub
> prompt for some reason. This needs some more investigation.

Yeah when I run 'make bringup' the consoles on all the guests are stuck
on "Booting `Debian GNU/Linux'".

I tried manually running virt-builder + virt-install using the same
image and I get the same result.  Granted, I've only ever used the
images in the libguestfs repos or images that I've built myself, so I
could be doing something wrong.

> 
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
>  kconfigs/Kconfig.defaults    |   4 ++
>  kconfigs/Kconfig.distro      |  12 ++++
>  kconfigs/Kconfig.guestfs     |  94 ++++++++++++++++++++++++-
>  scripts/bringup_guestfs.sh   | 132 +++++++++++++++++++++++++++++++++++
>  scripts/os-debian-version.sh |  19 +++++
>  5 files changed, 259 insertions(+), 2 deletions(-)
>  create mode 100755 scripts/os-debian-version.sh
> 
> diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults
> index 854a92e40ed2..41529dc86ce1 100644
> --- a/kconfigs/Kconfig.defaults
> +++ b/kconfigs/Kconfig.defaults
> @@ -4,6 +4,10 @@ config DEFAULT_BLKTRACE_URL
>  	string
>  	default "https://git.kernel.dk/blktrace"
>  
> +config DEFAULT_DATE_ISO8601
> +	string
> +	default $(shell, date -I)
> +
>  config DEFAULT_DBENCH_URL
>  	string
>  	default "https://github.com/linux-kdevops/dbench.git"
> diff --git a/kconfigs/Kconfig.distro b/kconfigs/Kconfig.distro
> index b064e31a7c29..18e582eda40b 100644
> --- a/kconfigs/Kconfig.distro
> +++ b/kconfigs/Kconfig.distro
> @@ -2,6 +2,18 @@ config DISTRO_DEBIAN
>  	bool
>  	default $(shell, scripts/os-release-check.sh debian)
>  
> +if DISTRO_DEBIAN
> +
> +config DISTRO_DEBIAN_BUSTER
> +	bool
> +	default $(shell, scripts/os-debian-version.sh buster)
> +
> +config DISTRO_DEBIAN_TRIXIE
> +	bool
> +	default $(shell, scripts/os-debian-version.sh trixie)
> +
> +endif
> +
>  config DISTRO_FEDORA
>  	bool
>  	default $(shell, scripts/os-release-check.sh fedora)
> diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> index 5838522908e8..5839fbedfd08 100644
> --- a/kconfigs/Kconfig.guestfs
> +++ b/kconfigs/Kconfig.guestfs
> @@ -1,5 +1,30 @@
>  if GUESTFS
>  
> +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> +	bool

It might be useful for these variables to have prompts so they show up
in 'make menuconfig'.  I have a whole bunch of workflows running for
RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
images and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
the latest bits.  If I could build my own images and dump them in a
directory without having to worry about updating the index file that
would probably speed things up a bit.

> +
> +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> +	bool
> +
> +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> +	string
> +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64

You need '.raw' at the end of the URL, right?

> +
> +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
> +	bool
> +
> +config GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL
> +	string
> +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
> +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/SHA512SUMS" if GUESTFS_DEBIAN_TRIXIE
> +
> +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING
> +	bool
> +
>  choice
>  	prompt "Guestfs Linux distribution to use"
>  	default GUESTFS_FEDORA if DISTRO_FEDORA || DISTRO_REDHAT
> @@ -33,16 +58,81 @@ endchoice
>  
>  if GUESTFS_DEBIAN
>  
> +choice
> +	prompt "Debian version"
> +	default GUESTFS_DEBIAN_BUSTER
> +
> +config GUESTFS_DEBIAN_TRIXIE
> +	bool "Debian 13 - Trixie"
> +	select GUESTFS_HAS_CUSTOM_RAW_IMAGE
> +	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> +	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS
> +	select GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING
> +	help
> +	  Select this for debian trixie, debian 13. Since debian does not yet
> +	  provie a virt-builder repo, we must download mock up our own URL
> +	  for trixie release, as the default guestfs repo only has debian buster.
> +	  We download the image for you once, and once its there we don't refresh
> +	  it. We can later add support to force a refresh using the latest URL,
> +	  as trixie is debian-testing and debian testing always has an new
> +	  up to date image release, that is, its a rolling distribution release.
> +
> +	  This doesn't work yet.. it gets stuck on the grub prompt.
> +
> +config GUESTFS_DEBIAN_BUSTER
> +	bool "Debian 12 - Buster"
> +	help
> +	  Select this for debian buster, debian 12.
> +
> +endchoice
> +
> +if GUESTFS_DEBIAN_TRIXIE
> +
> +choice
> +	prompt "Debian trixie flavor"
> +	default GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> +
> +config GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> +	bool "debian-13-generic-amd64-daily"
> +	help
> +	  Should run in any environment using cloud-init, for e.g. OpenStack,
> +	  DigitalOcean and also on bare metal.
> +
> +config GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> +	bool "debian-13-genericcloud-amd64-daily"
> +	help
> +	  Similar to generic. Should run in any virtualised environment. Is
> +	  smaller than generic by excluding drivers for physical hardware.
> +
> +config GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> +	bool "debian-13-nocloud-amd64-daily"
> +	help
> +	  Mostly useful for testing the build process itself. Doesn't have
> +	  cloud-init installed, but instead allows root login without a
> +	  password.
> +
> +endchoice
> +
> +endif # GUESTFS_DEBIAN_TRIXIE
> +
> +config GUESTFS_DEBIAN_IMAGE_NAME
> +	string
> +	default "debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> +	default "debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> +	default "debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> +	default "debian-12" if GUESTFS_DEBIAN_BUSTER
> +
>  config GUESTFS_DEBIAN_BOX_SHORT
>          string
> -        default "debian12" if GUESTFS_DEBIAN
> +        default "debian13" if GUESTFS_DEBIAN_TRIXIE
> +        default "debian12" if GUESTFS_DEBIAN_BUSTER
>  
>  endif # GUESTFS_DEBIAN
>  
>  config VIRT_BUILDER_OS_VERSION
>         string "virt-builder os-version"
>         default "fedora-39" if GUESTFS_FEDORA
> -       default "debian-12" if GUESTFS_DEBIAN
> +       default GUESTFS_DEBIAN_IMAGE_NAME if GUESTFS_DEBIAN
>         help
>           Have virt-builder use this os-version string to
>           build a root image for the guest. Run "virt-builder -l"
> diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> index f90ed499051b..514a26a60436 100755
> --- a/scripts/bringup_guestfs.sh
> +++ b/scripts/bringup_guestfs.sh
> @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
>  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
>  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
>  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> +
> +build_custom_source()
> +{
> +	SOURCE_TMP=$(mktemp)
> +	cat <<_EOT >$SOURCE_TMP
> +[local]
^^^
You probably want this to be unique based on the flavor.  If you switch
flavors, then you're going to wind up with multiple repo configs with the
same repo id and virt-builder will only pick up the last one it reads... 
the end result being that it probably won't find the template you want
to install.

> +uri=file:///${CUSTOM_INDEX}
> +proxy=off
> +_EOT
> +	sudo mv $SOURCE_TMP $CUSTOM_SOURCE
> +}
> +
> +build_custom_index()
> +{
> +	cat <<_EOT >$CUSTOM_INDEX
> +[$OS_VERSION]
> +file=${OS_VERSION}.raw
> +_EOT
> +}
> +
> +fetch_custom_image()
> +{
> +	wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL
> +	if [[ $? -ne 0 ]]; then
> +		echo -e "Could not download:\n$CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL"
> +		exit 1
> +	fi
> +}
> +
> +check_custom_image()
> +{
> +	SHA512SUMS_FILE="$(basename $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL)"
> +	CUSTOM_IMAGE_SHA512SUM="$CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE"
> +	if [[ ! -f $CUSTOM_IMAGE_SHA512SUM ]]; then
> +		wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL
> +		if [[ $? -ne 0 ]]; then
> +			echo "Could not get sha512sum file: $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL"
> +			exit 1
> +		fi
> +	fi
> +	echo "Checking $CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE"
> +
> +	# This subshell let's us keep below in the current directory.
> +	# sha512sum files are relative to the local directory
> +	(cd $CUSTOM_IMAGE_DIR && sha512sum --ignore-missing -c $SHA512SUMS_FILE)
> +	if [[ $? -ne 0 ]]; then
> +		echo "Invalid SHA512SUM checksum for $CUSTOM_IMAGE as per $SHA512SUMS_FILE"
> +		exit 1
> +	fi
> +	touch $CUSTOM_IMAGE_OK
> +}
> +
> +# Ensure folks are not surprised. If you're using rolling distros you know what
> +# you are doing. This gives us the right later to change this at will.
> +#
> +# In the future we can make this smoother, as we used to have it with vagrant
> +# update, but for now downloading *once* for a rolling distro seems ok to start.
> +# We give enough information so you can update.
> +build_warn_rolling_distro()
> +{
> +	echo "------------------------------------------------------------------"
> +	echo "This is a rolling distribution release! To upgrade just do:"
> +	echo
> +	echo "rm -rf ${CUSTOM_IMAGE}/*"
> +	echo "rm -f  ${CUSTOM_SOURCE}"
> +	echo "rm -f  ${CUSTOM_INDEX}"
> +	echo
> +	echo "Running guests always use their own copy. To rebuild your custom"
> +	echo "base image from the custom image, also remov the base image:"

spelling

> +	echo
> +	echo "rm -f  ${BASE_IMAGE}"
> +	echo
> +	echo "This can always be done safely without affecting running guests."
> +	echo "------------------------------------------------------------------"
> +}
> +
> +build_custom_image()
> +{
> +	CUSTOM_IMAGE_DIR="${STORAGEDIR}/custom_images/${OS_VERSION}"
> +	CUSTOM_IMAGE="${CUSTOM_IMAGE_DIR}/${OS_VERSION}.raw"
> +	CUSTOM_IMAGE_OK="${CUSTOM_IMAGE_DIR}.ok"
> +	CUSTOM_SOURCE="/etc/virt-builder/repos.d/kdevops-custom-images-${OS_VERSION}.conf"
> +	CUSTOM_INDEX="${CUSTOM_IMAGE_DIR}/index"
> +
> +	mkdir -p ${CUSTOM_IMAGE_DIR}
> +
> +	if [[ ! -f $CUSTOM_IMAGE && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL" == "y" ]]; then
> +		fetch_custom_image
> +	fi
> +
> +	if [[ ! -f $CUSTOM_IMAGE_OK && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS" == "y" ]]; then
> +		check_custom_image
> +	fi
> +
> +	if [[ ! -f $CUSTOM_IMAGE ]]; then
> +		echo "Custom image on path $CUSTOM_IMAGE not found"
> +		exit 1
> +	fi
> +
> +	if [[ ! -f $CUSTOM_SOURCE ]]; then
> +		build_custom_source
> +	fi
> +
> +	if [[ ! -f $CUSTOM_INDEX ]]; then
> +		build_custom_index
> +	fi
> +
> +	echo "Custom virt-builder source: $CUSTOM_SOURCE"
> +	echo "Custom virt-builder index:  $CUSTOM_INDEX"
> +	echo "Custom image source:        $CUSTOM_IMAGE"
> +
> +	if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING" == "y" ]]; then
> +		build_warn_rolling_distro
> +	fi
> +
> +	echo "Going to build index for $OS_VERSION ..."
> +	virt-builder-repository --no-compression $CUSTOM_IMAGE_DIR
> +	if [[ $? -ne 0 ]]; then
> +		echo "Failed to build repository ..."
> +		exit 1
> +	fi
> +
> +	# Note, we don't build $BASE_IMAGE, virt-builder does that later. We
> +	# just build $virt-builder, which is the pristine upstream image.
> +}
> +
>  mkdir -p $STORAGEDIR
>  mkdir -p $BASE_IMAGE_DIR
> +
>  if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then
>  	sudo chgrp -R $QEMU_GROUP $STORAGEDIR
>  	sudo chmod -R g+rw $STORAGEDIR
> @@ -41,6 +168,10 @@ fi
>  cmdfile=$(mktemp)
>  
>  if [ ! -f $BASE_IMAGE ]; then
> +	if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE" == "y" ]]; then
> +		build_custom_image
> +	fi
> +
>  	DO_UNREG=0
>  	if echo $OS_VERSION | grep -q '^rhel'; then
>  		if [ -n "$CONFIG_RHEL_ORG_ID" -a -n "$CONFIG_RHEL_ACTIVATION_KEY" ]; then
> @@ -77,6 +208,7 @@ _EOT
>  # Hope we get that interface name right!
>  	if echo $OS_VERSION | grep -q '^debian'; then
>  		cat <<_EOT >>$cmdfile
> +mkdir /etc/network/interfaces.d/
>  append-line /etc/network/interfaces.d/enp1s0:auto enp1s0
>  append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0
>  append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp
> diff --git a/scripts/os-debian-version.sh b/scripts/os-debian-version.sh
> new file mode 100755
> index 000000000000..f6b01d689b87
> --- /dev/null
> +++ b/scripts/os-debian-version.sh
> @@ -0,0 +1,19 @@
> +#!/bin/bash
> +DEBIAN_VERSION_FILE="/etc/debian_version"
> +
> +if [[ ! -f $DEBIAN_VERSION_FILE ]]; then
> +	echo n
> +fi
> +
> +check_debian_version()
> +{
> +	grep -qi $1 $DEBIAN_VERSION_FILE
> +	if [[ $? -eq 0 ]]; then
> +		echo y
> +		exit
> +	fi
> +	echo n
> +	exit
> +}
> +
> +check_debian_version $1
> -- 
> 2.43.0
> 
> 

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Luis Chamberlain]

On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> On Tue, 07 May 2024, Luis Chamberlain wrote:
> 
> > debian does not yet provide an index file for virt-builder, but we
> > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > not quite trivial. So we do the handy work to enable others to also
> > use custom URLs and build a virt-builder local source and index file
> > for you. All we need really, is to check the checksums.
> > 
> > Sadly this does not yet work, as it seems we get stuck on the grub
> > prompt for some reason. This needs some more investigation.
> 
> Yeah when I run 'make bringup' the consoles on all the guests are stuck
> on "Booting `Debian GNU/Linux'".

Is that with Trixie or also with y9our own custom ISO?

> I tried manually running virt-builder + virt-install using the same
> image and I get the same result.  Granted, I've only ever used the
> images in the libguestfs repos or images that I've built myself, so I
> could be doing something wrong.

I'm in hopes we can figure this out, there is likely something special
done on the images hosted by libguestfs.org which regular nocloud images
like debian's trixie is not doing:

https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw

Once we figure that out, we should be able to easily do that as a post
up with virt-builder.

Richard, any hints on what likely should help fix boot?

> > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > index 5838522908e8..5839fbedfd08 100644
> > --- a/kconfigs/Kconfig.guestfs
> > +++ b/kconfigs/Kconfig.guestfs
> > @@ -1,5 +1,30 @@
> >  if GUESTFS
> >  
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > +	bool
> 
> It might be useful for these variables to have prompts so they show up
> in 'make menuconfig'. 

That's a one line change, it just needs a description, for now its auto
because there is no custom raw image URL option, what you descrie seems
for us to want such a new entry and option. So sure, but I think that
can be done in subsequent patch?

> I have a whole bunch of workflows running for
> RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> images 

I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
but rather waht we want is just a drop down menu for distro to let you
select "custom", that way you could use a set of defconfigs which would
use the custom image URL, when that is enabled it selects
GUESTFS_HAS_CUSTOM_RAW_IMAGE.

> and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> the latest bits.

My patch after this "guestfs: add support to infer host distro
mirrororing optimizations" helps to infer when we want to copy a
custom sources list, based on a host's hop count. What you describe
seems to beg for another auto-infererence which could be based on
some target directory and based on on the custom URL. That could be
added and then you wouldn't even need to make any custom .config other
than having a few defconfigs for each distro point release.

> If I could build my own images and dump them in a
> directory without having to worry about updating the index file that
> would probably speed things up a bit.

I think this is possible, we could just look for directory we add to
.gitignore, and if the user has it as a symlink, we leverage that as
a key indicator for "hunting season open for custom images, please
populate my Kconfig with these other custom options".

> > +
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > +	bool
> > +
> > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > +	string
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> 
> You need '.raw' at the end of the URL, right?

Oops yes sorry.

> > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > index f90ed499051b..514a26a60436 100755
> > --- a/scripts/bringup_guestfs.sh
> > +++ b/scripts/bringup_guestfs.sh
> > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> >  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> >  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> >  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > +
> > +build_custom_source()
> > +{
> > +	SOURCE_TMP=$(mktemp)
> > +	cat <<_EOT >$SOURCE_TMP
> > +[local]
> ^^^
> You probably want this to be unique based on the flavor.  If you switch
> flavors, then you're going to wind up with multiple repo configs with the
> same repo id and virt-builder will only pick up the last one it reads... 
> the end result being that it probably won't find the template you want
> to install.

So this is for the source, not the index, the source is local, can we
not have multiple arbitrary sources with the same name? The indexes for
each file would be different. Here I just wanted to emphasize that the
source is local.

> 
> > +uri=file:///${CUSTOM_INDEX}
> > +proxy=off
> > +_EOT

See the source, points to a custom index, and so we'd have multiple
local sources, the index files do change per flavor.

But certainly, it would be good to get this right to make this scale.
For now to start off with trixie, and later to scale to whatever custom
setup we need for random builds.

Thoughts / advice?

Anyway, since this still has a big fat warning as "this doesn't work
yet", I've made the changes required so far and pushed these patches in,
we can make changes to scale this, now that at laest its easier to test
these things.

  Luis

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Richard W.M. Jones]

On Sat, May 11, 2024 at 04:46:48PM -0700, Luis Chamberlain wrote:
> On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> > On Tue, 07 May 2024, Luis Chamberlain wrote:
> > 
> > > debian does not yet provide an index file for virt-builder, but we
> > > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > > not quite trivial. So we do the handy work to enable others to also
> > > use custom URLs and build a virt-builder local source and index file
> > > for you. All we need really, is to check the checksums.
> > > 
> > > Sadly this does not yet work, as it seems we get stuck on the grub
> > > prompt for some reason. This needs some more investigation.
> > 
> > Yeah when I run 'make bringup' the consoles on all the guests are stuck
> > on "Booting `Debian GNU/Linux'".
> 
> Is that with Trixie or also with y9our own custom ISO?
> 
> > I tried manually running virt-builder + virt-install using the same
> > image and I get the same result.  Granted, I've only ever used the
> > images in the libguestfs repos or images that I've built myself, so I
> > could be doing something wrong.
> 
> I'm in hopes we can figure this out, there is likely something special
> done on the images hosted by libguestfs.org which regular nocloud images
> like debian's trixie is not doing:
> 
> https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw
>
> Once we figure that out, we should be able to easily do that as a post
> up with virt-builder.
> 
> Richard, any hints on what likely should help fix boot?

Sorry, it's not enough information to say.  Is it possible
to enable a serial console and find out where it's hanging?

Rich.

> > > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > > index 5838522908e8..5839fbedfd08 100644
> > > --- a/kconfigs/Kconfig.guestfs
> > > +++ b/kconfigs/Kconfig.guestfs
> > > @@ -1,5 +1,30 @@
> > >  if GUESTFS
> > >  
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	bool
> > 
> > It might be useful for these variables to have prompts so they show up
> > in 'make menuconfig'. 
> 
> That's a one line change, it just needs a description, for now its auto
> because there is no custom raw image URL option, what you descrie seems
> for us to want such a new entry and option. So sure, but I think that
> can be done in subsequent patch?
> 
> > I have a whole bunch of workflows running for
> > RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> > images 
> 
> I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
> but rather waht we want is just a drop down menu for distro to let you
> select "custom", that way you could use a set of defconfigs which would
> use the custom image URL, when that is enabled it selects
> GUESTFS_HAS_CUSTOM_RAW_IMAGE.
> 
> > and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> > the latest bits.
> 
> My patch after this "guestfs: add support to infer host distro
> mirrororing optimizations" helps to infer when we want to copy a
> custom sources list, based on a host's hop count. What you describe
> seems to beg for another auto-infererence which could be based on
> some target directory and based on on the custom URL. That could be
> added and then you wouldn't even need to make any custom .config other
> than having a few defconfigs for each distro point release.
> 
> > If I could build my own images and dump them in a
> > directory without having to worry about updating the index file that
> > would probably speed things up a bit.
> 
> I think this is possible, we could just look for directory we add to
> .gitignore, and if the user has it as a symlink, we leverage that as
> a key indicator for "hunting season open for custom images, please
> populate my Kconfig with these other custom options".
> 
> > > +
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	bool
> > > +
> > > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	string
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> > 
> > You need '.raw' at the end of the URL, right?
> 
> Oops yes sorry.
> 
> > > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > > index f90ed499051b..514a26a60436 100755
> > > --- a/scripts/bringup_guestfs.sh
> > > +++ b/scripts/bringup_guestfs.sh
> > > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> > >  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> > >  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> > >  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > > +
> > > +build_custom_source()
> > > +{
> > > +	SOURCE_TMP=$(mktemp)
> > > +	cat <<_EOT >$SOURCE_TMP
> > > +[local]
> > ^^^
> > You probably want this to be unique based on the flavor.  If you switch
> > flavors, then you're going to wind up with multiple repo configs with the
> > same repo id and virt-builder will only pick up the last one it reads... 
> > the end result being that it probably won't find the template you want
> > to install.
> 
> So this is for the source, not the index, the source is local, can we
> not have multiple arbitrary sources with the same name? The indexes for
> each file would be different. Here I just wanted to emphasize that the
> source is local.
> 
> > 
> > > +uri=file:///${CUSTOM_INDEX}
> > > +proxy=off
> > > +_EOT
> 
> See the source, points to a custom index, and so we'd have multiple
> local sources, the index files do change per flavor.
> 
> But certainly, it would be good to get this right to make this scale.
> For now to start off with trixie, and later to scale to whatever custom
> setup we need for random builds.
> 
> Thoughts / advice?
> 
> Anyway, since this still has a big fat warning as "this doesn't work
> yet", I've made the changes required so far and pushed these patches in,
> we can make changes to scale this, now that at laest its easier to test
> these things.
> 
>   Luis

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Luis Chamberlain]

On Mon, May 13, 2024 at 11:28 AM Richard W.M. Jones <rjones@redhat.com> wrote:
> On Sat, May 11, 2024 at 04:46:48PM -0700, Luis Chamberlain wrote:
> > Richard, any hints on what likely should help fix boot?
>
> Sorry, it's not enough information to say.  Is it possible
> to enable a serial console and find out where it's hanging?

That's the thing, that's already enabled, and we modified the grub
prompt to ensure we can access the console, we get a boot prompt, see
grub, but can't interact with it, we just see it try to boot in a
loop.

  Luis

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Scott Mayhew]

On Sat, 11 May 2024, Luis Chamberlain wrote:

> On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> > On Tue, 07 May 2024, Luis Chamberlain wrote:
> > 
> > > debian does not yet provide an index file for virt-builder, but we
> > > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > > not quite trivial. So we do the handy work to enable others to also
> > > use custom URLs and build a virt-builder local source and index file
> > > for you. All we need really, is to check the checksums.
> > > 
> > > Sadly this does not yet work, as it seems we get stuck on the grub
> > > prompt for some reason. This needs some more investigation.
> > 
> > Yeah when I run 'make bringup' the consoles on all the guests are stuck
> > on "Booting `Debian GNU/Linux'".
> 
> Is that with Trixie or also with y9our own custom ISO?

That was with the Trixie images from cloud.debian.org.

I poked around the image using guestfish, and it looks like it needs to use
UEFI boot... so I tried that:

$ virt-builder debian-13-genericcloud-amd64-daily -o debian13.qcow2 --format qcow2 --size 20G 
[   3.0] Downloading: file:////var/lib/libvirt/images/kdevops/smayhew/kdevops/guestfs/custom_images/debian-13-genericcloud-amd64-daily/debian-13-genericcloud-amd64-daily.raw
[   6.8] Planning how to build this image
[   6.8] Resizing (using virt-resize) to expand the disk to 20.0G
[  30.3] Opening the new disk
[  41.9] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
[  41.9] Setting the machine ID in /etc/machine-id
[  41.9] Setting passwords
virt-builder: Setting random password of root to LLq8WTqQJ4Hs3U2z
[  42.9] SELinux relabelling
[  42.9] Finishing off
                   Output file: debian13.qcow2
                   Output size: 20.0G
                 Output format: qcow2
            Total usable space: 19.5G
                    Free space: 18.9G (96%)

$ virt-install --connect qemu:///session --import --name debian13 --vcpus 2 --ram 2048 --disk /home/smayhew/debian13.qcow2 --network bridge=virbr0 --os-variant debian12 --graphics none --noautoconsole --noreboot --boot uefi

Starting install...
Creating domain...                                                                                                                                                                                                     |    0 B  00:00:00     
Domain creation completed.
You can restart your domain by running:
  virsh --connect qemu:///session start debian13

$ virsh -c qemu:///session start debian13
Domain 'debian13' started

$ virsh -c qemu:///session console debian13
Connected to domain 'debian13'
Escape character is ^] (Ctrl + ])

localhost login: root
Password: 
Linux localhost 6.7.12-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1 (2024-04-24) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@localhost:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux trixie/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=trixie
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@localhost:~# 
logout

So at least the VM booted.

Since kdevops using using virsh-define with xml templates instead of
virt-install, I guess you'll need a tempate with something like this
for those images?

$ virsh -c qemu:///session dumpxml --xpath '/domain/os' debian13
<os firmware="efi">
  <type arch="x86_64" machine="pc-q35-8.1">hvm</type>
  <firmware>
    <feature enabled="yes" name="enrolled-keys"/>
    <feature enabled="yes" name="secure-boot"/>
  </firmware>
  <loader readonly="yes" secure="yes" type="pflash" format="qcow2">/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
  <nvram template="/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2" format="qcow2">/home/smayhew/.config/libvirt/qemu/nvram/debian13_VARS.qcow2</nvram>
  <boot dev="hd"/>
</os>

$ virsh -c qemu:///session destroy debian13
Domain 'debian13' destroyed

And also you need to pass --nvram to get rid of the VMs afterward.

$ virsh -c qemu:///session undefine debian13
error: Failed to undefine domain 'debian13'
error: Requested operation is not valid: cannot undefine domain with nvram

$ virsh -c qemu:///session undefine --nvram debian13
Domain 'debian13' has been undefined

-Scott
> 
> > I tried manually running virt-builder + virt-install using the same
> > image and I get the same result.  Granted, I've only ever used the
> > images in the libguestfs repos or images that I've built myself, so I
> > could be doing something wrong.
> 
> I'm in hopes we can figure this out, there is likely something special
> done on the images hosted by libguestfs.org which regular nocloud images
> like debian's trixie is not doing:
> 
> https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw
> 
> Once we figure that out, we should be able to easily do that as a post
> up with virt-builder.
> 
> Richard, any hints on what likely should help fix boot?
> 
> > > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > > index 5838522908e8..5839fbedfd08 100644
> > > --- a/kconfigs/Kconfig.guestfs
> > > +++ b/kconfigs/Kconfig.guestfs
> > > @@ -1,5 +1,30 @@
> > >  if GUESTFS
> > >  
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	bool
> > 
> > It might be useful for these variables to have prompts so they show up
> > in 'make menuconfig'. 
> 
> That's a one line change, it just needs a description, for now its auto
> because there is no custom raw image URL option, what you descrie seems
> for us to want such a new entry and option. So sure, but I think that
> can be done in subsequent patch?
> 
> > I have a whole bunch of workflows running for
> > RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> > images 
> 
> I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
> but rather waht we want is just a drop down menu for distro to let you
> select "custom", that way you could use a set of defconfigs which would
> use the custom image URL, when that is enabled it selects
> GUESTFS_HAS_CUSTOM_RAW_IMAGE.
> 
> > and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> > the latest bits.
> 
> My patch after this "guestfs: add support to infer host distro
> mirrororing optimizations" helps to infer when we want to copy a
> custom sources list, based on a host's hop count. What you describe
> seems to beg for another auto-infererence which could be based on
> some target directory and based on on the custom URL. That could be
> added and then you wouldn't even need to make any custom .config other
> than having a few defconfigs for each distro point release.
> 
> > If I could build my own images and dump them in a
> > directory without having to worry about updating the index file that
> > would probably speed things up a bit.
> 
> I think this is possible, we could just look for directory we add to
> .gitignore, and if the user has it as a symlink, we leverage that as
> a key indicator for "hunting season open for custom images, please
> populate my Kconfig with these other custom options".
> 
> > > +
> > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	bool
> > > +
> > > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > > +	string
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> > 
> > You need '.raw' at the end of the URL, right?
> 
> Oops yes sorry.
> 
> > > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > > index f90ed499051b..514a26a60436 100755
> > > --- a/scripts/bringup_guestfs.sh
> > > +++ b/scripts/bringup_guestfs.sh
> > > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> > >  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> > >  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> > >  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > > +
> > > +build_custom_source()
> > > +{
> > > +	SOURCE_TMP=$(mktemp)
> > > +	cat <<_EOT >$SOURCE_TMP
> > > +[local]
> > ^^^
> > You probably want this to be unique based on the flavor.  If you switch
> > flavors, then you're going to wind up with multiple repo configs with the
> > same repo id and virt-builder will only pick up the last one it reads... 
> > the end result being that it probably won't find the template you want
> > to install.
> 
> So this is for the source, not the index, the source is local, can we
> not have multiple arbitrary sources with the same name? The indexes for
> each file would be different. Here I just wanted to emphasize that the
> source is local.
> 
> > 
> > > +uri=file:///${CUSTOM_INDEX}
> > > +proxy=off
> > > +_EOT
> 
> See the source, points to a custom index, and so we'd have multiple
> local sources, the index files do change per flavor.
> 
> But certainly, it would be good to get this right to make this scale.
> For now to start off with trixie, and later to scale to whatever custom
> setup we need for random builds.
> 
> Thoughts / advice?
> 
> Anyway, since this still has a big fat warning as "this doesn't work
> yet", I've made the changes required so far and pushed these patches in,
> we can make changes to scale this, now that at laest its easier to test
> these things.
> 
>   Luis
> 

Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs

[Luis Chamberlain]

On Mon, May 13, 2024 at 04:55:04PM -0400, Scott Mayhew wrote:
> On Sat, 11 May 2024, Luis Chamberlain wrote:
> 
> > On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> > > On Tue, 07 May 2024, Luis Chamberlain wrote:
> > > 
> > > > debian does not yet provide an index file for virt-builder, but we
> > > > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > > > not quite trivial. So we do the handy work to enable others to also
> > > > use custom URLs and build a virt-builder local source and index file
> > > > for you. All we need really, is to check the checksums.
> > > > 
> > > > Sadly this does not yet work, as it seems we get stuck on the grub
> > > > prompt for some reason. This needs some more investigation.
> > > 
> > > Yeah when I run 'make bringup' the consoles on all the guests are stuck
> > > on "Booting `Debian GNU/Linux'".
> > 
> > Is that with Trixie or also with y9our own custom ISO?
> 
> That was with the Trixie images from cloud.debian.org.
> 
> I poked around the image using guestfish, and it looks like it needs to use
> UEFI boot... so I tried that:

Score, and we can simplify this more by using automatic variables.
I just pushed the below patch. We're trixie-enabled now.

From d8a4dc36b64e5951c5d7d8b189eaf6aa00d912a1 Mon Sep 17 00:00:00 2001
From: Luis Chamberlain <mcgrof@kernel.org>
Date: Tue, 14 May 2024 04:59:55 -0700
Subject: [PATCH] guestfs: fix debian trixie support

First order of business to fix was the template image,
Scott Mayhew noted we needed to enable UEFI boot, and we
can just use a simple automatic UEFI firmware and secure boot
environment.

Next issue was networking, the dhcp package and scripts to
set up the network interface are needed. Finally, the last
step was to ensure we don't get prompted when we reconfigure
the openssh-server.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml | 3 ++-
 scripts/bringup_guestfs.sh                             | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
index eb5118a105f0..8f6b445da7b7 100644
--- a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
+++ b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
@@ -3,8 +3,9 @@
   <memory unit='MiB'>{{ libvirt_mem_mb }}</memory>
   <currentMemory unit='MiB'>{{ libvirt_mem_mb }}</currentMemory>
   <vcpu placement='static'>{{ libvirt_vcpus_count }}</vcpu>
-  <os>
+  <os firmware='efi'>
     <type arch='x86_64' machine='q35'>hvm</type>
+    <loader secure='yes'/>
     <boot dev='hd'/>
   </os>
   <features>
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index f8522eb549ee..fde19775a946 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -206,11 +206,12 @@ _EOT
 debian_pre_install_hacks()
 {
 	cat <<_EOT >>$cmdfile
+install isc-dhcp-client,ifupdown
 mkdir /etc/network/interfaces.d/
 append-line /etc/network/interfaces.d/enp1s0:auto enp1s0
 append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0
 append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp
-firstboot-command dpkg-reconfigure openssh-server
+firstboot-command DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true dpkg-reconfigure -p low --force openssh-server
 _EOT
 }
 
-- 
2.43.0