From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6740E1E4BE for ; Wed, 8 May 2024 17:30:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715189448; cv=none; b=FcE54301hrXmDRBC69Ok2VakBL4daCBt8GIUAa61Glkf6C5xamvmob1hgqcGAoFfuDuFlH71UM241ULkQCODdcqn/SXkWDXhcnqTeaoeAxFxsmPKJly02nkQ/fgNjuErpX7Lp8/LFLO3mRb5jzmpnTOUiocLvgSdAd+17WpHglE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715189448; c=relaxed/simple; bh=wwJ1an9nlQWUO5DEk14CplBKPlmN8VCr7dPFEtR7c9Q=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: In-Reply-To:Content-Type:Content-Disposition; b=NtD2ekCpCLow6uTq7vQxGsMqoM+33YLkPsw2nHNJ6mE0Ffci9VUSx3Jgt3fz8TpMvIRL/o5po0EoLUrpWLmgJ/PxcXDAKyzrfvryte9qcQBkCUnynf2qEfuHeAL7fvYI1e5Ur5fv/MFoFXrzDW3pvVrRjEBAvBbTBsTRw6Lwt+Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PexO+boL; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PexO+boL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1715189445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=x6ZXMbdP/U1S8ONkcYPqBplceqBpReTJ+g3uvy0IeUI=; b=PexO+boLg9TNADL7VwKYCgyijg6DyIhoeaoHBTa9kAOcCdG38nlxHSydbAbRJOJnY6kcue 9OSwxHmR1zHiNgqAjUTx46Puy+q2O5+S9FWHQvpnnvmglteTXAgYh/CFBoEsCt5pNsfNDd VAiTkqX+f+L2KhfVlbByBOFPoH8FXk4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-SYPMfog3O6u2yM6wdKs-Pg-1; Wed, 08 May 2024 13:30:43 -0400 X-MC-Unique: SYPMfog3O6u2yM6wdKs-Pg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4A2FF800262; Wed, 8 May 2024 17:30:43 +0000 (UTC) Received: from aion.redhat.com (unknown [10.22.16.67]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2FAA3492CAA; Wed, 8 May 2024 17:30:43 +0000 (UTC) Received: by aion.redhat.com (Postfix, from userid 1000) id C696114A6C3; Wed, 8 May 2024 13:30:42 -0400 (EDT) Date: Wed, 8 May 2024 13:30:42 -0400 From: Scott Mayhew To: Luis Chamberlain Cc: kdevops@lists.linux.dev Subject: Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs Message-ID: References: <20240508065039.3408637-1-mcgrof@kernel.org> <20240508065039.3408637-6-mcgrof@kernel.org> Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In-Reply-To: <20240508065039.3408637-6-mcgrof@kernel.org> X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, 07 May 2024, Luis Chamberlain wrote: > debian does not yet provide an index file for virt-builder, but we > have URLS with qcow2 and raw files. Using them in guestfs is actually > not quite trivial. So we do the handy work to enable others to also > use custom URLs and build a virt-builder local source and index file > for you. All we need really, is to check the checksums. > > Sadly this does not yet work, as it seems we get stuck on the grub > prompt for some reason. This needs some more investigation. Yeah when I run 'make bringup' the consoles on all the guests are stuck on "Booting `Debian GNU/Linux'". I tried manually running virt-builder + virt-install using the same image and I get the same result. Granted, I've only ever used the images in the libguestfs repos or images that I've built myself, so I could be doing something wrong. > > Signed-off-by: Luis Chamberlain > --- > kconfigs/Kconfig.defaults | 4 ++ > kconfigs/Kconfig.distro | 12 ++++ > kconfigs/Kconfig.guestfs | 94 ++++++++++++++++++++++++- > scripts/bringup_guestfs.sh | 132 +++++++++++++++++++++++++++++++++++ > scripts/os-debian-version.sh | 19 +++++ > 5 files changed, 259 insertions(+), 2 deletions(-) > create mode 100755 scripts/os-debian-version.sh > > diff --git a/kconfigs/Kconfig.defaults b/kconfigs/Kconfig.defaults > index 854a92e40ed2..41529dc86ce1 100644 > --- a/kconfigs/Kconfig.defaults > +++ b/kconfigs/Kconfig.defaults > @@ -4,6 +4,10 @@ config DEFAULT_BLKTRACE_URL > string > default "https://git.kernel.dk/blktrace" > > +config DEFAULT_DATE_ISO8601 > + string > + default $(shell, date -I) > + > config DEFAULT_DBENCH_URL > string > default "https://github.com/linux-kdevops/dbench.git" > diff --git a/kconfigs/Kconfig.distro b/kconfigs/Kconfig.distro > index b064e31a7c29..18e582eda40b 100644 > --- a/kconfigs/Kconfig.distro > +++ b/kconfigs/Kconfig.distro > @@ -2,6 +2,18 @@ config DISTRO_DEBIAN > bool > default $(shell, scripts/os-release-check.sh debian) > > +if DISTRO_DEBIAN > + > +config DISTRO_DEBIAN_BUSTER > + bool > + default $(shell, scripts/os-debian-version.sh buster) > + > +config DISTRO_DEBIAN_TRIXIE > + bool > + default $(shell, scripts/os-debian-version.sh trixie) > + > +endif > + > config DISTRO_FEDORA > bool > default $(shell, scripts/os-release-check.sh fedora) > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs > index 5838522908e8..5839fbedfd08 100644 > --- a/kconfigs/Kconfig.guestfs > +++ b/kconfigs/Kconfig.guestfs > @@ -1,5 +1,30 @@ > if GUESTFS > > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE > + bool It might be useful for these variables to have prompts so they show up in 'make menuconfig'. I have a whole bunch of workflows running for RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3 images and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to the latest bits. If I could build my own images and dump them in a directory without having to worry about updating the index file that would probably speed things up a bit. > + > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL > + bool > + > +config GUESTFS_CUSTOM_RAW_IMAGE_URL > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL > + string > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64 > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64 > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64 You need '.raw' at the end of the URL, right? > + > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS > + bool > + > +config GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL > + string > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/SHA512SUMS" if GUESTFS_DEBIAN_TRIXIE > + > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING > + bool > + > choice > prompt "Guestfs Linux distribution to use" > default GUESTFS_FEDORA if DISTRO_FEDORA || DISTRO_REDHAT > @@ -33,16 +58,81 @@ endchoice > > if GUESTFS_DEBIAN > > +choice > + prompt "Debian version" > + default GUESTFS_DEBIAN_BUSTER > + > +config GUESTFS_DEBIAN_TRIXIE > + bool "Debian 13 - Trixie" > + select GUESTFS_HAS_CUSTOM_RAW_IMAGE > + select GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL > + select GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS > + select GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING > + help > + Select this for debian trixie, debian 13. Since debian does not yet > + provie a virt-builder repo, we must download mock up our own URL > + for trixie release, as the default guestfs repo only has debian buster. > + We download the image for you once, and once its there we don't refresh > + it. We can later add support to force a refresh using the latest URL, > + as trixie is debian-testing and debian testing always has an new > + up to date image release, that is, its a rolling distribution release. > + > + This doesn't work yet.. it gets stuck on the grub prompt. > + > +config GUESTFS_DEBIAN_BUSTER > + bool "Debian 12 - Buster" > + help > + Select this for debian buster, debian 12. > + > +endchoice > + > +if GUESTFS_DEBIAN_TRIXIE > + > +choice > + prompt "Debian trixie flavor" > + default GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64 > + > +config GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64 > + bool "debian-13-generic-amd64-daily" > + help > + Should run in any environment using cloud-init, for e.g. OpenStack, > + DigitalOcean and also on bare metal. > + > +config GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64 > + bool "debian-13-genericcloud-amd64-daily" > + help > + Similar to generic. Should run in any virtualised environment. Is > + smaller than generic by excluding drivers for physical hardware. > + > +config GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64 > + bool "debian-13-nocloud-amd64-daily" > + help > + Mostly useful for testing the build process itself. Doesn't have > + cloud-init installed, but instead allows root login without a > + password. > + > +endchoice > + > +endif # GUESTFS_DEBIAN_TRIXIE > + > +config GUESTFS_DEBIAN_IMAGE_NAME > + string > + default "debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64 > + default "debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64 > + default "debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64 > + default "debian-12" if GUESTFS_DEBIAN_BUSTER > + > config GUESTFS_DEBIAN_BOX_SHORT > string > - default "debian12" if GUESTFS_DEBIAN > + default "debian13" if GUESTFS_DEBIAN_TRIXIE > + default "debian12" if GUESTFS_DEBIAN_BUSTER > > endif # GUESTFS_DEBIAN > > config VIRT_BUILDER_OS_VERSION > string "virt-builder os-version" > default "fedora-39" if GUESTFS_FEDORA > - default "debian-12" if GUESTFS_DEBIAN > + default GUESTFS_DEBIAN_IMAGE_NAME if GUESTFS_DEBIAN > help > Have virt-builder use this os-version string to > build a root image for the guest. Run "virt-builder -l" > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh > index f90ed499051b..514a26a60436 100755 > --- a/scripts/bringup_guestfs.sh > +++ b/scripts/bringup_guestfs.sh > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs" > OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION} > BASE_IMAGE_DIR="${STORAGEDIR}/base_images" > BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw" > + > +build_custom_source() > +{ > + SOURCE_TMP=$(mktemp) > + cat <<_EOT >$SOURCE_TMP > +[local] ^^^ You probably want this to be unique based on the flavor. If you switch flavors, then you're going to wind up with multiple repo configs with the same repo id and virt-builder will only pick up the last one it reads... the end result being that it probably won't find the template you want to install. > +uri=file:///${CUSTOM_INDEX} > +proxy=off > +_EOT > + sudo mv $SOURCE_TMP $CUSTOM_SOURCE > +} > + > +build_custom_index() > +{ > + cat <<_EOT >$CUSTOM_INDEX > +[$OS_VERSION] > +file=${OS_VERSION}.raw > +_EOT > +} > + > +fetch_custom_image() > +{ > + wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL > + if [[ $? -ne 0 ]]; then > + echo -e "Could not download:\n$CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_URL" > + exit 1 > + fi > +} > + > +check_custom_image() > +{ > + SHA512SUMS_FILE="$(basename $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL)" > + CUSTOM_IMAGE_SHA512SUM="$CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE" > + if [[ ! -f $CUSTOM_IMAGE_SHA512SUM ]]; then > + wget --directory-prefix=$CUSTOM_IMAGE_DIR $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL > + if [[ $? -ne 0 ]]; then > + echo "Could not get sha512sum file: $CONFIG_GUESTFS_CUSTOM_RAW_IMAGE_SHA512SUMS_URL" > + exit 1 > + fi > + fi > + echo "Checking $CUSTOM_IMAGE_DIR/$SHA512SUMS_FILE" > + > + # This subshell let's us keep below in the current directory. > + # sha512sum files are relative to the local directory > + (cd $CUSTOM_IMAGE_DIR && sha512sum --ignore-missing -c $SHA512SUMS_FILE) > + if [[ $? -ne 0 ]]; then > + echo "Invalid SHA512SUM checksum for $CUSTOM_IMAGE as per $SHA512SUMS_FILE" > + exit 1 > + fi > + touch $CUSTOM_IMAGE_OK > +} > + > +# Ensure folks are not surprised. If you're using rolling distros you know what > +# you are doing. This gives us the right later to change this at will. > +# > +# In the future we can make this smoother, as we used to have it with vagrant > +# update, but for now downloading *once* for a rolling distro seems ok to start. > +# We give enough information so you can update. > +build_warn_rolling_distro() > +{ > + echo "------------------------------------------------------------------" > + echo "This is a rolling distribution release! To upgrade just do:" > + echo > + echo "rm -rf ${CUSTOM_IMAGE}/*" > + echo "rm -f ${CUSTOM_SOURCE}" > + echo "rm -f ${CUSTOM_INDEX}" > + echo > + echo "Running guests always use their own copy. To rebuild your custom" > + echo "base image from the custom image, also remov the base image:" spelling > + echo > + echo "rm -f ${BASE_IMAGE}" > + echo > + echo "This can always be done safely without affecting running guests." > + echo "------------------------------------------------------------------" > +} > + > +build_custom_image() > +{ > + CUSTOM_IMAGE_DIR="${STORAGEDIR}/custom_images/${OS_VERSION}" > + CUSTOM_IMAGE="${CUSTOM_IMAGE_DIR}/${OS_VERSION}.raw" > + CUSTOM_IMAGE_OK="${CUSTOM_IMAGE_DIR}.ok" > + CUSTOM_SOURCE="/etc/virt-builder/repos.d/kdevops-custom-images-${OS_VERSION}.conf" > + CUSTOM_INDEX="${CUSTOM_IMAGE_DIR}/index" > + > + mkdir -p ${CUSTOM_IMAGE_DIR} > + > + if [[ ! -f $CUSTOM_IMAGE && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL" == "y" ]]; then > + fetch_custom_image > + fi > + > + if [[ ! -f $CUSTOM_IMAGE_OK && "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_SHA512SUMS" == "y" ]]; then > + check_custom_image > + fi > + > + if [[ ! -f $CUSTOM_IMAGE ]]; then > + echo "Custom image on path $CUSTOM_IMAGE not found" > + exit 1 > + fi > + > + if [[ ! -f $CUSTOM_SOURCE ]]; then > + build_custom_source > + fi > + > + if [[ ! -f $CUSTOM_INDEX ]]; then > + build_custom_index > + fi > + > + echo "Custom virt-builder source: $CUSTOM_SOURCE" > + echo "Custom virt-builder index: $CUSTOM_INDEX" > + echo "Custom image source: $CUSTOM_IMAGE" > + > + if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE_ROLLING" == "y" ]]; then > + build_warn_rolling_distro > + fi > + > + echo "Going to build index for $OS_VERSION ..." > + virt-builder-repository --no-compression $CUSTOM_IMAGE_DIR > + if [[ $? -ne 0 ]]; then > + echo "Failed to build repository ..." > + exit 1 > + fi > + > + # Note, we don't build $BASE_IMAGE, virt-builder does that later. We > + # just build $virt-builder, which is the pristine upstream image. > +} > + > mkdir -p $STORAGEDIR > mkdir -p $BASE_IMAGE_DIR > + > if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then > sudo chgrp -R $QEMU_GROUP $STORAGEDIR > sudo chmod -R g+rw $STORAGEDIR > @@ -41,6 +168,10 @@ fi > cmdfile=$(mktemp) > > if [ ! -f $BASE_IMAGE ]; then > + if [[ "$CONFIG_GUESTFS_HAS_CUSTOM_RAW_IMAGE" == "y" ]]; then > + build_custom_image > + fi > + > DO_UNREG=0 > if echo $OS_VERSION | grep -q '^rhel'; then > if [ -n "$CONFIG_RHEL_ORG_ID" -a -n "$CONFIG_RHEL_ACTIVATION_KEY" ]; then > @@ -77,6 +208,7 @@ _EOT > # Hope we get that interface name right! > if echo $OS_VERSION | grep -q '^debian'; then > cat <<_EOT >>$cmdfile > +mkdir /etc/network/interfaces.d/ > append-line /etc/network/interfaces.d/enp1s0:auto enp1s0 > append-line /etc/network/interfaces.d/enp1s0:allow-hotplug enp1s0 > append-line /etc/network/interfaces.d/enp1s0:iface enp1s0 inet dhcp > diff --git a/scripts/os-debian-version.sh b/scripts/os-debian-version.sh > new file mode 100755 > index 000000000000..f6b01d689b87 > --- /dev/null > +++ b/scripts/os-debian-version.sh > @@ -0,0 +1,19 @@ > +#!/bin/bash > +DEBIAN_VERSION_FILE="/etc/debian_version" > + > +if [[ ! -f $DEBIAN_VERSION_FILE ]]; then > + echo n > +fi > + > +check_debian_version() > +{ > + grep -qi $1 $DEBIAN_VERSION_FILE > + if [[ $? -eq 0 ]]; then > + echo y > + exit > + fi > + echo n > + exit > +} > + > +check_debian_version $1 > -- > 2.43.0 > >