[PATCH 00/11] build-appliance-image revival

[PATCH 00/11] build-appliance-image revival

[Trevor Woerner]

Revive the build-appliance.

The build-appliance-image produces an image that contains all the necessary
pieces in order to perform a bitbake build. This patch series contains a
number of small tweaks that were needed in order for the build of the
appliance to succeed, and then for a build within the appliance to succeed as
well.

Trevor Woerner (11):
  packagegroup-self-hosted: alphabetize
  packagegroup-self-hosted: add zstd
  build-appliance-image: add support for qemu
  build-appliance-image: set TERM
  build-appliance-image kernel: linux-yocto: qemuall: add taskstats
  build-appliance-image: fix HOMEPAGE
  build-appliance-image: add /lib64 symlink
  build-appliance-image: use a real /tmp
  build-appliance-image: QB_MEM: allow user config
  QB_SMP: allow user modification
  build-appliance-image: check for xattr feature

 meta/conf/machine/include/riscv/qemuriscv.inc |  2 +-
 .../conf/machine/include/x86/qemuboot-x86.inc |  2 +-
 meta/conf/machine/qemuarm.conf                |  2 +-
 meta/conf/machine/qemuarm64.conf              |  2 +-
 meta/conf/machine/qemuppc64.conf              |  2 +-
 .../images/build-appliance-image_15.0.0.bb    | 27 ++++++++++++++-----
 .../packagegroups/packagegroup-self-hosted.bb | 11 ++++----
 meta/recipes-kernel/linux/linux-yocto.inc     |  2 +-
 8 files changed, 33 insertions(+), 17 deletions(-)

-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 01/11] packagegroup-self-hosted: alphabetize

[Trevor Woerner]

Several of the lists of packages are alphabetized, but the alphabetization
was lost in a couple areas.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 .../packagegroups/packagegroup-self-hosted.bb          | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
index a1b0ee288362..529d4c5c5c6b 100644
--- a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
+++ b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
@@ -127,7 +127,6 @@ RDEPENDS:packagegroup-self-hosted-extended = "\
     grep \
     groff \
     gzip \
-    settings-daemon \
     libaio \
     libusb1 \
     libxml2 \
@@ -160,6 +159,7 @@ RDEPENDS:packagegroup-self-hosted-extended = "\
     readline \
     rpm \
     setserial \
+    settings-daemon \
     socat \
     subversion \
     sudo \
@@ -173,22 +173,22 @@ RDEPENDS:packagegroup-self-hosted-extended = "\
     wget \
     which \
     xinetd \
+    xz \
     zip \
     zlib \
-    xz \
     "
 
 
 RDEPENDS:packagegroup-self-hosted-graphics = "\
+    adwaita-icon-theme \
     builder \
+    l3afpad \
     libgl \
     libgl-dev \
     libglu \
     libglu-dev \
     libx11-dev \
-    adwaita-icon-theme \
-    xdg-utils \
-    l3afpad \
     pcmanfm \
     vte \
+    xdg-utils \
     "
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 02/11] packagegroup-self-hosted: add zstd

[Trevor Woerner]

Add missing tools necessary on the host for a build.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/packagegroups/packagegroup-self-hosted.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
index 529d4c5c5c6b..dd000fda49d1 100644
--- a/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
+++ b/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb
@@ -176,6 +176,7 @@ RDEPENDS:packagegroup-self-hosted-extended = "\
     xz \
     zip \
     zlib \
+    zstd \
     "
 
 
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 03/11] build-appliance-image: add support for qemu

[Trevor Woerner]

Add ext4 explicitly to the list of IMAGE_FSTYPES so the build appliance can be
run with runqemu.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index 064e7a1ed82f..e28096758b9b 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -1,5 +1,5 @@
 SUMMARY = "An image containing the build system itself"
-DESCRIPTION = "An image containing the build system that you can boot and run using either VirtualBox, VMware Player or VMware Workstation."
+DESCRIPTION = "An image containing the build system that you can boot and run using any of qemu, VirtualBox, VMware Player or VMware Workstation."
 HOMEPAGE = "http://www.yoctoproject.org/documentation/build-appliance"
 
 LICENSE = "MIT"
@@ -20,7 +20,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
 APPEND += "rootfstype=ext4 quiet"
 
 DEPENDS = "zip-native python3-pip-native"
-IMAGE_FSTYPES = "wic.vmdk wic.vhd wic.vhdx"
+IMAGE_FSTYPES = "ext4 wic.vmdk wic.vhd wic.vhdx"
 
 inherit core-image setuptools3
 
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 04/11] build-appliance-image: set TERM

[Trevor Woerner]

Choose a nicer terminal for the build appliance so that build warnings and
errors show up in colour instead of:

	$<50>Setscene tasks: 2844 of 2844
	$<50>WARNING: mesa-2_22.3.3-r0 do_fetch: Failed to fetch URL https://mesa.freedesktop.org/archive/mesa-22.3.3.tar.xz, attempting MIRRORS if available
	$<50>WARNING: glslang-native-1_1.3.236.0-r0 do_fetch: Failed to fetch URL git://github.com/KhronosGroup/glslang.git;protocol=https;branch=master, attempting MIRRORS if available

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index e28096758b9b..b000ea0c4d70 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -63,6 +63,7 @@ fakeroot do_populate_poky_src () {
 
 	echo "INHERIT += \"rm_work\"" >> ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
 	echo "export LC_ALL=en_US.utf8" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+	echo "export TERM=xterm-color" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
 
 	# Also save (for reference only) the actual SRCREV used to create this image
 	echo "export BA_SRCREV=${SRCREV}" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 05/11] build-appliance-image kernel: linux-yocto: qemuall: add taskstats

[Trevor Woerner]

Add the taskstats configuration to any kernel built for qemuall for bitbake's
benefit:

	WARNING: The Linux kernel on your build host was not configured to provide process I/O statistics. (CONFIG_TASK_IO_ACCOUNTING is not set)

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-kernel/linux/linux-yocto.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
index 091003ed8299..93eef9e94684 100644
--- a/meta/recipes-kernel/linux/linux-yocto.inc
+++ b/meta/recipes-kernel/linux/linux-yocto.inc
@@ -31,7 +31,7 @@ DEPENDS:append:arc = " libgcc"
 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
 
-KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
+KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc features/taskstats/taskstats.scc"
 
 KERNEL_FEATURES:append = " ${@bb.utils.contains('MACHINE_FEATURES', 'efi', 'cfg/efi.scc', '', d)}"
 KERNEL_FEATURES:append = " ${@bb.utils.contains('MACHINE_FEATURES', 'numa', 'features/numa/numa.scc', '', d)}"
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 06/11] build-appliance-image: fix HOMEPAGE

[Trevor Woerner]

Update the HOMEPAGE metadata.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index b000ea0c4d70..ad5ea92b3eda 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -1,6 +1,6 @@
 SUMMARY = "An image containing the build system itself"
 DESCRIPTION = "An image containing the build system that you can boot and run using any of qemu, VirtualBox, VMware Player or VMware Workstation."
-HOMEPAGE = "http://www.yoctoproject.org/documentation/build-appliance"
+HOMEPAGE = "https://docs.yoctoproject.org/overview-manual/yp-intro.html#archived-components"
 
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 07/11] build-appliance-image: add /lib64 symlink

[Trevor Woerner]

Add a /lib64 -> /lib symlink in case the build appliance is 64-bit.

Building rust-native requires cargo to be run. The native cargo is linked to
look for /lib64/ld-linux-x86-64.so.2 and can't be run without it. The build
produces the following error, even though that file does exist:

	FileNotFoundError: [Errno 2] No such file or directory: '/home/builder/poky/build/tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo'

Trying to run this binary on its own gives:

	qemux86-64:~/poky/build$ tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo
	-sh: tmp/work/x86_64-linux/rust-native/1.66.1-r0/rust-snapshot/bin/cargo: cannot execute: required file not found

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index ad5ea92b3eda..3a84de0a2b43 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -109,7 +109,13 @@ fakeroot do_populate_poky_src () {
 	chown -R builder:builder ${IMAGE_ROOTFS}/home/builder/.cache
 }
 
-IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; "
+fakeroot do_tweak_image () {
+	# add a /lib64 symlink
+	# this is needed for building rust-native on a 64-bit build appliance
+	ln -rs ${IMAGE_ROOTFS}/lib ${IMAGE_ROOTFS}/lib64
+}
+
+IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; do_tweak_image; "
 # For pip usage above
 do_image[network] = "1"
 
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 08/11] build-appliance-image: use a real /tmp

[Trevor Woerner]

Make /tmp a real directory in the root filesystem and not a link to an
in-memory tmpfs. Otherwise bitbake will run out of inodes on /tmp very quickly
and stop any build.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index 3a84de0a2b43..1f7b96a9fcf4 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -113,6 +113,12 @@ fakeroot do_tweak_image () {
 	# add a /lib64 symlink
 	# this is needed for building rust-native on a 64-bit build appliance
 	ln -rs ${IMAGE_ROOTFS}/lib ${IMAGE_ROOTFS}/lib64
+
+	# make /tmp a real directory and not a link to an in-memory tmpfs
+	# otherwise /tmp will run out of inodes very quickly when bitbaking
+	sed -i -e 's#l root root 1777 /tmp /var/tmp#d root root 1777 /tmp none#' ${IMAGE_ROOTFS}/etc/default/volatiles/00_core
+	mkdir -p ${IMAGE_ROOTFS}/tmp
+	chmod 1777 ${IMAGE_ROOTFS}/tmp
 }
 
 IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; do_tweak_image; "
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 09/11] build-appliance-image: QB_MEM: allow user config

[Trevor Woerner]

Allow the user to specify a QB_MEM value of their choosing, otherwise set it
to a default value.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index 1f7b96a9fcf4..aa1e87763d57 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -11,7 +11,7 @@ IMAGE_INSTALL = "packagegroup-core-boot packagegroup-core-ssh-openssh packagegro
 
 IMAGE_FEATURES += "x11-base package-management splash"
 
-QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
+QB_MEM ?= '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
 
 # Ensure there's enough space to do a core-image-sato build, with rm_work enabled
 IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 10/11] QB_SMP: allow user modification

[Trevor Woerner]

Allow a user to override the QM_SMP value giving them the opportunity to
select for themselves the number of CPUs to use in qemu.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/conf/machine/include/riscv/qemuriscv.inc  | 2 +-
 meta/conf/machine/include/x86/qemuboot-x86.inc | 2 +-
 meta/conf/machine/qemuarm.conf                 | 2 +-
 meta/conf/machine/qemuarm64.conf               | 2 +-
 meta/conf/machine/qemuppc64.conf               | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/conf/machine/include/riscv/qemuriscv.inc b/meta/conf/machine/include/riscv/qemuriscv.inc
index 1d32b4a58209..c977a266ca21 100644
--- a/meta/conf/machine/include/riscv/qemuriscv.inc
+++ b/meta/conf/machine/include/riscv/qemuriscv.inc
@@ -26,7 +26,7 @@ UBOOT_ENTRYPOINT:riscv32 = "0x80400000"
 UBOOT_ENTRYPOINT:riscv64 = "0x80200000"
 
 # qemuboot options
-QB_SMP = "-smp 4"
+QB_SMP ?= "-smp 4"
 QB_KERNEL_CMDLINE_APPEND = "earlycon=sbi"
 QB_MACHINE = "-machine virt"
 QB_DEFAULT_BIOS = "fw_jump.elf"
diff --git a/meta/conf/machine/include/x86/qemuboot-x86.inc b/meta/conf/machine/include/x86/qemuboot-x86.inc
index 3953679366d9..dd7e7c938413 100644
--- a/meta/conf/machine/include/x86/qemuboot-x86.inc
+++ b/meta/conf/machine/include/x86/qemuboot-x86.inc
@@ -1,6 +1,6 @@
 # For runqemu
 IMAGE_CLASSES += "qemuboot"
-QB_SMP = "-smp 4"
+QB_SMP ?= "-smp 4"
 QB_CPU:x86 = "-cpu IvyBridge -machine q35,i8042=off"
 QB_CPU_KVM:x86 = "-cpu IvyBridge -machine q35,i8042=off"
 
diff --git a/meta/conf/machine/qemuarm.conf b/meta/conf/machine/qemuarm.conf
index 1bd4e3e154fc..c5234231e2e5 100644
--- a/meta/conf/machine/qemuarm.conf
+++ b/meta/conf/machine/qemuarm.conf
@@ -16,7 +16,7 @@ SERIAL_CONSOLES_CHECK = "${SERIAL_CONSOLES}"
 QB_SYSTEM_NAME = "qemu-system-arm"
 QB_MACHINE = "-machine virt,highmem=off"
 QB_CPU = "-cpu cortex-a15"
-QB_SMP = "-smp 4"
+QB_SMP ?= "-smp 4"
 # Standard Serial console
 QB_KERNEL_CMDLINE_APPEND = "vmalloc=256"
 # For graphics to work we need to define the VGA device as well as the necessary USB devices
diff --git a/meta/conf/machine/qemuarm64.conf b/meta/conf/machine/qemuarm64.conf
index ffc51c803fa7..2f0f0e4f842f 100644
--- a/meta/conf/machine/qemuarm64.conf
+++ b/meta/conf/machine/qemuarm64.conf
@@ -16,7 +16,7 @@ SERIAL_CONSOLES_CHECK = "${SERIAL_CONSOLES}"
 QB_SYSTEM_NAME = "qemu-system-aarch64"
 QB_MACHINE = "-machine virt"
 QB_CPU = "-cpu cortex-a57"
-QB_SMP = "-smp 4"
+QB_SMP ?= "-smp 4"
 QB_CPU_KVM = "-cpu host -machine gic-version=3"
 # For graphics to work we need to define the VGA device as well as the necessary USB devices
 QB_GRAPHICS = "-device virtio-gpu-pci"
diff --git a/meta/conf/machine/qemuppc64.conf b/meta/conf/machine/qemuppc64.conf
index 770933931324..2fbd26a6f997 100644
--- a/meta/conf/machine/qemuppc64.conf
+++ b/meta/conf/machine/qemuppc64.conf
@@ -13,7 +13,7 @@ SERIAL_CONSOLES ?= "115200;hvc0"
 QB_SYSTEM_NAME = "qemu-system-ppc64"
 QB_MACHINE = "-machine pseries"
 QB_CPU = "-cpu POWER9"
-QB_SMP = "-smp 2"
+QB_SMP ?= "-smp 2"
 
 QB_NFSROOTFS_EXTRA_OPT = "wsize=524288,rsize=524288"
 QB_KERNEL_CMDLINE_APPEND = "console=hvc0 nohugevmalloc"
-- 
2.36.0.rc2.17.g4027e30c53

[PATCH 11/11] build-appliance-image: check for xattr feature

[Trevor Woerner]

The xattr DISTRO_FEATURE is required otherwise a bitbake build will fail due
to a failure with the 'cp' utility:

	Subprocess output:
	cp: cannot preserve extended attributes, cp is built without xattr support

	ERROR: Logfile of failure stored in: /home/builder/poky/build/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0-r83/temp/log.do_populate_lic.1001
	ERROR: Task (/home/builder/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_populate_lic) failed with exit code '1'

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/recipes-core/images/build-appliance-image_15.0.0.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
index aa1e87763d57..2321d3a51a24 100644
--- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
+++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
@@ -22,7 +22,9 @@ APPEND += "rootfstype=ext4 quiet"
 DEPENDS = "zip-native python3-pip-native"
 IMAGE_FSTYPES = "ext4 wic.vmdk wic.vhd wic.vhdx"
 
-inherit core-image setuptools3
+inherit core-image setuptools3 features_check
+
+REQUIRED_DISTRO_FEATURES += "xattr"
 
 SRCREV ?= "a5507f383cdab99806df131bf4aef191799c5153"
 SRC_URI = "git://git.yoctoproject.org/poky;branch=master \
-- 
2.36.0.rc2.17.g4027e30c53

Re: [OE-core] [PATCH 09/11] build-appliance-image: QB_MEM: allow user config

[Khem Raj]

On Tue, Feb 21, 2023 at 12:45 PM Trevor Woerner <twoerner@gmail.com> wrote:
>
> Allow the user to specify a QB_MEM value of their choosing, otherwise set it
> to a default value.
>
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> index 1f7b96a9fcf4..aa1e87763d57 100644
> --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> @@ -11,7 +11,7 @@ IMAGE_INSTALL = "packagegroup-core-boot packagegroup-core-ssh-openssh packagegro
>
>  IMAGE_FEATURES += "x11-base package-management splash"
>
> -QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
> +QB_MEM ?= '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
>

it would be good to see if we can say something like minimum memory
needed so it should be a fallback
if user sets higher value then it should use that. So for that matter
it looks its fine unless someone sets
QB_MEM < minmem then it will get some trouble.

>  # Ensure there's enough space to do a core-image-sato build, with rm_work enabled
>  IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
> --
> 2.36.0.rc2.17.g4027e30c53
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#177547): https://lists.openembedded.org/g/openembedded-core/message/177547
> Mute This Topic: https://lists.openembedded.org/mt/97118452/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH 08/11] build-appliance-image: use a real /tmp

[Trevor Woerner]

On Tue 2023-02-21 @ 03:44:42 PM, Trevor Woerner via lists.openembedded.org wrote:
> Make /tmp a real directory in the root filesystem and not a link to an
> in-memory tmpfs. Otherwise bitbake will run out of inodes on /tmp very quickly
> and stop any build.
> 
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> index 3a84de0a2b43..1f7b96a9fcf4 100644
> --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> @@ -113,6 +113,12 @@ fakeroot do_tweak_image () {
>  	# add a /lib64 symlink
>  	# this is needed for building rust-native on a 64-bit build appliance
>  	ln -rs ${IMAGE_ROOTFS}/lib ${IMAGE_ROOTFS}/lib64
> +
> +	# make /tmp a real directory and not a link to an in-memory tmpfs
> +	# otherwise /tmp will run out of inodes very quickly when bitbaking
> +	sed -i -e 's#l root root 1777 /tmp /var/tmp#d root root 1777 /tmp none#' ${IMAGE_ROOTFS}/etc/default/volatiles/00_core
> +	mkdir -p ${IMAGE_ROOTFS}/tmp
> +	chmod 1777 ${IMAGE_ROOTFS}/tmp
>  }

Richard, this feels rather "hacky" to me. I can't help wonder if it would be
better to implement a VOLATILE_TMP_DIR mechanism (along the same lines as
VOLATILE_LOG_DIR)?

https://docs.yoctoproject.org/dev/ref-manual/variables.html?highlight=volatile_log_dir#term-VOLATILE_LOG_DIR
https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/initscripts/initscripts_1.0.bb#n107

Re: [OE-core] [PATCH 08/11] build-appliance-image: use a real /tmp

[Richard Purdie]

On Tue, 2023-02-21 at 21:25 -0500, Trevor Woerner wrote:
> On Tue 2023-02-21 @ 03:44:42 PM, Trevor Woerner via lists.openembedded.org wrote:
> > Make /tmp a real directory in the root filesystem and not a link to an
> > in-memory tmpfs. Otherwise bitbake will run out of inodes on /tmp very quickly
> > and stop any build.
> > 
> > Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> > ---
> >  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > index 3a84de0a2b43..1f7b96a9fcf4 100644
> > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > @@ -113,6 +113,12 @@ fakeroot do_tweak_image () {
> >  	# add a /lib64 symlink
> >  	# this is needed for building rust-native on a 64-bit build appliance
> >  	ln -rs ${IMAGE_ROOTFS}/lib ${IMAGE_ROOTFS}/lib64
> > +
> > +	# make /tmp a real directory and not a link to an in-memory tmpfs
> > +	# otherwise /tmp will run out of inodes very quickly when bitbaking
> > +	sed -i -e 's#l root root 1777 /tmp /var/tmp#d root root 1777 /tmp none#' ${IMAGE_ROOTFS}/etc/default/volatiles/00_core
> > +	mkdir -p ${IMAGE_ROOTFS}/tmp
> > +	chmod 1777 ${IMAGE_ROOTFS}/tmp
> >  }
> 
> Richard, this feels rather "hacky" to me. I can't help wonder if it would be
> better to implement a VOLATILE_TMP_DIR mechanism (along the same lines as
> VOLATILE_LOG_DIR)?
> 
> https://docs.yoctoproject.org/dev/ref-manual/variables.html?highlight=volatile_log_dir#term-VOLATILE_LOG_DIR
> https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/initscripts/initscripts_1.0.bb#n107

When I saw this, I was wondering about that too...

I think I'll hold off this patch whilst we think about that a bit but
my memory on this area is taking a bit of paging back in.

Cheers,

Richard

Re: [OE-core] [PATCH 08/11] build-appliance-image: use a real /tmp

[Trevor Woerner]

On Wed 2023-02-22 @ 02:46:30 PM, Richard Purdie wrote:
> On Tue, 2023-02-21 at 21:25 -0500, Trevor Woerner wrote:
> > On Tue 2023-02-21 @ 03:44:42 PM, Trevor Woerner via lists.openembedded.org wrote:
> > > Make /tmp a real directory in the root filesystem and not a link to an
> > > in-memory tmpfs. Otherwise bitbake will run out of inodes on /tmp very quickly
> > > and stop any build.
> > > 
> > > Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> > > ---
> > >  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 6 ++++++
> > >  1 file changed, 6 insertions(+)
> > > 
> > > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > > index 3a84de0a2b43..1f7b96a9fcf4 100644
> > > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > > @@ -113,6 +113,12 @@ fakeroot do_tweak_image () {
> > >  	# add a /lib64 symlink
> > >  	# this is needed for building rust-native on a 64-bit build appliance
> > >  	ln -rs ${IMAGE_ROOTFS}/lib ${IMAGE_ROOTFS}/lib64
> > > +
> > > +	# make /tmp a real directory and not a link to an in-memory tmpfs
> > > +	# otherwise /tmp will run out of inodes very quickly when bitbaking
> > > +	sed -i -e 's#l root root 1777 /tmp /var/tmp#d root root 1777 /tmp none#' ${IMAGE_ROOTFS}/etc/default/volatiles/00_core
> > > +	mkdir -p ${IMAGE_ROOTFS}/tmp
> > > +	chmod 1777 ${IMAGE_ROOTFS}/tmp
> > >  }
> > 
> > Richard, this feels rather "hacky" to me. I can't help wonder if it would be
> > better to implement a VOLATILE_TMP_DIR mechanism (along the same lines as
> > VOLATILE_LOG_DIR)?
> > 
> > https://docs.yoctoproject.org/dev/ref-manual/variables.html?highlight=volatile_log_dir#term-VOLATILE_LOG_DIR
> > https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/initscripts/initscripts_1.0.bb#n107
> 
> When I saw this, I was wondering about that too...
> 
> I think I'll hold off this patch whilst we think about that a bit but
> my memory on this area is taking a bit of paging back in.

Sounds good. In that case I'll wait for the others to hit master then re-work
this patch to integrate a VOLATILE_TMP_DIR mechanism?

Re: [OE-core] [PATCH 03/11] build-appliance-image: add support for qemu

[Richard Purdie]

On Tue, 2023-02-21 at 15:44 -0500, Trevor Woerner wrote:
> Add ext4 explicitly to the list of IMAGE_FSTYPES so the build appliance can be
> run with runqemu.
> 
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> index 064e7a1ed82f..e28096758b9b 100644
> --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> @@ -1,5 +1,5 @@
>  SUMMARY = "An image containing the build system itself"
> -DESCRIPTION = "An image containing the build system that you can boot and run using either VirtualBox, VMware Player or VMware Workstation."
> +DESCRIPTION = "An image containing the build system that you can boot and run using any of qemu, VirtualBox, VMware Player or VMware Workstation."
>  HOMEPAGE = "http://www.yoctoproject.org/documentation/build-appliance"
>  
>  LICENSE = "MIT"
> @@ -20,7 +20,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
>  APPEND += "rootfstype=ext4 quiet"
>  
>  DEPENDS = "zip-native python3-pip-native"
> -IMAGE_FSTYPES = "wic.vmdk wic.vhd wic.vhdx"
> +IMAGE_FSTYPES = "ext4 wic.vmdk wic.vhd wic.vhdx"
>  
>  inherit core-image setuptools3
>  

I've held off this one since whilst it isn't wrong as such, the ext
images aren't exactly optimal. I think our hope was that we could use
the vmdk or vhdx images with qemu instead and then directly test the
ones we publicly share. Do you know if that could work? It may then
mean people don't need to handle multiple files too?

Cheers,

Richard

Re: [OE-core] [PATCH 03/11] build-appliance-image: add support for qemu

[Richard Purdie]

On Fri, 2023-02-24 at 16:05 +0000, Richard Purdie via
lists.openembedded.org wrote:
> On Tue, 2023-02-21 at 15:44 -0500, Trevor Woerner wrote:
> > Add ext4 explicitly to the list of IMAGE_FSTYPES so the build appliance can be
> > run with runqemu.
> > 
> > Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> > ---
> >  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > index 064e7a1ed82f..e28096758b9b 100644
> > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > @@ -1,5 +1,5 @@
> >  SUMMARY = "An image containing the build system itself"
> > -DESCRIPTION = "An image containing the build system that you can boot and run using either VirtualBox, VMware Player or VMware Workstation."
> > +DESCRIPTION = "An image containing the build system that you can boot and run using any of qemu, VirtualBox, VMware Player or VMware Workstation."
> >  HOMEPAGE = "http://www.yoctoproject.org/documentation/build-appliance"
> >  
> >  LICENSE = "MIT"
> > @@ -20,7 +20,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
> >  APPEND += "rootfstype=ext4 quiet"
> >  
> >  DEPENDS = "zip-native python3-pip-native"
> > -IMAGE_FSTYPES = "wic.vmdk wic.vhd wic.vhdx"
> > +IMAGE_FSTYPES = "ext4 wic.vmdk wic.vhd wic.vhdx"
> >  
> >  inherit core-image setuptools3
> >  
> 
> I've held off this one since whilst it isn't wrong as such, the ext
> images aren't exactly optimal. I think our hope was that we could use
> the vmdk or vhdx images with qemu instead and then directly test the
> ones we publicly share. Do you know if that could work? It may then
> mean people don't need to handle multiple files too?

I quickly tried:

qemu-system-x86_64 tmp/deploy/images/qemux86-64/build-appliance-image-qemux86-64.wic.vmdk -enable-kvm

and it nearly works (splash screen displayed but then went blank) so I
suspect runqemu could likely learn to run these relatively easily.

Cheers,

Richard

Re: [OE-core] [PATCH 03/11] build-appliance-image: add support for qemu

[Trevor Woerner]

On Fri 2023-02-24 @ 04:05:22 PM, Richard Purdie wrote:
> On Tue, 2023-02-21 at 15:44 -0500, Trevor Woerner wrote:
> > Add ext4 explicitly to the list of IMAGE_FSTYPES so the build appliance can be
> > run with runqemu.
> > 
> > Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> > ---
> >  meta/recipes-core/images/build-appliance-image_15.0.0.bb | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > index 064e7a1ed82f..e28096758b9b 100644
> > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
> > @@ -1,5 +1,5 @@
> >  SUMMARY = "An image containing the build system itself"
> > -DESCRIPTION = "An image containing the build system that you can boot and run using either VirtualBox, VMware Player or VMware Workstation."
> > +DESCRIPTION = "An image containing the build system that you can boot and run using any of qemu, VirtualBox, VMware Player or VMware Workstation."
> >  HOMEPAGE = "http://www.yoctoproject.org/documentation/build-appliance"
> >  
> >  LICENSE = "MIT"
> > @@ -20,7 +20,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
> >  APPEND += "rootfstype=ext4 quiet"
> >  
> >  DEPENDS = "zip-native python3-pip-native"
> > -IMAGE_FSTYPES = "wic.vmdk wic.vhd wic.vhdx"
> > +IMAGE_FSTYPES = "ext4 wic.vmdk wic.vhd wic.vhdx"
> >  
> >  inherit core-image setuptools3
> >  
> 
> I've held off this one since whilst it isn't wrong as such, the ext
> images aren't exactly optimal. I think our hope was that we could use
> the vmdk or vhdx images with qemu instead and then directly test the
> ones we publicly share. Do you know if that could work? It may then
> mean people don't need to handle multiple files too?

Okay, I'll take a look.

I did try the vmdk image and was surprised it didn't work. Then I noticed this
piece of code in the recipe:

	IMAGE_CMD:ext4:append () {
		# We don't need to reserve much space for root, 0.5% is more than enough
		tune2fs -m 0.5 ${IMGDEPLOYDIR}/${IMAGE_NAME}.rootfs.ext4
	}

	https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/images/build-appliance-image_15.0.0.bb#n39

and assumed ext4 was either an implicit or explicit artifact in the past.